x86_64-unknown-linux-musl with -Ctarget-feature=-crt-static links to glibc

[LunarLambda]

tested with rustc stable 1.83.0, on Arch Linux (kernel 6.6.69 LTS)

rustup install stable-x86_64-unknown-linux-gnu
rustup target add x86_64-unknown-linux-musl
cargo new musl-hm
cd musl-hm
cargo rustc --target x86_64-unknown-linux-musl -- -Ctarget-feature=-crt-static
ldd target/x86_64-unknown-linux-musl/debug/musl-hm
        linux-vdso.so.1 (0x00007ffc176a7000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x0000755e74d12000)
        libc.so.6 => /usr/lib/libc.so.6 (0x0000755e74b21000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x0000755e74dc4000)

The same thing happens with cargo build and RUSTFLAGS instead of cargo rustc.
This happens irrespective of whether a the musl system package is installed or not.

If you set target.x86_64-unknown-linux-musl.linker = "musl-gcc" then linking fails with -Ctarget-feature=-crt-static (complains about missing libgcc_s), and without -Ctarget-feature, the resulting binary crashes instantly (that's a duplicate of #95926), but is dynamically linked to the musl library and loader.

I think rustc should error out on musl targets if attempting to disable crt-static, since anything else produces wrong or broken binaries. If the idea is that -crt-static works on targets where musl is the system libc (e.g. Alpine), then perhaps there should be something that detects whether the system libc is musl, and errors (or ignores crt-static) if not.

[bjorn3]

(complains about missing libgcc_s)

You need to install libgcc_s for musl. Rust needs libgcc_s.so for the unwinder, but musl doesn't install it by default. When statically linking without musl-gcc, rustc's bundled copy of libgcc.a is used instead.

[LunarLambda]

musl oesn't use the system GCC libgcc_s, and doesn't use its own. There is no way for me to install one. Either way it's broken.

[bjorn3]

Musl itself doesn't use any unwinder. Programs that are compiled for musl however can use libgcc_s as unwinder if they want to. In that case you need to install a libgcc_s version compiled for musl. For example Alpine linux has the libgcc package for this: https://pkgs.alpinelinux.org/package/edge/main/x86_64/libgcc It seems like Archlinux doesn't have a package with a precompiled libgcc_s however. It doesn't have a package for any other unwinder that can be used either.

By the way I was wrong about rustc using libgcc by default on musl when statically linking. Libunwind is used instead. Archlinux doesn't have libunwind compiled for musl either however.

tl;dr: You will either have to statically link against rustc's bundled libunwind, you have to compile an unwinder for musl yourself or you have to compile on alpine with the libgcc package installed.

[LunarLambda]

Ultimately it doesn't matter as long as long as linking with musl-gcc is broken anyway (the missing libgcc_s only happens with musl-gcc as linker)

Still, I think either rustc should detect systems/configurations where -crt-static (without musl-gcc) is not supported instead of silently linking to the completely wrong libc, or it should be clearly documented that -crt-static is only supported on specific systems/configurations.

[workingjubilee]

musl-gcc?

[LunarLambda]

wrapper script around gcc shipped with musl that passes in a .specs file that sets up the musl libraries, interpreter and crt.

[workingjubilee]

@richfelker Okay, I know we already have busted compilation semantics for musl atm, but what exactly should I be doing here for the unwinder on musl? Should I just ship a Rust unwinding library for musl targets?

[bjorn3]

We already ship libunwind for musl when statically linking. When dynamically linking we can't safely use this however as there may only be a single unwinder in the whole program and one of the dynamic libraries the program depends on may already be linking against libgcc_s.so as unwinder, so we are forced to dynamically link against libgcc_s.so too as at least on Alpine linux that is the system unwinder that is used. Shipping our own libgcc_s.so copy would be an option, but that would only fix compilation. Not running it on Archlinux which should work if libgcc_s.so is part of LD_LIBRARY_PATH as the musl package also installs the correct dynamic linker and libc.so in the location where the executable would expect it. The correct solution I think would be for Archlinux to provide libgcc_s.so as part of their musl package or as separate libgcc-musl package.

[LunarLambda]

The correct solution I think would be for Archlinux to provide libgcc_s.so as part of their musl package or as separate libgcc-musl package.

I can take this to their bug tracker tomorrow, I think this is solvable on their side.

Where does that leave Rust? is musl-gcc the "correct" way to link to a dynamic musl & loader with -crt-static on systems where musl is not the system libc (and therefore -crt-static does not work out of the box)?

Also, as a side note, musl-gcc seems to want to pull in libgcc_eh if it exists, not libgcc_s. I'm not sure whether these are the same or at least compatible things.

[bjorn3]

I think libgcc_s.so is a dynamic library made by linking libgcc.a and libgcc_eh.a together. Statically linking libgcc_eh.a when dynamically linking musl may result in multiple unwinders in the same program, which is UB.

[richfelker]

I'm confused by this whole issue, especially text like:

I think rustc should error out on musl targets if attempting to disable crt-static, since anything else produces wrong or broken binaries.

A non-static musl target binary is only usable on a musl-based host or one where musl is installed alongside something else in /lib/ld-musl-$(ARCH).so.1. But in this context it should work fine regardless of how libgcc was linked, assuming the musl-based host/target-libs include a suitable libgcc.

is musl-gcc the "correct" way to link to a dynamic musl

No, musl-gcc is a wrapper for evaluation purposes and compiling/linking simple C-only programs on non-musl hosts. It's not a proper cross compiler and not the right way (not even present) on musl-based hosts. I think this whole issue is confusing 3 different scenarios:

1. Building native programs on a musl-based host.
2. Cross-compiling for a musl-based target, static or dynamic, from a non-musl-based host.
3. Building static binaries against musl to run on the non-musl-based host.

Case 3 seems to be what a lot of people are assuming when they hear "musl", but it shouldn't be considered exclusively at the expense of breaking the others.

Statically linking libgcc_eh.a when dynamically linking musl may result in multiple unwinders in the same program, which is UB.

This is not supposed to be the case. There is no good reason you can't have multiple unwinders in the same program, as long as they're all capable of processing the read-only unwind tables mapped at execution time. We do not intend to support the legacy "register eh frame" way of doing things; the unwinder is supposed to call dl_iterate_phdr to find the unwind tables statelessly without any registration. Only the legacy registration-based approach has issues with multiple instances.

If this is not working right, we should try to get it fixed.

In any case I don't see how it's relevant. If you're static linking musl, you have to static link everything. There is no such thing (in the context of musl) as a binary with static libc but dynamic linked to other things. If you try to make such a thing, it will always come out broken, and that seems to be what's going on in the original issue report here. If you dynamic link musl, you must have dynamic target libs and a dynamic musl execution environment (ld-musl in /lib, path config file in /etc if not the default, libgcc_s if you're running C++ binaries or other stuff with unwinding, etc.) on the machine you will run the binary on.

[bjorn3]

This is not supposed to be the case. There is no good reason you can't have multiple unwinders in the same program, as long as they're all capable of processing the read-only unwind tables mapped at execution time. We do not intend to support the legacy "register eh frame" way of doing things; the unwinder is supposed to call dl_iterate_phdr to find the unwind tables statelessly without any registration. Only the legacy registration-based approach has issues with multiple instances.

JIT's need to use __register_frame which only registers unwind tables with a single unwinder. Also AFAIK the unwinder state between different unwinder implementations is not ABI compatible, so unwinding from Rust code linked against one unwinder into C++ code linked against another unwinder would call _Unwind_Resume for the wrong unwinder and thus cause UB, right? Unwinding from Rust into C++ and vice versa is officially supported with the extern "C-unwind" ABI.

[richfelker]

I have some responses in mind to that but they're probably off-topic for this issue. If you believe it's necessary to use a single unwinder, then that's automatically achieved with static linking, and for dynamic linking - which is only valid on a musl-based host or if you're cross-compiling and have proper target libs for a musl-based environment - you just use whatever is the expected shared unwinder for the target system, most likely libgcc_s (built against musl and already there as part of the target environment).

[bjorn3]

Yeah, that is kind of the point I was trying to make. For dynamic linking of musl rustc can't workaround Archlinux's musl package missing libgcc_s.so by bundling it's own unwinder. It has to be fixed on Archlinux's side by shipping libgcc_s.so for musl too. (and same for every other glibc based distro with a musl package that is missing libgcc_s.so)

[LunarLambda]

So then, is or can -crt-static on non-musl based hosts be considered unsupported then? Can this be documented as such if that's the case?

I understand that I might've put too many unrelated cases/information into my initial report in an attempt to be comprehensive, and it's lead to a lot of discussion I'm not well versed in.

My primary issue is still that linking with -crt-static on a non-musl host silently links to glibc instead. I'm fine with accepting this as a wontfix, but I would like to have it visibly documented in the Rust documentation in that case.

[richfelker]

Yeah, that is kind of the point I was trying to make. For dynamic linking of musl rustc can't workaround Archlinux's musl package missing libgcc_s.so by bundling it's own unwinder. It has to be fixed on Archlinux's side by shipping libgcc_s.so for musl too. (and same for every other glibc based distro with a musl package that is missing libgcc_s.so)

This isn't specific to libgcc_s.so but applies to any library the program might be using. To run a dynamic linked program you need all the dynamic libraries (for the right ABI, installed in the right search path) it depends on.

The answer is probably not to bundle them with musl but provide packages for whichever ones the user is likely to need.

[richfelker]

So then, is or can -crt-static on non-musl based hosts be considered unsupported then? Can this be documented as such if that's the case?

If you're building to use musl on a non-musl-based host, dynamic linking generally does not make sense unless the user is prepared to setup an environment for running foreign dynamic linked binaries.

If "unsupported" is the word you want to use for that, ok, but it should be allowed for users who specifically want to, and of course makes sense for the case of cross compiling which is mechanically the same thing, or should be.

My primary issue is still that linking with -crt-static on a non-musl host silently links to glibc instead.

I don't understand how/why this is happening, but it suggests something is deeply broken in target handling/cross compiling. The host gcc compiler driver, library paths, etc. should not be used at all in this case.

[bjorn3]

I don't understand how/why this is happening, but it suggests something is deeply broken in target handling/cross compiling. The host gcc compiler driver, library paths, etc. should not be used at all in this case.

Unless you specifically override the linker, rustc will use the default linker for the target whether you are cross-compiling or not. This is fine when cross-compiling between 32bit and 64bit x86 for the same OS or when cross-compiling to a bare-metal target (where we set the default linker to lld and don't configure it to link against any C libraries) It also works fine when cross-compiling to statically linked musl as rustc tells the linker the directory to find libc.a and the crt objects for musl using -L/path/to/rustlib/x86_64-unknown-linux-musl/lib/self-contained as part of the link-self-contained feature. Using dynamic linking disables the link-self-contained feature for musl however, thus requiring something like gcc-musl when cross-compiling to correctly configure linking against musl instead. I think we could make rustc warn when cross-compiling with link-self-contained disabled when the linker is not explicitly specified.

[richfelker]

Unless you specifically override the linker, rustc will use the default linker for the target whether you are cross-compiling or not

gcc or ld should not be the default for a foreign (cross) target like x86_64-linux-musl. x86_64-linux-musl-gcc (if using the compiler driver) or x86_64-linux-musl-ld (if invoking linker directly, which requires knowing a lot of magic to pass to it) would be.

It also works fine when cross-compiling to statically linked musl...

It probably does, but this is a huge hack, not supported usage. It's almost surely broken on archs with specific ABI requirements that don't match glibc, and depends on having a suitable libgcc_eh.a and possibly other target libs (which you may be providing already).

I think we could make rustc warn when cross-compiling with link-self-contained disabled when the linker is not explicitly specified.

If Rust wants to have this self-contained mode, I think it should ignore -crt-static (i.e. always static link, refuse to support dynamic linking) in this mode, and it should be treated as a separate mode from cross compiling, which should always use the correct cross tools to link and assume they are providing the correct target libs.

[bjorn3]

and depends on having a suitable libgcc_eh.a and possibly other target libs (which you may be providing already).

We are providing libc.a, libunwind.a and the crt objects.

$ ls ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/
crt1.o  crtbegin.o  crtbeginS.o  crtend.o  crtendS.o  crti.o  crtn.o  libc.a  libunwind.a  rcrt1.o  Scrt1.o

If you try to use any other library you will get a broken executable.

gcc or ld should not be the default for a foreign (cross) target like x86_64-linux-musl. x86_64-linux-musl-gcc (if using the compiler driver) or x86_64-linux-musl-ld (if invoking linker directly, which requires knowing a lot of magic to pass to it) would be.

Yeah, I may make sense to make the self-contained mode invoke the linker directly. And in fact in that case it should be possible to use the rust-lld we ship to not have any dependency on a host toolchain. As you said, it requires a lot of magic flags, but in the end it would be much less of a hack than we currently have.

If Rust wants to have this self-contained mode, I think it should ignore -crt-static (i.e. always static link, refuse to support dynamic linking) in this mode, and it should be treated as a separate mode from cross compiling, which should always use the correct cross tools to link and assume they are providing the correct target libs.

I guess it would be possible to disable link-self-contained by default at the same time we change to -crt-static by default as the latter requires users to change their build process anyway if they are building static executables using musl (though I have no clue when we will make that change. I honestly expected it to be done years ago.). And yes, we should deny the combination of -crt-static and link-self-contained. Even on a musl based distro that doesn't do what it says it does as it will still use the system libc.so when linking, while link-self-contained is supposed to avoid using any system libraries.

[richfelker]

If you try to use any other library you will get a broken executable.

It should really be suppressing the host library paths so you get a link failure not a silently broken executable.

Yeah, I may make sense to make the self-contained mode invoke the linker directly. And in fact in that case it should be possible to use the rust-lld we ship to not have any dependency on a host toolchain. As you said, it requires a lot of magic flags, but in the end it would be much less of a hack than we currently have.

This, but also, cross mode should not be invoking the host linker at all. It should always be invoking a tuple-named one; or if using lld, lld in the correct cross mode.

I guess it would be possible to disable link-self-contained by default at the same time we change to -crt-static by default as the latter requires users to change their build process anyway if they are building static executables using musl (though I have no clue when we will make that change. I honestly expected it to be done years ago.).

I don't understand what you mean here, and again I'm a bit concerned that Rust is doing something wacky with musl, treating it as a second-class thing that's for making self-contained binaries on glibc hosts rather than a first-class target in itself. While making self-contained binaries like this is a usage we very much want to support, it's not fair to break musl as its own target for this.

Link-self-contained should not be a default, but if it's enabled, it should be highest priority. -crt-static should not disable it, but rather having link-self-contained enabled should suppress -crt-static (force +crt-static).

[bjorn3]

It should really be suppressing the host library paths so you get a link failure not a silently broken executable.

Is there a way to do that without also switching to directly invoking the linker? Or do we need to switch to directly invoking the linker in self-contained mode?

I don't understand what you mean here, and again I'm a bit concerned that Rust is doing something wacky with musl, treating it as a second-class thing that's for making self-contained binaries on glibc hosts rather than a first-class target in itself. While making self-contained binaries like this is a usage we very much want to support, it's not fair to break musl as its own target for this.

Currently +crt-static is the default for musl targets. The plan is to switch this to -crt-static by default to make life easier for musl based distros. As this forces everyone who uses musl to build statically linked executables to change anyway, my suggestion is to disable the link-self-contained mode by default at the same time. This forces anyone who cross-compiles from a glibc based distro to either explicitly enable link-self-contained or to use musl-gcc. And when building a static executable on a musl based distro, it would use the system libc.a copy by default rather than the bundled one unless you explicitly enable link-self-contained. In other words with my suggested we would dynamically link agaimst the system musl install by default, just like we do for every other libc implementation. And you can choose to either statically link, use the bundled libc or both without either option implicitly affecting the other. For as long as we don't bundle libc.so for musl, the combination of link-self-contained and -crt-static would emit an error rather than silently dynamically link glibc.

[richfelker]

Is there a way to do that without also switching to directly invoking the linker? Or do we need to switch to directly invoking the linker in self-contained mode?

Not without poking at gcc specfile stuff (passing a custom specfile). This is the whole point of the musl-gcc wrapper, suppressing the default gcc library paths.

As this forces everyone who uses musl to build statically linked executables to change anyway, my suggestion is to disable the link-self-contained mode by default at the same time

So is self-contained mode a thing that's also selected by default for musl targets now? If so, yes that should be changed. I thought it was only something that users explicitly request.

So the changes you're proposing sound like a real improvement that will both solve the issue under discussion here, and fix a lot of problems musl-based distros are having to work around with Rust.

[bjorn3]

So is self-contained mode a thing that's also selected by default for musl targets now?

Yes, it is currently selected on musl by default when statically linking, but not when dynamically linking:

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Lines 1877 to 1880 in 13f3924

	// FIXME: Find a better heuristic for "native musl toolchain is available",
	// based on host and linker path, for example.
	// (https://github.com/rust-lang/rust/pull/71769#issuecomment-626330237).
	LinkSelfContainedDefault::InferredForMusl => sess.crt_static(Some(crate_type)),

So the changes you're proposing sound like a real improvement that will both solve the issue under discussion here, and fix a lot of problems musl-based distros are having to work around with Rust.

Great.

FWIW while I'm on the compiler team, I'm speaking in personal capacity here.

[rhizoome]

The confusion with target-feature=-crt-static should be fixed soon. It has been planned since 2021. And was discussed for many years before. Musl targets should behave like other targets.

#133386
rust-lang/compiler-team#422

[JoelEarps]

Hello sorry to re-ask the question. But I have been recently tying to make a lightweight build image and then run the output on a scratch docker. Whilst I can get the code to compile I am having issues finding the binary on the scratch image. When I try and run the binary on an alpine image I seem to be getting issues with alpine looking for libgcc_s.so.1. Is this related to what is being discussed here with the unwinder? I am trying to compile with musl. So to compile statically, on alpine, I need to install glibc on the alpine build step and that should remove the dynamic link required in the final image? Or are there other steps I need to make sure I can do this?

Why am I still getting dynamic links even if I am telling the compiler to build statically, what is actually going on under the hood with the static build, are there any resources so I can learn what happens in this instance and how to make sure everything is statically linked?

Or is it one of the other libraries causing the problem and requirement for libgcc?

ldd /opt/rust_application 
        /lib/ld-musl-aarch64.so.1 (0xffff9e5c0000)
        libssl.so.3 => /usr/lib/libssl.so.3 (0xffff9dd79000)
        libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0xffff9d973000)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by /opt/rust_application)
        libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffff9e5c0000)
Error relocating /opt/rust_application: _Unwind_SetIP: symbol not found
Error relocating /opt/rust_application: _Unwind_GetRegionStart: symbol not found
Error relocating /opt/rust_application: _Unwind_GetTextRelBase: symbol not found
Error relocating /opt/rust_application: _Unwind_FindEnclosingFunction: symbol not found
Error relocating /opt/rust_application: _Unwind_Resume: symbol not found
Error relocating /opt/rust_application: _Unwind_DeleteException: symbol not found
Error relocating /opt/rust_application: _Unwind_RaiseException: symbol not found
Error relocating /opt/rust_application: _Unwind_GetIP: symbol not found
Error relocating /opt/rust_application: _Unwind_GetIPInfo: symbol not found
Error relocating /opt/rust_application: _Unwind_GetDataRelBase: symbol not found
Error relocating /opt/rust_application: _Unwind_SetGR: symbol not found
Error relocating /opt/rust_application: _Unwind_GetCFA: symbol not found
Error relocating /opt/rust_application: _Unwind_Backtrace: symbol not found
Error relocating /opt/rust_application: _Unwind_GetLanguageSpecificData: symbol not found

I am sorry if I am asking the same question I am just trying to wrap my head around whats happening, I cannot say I am an expert with cross compilation and it is really baffling me.

Is this why people say to avoid building on alpine?

[bjorn3]

Why am I still getting dynamic links even if I am telling the compiler to build statically, what is actually going on under the hood with the static build, are there any resources so I can learn what happens in this instance and how to make sure everything is statically linked?

How are you trying to tell rustc to link statically? Are you using RUSTFLAGS="-Ctarget-feature=+crt-static"?

So to compile statically, on alpine, I need to install glibc on the alpine build step and that should remove the dynamic link required in the final image?

There is absolutely no point in installing glibc when trying to build for musl.

[JoelEarps]

How are you trying to tell rustc to link statically? Are you using RUSTFLAGS="-Ctarget-feature=+crt-static"?

I am using the command :

RUSTFLAGS="-Ctarget-feature=-crt-static" cargo build --target ${PLAT_RUST}-unknown-linux-musl --release

There is absolutely no point in installing glibc when trying to build for musl.

Oh interesting, so what could be causing it to still be a dynamic link?

[bjorn3]

RUSTFLAGS="-Ctarget-feature=-crt-static"

Notice how I wrote +crt-static, not -crt-static. +crt-static statically links libc, while -crt-static dynamically links libc.

[JoelEarps]

Oh I am such an idiot, thank you!