static symbols alias despite symbol versioning across two libLLVM's

[vchuravy]

Bugzilla Link	48221
Version	11.0
OS	Linux
Attachments	Rough initial patch for ManagedStatic

Extended Description

Since many projects depend on LLVM, we recommend that distributions and downstream user use symbol versioning to avoid conflicts between multiple LLVM versions being loaded into the same process.

On Linux mesa implementations (OpenGL/OpenCL) use LLVM extensively and they are generally built against the system LLVM. On the other hand front-ends like Julia might want to use a version of LLVM that diverges from the system LLVM.

What we have observed happening for several years is that users will get errors like:

CommandLine Error: Option 'help-list' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

• CommandLine Error: Option 'help-list' registered more than once! JuliaGPU/OpenCL.jl#125
• LLVM ERROR: inconsistency in registered CommandLine options hughperkins/tf-coriander#69

Or segmentation faults:

• Blender segfaults on OpenCL render NixOS/nixpkgs#97401
• Using libstdc++.so.6.0.28 and libLLVM-10.so causes a segfault when creating windows JuliaLang/julia#37200 (comment)

What this boils down to is that despite symbol versioning wires get crossed and the state of the two loaded LLVM version aliases each other.

After I building mesa, my system LLVM and Julia's LLVM with debuginformation I get the following interesting backtrace:

julia: /home/vchuravy/src/julia/deps/srccache/llvm-11.0.0/include/llvm/Support/CommandLine.h:851: void llvm::cl::parser<DataType>::addLiteralOption(llvm::StringRef, const DT&, llvm::StringRef) [with DT = llvm::FunctionPass* (*)(); DataType = llvm::FunctionPass* (*)()]: Assertion `findOption(Name) == Values.size() && "Option already exists!"' failed.

Thread 1 "julia" received signal SIGABRT, Aborted.
0x00007ffff7dda615 in raise () from /usr/lib/libc.so.6
(gdb) bt
#&#8203;0  0x00007ffff7dda615 in raise () from /usr/lib/libc.so.6
#&#8203;1  0x00007ffff7dc3862 in abort () from /usr/lib/libc.so.6
#&#8203;2  0x00007ffff7dc3747 in __assert_fail_base.cold () from /usr/lib/libc.so.6
#&#8203;3  0x00007ffff7dd2bf6 in __assert_fail () from /usr/lib/libc.so.6
#&#8203;4  0x00007ffff0a8014e in llvm::cl::parser<llvm::FunctionPass* (*)()>::addLiteralOption<llvm::FunctionPass* (*)()> (this=0x7ffff72fce68 <RegAlloc+168>, 
    Name=..., 
    V=@0x7fffffff9bc0: 0x7fffa6d8ff40 <llvm::createBasicRegisterAllocator()>, 
    HelpStr=...)
    at /home/vchuravy/src/julia/deps/srccache/llvm-11.0.0/include/llvm/Support/CommandLine.h:851
#&#8203;5  0x00007ffff0a83884 in llvm::RegisterPassParser<llvm::RegisterRegAlloc>::NotifyAdd (this=0x7ffff72fce60 <RegAlloc+160>, N=..., 
    C=0x7fffa6d8ff40 <llvm::createBasicRegisterAllocator()>, D=...)
    at /home/vchuravy/src/julia/deps/srccache/llvm-11.0.0/include/llvm/CodeGen/MachinePassRegistry.h:162
#&#8203;6  0x00007fffa67b9bb9 in llvm::MachinePassRegistry<llvm::FunctionPass* (*)()>::Add (Node=0x7fffab14a900 <basicRegAlloc>, 
    this=0x7fffab14a8e0 <llvm::RegisterRegAllocBase<llvm::RegisterRegAlloc>::Registry>) at ../include/llvm/CodeGen/MachinePassRegistry.h:110
#&#8203;7  llvm::RegisterRegAllocBase<llvm::RegisterRegAlloc>::RegisterRegAllocBase (
    C=0x7fffa6d8ff40 <llvm::createBasicRegisterAllocator()>, 
    D=0x7fffa91c73b8 "basic register allocator", N=0x7fffa91c73b2 "basic", 
--Type <RET> for more, q to quit, c to continue without paging--
    this=0x7fffab14a900 <basicRegAlloc>)
    at ../include/llvm/CodeGen/RegAllocRegistry.h:37
#&#8203;8  llvm::RegisterRegAlloc::RegisterRegAlloc (
    C=0x7fffa6d8ff40 <llvm::createBasicRegisterAllocator()>, 
    D=0x7fffa91c73b8 "basic register allocator", N=0x7fffa91c73b2 "basic", 
    this=0x7fffab14a900 <basicRegAlloc>)
    at ../include/llvm/CodeGen/RegAllocRegistry.h:63
#&#8203;9  __static_initialization_and_destruction_0 (__initialize_p=1, 
    __priority=65535) at ../lib/CodeGen/RegAllocBasic.cpp:44
#&#8203;10 _GLOBAL__sub_I_RegAllocBasic.cpp(void) ()
    at ../lib/CodeGen/RegAllocBasic.cpp:333
#&#8203;11 0x00007ffff7fe12de in call_init.part () from /lib64/ld-linux-x86-64.so.2
#&#8203;12 0x00007ffff7fe13c8 in _dl_init () from /lib64/ld-linux-x86-64.so.2
#&#8203;13 0x00007ffff7ed80e5 in _dl_catch_exception () from /usr/lib/libc.so.6
#&#8203;14 0x00007ffff7fe5705 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#&#8203;15 0x00007ffff7ed8088 in _dl_catch_exception () from /usr/lib/libc.so.6
#&#8203;16 0x00007ffff7fe4f3e in _dl_open () from /lib64/ld-linux-x86-64.so.2
#&#8203;17 0x00007ffff7f8934c in ?? () from /usr/lib/libdl.so.2
#&#8203;18 0x00007ffff7ed8088 in _dl_catch_exception () from /usr/lib/libc.so.6
#&#8203;19 0x00007ffff7ed8153 in _dl_catch_error () from /usr/lib/libc.so.6
#&#8203;20 0x00007ffff7f89b89 in ?? () from /usr/lib/libdl.so.2
#&#8203;21 0x00007ffff7f893d8 in dlopen () from /usr/lib/libdl.so.2
#&#8203;22 0x00007fffac9a6b90 in loader_open_driver.constprop ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /usr/lib/libGLX_mesa.so.0
#&#8203;23 0x00007fffac99cbe0 in dri3_create_screen.lto_priv ()
   from /usr/lib/libGLX_mesa.so.0
#&#8203;24 0x00007fffac9856e9 in __glXInitialize () from /usr/lib/libGLX_mesa.so.0

In particular I find note-worthy how from the initializer:

#​10 _GLOBAL__sub_I_RegAllocBasic.cpp(void) ()
    at ../lib/CodeGen/RegAllocBasic.cpp:333

we meander to:

#&#8203;5  0x00007ffff0a83884 in llvm::RegisterPassParser<llvm::RegisterRegAlloc>::NotifyAdd (this=0x7ffff72fce60 <RegAlloc+160>, N=..., 
    C=0x7fffa6d8ff40 <llvm::createBasicRegisterAllocator()>, D=...)
    at /home/vchuravy/src/julia/deps/srccache/llvm-11.0.0/include/llvm/CodeGen/MachinePassRegistry.h:162

Now I am unclear on how the wires get crossed here by the dynamic linker. My assumption is this has to do with static variables, and perhaps those get merged across the libraries?

Working with that hypothesis I added a prefix to the static variables used by ManagedStatic and that got me further, but while writing this up I am unsure if that is a sufficient fix. I am baffled byt the behaviour since these static variables are not exported ...

Thinking about this more an looking at the particular global variable:
llvm::RegisterRegAllocBase<llvm::RegisterRegAlloc>::Registry

nm -D ~/builds/julia/usr/lib/libLLVM-11.0.0jl.so | ~/builds/julia/usr/tools/llvm-cxxfilt | grep "llvm::RegisterRegAlloc"
0000000007f71780 u llvm::RegisterRegAllocBase<llvm::RegisterRegAlloc>::Registry@@JL_LLVM_11.0
00000000016fe70e W void llvm::cl::printOptionDiff<llvm::RegisterPassParser<llvm::RegisterRegAlloc>, llvm::FunctionPass* (*)()>(llvm::cl::Option const&, llvm::cl::generic_parser_base const&, llvm::FunctionPass* (* const&)(), llvm::cl::OptionValue<llvm::FunctionPass* (*)()> const&, unsigned long)@@JL_LLVM_11.0
00000000016f7cc9 W void llvm::cl::apply<llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >, char [9], llvm::cl::OptionHidden, llvm::cl::initializer<llvm::FunctionPass* (*)()>, llvm::cl::desc>(llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >*, char const (&) [9], llvm::cl::OptionHidden const&, llvm::cl::initializer<llvm::FunctionPass* (*)()> const&, llvm::cl::desc const&)@@JL_LLVM_11.0
00000000016fa05c W void llvm::cl::apply<llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >, llvm::cl::initializer<llvm::FunctionPass* (*)()>, llvm::cl::desc>(llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >*, llvm::cl::initializer<llvm::FunctionPass* (*)()> const&, llvm::cl::desc const&)@@JL_LLVM_11.0
00000000016f9176 W void llvm::cl::apply<llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >, llvm::cl::OptionHidden, llvm::cl::initializer<llvm::FunctionPass* (*)()>, llvm::cl::desc>(llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >*, llvm::cl::OptionHidden const&, llvm::cl::initializer<llvm::FunctionPass* (*)()> const&, llvm::cl::desc const&)@@JL_LLVM_11.0
00000000016fa8c7 W void llvm::cl::apply<llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >, llvm::cl::desc>(llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >*, llvm::cl::desc const&)@@JL_LLVM_11.0
00000000016f7c62 W std::function<void (llvm::FunctionPass* (* const&)())>::function<llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >::Callback::'lambda'(llvm::FunctionPass* (* const&)()), void, void>(llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >::Callback::'lambda'(llvm::FunctionPass* (* const&)()))@@JL_LLVM_11.0
00000000016f7c62 W std::function<void (llvm::FunctionPass* (* const&)())>::function<llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >::Callback::'lambda'(llvm::FunctionPass* (* const&)()), void, void>(llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >::Callback::'lambda'(llvm::FunctionPass* (* const&)()))@@JL_LLVM_11.0
0000000007ec4160 V vtable for llvm::RegisterPassParser<llvm::RegisterRegAlloc>@@JL_LLVM_11.0
0000000007ec40f8 V vtable for llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >@@JL_LLVM_11.0

nm -D /usr/lib/libLLVM-11.0.0.so | ~/builds/julia/usr/tools/llvm-cxxfilt | grep "llvm::RegisterRegAlloc"
00000000052188e0 u llvm::RegisterRegAllocBase<llvm::RegisterRegAlloc>::Registry@@LLVM_11
0000000005142ec0 V typeinfo for llvm::RegisterPassParser<llvm::RegisterRegAlloc>@@LLVM_11
0000000005142f60 V typeinfo for llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >::Callback::'lambda'(llvm::FunctionPass* (* const&)())@@LLVM_11
0000000005142ef8 V typeinfo for llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >@@LLVM_11
000000000329e4c0 V typeinfo name for llvm::RegisterPassParser<llvm::RegisterRegAlloc>@@LLVM_11
000000000329e680 V typeinfo name for llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >::Callback::'lambda'(llvm::FunctionPass* (* const&)())@@LLVM_11
000000000329e500 V typeinfo name for llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >@@LLVM_11
00000000051434b8 V vtable for llvm::RegisterPassParser<llvm::RegisterRegAlloc>@@LLVM_11
0000000005143540 V vtable for llvm::cl::opt<llvm::FunctionPass* (*)(), false, llvm::RegisterPassParser<llvm::RegisterRegAlloc> >@@LLVM_11

To quote the nm man page:

           "u" The symbol is a unique global symbol.  This is a GNU extension
               to the standard set of ELF symbol bindings.  For such a symbol
               the dynamic linker will make sure that in the entire process
               there is just one symbol with this name and type in use.

So I would assume that issue might be that the dynamic linker ignores the symbol version.

[vchuravy]

So yeah looking at the glibc handling it does look like unique global symbols are merged by name and that ignores the version.

[vchuravy]

Got a recommendation to try:

-fno-gnu-unique

[chengjunlu]

Hi @vchuravy ,
I met the same issue.
I have two libraries both linked with the static llvm libraries. And they both have the same symbol name for the static global singleton registry.
000000000b89c490 u llvm::RegisterRegAllocBase<llvm::RegisterRegAlloc>::Registry
And I got the same error as you.

Can you explain more detail about the suggested solution?