[bug] Bugs in runtime_deployer

[valgur]

Describe the bug

Keeping the bug report info from #16527.

The runtime_deployer from #15382 is a very good addition to Conan, but the implementation currently has some defects:

• Only *.so files (e.g. libz.so) are copied, but not *.so.* (e.g. libz.so.1 and libz.so.1.3.1).
• shutil.copy2(..., follow_symlinks=symlinks) should be shutil.copy2(..., follow_symlinks=not symlinks).

How to reproduce it

No response

[johan-boule]

Another issue is that runtime_deploy does not keep any tree structure.
For example, the openssl package has lib/engines-3/capi.so.
Deployers should keep these trees intact.

I considered trying full_deploy instead and post-processing its output to merge all the packages together into one single tree, something like mv full_deploy/host/*/*/*/*/lib*/* lib64/ mv full_deploy/host/*/*/*/*/bin/* bin/

But there's also mystery with full_deploy, it doesn't seem to deploy the runtime libs:

$ ls full_deploy/host/*/*/*/*/lib/
full_deploy/host/benchmark/1.8.4/Release/x86_64/lib/:
full_deploy/host/bzip2/1.0.8/Release/x86_64/lib/:
full_deploy/host/fmt/10.2.1/Release/x86_64/lib/:
full_deploy/host/gtest/1.14.0/Release/x86_64/lib/:
full_deploy/host/libcurl/8.8.0/Release/x86_64/lib/:
full_deploy/host/libpq/15.5/Release/x86_64/lib/:
full_deploy/host/ocilib/4.7.4/Release/x86_64/lib/:
full_deploy/host/openssl/3.2.2/Release/x86_64/lib/:
cmake
full_deploy/host/oracle-instant-client/19.5.0/Release/x86_64/lib/:
full_deploy/host/pcre2/10.42/Release/x86_64/lib/:
full_deploy/host/poco/1.13.3/Release/x86_64/lib/:
full_deploy/host/pugixml/1.14/Release/x86_64/lib/:
full_deploy/host/spdlog/1.14.1/Release/x86_64/lib/:
full_deploy/host/xerces-c/3.2.5/Release/x86_64/lib/:
full_deploy/host/zlib/1.3.1/Release/x86_64/lib/:

This might be the same bug.
If we look at poco's package tree for example, it contains lib files all in this form:

libPocoCrypto.so -> libPocoCrypto.so.103
libPocoCrypto.so.103

So, all regular files are *.so.* and all symlinks are *.so
Maybe the deployer takes *.so BUT ignores symlinks ?

[memsharded]

Deployers should keep these trees intact.

I am not so sure about this.
The idea of the runtime deployer is that it contains all runtime dependencies in a single folder, so it is very easy to use them. For example, for a Windows application, having all DLLs in the same folder, copying them to the executable folder, for example, and everything works. If we have different DLLs in different subfolders, then it is necessary to add all of those to the PATH, and in order to do that, it would be needed some further scripting, etc.

This is related to @uilianries PR in #17848

[valgur]

That's not my experience for the most part. Subdirs in lib/ are nearly always used for runtime plugin libraries, at least on non-Windows-oriented projects. This also means that they almost always have a way for locating the relevant plugin dirs, either via an env var or via some path relative to the executable. By flattening these directories you are likely to break any existing assumptions or configs about the layout.

[johan-boule]

Deployers should keep these trees intact.

I am not so sure about this. The idea of the runtime deployer is that it contains all runtime dependencies in a single folder, so it is very easy to use them. For example, for a Windows application, having all DLLs in the same folder, copying them to the executable folder, for example, and everything works. If we have different DLLs in different subfolders, then it is necessary to add all of those to the PATH, and in order to do that, it would be needed some further scripting, etc.

This is related to @uilianries PR in #17848

Hi, to better describe what I mean by the tree structure we most likely want to have to run the application, I should probably have referred to it as the basic FHS layout, with a single bin dir, a lib or lib64 alongside, and possibly other standardized dirs.
Datafiles and loadable modules are a pain because they tend to rely on application-specific search paths, but at least it would be interesting if the basic case of normal shared libs worked out of the box without even having to temper the environment variables. This could involve having to rewrite the ELF RPATH entries to use $ORIGIN or such.
An alternative would be to automatically generate shell wrapper scripts for launching binaries, akin to what python venv does. I think Conan already supports generating them, but not for each individual program. It's more for env sourcing during development. I haven't tried much though.
Currently, I ended up writing my own deploy python method combined with some hackery that tells Conan what are the final bin and lib dirs via environment variable (there's no way to pass them to the deploy method).
We also had to customise our CMake scripts a lot to make sure the binaries we build/install use ELF RPATH with $ORIGIN.
At the end it works, but not at all out of the box and requires an amount of work that only few people are going to be willing to put these days (especially if they've enjoyed the ease of other ecosystems outside of the painful C/C++ world). We're talking about maybe 500 lines of build script CMake+Conan to deal with that very topic. That's a lot for supporting the de-facto standard way of working. This is causing headaches when we need to transfer this knowledge to other colleagues.

[memsharded]

At the end it works, but not at all out of the box and requires an amount of work that only few people are going to be willing to put these days (especially if they've enjoyed the ease of other ecosystems outside of the painful C/C++ world). We're talking about maybe 500 lines of build script CMake+Conan to deal with that very topic. That's a lot for supporting the de-facto standard way of working.

Could you please clarify the "de-facto standard way of working".
This thread is about a deployer, which is meant to extract things out of Conan. But Conan is a "development" package manager, and the most common way of working in development is not copying shared libs around, but using them from their installed location. Conan doesn't change here the standard practices by build systems.
Then, the deployment of those applications is a different story, and I agree it can be complicated. But I am not sure there is a "de-facto standard" there either. How users deploy their final applications might differ a lot, different users will chose different deployment and update of their applications, depending on how they are taking that to production, they might have desktop installers, system packages such as deb/rpm, be burning full images to be used by embedded systems, etc.

This is the reason why Conan made the deployers customizable by users. If the built-in deployers are not suitable for a use case, it is possible to create your own deployers, and make them available to other colleagues via conan config install/install-pkg with the rest of the configuration, so the knowledge to translate is that they need to call --deployer=mydeployer. Still, as commented above, this is intended to be a final deployment operation, not something that developers do on their normal development flow.

I am not opposed to change the current runtime_deployer to respect the existing folders. But we need to understand that it is almost sure that there are other users that will be inconvenienced by this change, and they will strongly think that the deployer should deploy all shared libs in a "flattened" single folder, because they are not concerned about plugins, and they want the convenience to have everything in a single folder, and they will strongly think that this is the way that the runtime_deployer should work. As a note, we didn't create this runtime_deployer, it was contributed by @fdgStilla with the "flatten" approach, maybe they also want to provide their feedback here.

[johan-boule]

By the way, I forgot to say thank you to the Conan team for their new Conan 2 design 👍. Maybe one still needs a half-thousand lines of scripts to get started with a working Conan+CMake integration up to the final app packaging but that's a BIG improvement compared to the older Conan 1