Introduce -Zsplit-metadata to split metadata out of rlibs/dylibs
#137535
+248
−54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
A-run-make
|
The job Click to see the possible cause of the failure (guessed by this bot) |
|
(The test failure is the one situation not yet implemented, which is described in the PR description - |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a continuation of #120855 (I was mentored by @bjorn3 to move it forward). Most of the original code was written by bjorn3, I tried to clean it up a bit and add some documentation and tests.
This PR introduces a new unstable compiler flag called
-Zsplit-metadata(see #57076 for context). When used, rustc will only store a small metadata stub inside rlibs/dylibs instead of the full metadata, to keep their size smaller. It should be used in combination with--emit=metadata, so that the users of such a compiled library can still read the metadata from the corresponding.rmetafile. This comment shows an example of binary/artifact size wins that can be achieved using this approach.Contrary to #120855, this PR only introduces the new flag, along with a couple of run-make tests and documentation, but does not yet use it in bootstrap to actually compile rustc. I plan to do that as a follow-up step (along with integration in Cargo, which should ideally just always pass this flag to reduce the size of target directories).
There is one unresolved thing that I wanted to discuss with the reviewer(s) first, and one smaller nit.
Lookup of
.rmetafilesWith
-Zsplit-metadata, when rustc tries to link against a rlib/dylib that only has the stub metadata, it needs to be able to find the corresponding .rmeta file somehow. Currently, there are two codepaths taken in rustc regarding this dependency lookup:If you don't specify
--extern, the usual list of possible dependency locations is searched (including-L). The compiler already does this and finds the corresponding .rmeta files. This already works out of the box also with-Zsplit-metadatain this PR.If you do specify
--extern, the step above is skipped, and the--externpaths are searched instead. This is not fully supported yet in this PR. What we do now is add a simple heuristic that tries to add a.rmetafile path located in the same directory as the corresponding rlib/dylib file to the search paths. This will probably work for the majority of use-cases, but is not a fully general solution. If the.rmetafile is not found at this location, we should probably invoke the general library search path lookup linked above. However, the locator code is currently structured in a way where this is not completely trivial. So I wanted to ask which approach do you consider to be better:find_commandline_library, and store the information whether we only found a stub or not. Then, if we only found a stub, but no.rmetafile, invokefind_library_crateto try to find a corresponding.rmetafile, and use it instead (if found).find_commandline_libraryis not exclusive withfind_library_crate, so that the library search paths and the--externpaths are first combined together (with--externprobably taking priority), and only then is the final SVH/metadata lookup performed. This would pretty much make this work also with split metadata. I found it quite surprising that when--externis passed, the-Lsearch logic is completely skipped, so this also looks like a nice cleanup. I'm not sure if it wouldn't have some unexpected behavior change though.Note that currently it is not possible to pass a full path to the corresponding
.rmetafile explicitly, outside of using-L, but I don't think that it should be needed. In theory, we could also literally store the (relative) path to the.rmetafile on disk in the stub metadata header, but that's probably also not ideal, because the.rmeta/rlib/dylib file could move independently of one another.Interaction with
--emit=metadataCurrently, if the user does not use
--emit=metadatawith-Zsplit-metadata, no metadata will be emitted (there will be no.rmetafile, and the rlib/dylib will only contain a stub). This is most likely a usage error, but in theory it could be a valid use-case (same as we e.g. allow--emit=metadatafor binaries, which just produces an empty.rmetafile without any warning). As an alternative, we could emit a warning/error about this, or automatically fill in--emit=metadatawhen-Zsplit-metadatais used if no--emitflag was specified.Fixes #23366
Closes #29511
Fixes #57076
Another attempt of #93945 and #120855.
r? @petrochenkov