-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
libstd::sys, the great libstd refactor #1502
Conversation
How are you planning on re-using code between different unix platforms (or even between e.g. 32bit/64bit linux)? A small example would be very helpful. |
@Stebalien sorry if it wasn't clear, I don't actually propose making changes to the current implementations for each platform and how that code is structured; that would be a ton of work, but if someone feels it needs to be done it'll be possible after this change! The goal here is only to define a common interface that libstd uses to talk to the underlying system agnostic to the OS, and move everything under that. It's simply proposing a high-level organizational change. Any unix-specific code will be moved under One of my initial attempts can be found at rust-lang/rust@28806e9 |
external operating-system provided libraries like libc, libpthread, etc. This in | ||
turn makes it difficult to begin work on implementing the standard library | ||
without using these libraries. For example, it is hard to determine whether it | ||
is better for the pc-windows-mscv targets to use libc or whether they should |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mscv
-> msvc
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
of course you would be the one to find that..!
I agree with the sentiment that the standard library should have a clear portability layer of some kind. The summary says to move platform specific code to "sys" so it would be easier to port, but this is pretty much how the standard library already is organized. See for example the Unix fs.rs vs. Windows fs.rs. Both modules implement a common interface that the rest of std relies on. It may be that the patterns to create these abstractions in std are not consistent, or complete, or easy to read in one place, but they are there. If we're going to add yet another abstraction layer, or drastically change the one exists, it needs to be clear why. The major technical detail I see is that what exists now will be encapsulated into traits. Using modules is though listed as an alternative, and I'd suggest that's how What is the granularity of the trait implementations? If there is one for every triple for example, that's a lot of duplication. An advantage of the current strategy of using modules is that there's not a lot of code duplication. All of Unix is handled in one module. The greatest benefit of having a clear platform abstraction layer would be to let people implement std out of tree by plugging in their version of I might suggest that the goal should be to take what exists in We should get feedback from redox since that is the biggest non-unix, non-windows platform I know of trying to implement std out of tree. Also from others that are trying to port std to non-unix, non-windows platforms. Who else is doing this? |
Right, and that's why I suggest repurposing the module. The hope is to reorganize it all into something sane and consistent.
It's more like fixing the one that's sort of supposed to already be there. The "why" is to rid most of libstd of scattered
As I said above, I don't propose deviating much from the current granularity for the initial sweep. One for unix, one for windows. I only wish to better define the abstractions it provides, while leaving things open to more changes down the line if they're deemed necessary.
But that's the whole point of these changes... Once done, you can swap out
This is mentioned under unresolved questions. It is how I originally implemented it and why the original PR/thread/etc. is called While I fully believe it should eventually be its own standalone crate, I'm also a bit wary of the performance implications of inhibiting cross-module inlining optimizations if it's shipped as two separate crates.
One of the big use cases I'm personally interested in is a Linux syscall implementation. The idea behind the changes I propose are in order to clean up |
I think a large part of this work—probably the initial part of the work—is simply making better use of the existing sys pattern. For example, see https://github.com/rust-lang/rust/blob/eac0a8bc3070e45047fff57e7b024a059289a36d/src/libstd/rand/os.rs. All the stuff conditional on
IMO, traits for this kind of abstraction don't make sense because (1) for any given platform, there would usually only be one implementation of the trait and (2) in fact, there is usually only a need for one instance of that one implementation. Traits are more suited for the case where there are multiple implementations of the trait that could be used at once. So, I think the current module-based approach should probably be chosen.
I see this as a stepping stone to that, or to something like it. Whether different ports are managed in-tree or out-of-tree is a false dichotomy and also IMO more of a political issue than a technical issue. I don't want those politics to get mixed up with the technical improvements that come with having a clearer and easier-to-maintain porting abstraction.
I think that would be a good stretch goal for this project. But, I think that doing less than that—in particular, making more consistent use of the |
Considering that this is all a static, compile-time abstraction... Whether it's implemented as a crate or module, using traits or not... is all just an implementation detail. The advantage of using traits is to allow the compiler to assert early that you're not breaking the abstractions. The advantage of using crates is to more strictly define the abstraction boundaries. Neither choice has much of an effect on anything besides diagnostics and enforced code structuring.
Mm, and Rust doesn't make it very easy to deal with these static traits in general (the associated type thing kills me tbh). They just improve diagnostics and warn you if a change would accidentally break other platforms without you having to actually compile and test against each one. They also make for good documentation about what exact method signatures and types need to be provided by a platform implementation. A mock impl for the module approach would give you the same guarantees, though would be more awkward to maintain and integrate into the build/test system.
The way I see it, this work is a preliminary step toward that. Making |
Also, right now, a lot of code lives in libstd that, if a new "bare metal" implementation takes off, should be moved to a new |
Via @brson:
Me, @sacooper, and @kanetkarster are working on a Xen unikernel in Rust here. We're very interested in implementing
I don't know whether this RFC is what will lead us there - I need more time to look at it - but the general goal of out-of-tree |
I support this and I'd like it to have the explict goal of moving Here is a patch that adds another (unix) platform to libstd (with just enough support to compile): AveryOS/rust@d9fb48b |
This was previously discussed here: #185 While I definitely agree the current situation is no good, frankly I don't like trying to shove all OS-specific stuff in one crate. To recap my arguments from that RFC: This implies that all platforms support the same functionality at the end of the day, and its just a matter of providing the right hooks. I rather see a renewed commitment to splitting functionality until multiple crates below the std facade. That way platforms that e.g. support networking but not filesystems can implement the subset of crates that makes sense. Of course each of {alloc, sync, fs, net} (or whatever the crates beneath the facade are) can be pared with a shim crate to provide the same benefits a libplatform or libsys would. |
And traits are ill-suited for this for exactly the reasons @briansmith mentions. #1408 is far more appropriate. |
Huh, it's unfortunate that seems to have been abandoned... It could've already been done by now!
I agree, and would certainly like to see libstd become more modular in the sense that you can implement only the functionality that makes sense on a given platform. I just really didn't want it to come up here because that in itself is a massive discussion... I'd like to fix libstd first so that we can actually start porting to other platforms, then these changes would be much more straightforward from there (and much less intrusive to the codebase as a whole).
But #1408 would require the use of traits..? The only difference is that with Meh, there's too much of a disconnect of goals in this conversation... |
Very true sadly
Well the whole point of the facade is to allow experimentation without breaking the stable interface, so I hope it wouldn't be so bad. My hunch is that std and ::sys as it exists may have some circular deps so as a practical matter both these goals might have to be implemented together.
Ok sorry what I said was wrong let me fix it :). Ideally we have some sort of parameterized crates so one can pass in the name of the "sys crate" for the platform in question, rather than hard code a bunch of |
Well, I'd argue that the two aspects at play here are:
The second is a higher level architectural change that I have many opinions on, but am expressly not making a goal of this proposal as it merits a much deeper discussion that I'm not confident I should be leading. The goal of this RFC is to tackle So yes, it's important to keep in mind that the eventual goal is something larger than what I've proposed, but I'm hoping that we can all come to an agreement on a short-term abstraction that is designed well enough that it works well now and can later be adapted with minimal effort. If it's decided that we should instead go straight to a final solution, then this RFC is not sufficient to cover that and another proposal should be made (may require informal discussion first as there are a number of approaches that could be proposed).
I've thus far implemented this RFC twice; first time using trait methods and a separate crate so that libstd didn't depend on libc directly. Second time using the
This may be more of a "nice to have" kind of thing regarding an implementation detail rather than something that would drive the discussion and decisions made regarding the overall architecture.
Cycles should certainly be avoidable, but it may be prudent to reorganize parts of libstd such that its convenient abstractions can still be used by platform-dependent code. Imagine libstd split up into
Neither does a trait-approach without using delayed resolution. The only real difference is that delayed resolution necessitates the use of traits while with a crate or module approach it's simply one option. Annointing with
"plain"? |
This would make porting libstd to Redox much, much easier! 👍 |
libstd, internally, is currently a mess. It makes a lot of sense to collect the platform specific functionality into one module. I'm very positive to this RFC, since it allows porting libstd to new platforms easily. |
I took some time today to try to better understand the porting I was mostly trying to understand the existing My main goal is to identify concretely some simple incremental steps Overall organizationThe It's organization is difficult to understand because there are a lot
Review of platform-specific code in
|
I suspect that
It may need to be split to reflect that. |
It's been a while since this RFC has been touched. As it stands, there is not enough detail in the RFC about the proposed design for somebody to implement it, nor is there agreement on what those details should be. I'd like to move this forward soon, either toward a mergable RFC or toward closing. A mergable RFC will describe the specifics of the final design, how it's different from today's and how we will get there. I've previously outlined some thoughts on these topics based on cursory research into the issue. |
Yeah, sorry, I've been very short on time lately... I'd certainly like to see it move forward. Your analysis was very helpful, and I like the suggestion for naming the abstraction layer I guess a good starting point would be to fill in some more details about what that interface will look like, and continue discussion then...
This is definitely part of what needs to be redefined and fixed. Moving |
Thanks for the update @arcnmx. |
🔔 This RFC is now entering its final comment period 🔔 The @rust-lang/libs team is inclined to close this RFC, given @brson's previous comments. @rust-lang/libs if you could check off your name below or write a comment below to the contrary: |
I tend to think ports like pull #1645 will resolve issues within |
IIUC we don't really need an RFC for this, since it involves moving around unstable things, but it would nice just to codify a plan that multiple people can work on. I hope things like rust-lang/rust#35021 will get the ball moving organically, and that deprecating the make system will make things easier. |
I think @brson's comments illustrate a proper resolution to the problem and acutely identify what needs changing inside of libstd. While an RFC isn't strictly necessary, it helps to get opinions on a final direction and I'm glad the discussion took place. Maybe it will happen eventually/organically, and this thread at least outlines how some parts around it should be designed. It'd be nice to get consensus on this with the appropriate changes made, but ugh the last time I had any time to do anything was a year ago... |
I'm also in favor of closing this. At one point, it looked like this might make it easier to refactor libstd, but it seems to not matter much. |
I'm surprised I didn't comment in here before. I support something like this RFC. I very often run into situations where I need parts of std but not others. This results in crates such as core_io, core_collections, core_rustc-serialize. I was hoping the refactoring proposed here would make it easier to have such independent modules. std currently contains data structures, abstractions and implementations. Generally only the implementations are platform-dependent. I want code that doesn't depend on some implementations to run on systems that don't have those. |
Ok, with the libs team weighing in, the decision was to close. Thanks regardless though for the RFC @arcnmx! |
Some people here might be interested in rust-lang/rust#37133 |
A proposal to refactor the platform-dependent parts of
libstd
into a form that is better suited for porting to new platforms. The idea is to isolate the functionality we require from the underlying OS (basically, a thin abstraction over libc) into a well-defined interface that can be filled out easily by any platform. Thanks go to @briansmith for helping me draft this up.See https://internals.rust-lang.org/t/libsystem-or-the-great-libstd-refactor/2765 for previous discussions and context.
Rendered