Tracking Issue for secure random data generation in std

[joboet]

Feature gate: #![feature(random)]

This is a tracking issue for secure random data generation support in std.

Central to this feature are the Random and RandomSource traits inside core::random. The Random trait defines a method to create a new random value of the implementing type from random bytes generated by a RandomSource. std also exposes the platform's secure random number generator via the DefaultRandomSource type which can be conveniently access via the random::random function.

Public API

// core::random

pub trait RandomSource {
    fn fill_bytes(&mut self, bytes: &mut [u8]);
}

pub trait Random {
    fn random(source: &mut (impl RandomSource + ?Sized)) -> Self;
}

impl Random for bool { ... }
impl Random for /* all integer types */ { ... }

// std::random (additionally)

pub struct DefaultRandomSource;

impl RandomSource for DefaultRandomSource { ... }

pub fn random<T: Random>() -> T { ... }

Steps / History

• ACP: Simple secure random number generation libs-team#393
• Implementation: std: implement the random feature (alternative version) #129201
• Final comment period (FCP)¹
• Stabilization PR

Unresolved Questions

• Naming: the ACP used gen_bytes and DefaultRng, the implementation PR uses fill_bytes and DefaultRandomSource (see arguments pro gen_bytes and pro fill_bytes)
• Concerns listed at Simple secure random number generation libs-team#393 (comment) should be addressed

Footnotes

1. https://std-dev-guide.rust-lang.org/feature-lifecycle/stabilization.html ↩

[newpavlov]

Disclaimer: I am one of the getrandom developers.

I think it's important for RandomSource methods to properly return potential errors. Getting randomness is an IO operation and it may fail. In some context it's important to process such errors instead of panicking. The error may be either io::Error or something like getrandom::Error (i.e. a thin wrapper around NonZeroU32).

It may be worth to add the following methods to RandomSource:

• fill_bytes which works with uninitialized buffers, e.g. based on BorrowedBuf. Yes, zeroization of buffers is usually a very small cost compared to a syscall, but it still goes against the zero-cost spirit.
• Generation of u32 and u64. Some platforms support direct generation of such values (e.g. RDRAND, WASI, etc.). Going through fill_bytes will be a bit less efficient in such cases.
• Methods for potentially "insecure" generation of random values, but which are less prone to blocking. The HashMap seeding is the most obvious use-case for this.

It's also not clear whether it's allowed to overwrite the default RandomSource supplied with std similarly to GlobalAlloc.

[joboet]

Would you consider rust-lang/libs-team#159 to be a better solution? That one used the Read trait to fulfil everything you mention.

[newpavlov]

No, I don't think it's an appropriate solution. Firstly, it relies on io::Error, while IIUC you intend for this API to be available in core. Secondly, it does not provide methods for generation of u32 and u64. As I wrote, going through the byte interface is not always efficient. Finally, most of io::Read methods are not relevant here.

For the last point I guess we could define a separate DefaultInsecureRandomSource type.

[bstrie]

Firstly, it relies on io::Error, while IIUC you intend for this API to be available in core.

I don't think this needs to be a blocker. IMO a lot of std::io should be moved into core--not the OS-specific implementations obviously, but all the cross-platform things like type definitions, same as what happened with core::net.

[wwylele]

fn random(source: &mut (impl RandomSource + ?Sized)) -> Self;

Would it be better to make this explicit with generics? Like fn random<R: RandomSource + ?Sized>(source: &mut R) -> Self;. This gives user the ability to specify the type name when needed.

[ericlagergren]

I think it's important for RandomSource methods to properly return potential errors.

I (mostly) disagree. CSPRNGs should almost never fail. When they do, users are almost never not qualified to diagnose the problem.

For example: golang/go#66821

A compromise is something like this:

trait RandomSource {
    type Error;
    fn fill_bytes(...) {
        self.try_fill_bytes(...).unwrap();
    }
    fn try_fill_bytes(...) -> Result<..., Self::Error>
}

This allows most CSPRNGs to use Error = Infallible, but still has support for weird HSMs, etc.

[newpavlov]

@ericlagergren
I've assumed this trait is for a "system" RNG, which will work together with a #[global_allocator]-like way to register implementation. I don't think that we need a general RNG trait in std/core as I wrote in this comment.

As for design of fallible RNG traits, see the new rand_core crate.

[dhardy]

pub trait Random {
    fn random(source: &mut (impl RandomSource + ?Sized)) -> Self;
}

This trait (and the topic of random value generation) should be removed from this discussion entirely in my opinion, focussing only on "secure random data generation" as in the title. Why: because (1) provision of secure random data is an important topic by itself (with many users only wanting a byte slice and with methods like from_ne_bytes already providing safe conversion) and (2) because random value generation is a whole other topic (including uniform-ranged samples and much more).

Disclaimer: I am one of the rand developers. rand originally had a similar trait which got removed; the closest surviving equivalent is StandardUniform.

[hanna-kruppe]

@newpavlov

It's also not clear whether it's allowed to overwrite the default RandomSource supplied with std similarly to GlobalAlloc.

Overriding the default source in an application that already has one from linking std seems questionable. It's not that I can't imagine any use case for it, but the established pattern for such overrides allows any crate in the dependency graph to do it (it's only an error if you link two such crates), instead of putting the leaf binary/cdylib/staticlib artifact in charge. As you articulated in the context of getrandom, that's a security risk for applications. So a RandomSource equivalent should probably be more restrictive in who can override it, but that's not the existing pattern. It also doesn't seem to fit with the proposed generalization of that pattern via "externally implementable functions" (rust-lang/rfcs#3632) -- if that RFC is accepted, any new API surface should use it instead of adding new one-off mechanisms.

If overriding the std source isn't supported, then it could work the same way as #[panic_handler]: you must supply an implementation if you don't link std, but if you do link std then supplying your own is an error. This would still be extremely useful. Currently, every crate that's (optionally) no_std and needs some randomness, most commonly for seeding Hashers, has to cobble together some sub-par ad-hoc solution to try and get some entropy from somewhere. There's a bunch of partial solutions that are better than nothing (const-random, taking addresses of global/local variables and praying that there's some ASLR, a global counter when atomics are available, cfg-gated access to target-specific sources like CPU cycle counters or x86 RDRAND) but:

1. How well this works ends up highly platform-specific, in particular none of them work well for wasm32-unknown-unknown and wasm32v1-none targets .
2. Applications that have access to a better source of entropy and (directly or transitively) use such libraries don't have a good way to enumerate them and make all of them use the better source.

This wouldn't be a problem if the entire ecosystem could agree to always delegate this problem to on one specific crate (version) with appropriate hooks, like getrandom, but evidently that's not happening. Putting this capability into core (or a new no_std sysroot crate, comparable to alloc) has a better chance of solving this coordination problem. Well, at least eventually, once everyone's MSRV has caught up.

Edit: almost forgot that even std::collections::Hash{Map,Set} depend on having a source of random seeds. A way to supply such a source without linking std could help with moving those types to alloc, although as #27242 (comment) points out, it's not backwards compatible to make such a source mandatory for no_std + alloc applications.

[newpavlov]

@hanna-kruppe

Overriding the default source in an application that already has one from linking std seems questionable.

There is a number of reasons to allow overriding:

1. An alternative interface may be more efficient than the default one (e.g. reading the RNDR register vs doing syscall)
2. It may help reduce binary size and eliminate potentially problematic fallback paths (e.g. if you know that you do not need the file fallback on Linux)
3. In some cases it's useful to eliminate non-deterministic inputs (testing, fuzzing)

So a RandomSource equivalent should probably be more restrictive in who can override it, but that's not the existing pattern.

Yes. How about following the getrandom path and allow override only when a special configuration flag is passed to the compiler?

Either way, overriding is probably can be left for later. I think we both agree that we need a way to expose "system" entropy source in std and a way to define this source for std-less targets.

It also doesn't seem to fit with the proposed generalization of that pattern via "externally implementable functions" -- if that RFC is accepted, any new API surface should use it instead of adding new one-off mechanisms.

I agree that ideally we need a unified approach for this kind of problem. I made a similar proposal once upon a time.

But I think it fits fine? Targets with std could implicitly use std_random_impl crate for "external implementation" of the getrandom-like functions and users will be able to override it in application crates if necessary.

How well this works ends up highly platform-specific, in particular none of them work well for wasm32-unknown-unknown and wasm32v1-none targets .

I believe that having std for wasm32-unknown-unknown was a big mistake in the first place and the wasm32v1-none target is a good step in the direction of amending it. So I hope we will not give too much attention to its special circumstances.

This wouldn't be a problem if the entire ecosystem could agree to always delegate this problem to on one specific crate (version) with appropriate hooks, like getrandom, but evidently that's not happening.

Well, it has happened, sort of. getrandom is reasonably popular in the ecosystem even after excluding rand users.

The problem is that std already effectively includes its variant of getrandom for HashMap seeding and people reasonably want to get access to that. And I think problem of getting "system" entropy is fundamental enough for having it in std (well, not in the std per se, let's say in the sysroot crate set).

A way to supply such a source without linking std could help with moving those types to alloc, although as #27242 (comment) points out, it's not backwards compatible to make such a source mandatory for no_std + alloc applications.

Can we add yet another sysroot crate for HashMap which will depend on both alloc and the hypothetical "system entropy" crate?

[hanna-kruppe]

But I think it fits fine? Targets with std could implicitly use std_random_impl crate for "external implementation" of the getrandom-like functions and users will be able to override it in application crates if necessary.

The RFC (and the competing ones I've looked at) only supports a default implementation in the crate that "declares" the externally-implementable thing. If that crate isn't std, then an implementation from std would not count as "default" but conflict with any other definition. So we'd need another special carve-out for std (the very thing we'd want to avoid by adding a general language feature), or the language feature needs to become much more general to support overrideable default implementations from another source.

I believe that having std for wasm32-unknown-unknown was a big mistake in the first place and the wasm32v1-none target is a good step in the direction of amending it. So I hope we will not give too much attention to its special circumstances.

I was specifically talking about no_std libraries, for which the two targets are basically equivalent. Both don't have any source of entropy implied by the target tuple (instruction set, OS, env, etc.), and if you want to add one it'll have to involve whatever application-specific interface the wasm module has with its host.

Well, it has happened, sort of. getrandom is reasonably popular in the ecosystem even after excluding rand users.

Not to point any fingers but a counter example that's fresh on my mind because I looked at its code recently is foldhash. As another example, ahash only uses getrandom optionally (though it's on by default). If you're only using ahash indirectly through another library that disables the feature, then it's not gonna use getrandom unless you happen to notice this and add a direct dependency to enable the feature. In that case there is a solution, at least, but it's still not discoverable.

Can we add yet another sysroot crate for HashMap which will depend on both alloc and the hypothetical "system entropy" crate?

Possibly, but people may object to a proliferation of sysroot crates so let's hope there's a better solution.

[newpavlov]

or the language feature needs to become much more general to support overrideable default implementations from another source.

Yes, it's not as if RFC is a technical specification which must be followed word-by-word. There is a number of cases where the original RFC vision has somewhat changed during implementation stages. If anything, I would say it's an oversight/deficiency of the RFC to not cover cases like this.

foldhash

If a crate aims to minimize its number of dependencies as far as possible even at the cost of code quality and security, it obviously will not depend on getrandom, despite it being the de facto standard for getting system entropy. I think most people will agree that hacks like this have a strong smell. The same applies to fastrand (amusingly, it still uses getrandom for Web WASM) and other "partial solutions" listed by you. As you can notice, both hack their way into using std to get system entropy and either use a fixed seed or pile even more hacks when std is not available.

[hanna-kruppe]

As I said, I have no intention of pointing fingers at any crates. They have to navigate tricky trade-offs and complexities due to Rust's standard library (as a whole, not just std) not yet providing any entropy access. Let's keep this issue focused on changing that.

[workingjubilee]

The Random trait seems weakly motivated in terms of coupling it to RandomSource, as its design seems like it will be a much more hotly contested space, and it is (mostly) unrelated to RandomSource.

[theemathas]

Note that @dhardy (maintainer of rand) wrote some criticism of this at rust-lang/libs-team#393 (comment)

[abgros]

Could Random be implemented for arrays? That way, we could write something like:

let random_array: [u64; 100] = random();

which is a lot more convenient than using the fill_bytes() method.

[sorairolake]

I think it would be useful to have a data type like rand::rngs::mock::StepRng to represent a source of randomness. This is useful for testing whether Random for an arbitrary data type T is implemented as expected.

[dhardy]

I think it would be useful to have a data type like rand::rngs::mock::StepRng to represent a source of randomness. This is useful for testing whether Random for an arbitrary data type T is implemented as expected.

I disagree that it is useful in testing implementations of traits like Random, since there should not be any implied restrictions on such a distribution except that outputs are uniformly distributed, given that the outputs of the random source are truly random. Testing the distribution of output values requires statistical tools, e.g. the KS test as we have here.

[hanna-kruppe]

As far as I know there’s no plan to make the RandomSource trait sealed, so other implementations can be provided by third party crates without having to debate whether it’s a good idea for std to include them.

[jstarks]

It seems like one might regret adding Random later--for some times (e.g., u32), there's a natural definition and implementation that produces a uniform distribution, but it happens to correspond to using RandomSource with something like zerocopy/bytemuck/the safe transmute effort. I.e., there's already a reasonable solution for these types.

For more complicated types (e.g., f32 or (T, U)), Random is under-defined. It is unclear what distribution is expected, and without implementations on tuples or a derive for enums/structs, there's really no "reference implementation" for third-party code to model.

It seems better to leave this out rather than leave it in this poorly specified state. Just stabilize RandomSource.

Although... even RandomSource is lacking, in that it doesn't support writing to an uninitialized buffer... Would it be better to wait until this problem is solved in Read (#78485) before committing to this new interface? Adding BorrowedBuf support later will be a headache.

[hanna-kruppe]

Would it be better to wait until this problem is solved in Read (#78485) before committing to this new interface?

I really don't think so. The current interface works perfectly fine, it's just slightly suboptimal in some use cases (fewer than in the context of general byte-centric I/O). Providing only a Read::read_buf-style API would be strictly worse for many common use cases, in that it would be more complicated to use for no benefit. Usually you either want to generate one conveniently-sized integer at a time (probably should have a dedicated trait method for performance reasons), or you want to fill a small fixed-size buffer completely (e.g., cryptographic key material). Scenarios where the cost of buffer initialization is significant, e.g. preparing a large ephemeral buffer but only filling a small part of it before discarding it, are rare.

So I think it's pretty clear that the method for filling a &mut [u8] should exist even if io::Read had already arrived at a stable solution for reading into uninitialized buffers. Adding the latter after stabilizing the former does pose some challenges, but io::Read already has to solve similar challenges -- more of them, actually, because it also has to work well for several layers of I/O adapters, while PRNGs are rarely composed in a way that stresses the performance of data flowing through several layers of composition.

[Kixunil]

@hanna-kruppe

Perhaps something like this would work:

// Users don't need to concern themselves with this type.
pub strut OutBuf([MaybeUninit<u8>]);

impl OutBuf {
    // various methods to construct it and write to it, no method to read from it
}

// Users don't need to worry about implementing this, just use it
pub unsafe trait AsOutBuf {
    type Init: ?Sized;

    fn as_out_buf(&mut self) -> &mut OutBuf;
    // Only allowed if the entire buffer was overwritten.
    unsafe fn assume_init(&mut self) -> &mut Self::Init;
}

impl AsOutBuf for [u8] { type Init = [u8]; ... }
impl AsOutBuf for [MaybeUninit<u8>] { type Init = [u8]; ... }
impl<const N: usize> AsOutBuf for [u8; N] { type Init = [u8; N]; ... }
impl<const N: usize> AsOutBuf for [MaybeUninit<u8>; N] { type Init = [u8; N]; ... }

// Users can pass in any byte slice, byte array or their uninitialized counterparts
// The entire value is guaranteed to get overwritten, so if using the returned value is undesirable due to borrowing issues the users can simply use the original value safely, if it was an initialized type to begin with or they may just soundly call `assume_init` on it.
pub fn fill_random_bytes<T: AsOutBuf>(buf: &mut T) -> &mut T::Init { ... }

This way it already supports all reasonable cases and people are not forced into using uninitialized API. They can write let mut buf = [0; 32]; fill_random_bytes(&mut buf); exactly as if they were using the simple API or calling Read::read_exact.

[hanna-kruppe]

That helps the simple call sites remain simple, but it has several downsides:

1. The more complicated signature makes it less obvious that it can be used like that for simple cases, both for users who aren't reading the rustdoc page from start to finish, and also to the compiler's type inference in some cases.
2. The reliance on generics means RandomSource is no longer dyn-compatible. This could be fixed by gating the AsOutBuf-using methods on Self: Sized and providing other methods that are dyn-compatible, but at that point you might as well provide two concrete methods for the initialized and possibly-uninitialized case.
3. The new abstractions compete with Borrowed{Buf,Cursor} and other proposed APIs for "writing into a possibly-uninitialized buffer". std should minimize the number of subtly different and incompatible abstractions for the same problem space, especially if it's something third party libraries would want to use as vocabulary type/trait. Going in this direction would give more reasons to block stabilization of secure random data generation on other APIs that have been in limbo for a long time.
4. Even if we ignore the previous point and focus only on the narrow use case of RandomSource, there's considerable design space to be explored there. For example, a downside of the specific approach you outlined is that it forces every RandomSource implementation to use unsafe internally while also requiring every caller who wants to use the MaybeUninit capabilities to use another bit of unsafe justified by "the implementation is correct" (and we'd have to make RandomSource unsafe to implement to make this sound). In contrast, the more complicated interface of Borrowed{Buf,Cursor} provides a bunch of useful functionality as safe APIs.

[zx2c4]

Have you forgot about other OSes? Or about older Linuxes which std still supports?

No, I have not.

• On normal Linux, FreeBSD (and some others, Solaris and Dragonfly maybe), use getrandom(..., ..., 0). If the vDSO has this function available, use it via that.
• On ancient Linux (before 3.17, iirc), poll() once per process for one byte on /dev/random, followed by reads to /dev/urandom.
• On macOS and OpenBSD use arc4random() or getentropy().
• On Windows call ProcessPrng() or RtlGenRandom().

This is really not a new problem. It might be instructive to look at what Go does, for example.

[zx2c4]

I don't think anyone was proposing that the standard library add its own user-space CSPRNG.

I dunno what's happening here, but usually when conversations about "how do I get random numbers?" veers into being super large with lots of diverse opinions, insanity reigns supreme and implementations quickly fly off the rails. So, seeing that Rust is kinda important to get right, I wanted to nip this tendency in the bud.

With that out of the way, and reading some comments above to the tune of, "oh no, but I heard this one interface was rumored to be somewhere somehow broken!", I'd like to suggest that you just stick to each OS's normal interface and not try to do anything fancy or weird. Implement it in the vanilla way, and then if there are bugs because of environments you didn't foresee, you can investigate that on a case-by-case basis and see what needs to change, if anything at all.

I wrote my suggestion for Linux (and other OSes) above. Just do that. If it doesn't work someplace, you won't be any different from most other runtimes and environments, and you can deal with the issue then.

As far as I can tell from watching this play out over a long period of time, the pitfall to watch out for is using a loop and checking for EINTR in the normal way. If you do that right, I think you'll basically be good.

[programmerjake]

what do we do on systems where there's intentionally no secure random source available, e.g. wasip2 with only wasi:random's insecure-seed (for hash map seeds) and not any other random interfaces -- this would be used for things where you want 100% deterministic reproducible execution. all of that is sufficient to implement std's stable API as it is now, but if you add a secure random requirement to std then it no longer is.

[zx2c4]

panic? Or refuse to compile? If the platform doesn't provide it, it's not like that's a real runtime error a programmer needs to care about.

[programmerjake]

I think there should be a cfg for secure randomness, similar to how Rust has #[cfg(target_has_atomic = "8")] for atomics.

[zx2c4]

Sounds like an utter disaster waiting to happen. Please don't do that.

[bstrie]

what do we do on systems where there's intentionally no secure random source available

First, let me say that I am explicitly trying to avoid wading into the argument over whether anyone, anywhere should be using io::Error for anything, because I do not care either way. Instead, I simply want to point out that io::Error has an Unsupported variant for just this sort of thing, and something analogous could easily be done here (and again, without trying to wade into the argument about whether reading from the system source of randomness should be fallible (because I do not care either way), if anyone's on the side of making this function infallible, then it's easy enough to make opening the entropy device into a fallible operation). I don't see this as a blocker; there are many solutions that are all mostly pretty good, and it's just a matter of choosing one.

[programmerjake]

Sounds like an utter disaster waiting to happen. Please don't do that.

why? it seems like the best option to me, since you know at compile-time if you're trying to use something that doesn't exist on your target, rather than finding out at runtime when code unexpectedly panics, or returns errors...

having a cfg also allows std's HashMap (and any other third-party code that has similar needs) to seamlessly switch to using insecure-seed (which is exactly what insecure-seed was designed for).

[bjoernager]

why? it seems like the best option to me, since you know at compile-time if you're trying to use something that doesn't exist on your target, rather than finding out at runtime when code unexpectedly panics, or returns errors...

IIRC using such flags isn't currently something that is done in other parts of the standard library (except atomics, but they also depend on the ISA rather than the OS). I do, however, find it worth noting that we will still have to handle edge cases where RNG is for whatever reason not available (be it by panicking, aborting, or something third), thus rendering the cfg flag only indicative and not really definitive.

[ericlagergren]

Sounds like an utter disaster waiting to happen. Please don't do that.

I think they mean "std::random only exists if the config is enabled, and it's only enabled for certain platforms" not "std::random is only a CSPRNG if you enable the config."

[programmerjake]

Sounds like an utter disaster waiting to happen. Please don't do that.

I think they mean "std::random only exists if the config is enabled, and it's only enabled for certain platforms" not "std::random is only a CSPRNG if you enable the config."

yes, or at least whatever functionality requires a secure random implementation; trait definitions (just the interface) and insecure randomness as a separate interface can still be enabled.

cfg in Rust is basically equivalent to #ifdef in C, so this would be like:

#ifdef SECURE_RANDOM_EXISTS
void fill_with_secure_random_bytes(void *buf, size_t size) { ... }
#endif

[dhardy]

Sounds like an utter disaster waiting to happen. Please don't do that.

why? it seems like the best option to me, since you know at compile-time if you're trying to use something that doesn't exist on your target, rather than finding out at runtime when code unexpectedly panics, or returns errors...

having a cfg also allows std's HashMap (and any other third-party code that has similar needs) to seamlessly switch to using insecure-seed (which is exactly what insecure-seed was designed for).

The standard library should just provide two interfaces here:

• A guaranteed secure source
• A best-effort potentially insecure source

In other words, the cfg selection should happen within libstd, not in user code.

Regarding error handling, clearly compile-time errors are superior, but are they sufficient? E.g. Linux's getrandom() interface can return an error code.

[Kixunil]

E.g. Linux's getrandom() interface can return an error code.

It can but there's no point having a function available if it's guaranteed to fail. Just as we don't have Windows API on Linux made to always fail even though on Windows it can sometimes fail.

[hanna-kruppe]

I don’t know if “has a source of entropy” makes a lot of sense as a static property of compiler targets. As discussed in earlier comments, it would be very useful if the “give me some entropy” API was available in no_std as well (with a link-time error if the binary/cdylib doesn’t provides an implementation). In this case, the API exists on every target anyway - the only difference is whether there’s a default implementation available. Making the re-export in std unavailable on some niche targets wouldn’t stop anyone from depending on the API, it would just make it harder to write portable libraries. (See: the many libraries who learned the hard way they need a conditional dependency on portable-atomic if they’re generally no_std-compatible but use some atomics somewhere.)

[joboet]

The policy in std so far has been to stub out unsupported functionality by either returning ErrorKind::Unsupported or panicking. While I do think that there are better ways to indicate this (I remember reading something about platform support levels somewhere...), we shouldn't depart from this pattern just for this feature. We should only make sure that the quality of the random bytes is sensible if the request for random bytes succeeds.

[Kixunil]

with a link-time error if the binary/cdylib doesn’t provides an implementation

I consider that compile-time category - as opposed to runtime. It's definitely still better than runtime.

Making the re-export in std unavailable on some niche targets wouldn’t stop anyone from depending on the API, it would just make it harder to write portable libraries.

I don't think it's that simple. If you don't use conditional compilation you're relying on either dead code elimination or assuming nobody needs to use the library without randomness. (IOW everyone is able to provide an impl.) Further, there are cryptographic algorithms that can make use of additional entropy to make side channel attacks harder/less likely but don't strictly require it. I co-maintain one such thing. It'd make a lot of sense for them to conditionally enable this protection depending on availability of RNG. (Though being able to emit a warning if it's off would be nice.)

I think what would make more sense is rather than link-time error provide the cfg but have some mechanism to turn it on during compilation by supplying an implementation. That way the library can attempt to get entropy.

See: the many libraries who learned the hard way they need a conditional dependency on portable-atomic if they’re generally no_std-compatible but use some atomics somewhere.

I co-maintain such library. Exactly, we learned by someone filing an issue - somebody who didn't need Arc anyway. But the reason is not that the conditional compilation exists It's that this specific condition was completely undocumented. We had no idea that Arc required ptr atomics and that cfg was even unavailable in older versions so we had to bump the MSRV of embedded. Secretly making it into linking error would not help. For all other cases such as operating system you can see very clear "only available on ..." note in the doc. Atomics or things using them don't have it.

The policy in std so far has been to stub out unsupported functionality by either returning ErrorKind::Unsupported or panicking.

I haven't seen this happen anywhere outside wasm. And it's a horrible approach. Things suddenly panicking which is annoying to debug, annoying to communicate downstream, annoying to work around. Funnily, this exactly happened in the above mentioned library exactly with RNG and exactly with optional hardening. A feature was added there to workaround it but it really wasn't a great solution.

Libraries need to know when some API they need is definitely unavailable. Sometimes because they can workaround things, sometimes because they can disable part of the functionality and still be useful and sometimes just to communicate it to the consumers properly.

[joboet]

On the subject of error handling:

• On Linux, getrandom(buf, len, 0) cannot reasonably fail on new systems.
• On Windows, ProcessPrng always succeeds.
• On macOS, CCRandomGenerateBytes internally calls getentropy, which cannot fail.
• Nearly all other UNIX systems use arc4random_buf, which cannot fail.

The only platforms where random data generation can reasonably fail are WASI and UEFI – and in the case of WASI, this is unlikely because the host platform is probably one of the above platforms, thus ensuring that the request succeeds. UEFI is the only environment where there may truly legitimate reasons for random number generation failure, as it lacks a way to block on the availability of sufficient entropy. On all other platforms, requests will either always succeed (potentially after blocking) or fail for unrelated reasons (e.g. restrictions imposed by seccomp or because the target just doesn't support random data generation).

In conclusion and after reading the above discussion, I think std should just expose a function like

// in std::io

fn read_random(bytes: &mut [u8]);

and call it a day. Everything else involves trade-offs between complexity, stability and security that conflict with the stability guarantees of the standard library. And quoting from stds crate documentation:

Besides basic data types, the standard library is largely concerned with abstracting over differences in common platforms

Providing only a getrandom-style platform abstraction fits this scope description very nicely.

[joboet]

I haven't seen this happen anywhere outside wasm.

Hermit (processes, pipes), SGX (filesystem, processes, UDP, pipes), SOLID (command line arguments, processes, pipes), TEEOS (nearly everything), UEFI (threads, pipes, other things are WIP), WASI (processes, pipes), Xous (filesystem, environment variables, pipes), ZKVM (nearly everything).

I agree that this isn't ideal, but this is not the issue for that discussion.

[zx2c4]

UEFI is the only environment where there may truly legitimate reasons for random number generation failure, as it lacks a way to block on the availability of sufficient entropy.

In my experience with reversing UEFI implementations, most of the time UEFI is just calling rdrand/rndr, which generally doesn't fail.

In conclusion and after reading the above discussion, I think std should just expose a function like

// in std::io

fn read_random(bytes: &mut [u8]);
and call it a day.

I agree.

And then compile time error if the platform doesn't implement it.

[dhardy]

In my experience with reversing UEFI implementations, most of the time UEFI is just calling rdrand/rndr, which generally doesn't fail.

Except for some known cases where RDRAND always returns 0xFFFFFFFF, you mean? getrandom now has non-trivial code to detect (known) RDRAND errors.

This issue is about secure random data generation, so if some platform happens to have known bugs in its random source, this cannot be ignored. I think it also cannot be handled as a compile-time error, thus run-time error handling is a must: either return an error code or panic.

[zx2c4]

In my experience with reversing UEFI implementations, most of the time UEFI is just calling rdrand/rndr, which generally doesn't fail.

Except for some known cases where RDRAND always returns 0xFFFFFFFF, you mean? getrandom now has non-trivial code to detect (known) RDRAND errors.

Ahh right the CPU bugs that get ironed out through microcode updates. If your hardware is broken, all bets are off. Just panic() and update your CPU microcode. This isn't something the programmer should need to account for. There are lots of CPU bugs that lead to all sorts of crazy behavior. This isn't Rust's responsibility.

Anyway, fortunately this isn't an issue on real OSes with a proper multisource RNG and such.

[programmerjake]

in the case of WASI, this is unlikely because the host platform is probably one of the above platforms, thus ensuring that the request succeeds.

wasi's randomness interfaces are specifically designed such that some wasi implementations simply don't have the secure random functions (if you try to use them it will fail to run your wasm with a link error), because the whole wasm engine is intentionally running in a deterministic-only mode, even if the host is unix or windows where secure randomness is available. rust std doesn't yet support that, but, unless we guarantee having secure randomness on wasi, it should be relatively easy to implement.

[hanna-kruppe]

That should probably be a separate target anyway. Even if std doesn’t assume it can import all of the randomness related WASI interface, third party crates like getrandom already do.

[newpavlov]

As noted above, panicking or returning an error on unsupported functionality in std is an unfortunate reality. Ideally, Rust should've introduced a set of PAL crates instead of a monolithic std, but alas...

The WASI/WASM case is yet another argument why we need to introduce an "insecure" randomness source. We can mandate that this source may use PRNG seeded with a fixed seed on platforms which do not provide proper system randomness.

[abgros]

In conclusion and after reading the above discussion, I think std should just expose a function like

// in std::io

fn read_random(bytes: &mut [u8]);
and call it a day.

This seems okay for general use but I think it could actually be very dangerous.

If I'm given an uninitialized buffer:

let mut buf = [MaybeUninit::<u8>::uninit(); 100];

I might be tempted to convert it to a &mut [u8] and use it in your function like so:

let slice = unsafe {
	std::slice::from_raw_parts_mut(buf.as_mut_ptr() as *mut u8, buf.len())
};
read_random(slice);

Unfortunately, calling from_raw_parts_mut on an uninitialized slice is immediate undefined behaviour. In fact, there's no way around this besides initializing buf with some garbage data.

I would recommend changing the signature to:

fn read_random(bytes: &mut [MaybeUninit<u8>]);

and also adding a guarantee that if this function succeeds, all the bytes in the slice will have been initialized (i.e. calling assume_init() is always sound).

EDIT: upon further discussion this might be a good use case for BorrowedBuf?

[hanna-kruppe]

Requiring every caller to write a bit of unsafe is a non-starter, even if the soundness argument is formulaic. As for BorrowedBuf or other ways of safely abstracting over both initialized and uninit buffers, this was discussed at length earlier and has some downsides plus relatively weak motivation (compared to other I/O).

I don’t find your new argument very convincing: casting an uninitialized slice to &mut [T] is always unsound, as you say. On some level we have to trust users who choose to invoke unsafe to actually read the preconditions of the functions they’re calling. That’s not to say unsafe code should be unnecessarily difficult to write correctly or be full of “gotchas”. For example , it might be a good idea to also add an example to read_random’s documentation calling out this specific case (if you have an uninit buffer, you must initialize it first before casting it to &mut [u8] and passing it to read_random). But making the API unsafe to use or more complicated for the common uses seems like a bad trade off to reduce the impact of “careless” deployment of unsafe.

[bjoernager]

What if we make the read_random function generic?

zerocopy and bytemuck, for example, define the FromBytes and AnyBitPattern traits, respectively. Maybe we could also borrow from them?

// core::marker

// Possibly as a language item:
pub /* unsafe */ trait FromBytes { } // Or with some other name...

// std::io

pub fn read_random<T: FromBytes>(buf: &mut T);

We could then add an extra promise that the buffer referenced by buf will be fully initialised after a call to read_random (regardless of whether T supports uninitialised states or not).

(edit: I would personally prefer the name fill_random in this case, but this was just to build upon the previous suggestion).

[abgros]

Requiring every caller to write a bit of unsafe is a non-starter, even if the soundness argument is formulaic. [...] making the API unsafe to use or more complicated for the common uses seems like a bad trade off [...]

To be fair, it seems like many people here are overtly hostile to the "common uses", e.g. see above

No, I expect a beginner to use a crate that provides a PRNG of some sort and sampling within a distribution using said PRNG.

But since every low-level OS API that I'm familiar with takes in a pointer and a length, it might be wise to have a lower-level function giving you direct access to it and avoiding any overhead or abstraction like:

pub unsafe fn read_bytes_raw(dst: *mut u8, len: usize);

and then also have higher-level APIs covering plenty of common stuff like generating numbers or vectors which don't require using unsafe.

---

What if we make the read_random function generic?

It seems like your proposed read_random() function is virtually identical to the exising std::random::random() which is generic over any type implementing Random.

[bjoernager]

It seems like your proposed read_random() function is virtually identical to the exising std::random::random() which is generic over any type implementing Random.

I would argue that it is different in that my proposal would randomise the raw representation of the buffer – regardless of how this effects its effective value (whereas the current random function seems to only support "raw" types, i.e. non-padded integers). The proposed trait would also fill a largely different need (although with notable overlap) than the existing Random trait, namely that it can be used with non-random sources.

[abgros]

Well, random() will support arrays when #136732 lands, which in principle should be optimal as long as the compiler realizes that it can reuse an existing allocation. But your method is admittedly more powerful since it lets you use a &mut anywhere, like a subslice of a larger allocation. Probably having both options would be useful.