Broadcast channel leak?

[CR7273868]

I noticed Fred uses tokio broadcast channel for it's pubsub. I only create a single single RX so it should (Clone) the value to one receiver. Could it be possible fred also uses Notification struct which also has over 300mb usage. Whenever I generate a flamegraph it reports Fred uses the most ram and keeps it allocated. I use Jemallocator. Other, allocators also do not work.

Reproducing this can be done with serde_json en deserializing an enum in my opinion. Nowhere, else is this leak found or no other part of my program uses over 50 mb except for fred that sits at 300MB+.

I use Redis 7.4.2 Cluster on Fred 10.0.4. Pool size of 10. And channel capacity set to 10000.

But the length does reach to 0 after it has processed and it keeps receiving. First fast, but there is something in between that prevents spam so eventually it becomes 0. Could you give me any pointers why Fred has too use so much ram? And why it all leads back to tokio sync broadcast channel when pubsub rx is used.

[aembke]

Interesting, how many clients are you using in total? Or if you're using a pool what's the pool size?

[CR7273868]

I'm using a single client to receive. My pool size is 10 in total.

[aembke]

Huh, that doesn't quite add up so something might be wrong here. Some back of the napkin math: sizeof::<Message>() = 136 * 7 channels per client (this should probably change) * 10 clients * 10_000 = 95_200_000 bytes preallocated for broadcast channel buffers.

I'll take a closer look over the next few days and let you know if anything jumps out.

[CR7273868]

Well it builds up very slow. Dhat-ra reports that tokio broadcast has it which is backed by fred.rs. Also in my project FuturesUnordered also took the crown because I also use for_each_concurrent for this:

https://docs.rs/tokio-stream/latest/tokio_stream/wrappers/struct.UnboundedReceiverStream.html

• But both the RX on fred and this for_each_concurrent all process succesfully to 0. So the buildup is very slow. But, dhat-rs reports in the flamegraph only-leaked that fred rs broadcast channel did hold it. My leak is now much less and a lot slowed down. But, maybe fred.rs holds something? I do deserialize message but drop implementation is called on Struct. However I didn't try for Message struct fred delivered me. So, it could he very well an issue of mine. But I've been at it for 2 weeks and ruled all possible side effects out even undoing my for_each_concurrent and the leak slowly build up.

[aembke]

Interesting, thanks for looking into it already with dhat. Can you describe the use case a bit? How many messages are you receiving roughly per second/minute/whatever, how big are the payloads, etc? I'll try to get a repro as well. And roughly how long did you have to wait to see a noticeable build up?

I don't think fred is holding on to anything here, especially since the frame decoding uses a zero-copy decoding implementation, but it's hard to say. The underlying payloads use Bytes and Str containers, which are just received on the socket, split_at the socket's BytesMut buffer, and then sent over the broadcast channel. Those types do have their own ref counting under the hood, but I can't think of any reason why the client would hold on to any of those values since there's not really anywhere to put them in the client internals other than the broadcast sender. And if the buffer splitting wasn't working correctly then I'd expect to see other issues where you'd see the same message over and over.

[CR7273868]

I identified a possible issue within my project. So this can be a false positive from my side. Fred isn't causing any issues in another of my 20 projects it's integrated in