BufReader: appending more data to buffer

[kex103]

Unless I'm missing something, after a fill_buf call, there isn't a way for me to request the BufReader to read again, appending the results to its internal buffer.

Would it be feasible to add a function like "append_buff" to the BufRead trait? Or to BufReader directly? This would call self.inner.read, passing to it the subslice of the remaining internal buffer self.buf.

The motivation for this is to read dynamically sized requests that don't have a message delimiter.
In my case, by parsing through the request, I can determine if the request is complete or not, but I would not be able to know the full size of the message beforehand.

[sfackler]

BufReader has a fixed internal size - you'll need to collect into a separate buffer if you want to be able to process messages of an arbitrary length.

[kex103]

Thanks for the response!

Yes, this proposed function won't help messages that are larger than the buffer size.

However, correct me if I'm wrong, but fill_buf doesn't necessarily fill up the buffer, despite the name. It just makes one read call, which needs to be consumed before it will read again

So my first fill_buf call will populate some of the buffer, but the rest of the request is not ready yet. After parsing through the partially-filled buffer, I want to be able to have the BufReader make another read call, and then parse through the combined message. In my case, the messages are much smaller than the internal buffer size, but they are still being fragmented. For context, the BufReader I'm using is wrapping a mio TcpStream, which I believe is a wrapper around a TcpStream set to non-blocking.

[sfackler]

BufReader it not really designed to be a universal solution to all read-to-buffer use cases. It's specifically designed to minimize the cost of repeated small reads.

[abonander]

@kex103 I created buf_redux as a more flexible alternative to the std::io buffered types. Take a look and let me know what you think.

[kex103]

@abonander Thanks, I'll check it out!

Closing this issue. I realize there are considerations with just adding an "append_buff" function, as now I'm getting clipped messages if I hit the end of the buffer.