Synchronizing multiple live streams with dynamic bitrate

[derekw0311]

Hi,

I'm working on building a custom Exoplayer that allows user to stream multiple live video in synchronize matter. I used the solution in #2855 which works great if all streams has the same bitrate; however if the bitrate start to fluctuate (some of the stream has dynamic bitrate), all streams will start to buffer frequent and becomes unwatchable.

Is there a flag I can set to have player to render frames as fast as they can without buffering?
My assumption is, as long as all streams start at the same time; the fps should stays the same across all streams.

Thanks in advance!

[AquilesCanta]

You mean you'd like the playback speed to depend on the available buffer?

This is not a supported feature, but I suppose you could try implementing this yourself.

My assumption is, as long as all streams start at the same time; the fps should stays the same across all streams.

The fps don't depend on the player or the start time, it depends on the stream you are playing. On the other hand, if you mean playback speed, synchronization should not be affected by an altered playback speed (speed is a player attribute, not a renderer attribute). Does this answer your question?

[derekw0311]

I'm not sure about changing playback speed would help me with this. The issue I'm trying to resolve is that the player have difficulty synchronizing two stream that have different bitrate using mergingMediaSource. (The bitrate isn't constant, it varies depends on the content of the video)

Thank you for the quick response.

[AquilesCanta]

Sounds like a bug to me, but in order to look into it we will need the information requested in the issue template

[derekw0311]

• Description
  Player having difficulty streaming multiple live streams (with different bitrate) simultaneously; the player tent to buffers frequent when there is a spike with the bitrate and eventually stuck at the buffering state.

• Reproduction steps
  Implement the solution proposed by @ojw28 in Synchronizing Multiple ExoPlayers #2855 and try to stream 3 or 4 live videos with different bitrate using MergingMediaSource.

1. Build a custom RenderersFactory that builds multiple video renderers.
2. Create multiple ExtractorMediaSources and combine them use MergingMediaSource
3. Build a custom TrackSelector to select correct tracks for each of the video renderers
4. Map textureSurface to the corresponding renderers using 'blockingSendMessages'

• Link to test content
  Unfortunately I could not provide the video links

• Version of Exoplayer: 2.7.1

Please let me know if you need more info, thanks

[AquilesCanta]

Does this reproduce locally?

Have you ruled out the possibilty of bandwidth just not being enough to handle that many videos?

eventually stuck at the buffering state

Stuck for how long? Endlessly or for a brief period of time?

Please let me know if you need more info, thanks

Please provide content that reproduces the issue, you can do this at [email protected] if you'd like it to remain undisclosed. It is not convenient for us to spend time searching for content that reproduces the issue, without the guarantee of it having the correct characteristics.

[derekw0311]

Does this reproduce locally?
Have you ruled out the possibility of bandwidth just not being enough to handle that many videos?

I tried streaming 4 identical live streams and it works pretty well so the bandwidth should not be the problem. It tend to happen when streaming multiple live stream with different source.

Stuck for how long? Endlessly or for a brief period of time?
The playback stuck indefinitely but in some cases it would resume but increase the delay from real time. (Since it's a live stream, we try to minimize the delay)

Do you think it has something to do with the allocator size that cause the infinite buffering?

I'm working on providing content for this, will email to you as soon as possible.

[derekw0311]

@AquilesCanta I have sent the content & sample project to [email protected]

Thanks

[derekw0311]

@AquilesCanta, just wanted to follow up with you on this issue, thanks!

[AquilesCanta]

I will do my best to look at this late next week. There are a few things I need to do with some urgency, first. Thanks for your patience!

[derekw0311]

Not a problem, thanks for looking into this.

[derekw0311]

Hi @AquilesCanta, I just sent another email to [email protected] with an updated project and streaming url; however the session with the url might expire, please let me know if it is and I will provide a new one.

Thanks

[AquilesCanta]

I am getting a 401 from the server for both URLs. I must insist that it might be faster if you try to diagnose this issue yourself. I am pretty confident that the issue will be easy to identify.

[derekw0311]

@AquilesCanta I have just send the updated content url to [email protected]. There is an expiration with the urls, it usually last ~20-24 hours. Please give it another try, I have been looking into this for weeks. If you could take a quick look at this would be greatly appreciated. Sorry for all the back and forth on this.

[AquilesCanta]

Please try the following and let me know if this fixes your issue:

• First, increase the DefaultLoadControl's maxBufferMs. The value it had is way too low and may mean you cannot fit an entire GOP, which could trigger endless stalling. I don't think this is affecting you, but as a general rule it's better to leave the default values, unless you have a good reason not to.
• When you instantiate the ExtractorMediaSources, make sure you set a smaller value for continueLoadingCheckIntervalBytes (for example 3*1024 - default value is 1024 * 1024). Like so:

ExtractorMediaSource.Factory factory = new ExtractorMediaSource.Factory(httpDataSourceFactory);
factory.setContinueLoadingCheckIntervalBytes(3 * 1024); // You can play around with this and see what works for you.
ExtractorMediaSource source_0 = factory.createMediaSource(Uri.parse(VIDEO_URL_0));
... // Other media sources.

This fix will not eliminate buffering, but it should fix your issue. Let me know if it does.

[derekw0311]

Adjusting max buffer and set a smaller value for continueLoadingCheckIntervalBytes works!
It solved my problem with the endless stalling and re-buffering issue but it seems like I lost the ability to control minimum buffer. The player will start the playback at an arbitrary amount of buffer. (Set minBuffer = 1.5 seconds but it'll start the playback when it buffer up to 4-5 or sometimes 6-7 sec.). The goal of setting minimum buffer low is to reduce real-time latency for the live streaming.

Again, thank you so much for your help.

[AquilesCanta]

Something you can try is implementing your own MergingMediaSource. The issue now is that CompositeSequenceableLoader tries to keep buffer approximately aligned. This is pointless in this scenario where the server is releasing content as it becomes available. In this case you pretty much want to buffer as much as you can (might need fine-tuning, but you can start with this approach). Also, it might take some initial time before the buffers stabilize, since the data received by each HTTP connection can be uneven, and this is out of our control. Once you observe that the buffer of every media period is healthy, you can slightly accelerate playback speed to shorten this gap, almost seamlessly to the user, to the point where every stream is as close to the live edge as you like. Of course, the closer you get, the higher the changes of entering buffering state. Can we close this issue?

[derekw0311]

Thank you for your thorough explanation and the investigation for the buffering issue I had. Yes, we can close this.