Camera streams do not show when more than 6 cameras in one dashboard with live option

[BoneheadFraggle]

Checklist

• I have updated to the latest available Home Assistant version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.

The problem

I have 13 cameras in Picture Glance Cards on a dashboard.
All cameras should show picture, but only 6 does. The other will not load.
It is not related to specific cameras, but the number (6) and the option camera_view: live

Expected behavior

The dashboard worked fine until 0.114.4, so I assume it is supposed work in 0.115 as well.

Steps to reproduce

1. Create a dashboard containing more than 6 Picture Glance Cards or Picture Entity Cards.
2. Configure each card with a camera and the option camera_view: live.
3. Only 6 streams are showing picture in the dashboard. The other cards shows no picture at all and won't even show the camera stream when clicked.

Environment

• Home Assistant release with the issue: 0.115.0b0 and .0b1
• Last working Home Assistant release (if known): 0.114.4
• Browser and browser version: Google Chrome Version 85.0.4183.83 (Official Build) (64-bit), Microsoft Edge Version 85.0.564.44 (Official build) (64-bit), Firefox 80.0.1 (64-bit)
• Operating system: Windows 10 / Android 10 App

State of relevant entities

Problem-relevant configuration

camera_image: camera.blueiris_maskinrum_pool
entities:
  - entity: switch.maskinrum_pool_torch
  - entity: switch.pool_pump
title: Maskinrum pool
type: picture-glance
camera_view: live

Javascript errors shown in your browser console/inspector

Additional information

[zsarnett]

@hunterjm Do you have an idea of what this could be?

[hunterjm]

I don't, I can't reproduce either by adding duplicate cameras. Two things:

1. My laptop is taking off like a jet engine trying to render all these live view videos at once.
2. There are 404's in the console downloading segments, but that could be due to having multiple of the same camera trying to render a live view at the same time?
   

[hunterjm]

Tried in the iOS app, and it can't render them all in the mobile application, some of them show an image that is frozen, others show the video, and others still are empty and not loading. Either way, clicking into the stream works when it's rendering just one video at a time :)

[zsarnett]

@BoneheadFraggle Do oyu have any other errors (other than the 404 segment errors) that are in your browser console?

[hunterjm]

Also, do any of your cameras have audio? Since that was introduced in 0.115 so could be causing issues if multiple video players are trying to play audio at the same time... maybe?

[BoneheadFraggle]

@hunterjm Yes, 8 of the 13 cameras have audio.
@zsarnett Sorry, saw this to late, already reverted back, can't say for sure.

Unfortunately, I had to revert back to 0.114.4 due to low Family Acceptance Factor in the beta, so we may need to put this on hold until I get a test system up and running and can reproduce it again.

Should I close the issue this in the mean time?

[BoneheadFraggle]

Back on 0.115, now on 0.115.0b4.
After upgrading and activating stream: I'm now experiencing home-assistant/core#39850 instead, so let's close this and focus on that one instead.

[BoneheadFraggle]

@hunterjm
I am now using 0.115.0b11. A lot of testing later, it seems it is related to if using Nginx proxy or not.

When using ordinary SSL/443 with Let's encrypt the above occurs, only 6 cameras load.
When using the exact same config, except using Nginx for proxy, all cameras load fine.

It sounds strange, but I have tried back and forth 3 times now. With different browsers and cleared cache in between.
It would be really interresting if someone else could reproduce it. If not, I guess it's just something in my setup.

[uvjustin]

Since @hunterjm doesn't seem to have the same limit and because you don't see this issue when using NGINX, this appears to be a resource issue on the server. NGINX will help by offloading the web serving/encryption from HA onto a different process/core.
As for why it worked in 0.114 and not in 0.115, it is possible this is due to more processing of audio packets. You could test this by disabling audio on your camera RTSP streams and seeing if there are any changes to this limit. It might also be due to other changes in 0.115 which seem to have increased CPU usage. See home-assistant/core#40292 and home-assistant/core#40311

[BoneheadFraggle]

Just tried this again, now with 0.117.0b5.

• Accessing camera view over https/nginx = works fine.
• Accessing same page at the same time over http (HA directly, no nginx) = only 6 cameras. The 6 cameras work fine though.

Not using stream:.

All of them are setup like this:

- platform: mjpeg
  mjpeg_url: http://mycam.local:80/mjpg/cam01/video.mjpg
  name: mjpeg_cam01
  username: !secret cam_user
  password: !secret cam_pwd

Nothing in the logs. Anything else I can try to provide more info?

[uvjustin]

@BoneheadFraggle It's possible that when HA serves the connections to your browser end to end, the number of connections from HA to your browser may get saturated (most browsers have a limit of 6 simultaneous connections to a domain. When you use nginx as a proxy, especially if nginx and HA are on the same machine, nginx can essentially act as a buffer, allowing connections to catch up even if they periodically get bogged down.

[BoneheadFraggle]

If you are correct (which I think you are), shouldn't it be quite common for this to occur?

This is not a problem for me anymore since I worked around it with nginx, but I am still curious so I just dug up an old Internet Explorer to try if I got 11 cams. But HA didn't work at all in IE so that didn't give any more info. Tried Edge on Android and Firefox on Android outside nginx and only 6 cams are loaded. So the "error" is consistent, and that's good.

Feel free to ping me if you want me to test something, otherwise I think you can close it with a probable reason to the phenomenon.

[uvjustin]

If things are fast enough the connection limit shouldn't matter, so maybe it's related to some combination of video size, network bandwidth, and processing/serving speed?
Glad that you were able to work around it with nginx. Using nginx is probably a better overall setup anyway. I don't really know what else to test - let me know if you find out anything more the issue.