feat(client): Client-side Snapdragon Game Super Resolution #2723
Conversation
zeroxoneafour
changed the title
|
There are a few problems with this PR:
If you intended to just using this as a sharpening algorithm, move it to the server side. If instead you want to upscale the image you need to increase the final resolution. |
|
I moved the shader to the stream pass and exposed more parameters. Staging resolution is now also scaled up, but I'm not 100% sure if thats the right way to go about it. The problem I'm running into is passing the original texture dimensions into the wgsl from the staging resolution before scaling it up ( |
|
Ok I think it's actually working now. An fxaa pass might be needed before using sgsr as it's hard to see tangible improvements especially on the edges of the screen below a 2x dimension multiplier, but it should work good enough without it. Might try using 3-4x "ultra performance" and see if that works any better without burning my headset to a crisp. |
|
So as it turns out using 4x dimensional multiplier on a Quest 3 does in fact burn it to a crisp. It could be left in in case someone is using ex. potato settings with an AVP, for some reason. |
|
Yeah, i am worried about the headset not handling the final resolution. 4x actually means 16x the amount of pixels to render, which would definitely not work on any current headset. I think Virtual Desktop handles this by reinterpreting the resolution selector by including the scale factor of upscaling, which means that with same resolution, upscaling would reduce bitrate usage. This is basically what we are doing with foveated encoding. It's not trivial to choose what to do, any opinions from others? @Meister1593 @The-personified-devil |
|
I tried playing bonelab on 2x upscale factor/low resolution and client fps interestingly dropped to 45. It seems that 2x is the limit to what Quest 3 can do without being put into some form of ASW. Pc fps stayed stable at 90. Maybe some sort of scaling based on target resolution should be used instead, but this could limit certain outlying setups with large disparities between their headset and PC power. |
|
I suggest doing what i said, that is keeping the same final resolution and scale down the transcoding resolution. Also you're saying the client drops to 45 fps, what is the client compositor latency in the graphs (the red strip)? |
|
Client compositor latency is still fairly low (averaging around 1ms). In my experience, the system latency goes up rather than the client latency for whatever reason. Foveated rendering client-side seems to fix it. As for scaling down the transcoding resolution, I don't really think that's a good approach. Virtual Desktop, according to their discord, implements SGSR by rescaling whatever input they get in to the headset panel resolution. The game runs at whatever resolution is selected, and the upscaler just adds a bit on top of it. This is what I set out to do by introducing upscaling, and in order to keep it independent of SteamVR, I think it's better treated as an addition on top of whatever resolution you're running for a bit more PC performance. The slider of course adds a bit more user configurability to the upscaler, but silently changing the render resolution when using upscaling may cause upscaled output to appear worse from the lower res. |
|
Ok. If you think that it causes non trivial performance issues, then let's keep it off by default as you're doing. but if instead we would keep it on by default then we would probably need to scale down the default resolution presets |
|
I think the biggest problem here is that there's no default setting that would work well because it's dependent on both render resolution and headset speed. I tested with a Quest 3 on low res/90hz, so a Quest 2 would probably only benefit from 1.3-1.5x while an AVP on the same settings could probably push 3x and still see gains. The most important thing is probably just being explicit in the tooltips/guides so users know how to tweak the settings to their hardware. |
|
For foveated encoding we decided to have it enabled by default, as that would only reduce the transcoding resolution, and while we found it caused some issues on low spec android phones (when used for PhoneVR), and then thankfully fixed, we found that generally having it enabled by default was the better choice. I say that if the performance issues are only dependent on final resolution (and not resolution-independent shader issues) we should think of reducing the transcoding resolution proportional to the upscaling increase, like foveated encoding, and enabling upscaling by default. If you say that the performance issues you face are unique to the upscaling algorithm (beyond an equivalent native resolution), then let's keep it off. In any case, the resolution and framerate defaults are themselves already a compromise between different headsets hardware, this is nothing new. We should just be careful of not having defaults that make the experience unusable on low spec headsets |
|
Yeah I think off by default is the best solution. ALVR already pushes the client, this is just something to try if you think you have a bit of extra client side processing headroom but your PC is maxing out. Plus some people are more sensitive to upscaling artifacts than they are to blurred screens. |
|
Sounds good to me. Does VD default to it on or off? |
|
Off I believe, it also doesn't have any of these config options and it limits you from using it to upscale/sharpen on resolutions that are too high. In VD its just a toggle. |
Not sure if i have much to say about this to be completely honest... As someone who doesn't know much about this feature, i think it needs to be properly explained what it does, how to use and what implications it has. Notice (yellow box) would be nice for sure. |
|
Rebased and squashed. Should be good to go |
|
A lints need to be fixed |
There was a problem hiding this comment.
I just tested this, the effect is great, although this also sharpens the pixelation of the foveation areas. I understand that this is not easy to fix, we would probably need to implement a separate pass. I think this is usable and useful as it is, any improvements can be done later.
I'll wait another day before merging if anyone else wants to chime in for a review
|
LGTM. Any more fixes can be done in a future PR |
* add broken sgsr * working now * Rebase with SGSR * renamed some values and it stopped working * minor rework but still nroken * trying to pass upscale settings into alvr * upscaling achieved yippee * forgot to pass upscale factor oops * allow excessive upscale factor (not necessarily recommended) * better defaults centered around FSR balanced * default to 1.5x dimensional, better factor tooltip, use builtin as_vec2 * flattened shader code and addded scale_resolution() to handle upscaling calculations * fix up functions * adjust resolution handling * increase upscale power on edges when foveated rendering * ok rust * add help to edge direction
* add broken sgsr * working now * Rebase with SGSR * renamed some values and it stopped working * minor rework but still nroken * trying to pass upscale settings into alvr * upscaling achieved yippee * forgot to pass upscale factor oops * allow excessive upscale factor (not necessarily recommended) * better defaults centered around FSR balanced * default to 1.5x dimensional, better factor tooltip, use builtin as_vec2 * flattened shader code and addded scale_resolution() to handle upscaling calculations * fix up functions * adjust resolution handling * increase upscale power on edges when foveated rendering * ok rust * add help to edge direction
Could offer improvement in visual quality, especially around the edges. I don't really know how to fine-tune the settings for SGSR so more improvement could probably be made on top of upscaling already. Of course this is toggleable too so shouldn't be a big issue if it doesn't look better for a lot of people.