Disable Pico face tracking lipsync mode #2738
Conversation
Pico has a "Lipsync" face tracking mode which uses headset audio to drive face tracking blendshapes when it detects audio. Typically applications (such as VRChat) already have this sort of functionality, and face tracked avatars typically have options to enable/disable it in-game. Have face tracking blendshapes overridden by the headset without any user control is not expected behavior and causes confusion when troubleshooting avatar face tracking.
| if audio { | ||
| tracking_flags |= TRACKING_MODE_FACE_LIPSYNC | TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES; | ||
| } | ||
| tracking_flags |= TRACKING_MODE_FACE_BIT; |
There was a problem hiding this comment.
There are bools in constructors for that purpose. IMO this should not be just removed but implemented as switches in dashboard for both FaceTracker2Fb and FaceTrackingPico. See curoviyxru@b6aa99f.
There was a problem hiding this comment.
Agreed. instead of removing audio input at all, we should only remove the flag TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES
There was a problem hiding this comment.
we should only remove the flag TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES
but it matches FaceTracker2Fb's audio face tracking, doesn't it?
There was a problem hiding this comment.
Meta has VIDEO and AUDIO flags, which control the sources of tracking. the change we should do here is to not generate visemes/blend shapes, which regards tracking output, not input
There was a problem hiding this comment.
is to not generate visemes/blend shapes
but what does AUDIO flag on Meta do then if not generating blend shapes based on audio input?
There was a problem hiding this comment.
Ah right. I think there is something we are missing, Pico API might be a bit confusing. What we want is to maybe allow audio source but not generate visemes. You could look up how to actually do this.
There was a problem hiding this comment.
allow audio source but not generate visemes
I'm afraid that there is no such thing.
There was a problem hiding this comment.
Then it's a matter of sensible defaults. If generating visemes is more resource intensive, probably we should disable the audio source by default. Furthermore we should not send the visemes array range, and we should add another rule on the server-side to detect the Pico face tracking without visemes. As for VRCFaceTracking, i'm afraid we can't do much unless we make a breaking change. It should be fine to just send zeroes for the visemes range
|
I agree, we probably don't need to poll visemes. That's more of a feature for standalone games that don't have their own visemes engine. |
|
@dirtball-official The lints must be resolved for the checks to pass |
Pico has a "Lipsync" face tracking mode which uses headset audio to drive face tracking blendshapes when it detects audio. Typically applications (such as VRChat) already have this sort of functionality, and face tracked avatars typically have options to enable/disable it in-game. Having face tracking blendshapes overridden by the headset without any user control is not expected behavior and causes confusion when troubleshooting avatar face tracking.
See the Pico documentation for details.
https://developer.picoxr.com/document/unity-avatar/face-tracking/