Publish tracking data through VMC Protocol #2461
Conversation
|
I haven't tested myself, at least seems it doesn't break anything else. |
|
https://gist.github.com/grillo-delmal/b7712e4935f017067ced3d0428536a01 here is a short python script if you want to see the data being communicated :) (requires pyliblo3) |
grillo-delmal
changed the title
|
I moved it to draft to correct the pose orientation, tested a model with vseeface and it's not good... current state: |
grillo-delmal
changed the title
|
Ok, got orientations fixed. Had to build some tooling to record the VMC communication and transform the output on the go. One thing I learned is that VMC expects for the data to be sent in Unity orientation to work correctly. Also, the actual math to make this work, I had to apply different rotations to each bone, but all where some multiple of pi/2, so it wasn't that bad. It didn't help that the model that I have available is a free one generated with VRoid, which is not intended for this purpose XD. But at least I'm happy with the end result ^^. Screencast.From.2024-10-21.02-40-01.mp4 |
| })) | ||
| .unwrap(), | ||
| ) | ||
| .ok(); |
There was a problem hiding this comment.
Might want to log errors that occur here
There was a problem hiding this comment.
I based myself on the one implemented here https://github.com/alvr-org/ALVR/blob/master/alvr/server_core/src/tracking/body.rs#L48-L60
So if it's really needed, adding error logging there could also be helpful.
There was a problem hiding this comment.
Tbh it's probably fine without error logging, was just a hunch
| Some(( | ||
| *id, | ||
| tracking_manager_lock | ||
| .get_device_motion(*id, timestamp) |
There was a problem hiding this comment.
Don't really have the time to actually look right now, how does this interact with hand tracking/multi-input protocol (I forgot what it's called sry). I.e. does it miss the hand poses? And is this before and after the removal of the hand poses in the case of protocol compat?
There was a problem hiding this comment.
You can see the multimodal compat is at line 352. that already handles everything we need. after that point you don't care if the client does or doesn't support multimodal protocol, it will always be used.
There was a problem hiding this comment.
Great. Well then I think hands won't get exported properly if you're currently using hand tracking. Or I'm missing something
There was a problem hiding this comment.
This is only sending hand position/rotation information. Even if the protocol supports hand poses (through bone position/rotation), I haven't checked how to integrate that. So if there is any data about that, the sink should just ignore it for now.
There was a problem hiding this comment.
The issue is that if you're using hand tracking then the palm/wrist pose is carried as part of the hand tracking info, meaning your code will miss it
There was a problem hiding this comment.
I'll look how it behaves with hand tracking when I get home and report back then.
There was a problem hiding this comment.
@The-personified-devil Ah i got it now, you're right. if hand tracking is active, we don't send hand DeviceMotions, you should at least get the palm pose (first element of the hand skeleton array)
There was a problem hiding this comment.
Now I see what you mean! I started toying with hand tracking and integrating it wasn't hard, but noticed that they require their own rotation corrections :p
There was a problem hiding this comment.
That's odd. controllers do need a pose offset fix, which we do from settings. since you are getting the data from the tracking_manager then the pose is already fixed, and also recentered. same for hands. so i'm not sure a separate pose fix for hands
There was a problem hiding this comment.
in the end it was an off calibration of the Right Elbow what was off. It's kinda hard to calibrate without shoulder/upper arm data, but now it should be fine.
grillo-delmal
force-pushed
the
vmc_protocol
branch
from
2d4d3bd to
6e23ce2
Compare
|
@grillo-delmal are you sure what you are doing is right? shouldn't you just follow clippy suggestion? |
|
@grillo-delmal your link is unrelated lol. in any case the PR looks good to me |
Thx for the catch :) (deleted old message cos of the wrong link, here is what actually said xD)
|
* VMC Protocol implementation * VMC Protocol: add configuration parameters * VMC Protocol: work with id's instead of strings * fix formating * Merge VMCSinkConfig into VMCConfig * Apply bone rotation comply with VMC standard. * VMC: Add toggle for orientation correction * VMC: set the correct default port * VMC: Add hand tracking * VMC: Update rotations * VMC: Weird constant collision error
* VMC Protocol implementation * VMC Protocol: add configuration parameters * VMC Protocol: work with id's instead of strings * fix formating * Merge VMCSinkConfig into VMCConfig * Apply bone rotation comply with VMC standard. * VMC: Add toggle for orientation correction * VMC: set the correct default port * VMC: Add hand tracking * VMC: Update rotations * VMC: Weird constant collision error
* wiki updates for flatpak stuff (#2401) * wiki updates for flatpak stuff * explain difference between native and flatpak steam more clearly * give examples of using hybrid graphics with other fixes * notes about wlx-overlay-s testing --------- Co-authored-by: failboat78 <[email protected]> * refactor: 🚨 Use "crate-type" * ci: ⬆️ Update platform tools to v32 * refactor(streamer): ♻️ Clarify distinction between bitrate and throughput (#2423) * chore: ⬆️ Update some dependencies * fix(client_core): 🐛 Fix potential blank render bug This bug was already mitigated on #2397 but this is the proper fix. Discovered by @nowak-pl * Add Quest 3S support * refactor(client): ♻️ Use VIEW reference space; tweak client API * feat(client): ✨ Add platform vendor shortcuts * refactor(client): ♻️ Move decoder out of ClientCoreContext; fix IDR race condition (#2421) * feat(client_core): ✨ Add `alvr_initialize_logging()` (#2439) * fix(client_openxr): 🐛 Fix crash on Focus 3 with 0 timestamp (#2440) * feat(server_openvr) NVENC AV1 support (#2438) * Initial attempt at AV1, ffplay is happy with the AV1 stream, but the Quest 3 is not. * Trim out IVF bits for NVENC AV1 * Working OSLog for iOS/macOS * Not AMD only any more * One more bounds check * Fix writing AV1 to files * Remove logging changes * No hex * tracking rewrite(1): client API * fix(server): 🐛 Fix IDR resend logic (#2403) Co-authored-by: zarik5 <[email protected]> * build: ✨ Support aarch64 Windows host * fix(client_core): 🐛 Fix wrong controller timestamp Fixes #2445 * Revert "chore: ⬆️ Update some dependencies" This reverts commit c08e67e. * fix(server_core): 🐛 Fix tilt mode origin; refactoring * tracking rewrite (2): server API * Negotiated encoding settings for OLED-based headsets (#2442) * fix(server_core): 🐛 Do not crash with Custom controllers profile * fix(client_openxr): 🐛 Fix blank flashes * feat(client): ✨ Add Pico 4 Ultra support (#2466) * fix(client_openxr): 🐛 Fix controllers profile for Pico 4 Ultra (#2470) * feat: ✨ Use generic loader for Pico >= 4 (#2471) * Create iptables folder during firewall set up if it doesn't exist. (#2457) * Publish tracking data through VMC Protocol (#2461) * VMC Protocol implementation * VMC Protocol: add configuration parameters * VMC Protocol: work with id's instead of strings * fix formating * Merge VMCSinkConfig into VMCConfig * Apply bone rotation comply with VMC standard. * VMC: Add toggle for orientation correction * VMC: set the correct default port * VMC: Add hand tracking * VMC: Update rotations * VMC: Weird constant collision error * refactor(server_openvr): ♻️ Refactor around TrackedDevice * refactor(server_openvr): ♻️ Use TrackedDevice for FakeViveTracker * fix(client_core): 🐛 Fix Pico 4 detection; add Focus Vision (#2479) * Fix SteamVR restart loops on AVP (#2484) * feat-fix(server_openvr): Add timewarped compositing to Windows OpenVR compositor (#2472) * WIP * cleanup * done * Fix linux build * nits * take 2 * refactor(client): ♻️ Move platform to alvr_system_info; rename video decoder (#2488) * feat: ✨ Reduce session file length (#2480) * feat(client): ✨ Make lobby use passthrough (#2485) * feat(server_openvr): Add Quest Pro controller and headset emulation; Use VRLink hand tracking icons for the hand tracker controllers; Misc refactoring to reduce prop.rs linecount (#2489) * feat: integrate adb port forwarding (#2451) * fix(server_openvr): Fix CPU/GPU deadlocking at 100% usage before the headset connects (#2495) * Fix CPU/GPU deadlocking at 100% usage before the headset connects * Temporary stats manager and nits * Remove optimize_game_render_latency * fix(server_openvr): 🐛 Fix prop parse fail; use key name for log * refactor: 💄 Fix dashboard spacing; refactor alvr_adb * refactor(client_openxr): ♻️ Remove redundant state and fix potential refactoring pitfall * docs: 📝 Add Steam store link * feat(xtask): ✨ Add Pico store build; fix Pico 4e and G3 * fix(adb): fix device serial parsing and use TCP (#2507) * Fix device serial number parsing * Force stream protocol to TCP when wired * feat(server_openvr): ✨ Add option to disable hand skeleton prediction (#2517) * feat(server_openvr): ✨ Add option to disable hand skeleton prediction * Add `predict` help string * feat(adb): add client flavors and autolaunch (#2515) * Add client flavours * Add client autolaunch * Apply requested changes * Fix wired connection status message * Fix wired client type help string * fix(sockets): 🐛 Ignore error 997 (overlapped IO) (#2522) * refactor(client_core): ♻️ Port FFE to Rust and remove all C++ code (#2523) * build(xtask): 💚 Pin cargo-about (#2524) * build(xtask): 💚 Really fix cargo about (#2525) * feat(client): ✨ Add passthrough alpha support (#2527) * refactor(launcher): refactor client APK installation from launcher (#2531) * Refactor client APK installation from launcher * Apply requested changes * refactor(client_openxr): ♻️ Do not recreate OpenXR session (#2519) * refactor(client_openxr): ♻️ Do not recreate OpenXR session * Address comments * docs(linux): update wiki (#2512) * docs(linux): Updates for linux wiki, make sure user knows where wiki for linux is * docs(linux): finalize wiki update --------- Co-authored-by: Leonhard Saam <[email protected]> * docs: fix wiki breaking yet again (#2540) * docs: temp fix for quest framerate throttling (#2538) * fix(xtask): 🐛 Fix package commands; update release ci (#2545) * build(xtask): 🔧 Fix Flatpak script (#2547) * chore: recreate logos as SVG (#2548) * chore: recreate logos as SVG * fix: update image link in readme to new logos * fix: convert text to paths in SVG logo * fix: 🐛 Fix package-launcher requiring openvr submodule (#2549) * fix: 🐛 Fix package-launcher requiring openvr submodule * Do not delete launch build dir with package-launcher; fix launcher installations dir * fix(dashboard): 🐛 Reject unsupported servers (#2536) * feat(client): ✨ Add reprojection support for C API (#2543) * feat(client): ✨ Add reprojection support for C API * Address comment * chore(ci): update deprecated actions (#2554) * chore(ci): update deprecated actions * chore(ci): Fancy annotations uwu * chore(ci): update ci some more * chore(xtask): Dont run submodule update (#2556) * chore(xtask): Dont run submodule update * fix(linux): Fix compilitation error without jack installed * chore: Add composited background resource (#2561) * docs(linux): fix for no steamvr dashboard on hybrid graphics (#2511) * docs(linux): fix for no steamvr dashboard on hybrid graphics * docs(linux) specify dashboard fix is for nvidia * docs(linux): note to first close the steam client before opening it with render offload * chore(cpp): Fix warnings from openvr headers * chore(xtask): Fully remove windows installer * chore: Bump rust msrv to 1.82 * chore: Fix clippy lints * chore(ci): fix cargo msrv and print clippy output * feat(common): ✨ Add `Pose * DeviceMotion` operator (#2574) * refactor(client_openxr): ♻️ Refactor around reference spaces (#2576) * refactor(client_openxr): ♻️ Refactor around reference spaces * Fix clippy * fix(client_openxr): 🐛 Fix head angular velocity (#2577) * chore: 💚 Fix CI build (#2578) * fix launcher on deck (#2585) * Made the dashboard and launcher report respective appids to the system (#2595) * fix(server_openvr): 🐛 Fix tracked toggle for controllers (#2604) * fix(server_core): 🐛 Don't trigger ALVR gestures if holding controllers (#2606) * Full range wgpu partial fix (#2597) * Full range HDR works? * Fix full range for non-HDR * Cleanup * Adjust this note * Fix CI * Fix server * Return fractions * fix(server_core): 🐛 Fix controllers disabling with lag (#2609) * fix(server_core): 🐛 Fix controllers disabling with lag * clippy * fix(client): 🐛 Fix controller angular velocity for some devices; show velocities in lobby (#2608) * fix(client): 🐛 Fix controller angular velocity for some devices; show velocities in lobby * Draw velocities only in debug builds * Add push constants consts and bound checks * Add support for Virtual Audio Cable (#2483) * docs: 📝 Mark Vive Focus Vision as supported (#2613) * fix(client): 🐛 Fix Pico wrong prediction; refactoring (#2612) * fix(server_core): 🐛 Fix server locking the session unnecessarily (#2619) * fix(client_openxr): 🐛 Fix crash on unsupported eye tracking permission on Pico (#2618) * feat: Make on connect script toggle only + disable remote dash by default (#2621) * Bump to v20.12.0 * chore: fix prepare-release ci (#2624) --------- Co-authored-by: failboat78 <[email protected]> Co-authored-by: failboat78 <[email protected]> Co-authored-by: Max Thomas <[email protected]> Co-authored-by: Vixea <[email protected]> Co-authored-by: Simon Storfors <[email protected]> Co-authored-by: Grillo <[email protected]> Co-authored-by: E1int <[email protected]> Co-authored-by: Meister1593 <[email protected]> Co-authored-by: Leonhard Saam <[email protected]> Co-authored-by: Leonhard Saam <[email protected]> Co-authored-by: Ozzy Helix <[email protected]> Co-authored-by: Issac Dowling <[email protected]> Co-authored-by: yobson1 <[email protected]> Co-authored-by: Kirottu <[email protected]> Co-authored-by: Sobble Entertainment Inc. <[email protected]>
The VMC Protocol is a protocol to send motion tracking data through OSC for avatar motion applications. It's implemented in software like VSeeFace and others.
Here is a test of it being used through the inochi-session app
Screencast.From.2024-10-16.00-05-31.mp4