Popular GitHub Repositories for Mixed Reality

NeRF (Neural Radiance Fields) and NeRF in the Wild using pytorch-lightning

Official Code for ECCV 2022 paper "AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing"

Proof of concept of a mixed reality application for the Microsoft HoloLens 2 integrating object recognition using cloud computing and real-time on-device markerless object tracking. The augmented objects provide interaction using hand input, eye-gaze, and voice commands to verify the tracking result visually.

Swift AI Code & Chat for OpenAI API

[ICCV 2023] Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling

Real-time translation services in Mixed Reality. Core pieces to get audio from the mic in Unity and stream over a websocket to the Azure Translator API and process the responses - as you speak!

Running speech-to-text in a Meta Quest headset using OpenAI's Whisper tiny model

Ubiq-Genie is an open-source framework for integrating generative AI capabilities into social VR platforms. It offers a modular approach for server-assisted VR applications, enabling new types of collaborative VR experiences such as conversational agents driven by large language models and voice-driven diffusion-based texture generation.

[ECCV 2024] PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance

Source code for the BLEARVIS prototype combining CV-based and RF-based object detection.

An innovative mixed reality (MR) pipeline that integrates real-time instance segmentation and speech-guided natural language interaction. It aims to create a more intuitive and immersive experience for users interacting with virtual and real-world environments.

NeRFusion: Towards Understanding Diffusion and NeRF in Spatial Computing (Work in progress)