Researchers from the University of California, San Diego, and the Massachusetts Institute of Technology have developed a project called Open-TeleVision, which sounds pretty cool. This thing is an open-source remote operating system that boasts the ability to easily control robots from 3,000 miles away, allowing precise manipulation of various objects, just like the high-tech scenes in the movie "Avatar".

Firstly, let's talk about the adaptability of this system, which is truly unmatched. With any of these devices—Vision Pro, Quest, Mac, iPad, or iPhone—you can easily get started. It supports real-time stereoscopic video streams, allowing you to clearly see space and depth, making the control smooth and almost addictive.

Core Concept

The biggest selling point of the Open-TeleVision system is that you can remotely control robots through VR headsets without needing any additional wearable devices. It simulates human binocular stereoscopic vision and active neck movement, making you feel like you are the robot, with an immersive experience that's overwhelming.

Technical Highlights

  • Immersive Experience: Say goodbye to the mediocrity of traditional 2D videos, as VR headsets give you 3D videos with different images for each eye, creating a natural stereoscopic vision and precise spatial distance.

  • Active Neck: The robot has an active neck that simulates the focusing method of the human eye. You can simply turn your head to observe different areas, making the operation as intuitive as playing a game.

  • Computational Efficiency: Focusing on a small part of the image center, it reduces the processing demand for high-resolution wide-angle images, resulting in a direct increase in computational efficiency and system frame rate.

Implementation Method

  • Using inverse kinematics algorithms to map the API provided by the VR headset to the robot, achieving precise transmission of actions.

  • The system is based on the Web platform, allowing access through any device's browser at any time, with low latency and high efficiency, making remote operation stress-free.

Challenges and Solutions of Teleoperation

  • Freedom of Movement Matching: Solving the issue of mismatched freedom of movement between humanoid robots and humans through the mapping process, making it possible for almost any robot to find a suitable mapping method.

  • Data Collection and Learning: Collecting data through teleoperation and training the robot to complete tasks autonomously, with significant effects, making the system's reliability and generalization ability excellent.

Project Address: https://github.com/Improbable-AI/VisionProTeleop