For a long time, we have dreamed of having robots as intelligent as humans, capable of helping with household chores, engaging in conversation, and even possessing the all-encompassing abilities of Iron Man's Jarvis. However, the ideal is rich, but reality is lean. Teaching a robot to perform tasks is no easier than trying to reason with your girlfriend—it's laborious and may not yield results.
Why is this? Because the real world is complex, filled with unexpected events and changes. Just think, even teaching your girlfriend a simple principle requires a lot of effort. How much more difficult would it be to teach a robot without human thought?
Traditional methods of robot training are either too costly, requiring repeated trials in the real world and potentially causing safety issues, or they are ineffective, with robots trained in simulated environments struggling when faced with reality, much like a child with cognitive disabilities.
To address this issue, researchers at Stanford University came up with a brilliant idea: digital cousins.
What are digital cousins?
Simply put, digital cousins are virtual stand-ins for real-world objects. You can think of them as highly realistic digital models that look and function similarly to real objects but do not need to be identical.
For example, a digital cousin of a real-world cabinet should have similar handles and drawer arrangements, but the materials and details can differ. Similarly, a digital cousin of a real-world kitchen should have similar furniture placements, but the specific models can vary slightly.
Why create digital cousins? Because they have two significant advantages:
Cost reduction: Digital cousins do not need to replicate the real world as precisely as digital twins, making them simpler and cheaper to produce.
Enhanced robustness: A single real object can have multiple digital cousins, each with slight differences. This provides robots with more diverse training data, enabling them to learn to handle various changes.
How are digital cousins automatically generated?
Stanford University researchers have developed a system called ACDC, which can automatically generate digital cousin scenes from a single RGB image. This system is a boon for the lazy; you only need to take a photo, and it will generate a virtual training ground for your robot to play in.
The ACDC system's workflow is roughly divided into three steps:
Extract information: Extract object masks, depth information, etc., from the input RGB image.
Match cousins: Based on the extracted information, find the most similar digital model from the database and adjust the model's size and orientation according to the object category and features.
Generate scene: Combine the matched digital models to create a complete virtual scene and make physical adjustments to ensure the scene's stability and rationality.
Are digital cousins really useful?
Stanford University researchers conducted a series of experiments, and the results showed that robots trained with digital cousins performed better:
Simulated environment: In simulated environments, robots trained with digital cousins had higher success rates in tasks such as opening doors, drawers, and placing dishes, and were more adaptable to different types of furniture. In contrast, robots trained with digital twins tended to falter when faced with unfamiliar furniture.
Real world: In the real world, robots trained with digital cousins could be directly applied to real-world scenarios without additional fine-tuning. Robots trained with digital twins, however, needed extra adjustments to adapt to real-world differences.
The emergence of digital cousin technology opens a new door for robot learning. Future robots will be smarter and more flexible, better able to adapt to the complex and ever-changing real world.
Of course, this technology currently has some limitations, such as the limited number and variety of models in the database, and imperfect handling of some special cases. However, with technological advancements and data accumulation, these issues will be gradually resolved.
In summary, digital cousin technology holds a bright future and will propel robot technology to new heights. In the not-too-distant future, we may truly have robot companions as intelligent as humans.
Project link: https://digital-cousins.github.io/
Paper link: https://arxiv.org/pdf/2410.07408