Recently, Tencent AI Lab and Tencent PCG's ARC Lab jointly launched a new framework called StereoCrafter, which can convert ordinary 2D videos into high-fidelity stereoscopic 3D videos.
This innovation responds to the growing demand for 3D content, particularly in the immersive experience field. StereoCrafter fully leverages the advantages of foundational models, overcoming the limitations of traditional conversion methods, significantly enhancing the generated effects, and ensuring that the generated content meets the high-fidelity requirements of various display devices.
The core of the system is divided into two main steps. The first step involves video remapping based on depth information, extracting occlusion information while performing video transformation; the second step is the repair of the stereoscopic video. The system uses a pre-trained stable video diffusion model as its foundation and introduces a fine-tuning protocol for the stereoscopic video repair task. To handle video inputs of varying lengths and resolutions, the team also explored autoregressive strategies and slicing techniques, ensuring that the system can flexibly adapt to various input conditions.
To support training, the team established a complex data processing pipeline, generating a large-scale, high-quality dataset. During the dataset construction process, the research team selected from a vast number of stereoscopic videos and generated corresponding video depth, transformed videos, and occlusion information, ensuring that the right-side video serves as a true benchmark. These innovative methods provide a practical solution for converting 2D videos into 3D videos, enabling devices like Apple Vision Pro and other 3D displays to deliver more exciting immersive experiences.
StereoCrafter not only achieves breakthroughs in technology but also brings potential transformations to the way digital media is experienced, possibly changing how we watch and engage with digital content.
Project Link: https://stereocrafter.github.io/
Key Points:
🌟 StereoCrafter efficiently converts 2D videos into immersive stereoscopic 3D videos using new technology.
🖥️ The system consists of two main steps: depth video reconstruction and stereoscopic video repair, enhancing the generated effects.
📊 The research team constructed a high-quality dataset to support algorithm training, ensuring output quality.
