Diffusion as Shader
A unified architectural model supporting various video generation control tasks.
CommonProductVideoVideo generation3D perception
Diffusion as Shader (DaS) is an innovative video generation control model designed to achieve diverse control over video generation through a 3D perception diffusion process. This model utilizes 3D tracking videos as control inputs, supporting multiple video control tasks under a unified architecture, including mesh-to-video generation, camera control, motion transfer, and object manipulation. The main advantage of DaS lies in its 3D perception capabilities, significantly enhancing the temporal consistency of generated videos and demonstrating powerful control abilities with minimal data and short tuning times. Developed collaboratively by research teams from institutions like the Hong Kong University of Science and Technology, the model aims to advance video generation technology, providing more flexible and efficient solutions for fields such as filmmaking and virtual reality.