Recently, researchers from the National University of Singapore and Purdue University have successfully developed PAB technology, enabling real-time processing of video generation based on diffusion transformers.
Product Entry:https://top.aibase.com/tool/pab
This technology represents the first attempt at a video generation model based on Diffusion Transformer (DiT), achieving a generation speed of up to 21.6 frames per second by reducing redundant attention calculations, a 10.6-fold acceleration, and applicable to multiple popular DiT video generation models, including Open-Sora, Open-Sora-Plan, and Latte, without compromising quality. PAB is a training-free method that can endow future DiT video generation models with real-time generation capabilities.
Key Features:
PAB enhances video generation speed by reducing redundant attention calculations, enabling real-time generation.
PAB sets different broadcast ranges for different types of attention based on stability and variability, minimizing quality loss while ensuring computational efficiency.
By improving sequence parallel processing technology, PAB reduces communication overhead between multiple GPUs, further enhancing the speed and efficiency of video generation.
Researchers discovered significant differences in the attention mechanisms between time steps in video diffusion transformer models, leading to the proposal of PAB to mitigate unnecessary attention calculations. In stable intermediate sections, PAB broadcasts the attention output of one diffusion step to subsequent steps, significantly reducing computational costs. Additionally, different broadcast ranges are set for different attention types to achieve more efficient calculations and minimize quality loss.
To further enhance video generation speed, researchers improved the parallel processing method based on Dynamic Sequence Parallelism (DSP), eliminating most communication overhead by broadcasting temporal attention, achieving over 50% reduction in communication overhead, and providing more efficient distributed inference capabilities for real-time video generation.
Key Points:
⭐ PAB technology enables real-time video generation with a 10.6-fold increase in processing speed.
⭐ By observing differences in the attention mechanisms of video diffusion transformer models, PAB was proposed to reduce unnecessary attention calculations.
⭐ Improved parallel processing methods significantly reduced communication overhead, providing more efficient distributed inference capabilities for real-time video generation.
