MorphCut
Smooth jump cuts, eliminate filler words, output smooth video
CommonProductVideoVideo processingJump cut smoothing
This product offers a novel framework for smoothing jump cuts, especially in dialogue videos. It leverages the appearance of the subject in the video, using DensePose keypoints and facial landmark-driven intermediate representations to fuse information from other source frames. To achieve motion, it interpolates keypoints and landmarks between surrounding end frames. Then, it uses an image transformation network to synthesize pixels from the keypoints and source frames. Since keypoints may contain errors, a cross-modal attention mechanism is proposed to select and choose the most suitable source for each keypoint. By utilizing this intermediate representation, our method can achieve stronger results compared to strong video interpolation baselines. We demonstrate our method on various jump cuts in dialogue videos, such as removing filler words, pauses, and even random cuts. Our experiments show that we can achieve seamless transitions, even in challenging cases like head rotations or intense movements in dialogue heads.