Diffusion-Vas: Tracking Video Objects with Occlusion Completion

AIbase基地

Published inAI News · 5 min read · Dec 17, 2024

241

In the field of video analysis, the persistence of objects is an important clue for humans to understand that objects still exist even when completely occluded. However, current object segmentation methods mostly focus on visible (modal) objects and lack the handling of non-modal (visible + invisible) objects.

To address this issue, researchers have proposed a two-stage method based on diffusion priors called Diffusion-Vas, aimed at improving the performance of non-modal segmentation and content completion in videos. This method can track specified targets in the video and then use a diffusion model to complete the occluded parts.

The first stage of this method involves generating non-modal masks for objects. Researchers infer the occlusion of object boundaries by combining visible mask sequences with pseudo-depth maps. The pseudo-depth maps are obtained through monocular depth estimation of RGB video sequences. The goal of this stage is to determine which parts of the objects may be occluded in the scene, thereby expanding the complete outline of the objects.

Based on the non-modal masks generated in the first stage, the second stage is responsible for content completion in the occluded areas. The research team utilizes modal RGB content and applies conditional generative models to fill in the occluded regions, ultimately generating complete non-modal RGB content. The entire process employs a conditional latent diffusion framework with a 3D UNet backbone, ensuring high fidelity of the generated results.

To validate its effectiveness, the research team benchmarked the new method on four datasets, and the results indicated that it improved the accuracy of non-modal segmentation in occluded areas by up to 13% compared to various advanced methods. Particularly in complex scenes, the research method demonstrated excellent robustness, effectively coping with strong camera motion and frequent complete occlusions.

This research not only enhances the accuracy of video analysis but also provides a new perspective on understanding the existence of objects in complex scenes. In the future, this technology is expected to be applied in various fields such as autonomous driving and surveillance video analysis.

Project: https://diffusion-vas.github.io/

Key Points:
🌟 The research proposes a new method for achieving non-modal segmentation and content completion in videos through diffusion priors.
🖼️ The method is divided into two stages: first generating non-modal masks, and then completing the content of occluded areas.
📊 In multiple benchmark tests, this method significantly improved the accuracy of non-modal segmentation, especially performing well in complex scenes.

Google AI Introduces a Fundamental Framework for Scaling Inference Time in Diffusion Models

A research team from New York University, MIT, and Google has recently proposed an innovative framework aimed at addressing the bottleneck of inference time scaling in diffusion models. This groundbreaking research transcends the traditional method of simply increasing the denoising steps, opening up new avenues for improving generative model performance. The framework primarily unfolds from two dimensions: one is leveraging feedback from validators, and the second is implementing algorithms to discover better noise candidates. The research team builds on a pre-trained SiT-XL model with a resolution of 256×256, maintaining 250 fixed denoising steps.

Microsoft Launches Innovative AI Model MatterGen to Create New Materials Based on Specific Needs

Microsoft Research has launched a powerful AI system called MatterGen that can generate new materials with specific properties, potentially accelerating the development of key technologies such as batteries and solar panels. The introduction of MatterGen signifies a fundamental shift in how scientists discover new materials. Unlike traditional methods that screen millions of existing compounds, MatterGen directly generates new materials based on desired characteristics, akin to how AI image generators create visuals from textual descriptions.

New Technology for Coloring Line Art MangaNinja: Input Line Art and Reference Images for Accurate Coloring

Recently, a line art coloring method called MangaNinja has garnered widespread attention, allowing users to color target line art based on reference images with just an input of the line art and the reference image. This technology is based on a diffusion model and focuses on reference-image-guided line art coloring, significantly enhancing the accuracy of coloring and interactive control capabilities. The research team employed two innovative designs to ensure precise transfer of character details. First, they introduced a patch rearrangement module to facilitate correspondence learning between the reference color images and the target line art. Secondly, they adopted a point-driven approach.

Single Image Transforms into 3D Wonderland: Efficiently Building High-Quality 3D Scenes with Wonderland

For a long time, efficiently generating high-quality, wide-angle 3D scenes from a single image has been a challenge faced by researchers. Traditional methods often rely on multi-view data or require time-consuming per-scene optimization, and they struggle with background quality and reconstruction of unseen areas. Existing technologies, when dealing with single-view 3D scene generation, often produce errors or distortions in occluded areas due to insufficient information, along with blurry backgrounds and difficulties in inferring the geometric structure of unseen regions. While regression-based models can synthesize new views in a feedforward manner, they struggle with complex...