Video-Foley

A system for synchronized generation of sound from video

CommonProductProductivityVideo Sound SynthesisSelf-Supervised Learning

Video-Foley is an innovative system for generating sound from video. It employs root mean square (RMS) as a temporal event condition, combined with semantic tonal prompts (audio or text), to achieve high control and synchronization in video sound synthesis. The system utilizes an unsupervised learning framework that requires no manual labeling, consisting of two stages: Video2RMS and RMS2Sound, incorporating novel concepts such as RMS discretization and RMS-ControlNet, in conjunction with a pre-trained text-to-audio model. Video-Foley achieves state-of-the-art performance in aligning and controlling sound timing, intensity, timbre, and detail.

Utilizes root mean square (RMS) as a temporal feature for high control and synchronization in video sound synthesis.
Requires no manual labeling
employing a self-supervised learning framework to reduce costs and increase efficiency.
RMS-ControlNet
in conjunction with a pre-trained text-to-audio model
enables controllable audio generation.
Controls audio semantics through text prompts
such as sound sources
timbres
and details.

\Video-Foley is primarily designed for multimedia producers
video editors
and sound designers who need to synchronize audio and video during the production process to enhance user experience. This system automates the cumbersome Foley sound generation process
providing high control and flexibility
making it suitable for professional users who require precise audio synchronization and rich tonal expression.\

A video editor uses Video-Foley to generate corresponding meowing sounds for a quiet video of a cat.
A sound designer utilizes the system to create sound effects for a game with specific RMS characteristics.
Multimedia producers generate realistic keyboard typing sounds for a typing video.

Visit the DEМО page for Video-Foley.
Select or input the video and text prompts as needed.
Adjust the RMS conditions to control the intensity and characteristics of the sound.
Click the generate button
and the system will automatically produce sounds synchronized with the video.

Visit

Video-Foley Alternatives

Sparsh — Self-supervised tactile representation for vision-based tactile sensing.

Image•Self-supervised learning•Tactile sensing

HOI-Swap — Hand-Object Interaction Awareness in Video Editing

Video•Video Editing•Hand-Object Interaction

138

MimicBrush — Zero-shot image editing, mimic the style of reference images with one click

Image•Image Editing•Zero-Shot Learning

7050

DenseAV — A self-supervised audio-visual feature alignment model.

Video•Self-Supervised Learning•Audio-Visual Alignment

198

AniTalker — Transforms static portrait images and input audio into vibrant animated dialogue videos

Video•Animation•Facial Expressions

1458

Ramban.AI — An all-in-one AI platform empowering content creation, multimedia production, and intelligent collaboration.

Productivity•Artificial Intelligence•Content Creation

AV-HuBERT — A state-of-the-art auto-referenced framework for agricultural, environmental, and energy innovations.

Programming•Audio-visual processing•Self-supervised learning

378

miqu-1-70b — Miqu 1-70b is an open-source large language model.

Productivity•Transformer•GPT

1578

A Vision Check-up — Learns string relationships between models, examines the visual world

Image•Language Model•Vision

Best AI Websites & Tools