Recently, a research team from the University of Illinois at Urbana-Champaign, Sony AI, and Sony Group introduced a new technology called MMAudio, which aims to achieve high-quality audio synthesis from videos through multi-modal joint training. The core innovation of MMAudio is its ability to generate synchronized audio using video and text inputs, thereby expanding the application scenarios for audio generation. MMAudio is designed to support generating sound effects that align with video content from either video or text inputs.

Recently, Johns Hopkins University and Tencent AI Lab jointly launched a new text-to-audio generation model named EzAudio. This technology promises unprecedented efficiency and high-quality text-to-speech conversion, marking a significant leap in artificial intelligence and audio technology. EzAudio operates by utilizing the latent space of audio waveforms instead of traditional spectrograms, an innovation that allows it to work at high temporal resolution without the need for an additional neural vocoder. The architecture of EzAudio is referred to as EzAu.