Recently, Meta AI open-sourced a foundational multimodal language model named SPIRIT LM, which can freely mix text and speech, opening new possibilities for multimodal tasks involving audio and text. SPIRIT LM is based on a pre-trained text language model with 7 billion parameters, which has been continuously trained on text and speech units, expanding into the speech modality. It can understand and generate text like a large text model, while also being capable of understanding and generating speech, and even mixing text and speech to create various forms of expression.

Recently, Oute AI released a novel text-to-speech synthesis method called OuteTTS-0.1-350M. This method utilizes pure language modeling without the need for external adapters or complex architectures, offering a simplified TTS approach. OuteTTS-0.1-350M is based on the LLaMa architecture, using WavTokenizer to directly generate audio tokens, making the process more efficient. The model features zero-shot voice cloning capability, requiring only a few seconds of reference audio.

In today's rapidly advancing technology, speech synthesis technology is also progressing, especially in the field of restoring lost voices. Recently, Google researchers introduced a new technology called 'Zero-shot Voice Transfer' which can be directly integrated with state-of-the-art Text-to-Speech (TTS) systems to help those who have lost their voices due to illness or accidents regain their 'voice memory'. The core of this technology is its 'zero-shot' capability, meaning that we do not need a large number of samples to achieve this.

At today's 2024 Volcano Engine AI Innovation Tour, in addition to the video generation model, ByteDance also launched the Doubao Music Model and Doubao Simultaneous Interpretation Model, announcing significant upgrades to the Doubao General Model Pro, Text-to-Image Model, Speech Synthesis Model, and other specialized models. The introduction of the Doubao Music Model signifies Volcano Engine's deep commitment to the field of music creation. Supported by powerful algorithms, this model enables high-quality music creation freely. For lyrics generation, it can quickly generate emotional lyrics based on just a few simple input words.

The release of Fish Speech 1.4 marks a significant breakthrough in multilingual support and performance for this open-source text-to-speech (TTS) model. As an innovative solution committed to providing high-quality and natural-sounding speech synthesis experiences, Fish Speech demonstrates its strong technical capabilities and broad application prospects in this update. The most notable feature of Fish Speech 1.4 is its powerful multilingual support capability: the training data volume has doubled, enhancing the model's

Alibaba's latest CosyVoice speech synthesis model and SenseVoice speech recognition model together form the FunAudioLLM framework, aimed at enhancing human-computer interaction experience. CosyVoice, with its realistic voice generation capability, can mimic voices of different genders, ages, and personalities, adding emotions and styles while even simulating natural features such as laughter, coughing, and breathing. SenseVoice focuses on high-precision multilingual speech recognition, emotion detection, and audio event detection, supporting over 50 languages.

AI speech synthesis startup ElevenLabs has completed $80 million in Series B funding led by Andreessen Horowitz, Nat Friedman, and others. The funding will be used for product development, expanding infrastructure and teams, AI research, and enhancing security measures. ElevenLabs is valued at over $1 billion.

Bilibili UP主 'XiaoChongGe_' released a fan-made video of Genshin Impact, featuring character imitation of Frowning Na, retaining the original voice, with over 1.36 million views. The creator used the VITS speech synthesis model to upload voice text and extract language features, preserving the unique vocal characteristics of the Genshin Impact character. AI voice cloning technology has become widespread, with tools like HeyGen and AI Dubbing lowering the barriers to entry. The upgrade of AI technology has attracted the attention of creators, emphasizing the importance of producing unique works in both content and form. The article discusses the principles of the VITS model and briefly explains speech synthesis.

NetEase Youdao has launched the 'Yimosheng' open source speech synthesis engine, supporting both Chinese and English languages with over 2000 different voice tones. This engine features a distinctive emotional synthesis capability, able to synthesize voices that convey a wide range of emotions including happiness, excitement, sadness, and anger. Users can download and use it for free on GitHub and implement emotional synthesis and applications through the provided interfaces and script APIs. This project aims to help developers and content creators expand the application scope of high-quality TTS. NetEase Youdao has also introduced voice customization and voice replication.