FunAudioLLM: Alibaba's Open-Source Audio Generation Model Supports Emotional Voice Dialogues and Audiobooks

AIbase

Published inAI News · 3 min read · Jul 8, 2024

510

The AliTongyi Lab recently opened sourced a large-scale audio generation model project called FunAudioLLM, aiming to enhance the natural voice interaction experience between humans and Large Language Models (LLMs). The project consists of two core models: SenseVoice and CosyVoice.

CosyVoice focuses on natural voice generation, featuring multi-language support, voice and emotion control functions, and excels in multi-language voice generation, zero-shot voice generation, cross-language voice synthesis, and command execution. Trained on 150,000 hours of data, it supports Chinese, English, Japanese, Cantonese, and Korean languages, can quickly simulate voice timbres, and provide fine-grained control over emotion and rhythm.

SenseVoice is dedicated to high-precision multi-language speech recognition, emotion recognition, and audio event detection. Trained on 400,000 hours of data, it supports over 50 languages, achieving recognition results superior to the Whisper model, with improvements over 50% in Chinese and Cantonese. SenseVoice also features emotion recognition and sound event detection capabilities, as well as rapid reasoning speed.

WeChat Screenshot_20240708084503.png

FunAudioLLM supports various human-computer interaction scenarios, such as multi-language translation, emotional voice conversations, interactive podcasts, and audiobooks. By combining SenseVoice, LLMs, and CosyVoice, it can achieve seamless voice-to-voice translation, emotional voice chat applications, and interactive podcast radio stations.

In terms of technical principles, CosyVoice is based on voice quantization encoding, supporting natural and smooth voice generation, while SenseVoice provides comprehensive voice processing functions, including automatic speech recognition, language recognition, emotion recognition, and audio event detection.

The open-source models and code have been released on ModelScope and Huggingface, and training, inference, and fine-tuning code is also available on GitHub. Both the CosyVoice and SenseVoice models have online experiences on ModelScope, making it convenient for users to directly try these advanced voice technologies.

Project Address: https://github.com/FunAudioLLM

Samsung Launches New Gauss 2 AI Model, Expected to Be the Next Galaxy Brain

At this year's Samsung Developers Conference, Samsung officially unveiled the new Gauss 2 AI model, marking another significant advancement in its artificial intelligence endeavors. The Gauss 2 model upgrades in performance and efficiency, with applications ranging from smartphones to home appliances, encompassing a variety of smart devices. Gauss 2 has multimodal capabilities, allowing it to process images, text, and code simultaneously, enhancing the integration of AI into various device ecosystems. Samsung has developed three versions: Compact, Balanced, and

Stronger Writing Skills! OpenAI's Major Upgrade to GPT-4o Reclaims the Title of Best AI Model

OpenAI has announced a significant upgrade to its flagship model GPT-4o, further enhancing its performance in reasoning, cross-media, and conversational abilities. This update is referred to as ChatGPT-4o (20241120) and aims to provide users with a more natural and engaging text generation experience. Prior to the upgrade, OpenAI conducted rigorous performance testing on GPT-4o in a large language model (LLM) environment known as the 'Chatbot Arena'.

Apple is Developing a More Conversational Siri Voice Assistant Set to Launch in 2026

Recently, according to Bloomberg, Apple is developing a new voice assistant, Siri, which will utilize advanced large language models (LLMs) technology to achieve a more natural conversational experience. This move aims to close the gap with competitors, as products like Google's Gemini Live have demonstrated higher naturalness in conversational capabilities in recent years. According to sources, the new Siri assistant will completely replace the current Siri interface used by customers, with a planned release in 2026.

By 2025, Data Quality Will Become the Key to AI Success, More Important than Generative Technologies

With the rapid development of artificial intelligence (AI) technologies in industrial sectors, experts indicate that high-quality data and data governance will be more important than generative technologies. By 2025, companies must focus more on scalable and flexible solutions when adopting AI, rather than solely relying on Generative AI (GenAI). Image note: The picture is generated by AI, authorized by the image service provider Midjourney, based on the analysis by Qlik, the key to fully harnessing AI's potential lies in companies' investments in high-quality, real-time data.