Rhymes AI Open Source Video Generation Model Allegro: Transform Text into HD Videos in Seconds

AIbase基地

Published inAI News · 4 min read · Oct 22, 2024

352

Recently, AI company Rhymes AI has officially open-sourced its advanced text-to-video model, Allegro. Allegro enables users to transform simple text descriptions into high-quality short video clips, opening up new possibilities for creators, developers, and researchers in the field of AI-generated video.

Allegro can generate high-quality videos at 720p resolution, 15 frames per second, and 6 seconds in length based on user-provided text prompts, covering a variety of film themes, from close-ups of people and animals to action scenes in various settings, almost any scene described in text.

The core technologies of Allegro include large-scale video data processing, compressing raw videos into visual tokens, and an extended video diffusion Transformer.

In terms of large-scale video data processing, Rhymes AI has designed a systematic data processing and filtering pipeline to extract training videos from raw data and developed a structured data system for multidimensional classification and clustering of the data, facilitating model training and fine-tuning.

For compressing videos into visual tokens, Allegro uses a Video Variational Autoencoder (VideoVAE) to compress raw videos into smaller visual tokens while retaining necessary details, achieving smoother and more efficient video generation. VideoVAE is built on a pre-trained image VAE and extends the spatial-temporal modeling layers, effectively leveraging spatial compression capabilities.

Regarding the extended video diffusion Transformer, Allegro's core is its extended diffusion Transformer architecture, which applies diffusion models to generate high-resolution video frames, ensuring the quality and fluidity of video motion. Allegro's backbone network is built on the DiT (Diffusion Transformer) architecture, featuring 3D RoPE position embeddings and a 3D full attention mechanism. Compared to traditional diffusion models using a UNet architecture, the Transformer structure is more conducive to model scaling. By leveraging the 3D attention mechanism, DiT can simultaneously process the spatial dimensions of video frames and their temporal evolution, providing a more nuanced understanding of motion and context.

Rhymes AI states that Allegro is just the beginning, and the team is actively developing more advanced features, including image-to-video generation, motion control, and support for longer, narrative-based, storyboard-style video generation.

To make AI-driven video creation more accessible to a wider audience, Rhymes AI has open-sourced Allegro's model weights and code, encouraging the community to explore, unleash creativity, and build upon it, aiming for collaborative progress in AI-generated video technology.

Project link: https://github.com/rhymes-ai/Allegro

Major Overhaul of US Technology Policy! AI Experts Take Charge of the White House, Trump Restructures Technology Team

The US technology policy sector is undergoing significant adjustments. Recently, Michael Kratsios, a researcher at the Stanford Institute for Human-Centered Artificial Intelligence (Stanford HAI), was appointed as the Director of the White House Office of Technology Policy, and will also serve as the incoming Presidential Technology Advisor for the Trump administration. This appointment has garnered widespread attention, marking the growing influence of top AI researchers on national technology policy. Trump is bringing back former officials to restructure the technology team, in addition to Michael Kratsios.

AI 'Artists' Break Through! Works Auctioned for Millions, Disrupting the Traditional Art World

The future of art may no longer belong solely to humans. This year, the application of artificial intelligence (AI) has permeated the daily lives of millions of people worldwide, transforming from a science fiction concept into a practical tool that simplifies everyday activities and optimizes user time. Currently, a new phenomenon is emerging: AI 'artists' are raising millions of dollars by 'creating works that are more interesting than those of humans,' undoubtedly disrupting traditional modes of artistic creation. An innovative AI design program called Botto has sold its generated computer works for millions of dollars, which sets the scene for a revolution in the art world.

AI Model's 'Hallucination' Becomes an Innovation Driver, Helping Scientists Win Nobel Prize

A characteristic widely regarded as a flaw of artificial intelligence - 'hallucination' - has surprisingly become key to scientific breakthroughs. According to Fortune magazine, Professor David Baker, one of the winners of the 2024 Nobel Prize in Chemistry, achieved significant research breakthroughs using the 'hallucination' feature of AI models. Professor Baker, from the University of Washington, is the only American scientist among this year's three Nobel Prize winners in Chemistry. He successfully invented a novel protein using computer software, an achievement that ultimately earned him this honor. Image source note: The image is AI-generated and authorized.

Indeed the Strongest! OpenAI's New Model o3 Sets a Record Score in ARC-AGI Benchmark Test

The latest model o3 released by OpenAI has achieved remarkable results in the ARC-AGI benchmark test, scoring an impressive 75.7% under standard computing conditions, and even reaching 87.5% under high-computation versions. This achievement has surprised the AI research community, but it still does not prove that artificial general intelligence (AGI) has been solved. The ARC-AGI benchmark test is based on the Abstract Reasoning Corpus and aims to evaluate AI systems' adaptability to new tasks.