OpenEMMA

An open-source end-to-end multimodal model for autonomous driving.

CommonProductProductivityAutonomous DrivingEnd-to-End Model

OpenEMMA is an open-source project that replicates Waymo's EMMA model, providing an end-to-end framework for motion planning of autonomous vehicles. This model utilizes pre-trained Vision-Language Models (VLMs) such as GPT-4 and LLaVA to integrate text and forward-facing camera inputs, achieving precise predictions of future path points and providing decision rationale. OpenEMMA aims to provide accessible tools for researchers and developers to advance research and applications in autonomous driving.

• Integrate text and visual inputs using pre-trained Vision-Language Models (VLMs)
• Accurately predict future path points for autonomous vehicles
• Provide rationale and explanations for model decisions
• Support external tools like YOLO-3D for key object detection
• Work with various models such as GPT-4
LLaVA
Llama
and Qwen2
• Generate visualizations of predicted paths and compiled videos
• Support the nuScenes dataset for model training and testing

The target audience includes researchers and developers in the field of autonomous driving who need an end-to-end framework to implement and test driving algorithms. The open-source tools provided by OpenEMMA enable them to swiftly build their own autonomous driving systems and accelerate the R&D process through pre-trained models.

Researchers use OpenEMMA to test new autonomous driving algorithms on the nuScenes dataset.
Developers utilize the framework provided by OpenEMMA to build their own autonomous driving decision systems.
Educational institutions employ OpenEMMA as a teaching tool to demonstrate practical applications of autonomous driving technology.

1. Set up and activate the Conda environment: conda create -n openemma python=3.8; conda activate openemma
2. Clone the OpenEMMA repository: git clone git@github.com:taco-group/OpenEMMA.git; cd OpenEMMA
3. Install dependencies: pip install -r requirements.txt
4. Set up GPT-4 API access: export OPENAI_API_KEY=\your_openai_api_key\
5. Prepare input data: download and extract the nuScenes dataset

Visit

OpenEMMA Visit Over Time

Monthly Visits

494758773

Bounce Rate

37.69%

Page per Visit

5.7

Visit Duration

00:06:29

OpenEMMA Visit Trend

OpenEMMA Visit Geography

OpenEMMA Traffic Sources

OpenEMMA Alternatives

Valley 2.0 — A multimodal large language model that enhances the ability to process text, image, and video data.

Others•Multimodal•Large Language Model

108

Gemini Multimodal Live + WebRTC — A single-file application that integrates Gemini's multimodal live streaming and WebRTC technology.

Programming•WebRTC•Gemini

InternVL2_5-26B — A large multimodal language model that integrates visual and linguistic understanding.

Image•Multimodal•Large Language Model

114

Aria-Base-64K — Multimodal native Mixture-of-Experts model

Productivity•Multimodal•Long text processing

DiffusionDrive — A truncated diffusion model for real-time end-to-end autonomous driving.

Productivity•Autonomous Driving•End-to-End

NVLM 1.0 — Cutting-edge multimodal large language model

Productivity•Multimodal•Large Language Model

LLaMA-Omni — A low-latency, high-quality end-to-end speech interaction model

chatting•Speech Interaction•End-to-End Model

180

OLMoE — An open-source expert mixture language model with 130 million active parameters.

Productivity•Natural Language Processing•Expert Mixture Model

Mini-Omni — An open-source multimodal large language model that supports real-time voice input and streaming audio output.

Productivity•Multimodal•Speech Recognition

474

YAYI-UIE Information Extraction Large Model — High-quality information extraction model based on massive data

Programming•Information Extraction•Natural Language Processing

546

GLM-4V-9B — Open-source multimodal pre-trained model with English and Chinese dialogue capabilities.

InternationalSelection•Multimodal•Pre-trained Model

708

GLM-4 Series — Open-source multilingual multimodal dialogue model

Programming•Multilingual•Multimodal

360

CogVLM2 — Second-generation multimodal pre-trained dialogue model

Productivity•Multimodal•Pre-trained model

468

Meta Llama 3 — Meta's new generation of open-source large language model with excellent performance

GlobalTrending•Large Model•Open Source

4692

Gemma-2b — An open-source pre-trained language model released by Google

Productivity•Open Source•Pre-trained Model

2532

InternVL2_5-26B-MPO — A multimodal large language model that enhances the interaction between visual and linguistic data.

Image•Multimodal•Large Language Model

InternVL2_5-8B-MPO-AWQ — A multimodal large language model enhancing visual and linguistic interaction capabilities.

Image•Multimodal•Large Language Model

GitDiagram — Convert any GitHub repository into an interactive chart for project visualization.

Programming•GitHub•Project Visualization

TangoFlux — An efficient text-to-audio generation model

Music•Text-to-audio•Audio generation

InternVL2_5-8B-MPO — A large multimodal language model showcasing exceptional overall performance.

Image•Multimodal•Large Language Model

HuatuoGPT-o1-8B — An advanced large language model for the medical field.

Others•Healthcare•Reasoning

DiffSensei — Customized comic generation model, connecting multimodal LLMs and diffusion models.

Image•Comic Generation•Multimodal

144

HuatuoGPT-o1 — A large language model for complex reasoning in the medical field

Education•Medical•Complex Reasoning

InternVL2_5-4B-MPO-AWQ — A multimodal large language model designed to enhance image and text interaction capabilities.

Image•Multimodal•Large Language Model

BetterWhisperX — An automatic speech recognition tool providing word-level timestamps and speaker identification.

Programming•Automatic Speech Recognition•Word-Level Timestamps

114

VidTok — A family of open-source video segmenters from Microsoft.

Video•Video Segmentation•Video Compression

AIGCPanel Open Source AI Digital Human System — One-stop AI Digital Human System that supports video synthesis, audio synthesis, and voice cloning.

Productivity•Digital Human•Video Synthesis

162

AI Digital Avatar Synchronization System — AI Digital Lip Sync Technology, limitless white-label OEM solutions

Others•Digital Persona•Lip Sync

384

RAG-logger — An open-source logging tool for RAG applications

Programming•RAG•Logging

Best AI Websites & Tools

OpenEMMA

OpenEMMA Visit Over Time

OpenEMMA Visit Trend

OpenEMMA Visit Geography

OpenEMMA Traffic Sources

OpenEMMA Alternatives

Valley 2.0 — A multimodal large language model that enhances the ability to process text, image, and video data.

Gemini Multimodal Live + WebRTC — A single-file application that integrates Gemini's multimodal live streaming and WebRTC technology.

InternVL2_5-26B — A large multimodal language model that integrates visual and linguistic understanding.

Aria-Base-64K — Multimodal native Mixture-of-Experts model

DiffusionDrive — A truncated diffusion model for real-time end-to-end autonomous driving.

NVLM 1.0 — Cutting-edge multimodal large language model

LLaMA-Omni — A low-latency, high-quality end-to-end speech interaction model

OLMoE — An open-source expert mixture language model with 130 million active parameters.

Mini-Omni — An open-source multimodal large language model that supports real-time voice input and streaming audio output.

YAYI-UIE Information Extraction Large Model — High-quality information extraction model based on massive data

GLM-4V-9B — Open-source multimodal pre-trained model with English and Chinese dialogue capabilities.

GLM-4 Series — Open-source multilingual multimodal dialogue model

CogVLM2 — Second-generation multimodal pre-trained dialogue model

Meta Llama 3 — Meta's new generation of open-source large language model with excellent performance

Gemma-2b — An open-source pre-trained language model released by Google

InternVL2_5-26B-MPO — A multimodal large language model that enhances the interaction between visual and linguistic data.

InternVL2_5-8B-MPO-AWQ — A multimodal large language model enhancing visual and linguistic interaction capabilities.

GitDiagram — Convert any GitHub repository into an interactive chart for project visualization.

TangoFlux — An efficient text-to-audio generation model

InternVL2_5-8B-MPO — A large multimodal language model showcasing exceptional overall performance.

HuatuoGPT-o1-8B — An advanced large language model for the medical field.

DiffSensei — Customized comic generation model, connecting multimodal LLMs and diffusion models.

HuatuoGPT-o1 — A large language model for complex reasoning in the medical field

InternVL2_5-4B-MPO-AWQ — A multimodal large language model designed to enhance image and text interaction capabilities.

BetterWhisperX — An automatic speech recognition tool providing word-level timestamps and speaker identification.

VidTok — A family of open-source video segmenters from Microsoft.

AIGCPanel Open Source AI Digital Human System — One-stop AI Digital Human System that supports video synthesis, audio synthesis, and voice cloning.

AI Digital Avatar Synchronization System — AI Digital Lip Sync Technology, limitless white-label OEM solutions

RAG-logger — An open-source logging tool for RAG applications