ModernBERT-base

Efficient bidirectional encoder model for processing long texts.

CommonProductProgrammingBERTLong Text Processing

ModernBERT-base is a modern bidirectional encoder Transformer model pretrained on 2 trillion English and code samples, natively supporting up to 8192 tokens of context. The model incorporates cutting-edge architectural improvements such as Rotary Positional Embeddings (RoPE), Local-Global Alternating Attention, and Unpadding, showing exceptional performance on long-text processing tasks. It is ideal for processing long documents for tasks such as retrieval, classification, and semantic search within large corpuses. Since the training data is primarily in English and code, its performance may be reduced when handling other languages.

Supports long-text processing for sequences up to 8192 tokens
Rotary Positional Embeddings (RoPE) for extended context support
Local-Global Alternating Attention for improved efficiency on long inputs
Unpadding and Flash Attention for optimized inference speed
Pretrained on extensive text and code datasets
Eliminates the need for token type IDs
simplifying downstream tasks
Supports Flash Attention 2 for greater efficiency

The target audience includes developers
data scientists
and researchers who need to handle long textual data. ModernBERT-base is particularly suited for natural language processing
code retrieval
and hybrid (text + code) semantic search scenarios due to its capabilities in processing lengthy texts and optimization for English and code data.

Information retrieval from large-scale documents
Semantic search within codebases to find related functions or modules
Text classification and semantic search within large corpuses

1. Install the transformers library: Use pip to install git+https://github.com/huggingface/transformers.git.
2. Load the model and tokenizer: Use AutoTokenizer and AutoModelForMaskedLM to load the tokenizer and model from the pretrained model.
3. Prepare the input text: Feed the text to be processed into the tokenizer to obtain the input format required by the model.
4. Model inference: Pass the processed input data to the model for inference.
5. Obtain prediction results: For Masked Language Model tasks

Visit

ModernBERT-base Visit Over Time

Monthly Visits

20899836

Bounce Rate

46.04%

Page per Visit

5.2

Visit Duration

00:04:57

ModernBERT-base Visit Trend

ModernBERT-base Visit Geography

ModernBERT-base Traffic Sources

ModernBERT-base Alternatives

EXAONE 3.5 — State-of-the-art AI model offering exceptional instruction-following and long text processing capabilities.

Programming•AI Models•Long Text Processing

Qwen2.5-Coder-3B-Instruct-GPTQ-Int4 — 3B parameter instruction-tuned model within the Qwen2.5-Coder series

Programming•Code Generation•Code Inference

Qwen2.5-Coder-32B-Instruct-GPTQ-Int8 — Open-source code generation language model

Programming•Code Generation•Long Text Processing

Qwen2.5-Coder-32B-Instruct-GPTQ-Int4 — An open-source language model for code generation with 3.25 billion parameters.

Programming•Code Generation•Code Inference

Qwen2.5-Coder-32B-Instruct-AWQ — Open-source large language model for code generation

Programming•Code Generation•Code Inference

Qwen2.5-Coder-32B — The open-source code generation model in the Qwen2.5-Coder series with the most parameters.

Programming•Code Generation•Code Reasoning

kelindar/search — A Go language library for embedded vector search and semantic embedding.

Programming•Search Engine•Artificial Intelligence

Mistral-Small-Instruct-2409 — High-performance instruct-tuned AI model

Programming•Artificial Intelligence•Natural Language Processing

Reader-LM — Converts web HTML content into clear Markdown format.

Productivity•HTML Conversion•Markdown

AI21-Jamba-1.5-Mini — High-performance long text processing AI model

Productivity•Text generation•Long text processing

Jamba 1.5 Open Model Family — High-performance AI model for long text processing

InternationalSelection•AI models•Long text processing

150

IPAdapter-Instruct — A model for image generation.

Image•Image Generation•Transformer Model

438

MusiConGen — A Transformer-based text-to-music generation model

Music•Text-to-music•Transformer model

276

ViTMatte — Enhanced Image Segmentation with a Pretrained Pure Vision Transformer

Image•Image Segmentation•Vision Transformer

258

InternLM-XComposer2.5 — 7B parameters text-image understanding and synthesis model

Productivity•Text-Image Understanding•Long Text Processing

270

GenAU — Audio Generation and Automatic Captioning Model

Productivity•Audio Generation•Automatic Captioning

120

InternLM2.5-7B-Chat-1M — A 7 billion parameter long-context dialogue model

chatting•Dialogue Model•Long Text Processing

156

InternLM2.5-7B-Chat — A high-performance 7 billion parameter dialogue model

chatting•Dialogue Model•Reasoning Ability

174

4M — Multi-modal and Multi-task Model Training Framework

InternationalSelection•Multi-modal learning•Transformer model

114

VideoLLaMA2-7B-Base — A large video language model that provides visual question answering and video captioning capabilities.

Video•Video Analysis•Multi-Modal Learning

666

GLM-4-9B-Chat — A new generation of multilingual pre-trained model, supporting long text and code execution.

Programming•Pre-trained model•Multilingual support

378

Llama-3 70B Gradient 524K Adapter — The Llama-3 70B LoRA Adapter, extending context length beyond 524K.

Programming•LoRA•Transformers

Llama-3 70B Instruct Gradient 1048k — A high-performance language model developed by the Gradient AI team, supporting long text generation and dialogue.

Programming•Language Model•Long Text Processing

240

Unichat-llama3-Chinese — The industry's first Chinese instruction fine-tuning model based on the Llama3, supporting long text input and achieving high-quality Chinese question and answer.

chatting•Artificial Intelligence•Natural Language Processing

330

Llama-3 8B Instruct 262k — A high-performance text generation model developed by the Gradient AI team.

Productivity•Text Generation•Long Text Processing

258

GRM — GRM is a large-scale Gaussian reconstruction model for high-quality and efficient 3D reconstruction and generation.

Design•3D Reconstruction•3D Generation

300

Best AI Websites & Tools

ModernBERT-base

ModernBERT-base Visit Over Time

ModernBERT-base Visit Trend

ModernBERT-base Visit Geography

ModernBERT-base Traffic Sources

ModernBERT-base Alternatives

EXAONE 3.5 — State-of-the-art AI model offering exceptional instruction-following and long text processing capabilities.

Aria-Base-64K — Multimodal native Mixture-of-Experts model

LLNL/LUAR — A Transformer-based author representation learning model

Qwen2.5-Turbo — An advanced language model for efficient long text processing.

Qwen2.5-Coder-3B-Instruct-GPTQ-Int4 — 3B parameter instruction-tuned model within the Qwen2.5-Coder series

Qwen2.5-Coder-32B-Instruct-GPTQ-Int8 — Open-source code generation language model

Qwen2.5-Coder-32B-Instruct-GPTQ-Int4 — An open-source language model for code generation with 3.25 billion parameters.

Qwen2.5-Coder-32B-Instruct-AWQ — Open-source large language model for code generation

Qwen2.5-Coder-32B — The open-source code generation model in the Qwen2.5-Coder series with the most parameters.

kelindar/search — A Go language library for embedded vector search and semantic embedding.

Mistral-Small-Instruct-2409 — High-performance instruct-tuned AI model

Reader-LM — Converts web HTML content into clear Markdown format.

AI21-Jamba-1.5-Mini — High-performance long text processing AI model

Jamba 1.5 Open Model Family — High-performance AI model for long text processing

IPAdapter-Instruct — A model for image generation.

MusiConGen — A Transformer-based text-to-music generation model

ViTMatte — Enhanced Image Segmentation with a Pretrained Pure Vision Transformer

InternLM-XComposer2.5 — 7B parameters text-image understanding and synthesis model

GenAU — Audio Generation and Automatic Captioning Model

InternLM2.5-7B-Chat-1M — A 7 billion parameter long-context dialogue model

InternLM2.5-7B-Chat — A high-performance 7 billion parameter dialogue model

4M — Multi-modal and Multi-task Model Training Framework

VideoLLaMA2-7B-Base — A large video language model that provides visual question answering and video captioning capabilities.

GLM-4-9B-Chat — A new generation of multilingual pre-trained model, supporting long text and code execution.

Llama-3 70B Gradient 524K Adapter — The Llama-3 70B LoRA Adapter, extending context length beyond 524K.

Llama-3 70B Instruct Gradient 1048k — A high-performance language model developed by the Gradient AI team, supporting long text generation and dialogue.

Unichat-llama3-Chinese — The industry's first Chinese instruction fine-tuning model based on the Llama3, supporting long text input and achieving high-quality Chinese question and answer.

Llama-3 8B Instruct 262k — A high-performance text generation model developed by the Gradient AI team.

GRM — GRM is a large-scale Gaussian reconstruction model for high-quality and efficient 3D reconstruction and generation.