SpacTor-T5

Pre-trained T5 model using a combination of span corruption (SC) and replacement tag detection (RTD).

CommonProductProgrammingNLPPre-trained model

SpacTor is a new training procedure that includes (1) a mixed objective combining span corruption (SC) and replacement tag detection (RTD), and (2) a two-stage curriculum that optimizes the mixed objective in the initial \tau iterations and then transitions to standard SC loss. Experiments on various NLP tasks, using the encoder-decoder architecture (T5), show that SpacTor-T5 achieves comparable downstream performance to standard SC pre-training while reducing the pre-training iterations by 50% and the total FLOPs by 40%. Additionally, under the same computational budget, we find that SpacTor can significantly improve downstream benchmark performance.

Best AI Websites & Tools

SpacTor-T5

SpacTor-T5 Visit Over Time

SpacTor-T5 Visit Trend

SpacTor-T5 Visit Geography

SpacTor-T5 Traffic Sources

SpacTor-T5 Alternatives

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

InternVL2_5-26B — A large multimodal language model that integrates visual and linguistic understanding.

FineWeb2 — Multilingual Pretrained Dataset

Meta Llama 3.3 — A multilingual large pre-trained language model with 70 billion parameters.

Aria-Base-64K — Multimodal native Mixture-of-Experts model

ViTLP — A visually guided generative text layout pre-trained model for document intelligence.

Extractous — A fast and efficient tool for unstructured data extraction

Qwen2.5-Coder-3B — The 3B parameter model in the Qwen2.5-Coder series focuses on code generation and comprehension.

Prompt Engineering — A comprehensive resource library for Prompt Engineering techniques.

GenAI_Agents — A comprehensive resource library for the development and implementation of generative AI agents.

DTLR — Handwritten text recognition and character detection model.

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

OLMoE — An open-source expert mixture language model with 130 million active parameters.

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

Meta Llama 3.1-405B — Large multilingual pre-trained language model

Index-1.9B-Pure — A lightweight large language model focused on text generation.

Index-1.9B-Chat — A 1.9B parameter dialogue generation model

YAYI-UIE Information Extraction Large Model — High-quality information extraction model based on massive data

Qwen2 — A next-generation multilingual pre-trained model with exceptional performance.

GLM-4V-9B — Open-source multimodal pre-trained model with English and Chinese dialogue capabilities.

GLM-4-9B-Chat-1M — A new generation of open-source pre-trained model supporting multi-turn dialogue and multilingualism.

GLM-4-9B-Chat — A new generation of multilingual pre-trained model, supporting long text and code execution.

GLM-4 Series — Open-source multilingual multimodal dialogue model

CogVLM2 — Second-generation multimodal pre-trained dialogue model

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.

Llama-Chinese — Building the best Chinese Llama large model, open source and available for commercial use.