SpacTor-T5

Pre-trained T5 model using a combination of span corruption (SC) and replacement tag detection (RTD).

CommonProductProgrammingNLPPre-trained model

SpacTor is a new training procedure that includes (1) a mixed objective combining span corruption (SC) and replacement tag detection (RTD), and (2) a two-stage curriculum that optimizes the mixed objective in the initial \tau iterations and then transitions to standard SC loss. Experiments on various NLP tasks, using the encoder-decoder architecture (T5), show that SpacTor-T5 achieves comparable downstream performance to standard SC pre-training while reducing the pre-training iterations by 50% and the total FLOPs by 40%. Additionally, under the same computational budget, we find that SpacTor can significantly improve downstream benchmark performance.

Best AI Websites & Tools

SpacTor-T5

SpacTor-T5 Visit Over Time

SpacTor-T5 Visit Trend

SpacTor-T5 Visit Geography

SpacTor-T5 Traffic Sources

SpacTor-T5 Alternatives

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

OLMoE — An open-source expert mixture language model with 130 million active parameters.

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

Meta Llama 3.1-405B — Large multilingual pre-trained language model

Index-1.9B-Pure — A lightweight large language model focused on text generation.

Index-1.9B-Chat — A 1.9B parameter dialogue generation model

YAYI-UIE Information Extraction Large Model — High-quality information extraction model based on massive data

Qwen2 — A next-generation multilingual pre-trained model with exceptional performance.

GLM-4V-9B — Open-source multimodal pre-trained model with English and Chinese dialogue capabilities.

GLM-4-9B-Chat-1M — A new generation of open-source pre-trained model supporting multi-turn dialogue and multilingualism.

GLM-4-9B-Chat — A new generation of multilingual pre-trained model, supporting long text and code execution.

GLM-4 Series — Open-source multilingual multimodal dialogue model

CogVLM2 — Second-generation multimodal pre-trained dialogue model

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.

Llama-Chinese — Building the best Chinese Llama large model, open source and available for commercial use.

Pile-T5 — A T5 model trained on the Pile dataset

Mixtral-8x22B — A large language model based on a sparse expert framework.

Qwen1.5-32B — A series of Transformer-based pre-trained language models

HPT — HPT is an innovative multi-modal LLM framework launched by HyperGAI, designed to understand and process various input modalities including text, images, and videos.

Chronos — A pre-trained time series forecasting model based on a language model architecture

speakai.co — Transcription, Research, Data Analysis and NLP software

DataDreamer — A tool for generating synthetic data to train and align models.

Gemma-7B — A 70-Billion Parameter Language Model by Google

Gemma-2b — An open-source pre-trained language model released by Google

minbpe — A minimalist code project for BPE algorithm introduced by Karpathy