TAG-Bench

Natural language processing benchmark for database queries

CommonProductProgrammingNatural Language ProcessingDatabase Queries

TAG-Bench is a benchmark for evaluating and researching the performance of natural language processing models in answering database queries. It is built on the BIRD Text2SQL benchmark, enhancing query complexity by incorporating semantic reasoning that leans on world knowledge or goes beyond the explicit information in the database. TAG-Bench aims to foster the integration of AI and database technologies by simulating realistic database query scenarios, providing researchers with a platform to challenge existing models.

Offers 80 complex queries based on the BIRD Text2SQL benchmark
covering matching
comparison
ranking
and aggregation queries.
Requires models to possess world knowledge or perform semantic reasoning beyond database information.
Supports the use of Pandas DataFrames to simulate a database environment.
Recommends using GPU for creating table indexes to enhance query efficiency.
Provides detailed setup guidelines
including environment creation

\TAG-Bench is primarily designed for researchers and developers in the fields of natural language processing and database studies. It is suitable for professionals looking to evaluate and enhance model performance in handling complex database queries. By utilizing TAG-Bench
they can gain insights into the strengths and weaknesses of their models and explore new algorithms and techniques to improve reasoning and query processing capabilities.\

Researchers use TAG-Bench to assess the performance of their newly developed natural language processing models in handling complex database queries.
Developers leverage TAG-Bench to test and optimize their database query processing systems to enhance their performance in real-world applications.
Educational institutions utilize TAG-Bench as a teaching tool to help students understand the application of natural language processing in database queries.

Create a conda environment and download dependencies.
Download the BIRD database and convert it to Pandas DataFrames.
Create indexes for each table (GPU usage is recommended).
Obtain Text2SQL prompts and modify the tag_queries.csv file.
Run the evaluation command in the tag directory to reproduce the results from the paper.

Visit

TAG-Bench Visit Over Time

Monthly Visits

494758773

Bounce Rate

37.69%

Page per Visit

5.7

Visit Duration

00:06:29

TAG-Bench Visit Trend

TAG-Bench Visit Geography

TAG-Bench Traffic Sources

TAG-Bench Alternatives

Large Concept Models — Language modeling in the sentence representation space

Programming•Natural Language Processing•Multilingual

OmniAudio-2.6B — The fastest edge-deployed audio language model in the world.

Productivity•Audio Processing•Edge Computing

DeepSeek-VL2-Tiny — Advanced Large-scale Mixture of Experts Visual Language Model

Image•Visual Language Model•Mixture of Experts

Megrez-3B-Omni — Open-source full-modal understanding model for edge deployment

Productivity•Full-modal understanding•Image recognition

Q-RWKV-6 32B Instruct Preview — The most powerful variant of the RWKV model, breaking multiple English benchmark tests.

Programming•Machine Learning•Natural Language Processing

CosyVoice Speech Generation Model 2.0-0.5B — Efficient, multilingual speech synthesis model

Music•Speech Synthesis•Artificial Intelligence

126

WePOINTS — WePOINTS project offers a unified framework for multimodal models.

Programming•Multimodal•Artificial Intelligence

tulu-3-sft-olmo-2-mixture — A large-scale multilingual text dataset.

Others•Multilingual•Text Dataset

Devin.ai — AI programming assistant that aids engineering teams in increasing efficiency.

InternationalSelection•AI programming assistant•code migration

102

OLMo-2-1124-7B-RM — A large language model for text generation and classification.

Productivity•Artificial Intelligence•Natural Language Processing

Project Astra — Explore the capabilities of a future general-purpose AI assistant.

InternationalSelection•Assistant•Natural Language Processing

150

InternVL2_5-38B — Advanced Multimodal Large Language Model Series

Image•Multimodal•Large Language Models

108

mwp_ReFT — A deep reinforcement learning-based model fine-tuning framework

Programming•Natural Language Processing•Deep Learning

Meta Llama 3.3 — A multilingual large pre-trained language model with 70 billion parameters.

Programming•Multilingual•Pre-trained Model

Florence-VL — Enhancement tool for visual language models, combining generative visual encoders and deep breadth fusion technology.

Programming•Visual Language Models•Multimodal Learning

ChatGPT Pro — Scalable access plan for cutting-edge AI models.

InternationalSelection•Machine Learning•Data Science

252

PaliGemma 2 — PaliGemma 2 is a powerful visual language model that is easy to fine-tune.

Productivity•Visual Language Model•Machine Learning

OLMo 2 1124 7B Preference Mixture — A large-scale textual dataset for preference mixture research.

Others•Natural Language Processing•Text Dataset

OLMo 2 1124 13B Preference Mixture — Large-scale multilingual preference mixture dataset

Others•Dataset•Multilingual

OLMo-2-1124-7B-SFT — High-performance English text generation model

Productivity•Text Generation•Natural Language Processing

OLMo-2-1124-13B-SFT — Advanced text generation model

Productivity•Text Generation•Chat

INTELLECT-1-Instruct — A language model with 1 billion parameters for English text and code.

Programming•Text Generation•Distributed Training

OLMo-2-1124-7B-DPO — An advanced text generation model supporting diverse task handling.