OpenScholar_ExpertEval

Expert evaluation interface and data assessment scripts

CommonProductEducationExpert evaluationData assessment

OpenScholar_ExpertEval is a collection of interfaces and scripts for expert evaluation and data assessment, designed to support the OpenScholar project. This project provides detailed human evaluation of model-generated text through retrieval-augmented language model synthesis of scientific literature. The background of this product is based on research projects by AllenAI, offering significant academic and technical value to help researchers and developers better understand and enhance language models.

Provides a manual evaluation annotation interface for experts to assess text generated by models.
Supports RAG evaluation
capable of evaluating retrieval-augmented generation models.
Fine-grained evaluation: Allows experts to conduct more detailed assessments.
Data preparation: Requires evaluation instances in the specified folder
supporting JSONL format.
Results database storage: Evaluation results are stored by default in a local database file.
Results export: Supports exporting evaluation results to Excel files.
Evaluation metric computation: Provides scripts to calculate evaluation metrics and consistency.
Interface sharing: Supports deployment on cloud services for sharing the evaluation interface.

The target audience includes researchers
developers
and educators
particularly those working in the fields of natural language processing and machine learning. This product is well-suited for them as it provides a platform to evaluate and improve the performance of language models
especially in synthesizing scientific literature.

Researchers use this tool to assess the accuracy and reliability of scientific literature generated by different language models.
Educators can leverage this tool to teach students how to evaluate AI-generated content.
Developers can use this tool to test and improve their own language models.

1. Set up the environment: Follow the instructions in the README to create and activate a virtual environment
and install the dependencies.
2. Prepare data: Place the evaluation instances into the `data` folder
ensuring each instance includes prompts and the completion results of two models.
3. Run the application: Start the evaluation interface using the command `python app.py`.

Visit