CogAgent

An open-source end-to-end visual language model (VLM) based GUI agent

CommonProductProgrammingVisual Language ModelGUI Agent

CogAgent is a GUI agent based on visual language models (VLM) that facilitates bilingual (Chinese and English) cloud interaction through screenshots and natural language. CogAgent has made significant advancements in GUI perception, inference prediction accuracy, operational space integrity, and task generalization. The model has been applied in ZhipuAI's GLM-PC product, with the aim of aiding researchers and developers in advancing the research and application of GUI agents based on visual language models.

Supports bilingual (Chinese and English) cloud interaction via screenshots and natural language.
Demonstrates significant advantages in GUI perception
inference prediction accuracy
operational space integrity
and task generalization.
The CogAgent-9B-20241220 model is based on GLM-4V-9B
a bilingual open-source VLM foundational model.
Supports multi-stage training and policy improvement to achieve accuracy in GUI perception and inference prediction.
Model outputs follow strict formatting and are returned as strings; JSON output is not supported.
Does not support continuous dialogue but allows for sequential execution of history.

\The target audience includes researchers and developers
particularly those seeking efficient solutions in the fields of GUI automation
visual language models
and natural language processing. The advanced technology provided by CogAgent can assist them in developing and researching GUI agents based on visual language models
advancing the development and application of related technologies.\

Researchers conducting experiments on GUI perception and inference prediction using the CogAgent model.
Developers utilizing CogAgent to automate operations in desktop applications.
Businesses employing the CogAgent model to optimize customer service processes
enhancing efficiency through automated GUI operations.

1. Ensure that Python 3.10.16 or higher is installed and install the dependencies listed in requirements.txt.
2. Run the model using appropriate command-line parameters based on the desired output format and platform.
3. Provide the required input images to the model and receive the output containing operational instructions.
4. If the model returns results with bounding boxes
the output will include images indicating the execution areas.

Visit

CogAgent Visit Over Time

Monthly Visits

494758773

Bounce Rate

37.69%

Page per Visit

5.7

Visit Duration

00:06:29

CogAgent Visit Trend

CogAgent Visit Geography

CogAgent Traffic Sources

CogAgent Alternatives

Vanna — Simplify database interactions through SQL query generation with natural language.

Programming•SQL•Natural Language Processing

618

Quizify — An AI-powered quiz generator that simplifies the assessment creation process for educators.

Productivity•Artificial Intelligence•Natural Language Processing

126

FYRAN — Future Chatbot

Productivity•Chat•Artificial Intelligence

BotSquare — Artificial Intelligence Software Development Company

Productivity•Artificial Intelligence•Software Development

FlagAI — A comprehensive open-source project for large model algorithms, models, and optimization tools.

Programming•Artificial Intelligence•Large Models

Video Note Generator — Convert videos into high-quality Xiaohongshu notes with one click.

Writing•Video to Text•Content Optimization

138

DeepSeek-VL2-Tiny — Advanced Large-scale Mixture of Experts Visual Language Model

Image•Visual Language Model•Mixture of Experts

CosyVoice Speech Generation Model 2.0-0.5B — Efficient, multilingual speech synthesis model

Music•Speech Synthesis•Artificial Intelligence

210

WePOINTS — WePOINTS project offers a unified framework for multimodal models.

Programming•Multimodal•Artificial Intelligence

Devin.ai — AI programming assistant that aids engineering teams in increasing efficiency.

InternationalSelection•AI programming assistant•code migration

126

OLMo-2-1124-7B-RM — A large language model for text generation and classification.

Productivity•Artificial Intelligence•Natural Language Processing

PaliGemma 2 — PaliGemma 2 is a powerful visual language model that is easy to fine-tune.

Productivity•Visual Language Model•Machine Learning

OLMo-2-1124-13B-DPO — High-performance English language model suitable for diverse tasks.

Programming•Language Model•Natural Language Processing

ProactiveAgent — A proactive agent based on large language models that predicts user needs and offers assistance.

Programming•Artificial Intelligence•Machine Learning

Ai2 OpenScholar — Enhanced retrieval-based language model for scientific literature.

Productivity•Artificial Intelligence•Scientific Literature

OLMo 2 — State-of-the-art fully open language model

Programming•Language Model•Natural Language Processing

Nfig — API for executing tasks on the web via AI agents

InternationalSelection•AI Agents•Automation

234

Llama-3.1-Tulu-3-8B — An advanced instruction-following model that provides open-source data and code.

Programming•Natural Language Processing•Text Generation

LLaVA-o1 — A visual language model capable of step-by-step reasoning.

Productivity•Visual Language Model•Step-by-Step Reasoning

Sandra AI — Sandra AI, an intelligent voice agent designed specifically for automotive distribution.

Business•Artificial Intelligence•Voice Agent

Qwen Turbo 1M Demo — The Qwen Turbo 1M Demo is a Hugging Face space provided by Qwen.

Productivity•Natural Language Processing•Machine Learning

LLaMA-Mesh — Unified 3D Mesh Generation with Language Models

Productivity•3D Modeling•Artificial Intelligence

120

CR-Mentor — AI-driven code review mentor that enhances code quality and review efficiency.

Programming•Code Review•Artificial Intelligence

216

Squire AI — A new approach to code review, enhancing code quality and consistency.