The GLM-PC base model CogAgent-9B under Zhipu AI has now been open-sourced to promote the development of the large model Agent ecosystem. CogAgent-9B is a specialized Agent task model trained based on GLM-4V-9B, capable of predicting the next GUI action based solely on a screenshot input, combining user-specified tasks with historical operations. This model's versatility makes it widely applicable to various GUI interaction scenarios such as personal computers, smartphones, and vehicle systems.

The Tsinghua University ZhiPu AI team has released a new visual language model called CogAgent, which focuses on understanding and navigating graphical user interfaces (GUIs). CogAgent uses a dual-encoder system to process complex GUI elements and text, showing outstanding performance with high-resolution inputs of 1120x1120 pixels. The model outperforms existing LLM methods in GUI navigation tasks on PC and Android platforms, and also excels in text and visual question-answering benchmarks. Potential applications include automated GUI operations, providing G

Zhipu AI has open-sourced CogAgent, a visual language model with 18 billion parameters. CogAgent excels in GUI understanding and navigation, achieving state-of-the-art general performance across multiple benchmark tests. The model supports high-resolution visual input and dialog Q&A, and can answer questions based on any GUI screenshot. CogAgent also supports OCR-related tasks, with its capabilities significantly enhanced through pre-training and fine-tuning.