The Tsinghua University ZhiPu AI team has released a new visual language model called CogAgent, which focuses on understanding and navigating graphical user interfaces (GUIs). CogAgent uses a dual-encoder system to process complex GUI elements and text, showing outstanding performance with high-resolution inputs of 1120x1120 pixels. The model outperforms existing LLM methods in GUI navigation tasks on PC and Android platforms, and also excels in text and visual question-answering benchmarks. Potential applications include automated GUI operations, providing G

Zhipu AI has open-sourced CogAgent, a visual language model with 18 billion parameters. CogAgent excels in GUI understanding and navigation, achieving state-of-the-art general performance across multiple benchmark tests. The model supports high-resolution visual input and dialog Q&A, and can answer questions based on any GUI screenshot. CogAgent also supports OCR-related tasks, with its capabilities significantly enhanced through pre-training and fine-tuning.