Zhipu GLM-PC Open Experience: Upgraded Multimodal Agent for Independent Computer Operations

AIbase基地

Published inAI News · 4 min read · Jan 23, 2025

141

Beijing Zhipu Huazhang Technology Co., Ltd. recently announced that its Zhipu GLM-PC intelligent agent has been upgraded and is now officially open for public experience. As the world's first multimodal intelligent agent capable of independently operating a computer, the technology behind GLM-PC is based on Zhipu's multimodal large model, CogAgent. Users can experience this revolutionary computer assistant with just a simple press of the enter key.

Since the release of GLM-PC v1.0 on November 29, 2024, it has been in a beta testing phase. This version introduced the "Deep Thinking" mode, added logical reasoning and code generation capabilities, and also supports the Windows operating system. GLM-PC's capabilities encompass code generation, logical execution, graphical user interface (GUI) understanding, and more, showcasing its strong potential in intelligent operations.

In terms of code generation and logical execution, GLM-PC has the ability to comprehensively analyze goals and resources, generating execution roadmaps that break down large tasks into smaller, manageable sub-tasks for efficient task planning. Once the task planning is complete, the intelligent agent can activate the code generation module for iterative execution, ensuring precise task completion. Additionally, GLM-PC possesses long-thinking capabilities, allowing it to adjust and reflect in real-time, interact with users, and optimize solutions.

In the realm of image and GUI cognition, GLM-PC can accurately identify and understand elements within graphical interfaces, such as buttons and icons, and provide intelligent recommendations based on users' historical operation information. Its image semantic analysis feature can delve into complex images to extract key information, such as trends and indicators. Moreover, GLM-PC can integrate image and text information to provide users with comprehensive perceptual results, assisting them in formulating precise operational plans.

With the continuous development of artificial intelligence technology, the launch of Zhipu GLM-PC undoubtedly offers users a more efficient and intelligent computer experience, marking an important advancement in human-computer interaction.

Open Experience of GLM-PC Computer Intelligent Agent Based on CogAgent!

On January 23, 2025, the world's first publicly accessible, ready-to-use computer intelligent agent, GLM-PC, was upgraded again, attracting widespread attention. GLM-PC is based on the multimodal large model CogAgent, capable of 'observing' and 'operating' the computer like a human, assisting users in efficiently completing various computer tasks.

Zhipu AI Open Source Agent Task Model CogAgent-9B: Predicting Actions Through Screenshots

The GLM-PC base model CogAgent-9B under Zhipu AI has now been open-sourced to promote the development of the large model Agent ecosystem. CogAgent-9B is a specialized Agent task model trained based on GLM-4V-9B, capable of predicting the next GUI action based solely on a screenshot input, combining user-specified tasks with historical operations. This model's versatility makes it widely applicable to various GUI interaction scenarios such as personal computers, smartphones, and vehicle systems.

Tsinghua University Develops New Visual Language Model CogAgent to Enhance GUI Understanding and Navigation

The Tsinghua University ZhiPu AI team has released a new visual language model called CogAgent, which focuses on understanding and navigating graphical user interfaces (GUIs). CogAgent uses a dual-encoder system to process complex GUI elements and text, showing outstanding performance with high-resolution inputs of 1120x1120 pixels. The model outperforms existing LLM methods in GUI navigation tasks on PC and Android platforms, and also excels in text and visual question-answering benchmarks. Potential applications include automated GUI operations, providing G

Zhipu AI Open-Source Visual Language Model CogAgent Supports GUI Graphic Interface Q&A

Zhipu AI has open-sourced CogAgent, a visual language model with 18 billion parameters. CogAgent excels in GUI understanding and navigation, achieving state-of-the-art general performance across multiple benchmark tests. The model supports high-resolution visual input and dialog Q&A, and can answer questions based on any GUI screenshot. CogAgent also supports OCR-related tasks, with its capabilities significantly enhanced through pre-training and fine-tuning.