Zhipu AI Open Source Agent Task Model CogAgent-9B: Predicting Actions Through Screenshots

AIbase基地

Published inAI News · 3 min read · Dec 27, 2024

222

The GLM-PC foundational model CogAgent-9B from Zhipu AI has now been open-sourced to promote the development of the large model Agent ecosystem. CogAgent-9B is a specialized Agent task model trained based on GLM-4V-9B, capable of predicting the next GUI operation based solely on a screenshot input, combined with historical actions, according to any task specified by the user. The versatility of this model allows it to be widely applied in various GUI interaction scenarios, including personal computers, mobile phones, and in-car devices.

WeChat Screenshot_20241227091131.png

Compared to the first version of the CogAgent model open-sourced in December 2023, CogAgent-9B-20241220 has shown significant improvements in GUI perception, inference prediction accuracy, action space completeness, task universality, and generalization. It supports bilingual interactions in both Chinese and English via screenshots. The input for CogAgent includes only the user's natural language instructions, a record of executed historical actions, and GUI screenshots, without the need for any layout information or additional element tags in text form. The output includes the thought process, a natural language description of the next action, a structured description of the next action, and a sensitivity assessment of the next action.

In performance testing, CogAgent-9B-20241220 achieved leading results across multiple datasets, showcasing its advantages in GUI localization, single-step operations, Chinese step-wise rankings, and multi-step operations. This initiative by Zhipu Technology not only advances the development of large model technology but also provides new tools and possibilities for visually impaired IT professionals.

Code:
https://github.com/THUDM/CogAgent
Model:
Huggingface: https://huggingface.co/THUDM/cogagent-9b-20241220
Modao Community: https://modelscope.cn/models/ZhipuAI/cogagent-9b-20241220

Zhipu AI Becomes the First Chinese Large Model Company Listed on the US Entity List, Responds That It Will Not Be Affected

According to reports, the U.S. Department of Commerce has added Zhipu AI to its entity list for export controls, accusing it of promoting China's military modernization and infringing on U.S. national security and diplomatic interests. ** This decision makes Zhipu AI the first Chinese large model company to be added to the U.S. entity list, marking a new round of suppression by the U.S. on China's artificial intelligence industry. Zhipu AI subsequently issued a statement emphasizing that the company is based on technology transfer from Tsinghua University and provided a detailed review of its technological development history. Zhipu AI started developing the GLM pre-training architecture in 2020 and launched it in 2022.

Zhipu AI Secures 3 Billion RMB Financing to Advance Large Model Technology Development and Commercialization

Recently, Zhipu Company successfully completed a new round of financing, raising 3 billion RMB, attracting participation from multiple strategic investors and state-owned institutions, with existing shareholders such as Guanjun Capital also increasing their investment. The funds will primarily be used for the research and development upgrades of Zhipu's base large model, further enhancing its capabilities in complex reasoning, multimodal task solving, and promoting innovation and development in the industry ecosystem. This marks Zhipu's second round of financing this year. Previously, in September, Zhipu received funding support led by Zhongguancun Science City, becoming the first large model startup to achieve a valuation exceeding 20 billion RMB.

Zhipu AI Launches Free Multimodal Model GLM-4V-Flash: Enhancing Image Processing Accuracy

Beijing Zhipu Huazhang Technology Co., Ltd. announced that its Zhipu Open Platform BigModel has launched the first free multimodal API—GLM-4V-Flash. This new model leverages the excellent capabilities of the 4V series, achieving improved accuracy in image processing and further lowering the barriers for developers to delve deeper into large models across various fields.

Zhipu AI Open Source End-Side Large Language and Multimodal Model GLM-Edge Series

Zhipu Technology recently announced the open source of its end-side large language and multimodal model GLM-Edge series, marking an important attempt by the company in real-world use cases at the end side. The GLM-Edge series consists of four different model sizes, including GLM-Edge-1.5B-Chat, GLM-Edge-4B-Chat, GLM-Edge-V-2B, and GLM-Edge-V-5B, which are optimized for mobile platforms such as smartphones and vehicle systems, as well as desktop platforms like PCs.