Microsoft Releases OmniParser V2.0: Converting Screenshots into Structured Formats for LLM Processing

AIbase基地

Published inAI News · 5 min read · Feb 17, 2025

596

Microsoft recently released OmniParser V2.0, a new parsing tool designed to convert user interface (UI) screenshots into structured formats. OmniParser can enhance the performance of UI agents based on large language models (LLM), helping users better understand and interact with information on the screen.

The tool's training dataset includes an interactive icon detection dataset, carefully selected and automatically annotated from popular web pages to highlight clickable and actionable areas. Additionally, there is an icon description dataset aimed at linking each UI element with its corresponding functionality.

In version V2.0, OmniParser has undergone significant improvements, with an updated dataset that is larger and cleaner, and the description and localization of icons improved by 60%. According to tests, the average latency of this version has also been greatly reduced, approximately 0.6 seconds per frame on A100 devices and 0.8 seconds per frame on a single 4090 graphics card. In terms of performance, OmniParser achieved an average accuracy of 39.6 in the ScreenSpot Pro test.

Users can control a Windows 11 virtual machine using the OmniTool, which works in conjunction with OmniParser, allowing users to select suitable visual models. Currently, OmniTool supports various large language models, including multiple versions of OpenAI, DeepSeek (R1), Qwen (2.5VL), and Anthropic Computer Use, making it convenient for users to perform various operations.

OmniParser is designed to convert unstructured screenshot images into a structured list of elements, including the locations of interactive areas and potential functional descriptions of icons. Users of this tool need to possess basic analytical skills and critical thinking, as while OmniParser can extract information, the final judgment must still be made by the user. This tool can be used for various types of screenshots, including PC and mobile interfaces, demonstrating strong adaptability.

However, it is also important to note the limitations of OmniParser. The tool does not detect harmful content in the input, so users should be cautious when providing input to ensure it does not contain harmful information. Furthermore, although OmniParser only converts screenshots to text, it can still be used to build operational graphical user interface agents. Developers using OmniParser to build and operate agents must adhere to safety standards and ethical guidelines.

Model: https://huggingface.co/microsoft/OmniParser-v2.0

Project: https://github.com/microsoft/OmniParser/tree/master

Highlights:

🔍 OmniParser V2.0 is an intelligent parsing tool that converts UI screenshots into structured information, enhancing user experience.

⚡ The new version has significant improvements, with average latency reduced to 0.6 seconds per frame and an accuracy rate of 39.6%.

🔐 Users should be mindful of the safety of the input content, and developers should follow safety standards and ethical guidelines.

Tencent Releases New Patent on Training Large Language Models to Enhance Generalization and Accuracy

Recently, Tencent Technology (Shenzhen) Co., Ltd. published a patent regarding a training method and related equipment for large language models on the Tianyancha app. The patent is titled 'Training Method, Device, Computer Equipment, and Storage Medium for Large Language Models' and aims to enhance the learning capacity and accuracy of large language models through innovative training methods. In the training process of large language models, traditional methods often rely on a single text summary, which may lead to model overfitting and negatively impact the accuracy and diversity of generated content. However, Tencent's new...

Realbotix Launches Third-Party AI Integration to Enhance Humanoid Robot Interaction Experience

Realbotix, a leading manufacturer of humanoid robots and companion AI, is further expanding its technological capabilities by launching large language model (LLM) integration and advanced customization features. These new functionalities are set to debut in February 2025, allowing users to seamlessly connect with a variety of mainstream AI platforms including OpenAI's ChatGPT, Meta's Llama, Google's Gemini, and the newly launched DeepSeek R1.

AI Has Fatal Flaws in This Area! Experts: AI Should Not Be Used in Critical Applications

In the current technological environment, artificial intelligence (AI) has sparked widespread discussion. Eerke Boiten, a professor of cybersecurity at De Montfort University, states that existing AI systems have fundamental flaws in management and reliability, and therefore should not be used in critical applications. Professor Boiten points out that most current AI systems rely heavily on large neural networks, particularly generative AI and large language models (like ChatGPT). The workings of these systems are relatively complex, even though each neural...

ByteDance Launches PaSa: An Intelligent Academic Paper Search Agent Based on Large Language Models

In the field of academic research, literature retrieval is a complex and crucial information acquisition task. Researchers need the capability to handle intricate, domain-specific search queries to meet detailed research requirements. However, existing academic search platforms, like Google Scholar, often struggle to address these complex research queries. For example, queries related to non-stationary reinforcement learning using the UCB method require stronger computational and analytical capabilities. Moreover, researchers typically need to spend a significant amount of time and effort manually browsing extensive academic literature when conducting literature reviews.