Apple's New AI System Ferret-UI 2 Refreshes UI Interaction Experience, Effortlessly Control Your Devices

AIbase基地

Published inAI News · 4 min read · Oct 28, 2024

338

Apple has recently unveiled the new generation artificial intelligence system, Ferret-UI2. This cross-platform AI assistant has made significant strides in UI element recognition, scoring 89.73 in tests, significantly ahead of GPT-4V's 77.73, showcasing exceptional performance.

The standout feature of this system lies in its intelligent understanding of user intent. Unlike traditional coordinate-based click operations, Ferret-UI2 can automatically locate and execute actions based on natural language commands from users. The research team has leveraged GPT-4V's visual capabilities to generate training data, enabling the system to better understand the spatial relationships between interface elements.

In terms of technical architecture, Ferret-UI2 employs an adaptive design, accurately recognizing UI elements across multiple platforms including iPhone, iPad, Android devices, web browsers, and Apple TV. The system is equipped with intelligent algorithms that automatically adjust image resolution and processing needs according to different platforms, ensuring local computational efficiency while maintaining information integrity.

Actual test data shows that the system performs excellently across various platforms: smooth operation on iPhone, 68% accuracy on iPad, and a success rate of 71% on Android devices. However, challenges remain in cross-device scenarios, such as switching between mobile devices and TVs or web interfaces, primarily due to differences in interface layouts across platforms.

It is worth noting that the competition in the UI interaction AI field is intensifying. Anthropic has recently upgraded Claude3.5Sonnet's UI interaction capabilities, while Microsoft has open-sourced the OmniParser tool, dedicated to converting screen content into structured data.

Apple has also introduced the CAMPHOR framework, which, through the collaboration of professional AI agents and master inference agents, further enhances the system's ability to handle complex tasks. This means that future voice assistants like Siri will be able to more intelligently complete complex tasks such as restaurant reservations without user manual intervention.

This technological breakthrough not only elevates the level of cross-device operation intelligence but also outlines a clear development roadmap for the next generation of human-computer interaction. With ongoing technological evolution, smarter and more natural human-computer interaction experiences are within reach.

Apple's 'Multimodal Alchemy Furnace' Upgraded Again! MM1.5 Enhances Text Density and Multimodal Understanding

Apple Inc. recently launched a significant update for its multimodal AI model MM1, upgrading it to version MM1.5. This upgrade represents more than just a simple change in version number; it is a comprehensive enhancement of capabilities, showcasing the model's stronger performance across various fields. The core upgrade of MM1.5 lies in its innovative data processing methods. The model utilizes a data-centric training approach, carefully selecting and optimizing the training dataset. Specifically, MM1.5 employs high-definition OCR data and synthetic image descriptions, along with optimizations.

Apple AI Research Team Discovers Limitations of Large Model Inference, Rendering OpenAI's o1 Ineffective with Just One Sentence

In the world of artificial intelligence, the reasoning capabilities of machine learning models, particularly large language models (LLMs), have been a focal point for scientists. Recently, Apple's AI research team published a paper titled 'Understanding the Limitations of Large Language Models in Mathematical Reasoning,' shedding light on these models' limitations when addressing logical problems. In the paper, researchers demonstrate this through a simple mathematical question. They first posed a question about Oliver picking kiwis: as follows: Oliver picked 44 on Friday.

Apple Launches New AI Model Depth Pro: 0.3 Seconds for Ultra-Fast 3D Depth Map Generation

Recently, Apple's AI research team released a new model called Depth Pro, marking a significant breakthrough in the field of depth estimation! This model can quickly generate high-resolution 3D depth maps from a single 2D image, and astonishingly, it does not require any camera metadata, which is a rarity in previous technologies. Depth Pro works incredibly fast, generating depth maps in just 0.3 seconds. The model can create maps with outstanding clarity at 2.25MP.

Apple Steps Back from $6.5 Billion Funding: OpenAI Faces Risks in Large-Scale Financing

Recently, OpenAI is undergoing a critical transformation, planning to shift from a non-profit lab to a for-profit company and preparing for a new round of massive funding. However, the company has been facing internal turmoil, with several executives including the Chief Technology Officer and Chief Research Officer resigning, raising concerns about OpenAI's future direction. Reportedly, Apple unexpectedly decided not to participate in OpenAI's latest $6.5 billion funding round, leading to a cooling of the financing plan. Despite this, OpenAI expects revenue to reach $300 million this year, a 17% year-on-year increase.