Apple Inc. recently launched the next generation AI system Ferret-UI 2. This cross-platform AI assistant has made significant breakthroughs in UI element recognition, achieving a test score of 89.73, far surpassing GPT-4V's score of 77.73, demonstrating exceptional performance. The most notable feature of this system is its intelligent ability to understand user intent. Unlike traditional coordinate-based interaction methods, Ferret-UI 2 can automatically locate and execute relevant actions based on users' natural language commands.

Apple, in collaboration with Cornell University, has released the open source multimodal machine learning model Ferret. Ferret is a system that can reference and locate any point in an image at any precision using any shape of area. It can identify elements within images and respond as part of the query. This open source release demonstrates Apple's commitment to impactful AI research and its willingness to collaborate more openly in AI work.

Researchers from Apple and Columbia University have developed the Ferret multimodal language model, aimed at achieving advanced image understanding and description. The Ferret model possesses strong global understanding capabilities, able to handle free text and referenced regions, outperforming traditional models. The researchers created the GRIT dataset, which includes 1.1 million samples, to guide the model in reference and localization tasks. Ferret-Bench evaluations show that Ferret outperforms the best MLLM models by an average of 20.4% while reducing object-related errors.