Siri's New Feature: Use ChatGPT to Ask About On-Screen Content

AIbase基地

Published inAI News · 4 min read · Oct 25, 2024

Apple has released the first developer beta of iOS 18.2, featuring numerous new Apple Intelligence capabilities. One of the highly anticipated additions is the integration with ChatGPT, allowing users to ask Siri for detailed information about content displayed on their device screens.

By integrating ChatGPT into Siri, Apple's virtual assistant will utilize the OpenAI platform to answer complex queries. Additionally, it can read content from the screens of iPhones, iPads, or Macs. For instance, if you're watching a video or viewing someone's photo on Instagram, you can ask questions like "Where is this?" or "What building is in that photo?"

Source: 9to5mac

Siri captures a screenshot and uploads it to ChatGPT, which can understand the context and provide answers using its database. Of course, for privacy reasons, Siri always seeks your permission before capturing a screenshot of your device and sending it to ChatGPT.

This feature is particularly intriguing for those without the iPhone 16, as the iPhone 16 is currently the only device equipped with Visual Intelligence. With this feature, users can press "Camera Control" to get information about what they see through the camera. Visual Intelligence leverages Apple Intelligence, ChatGPT, and Google to provide answers.

Apple is also developing its own screen perception feature using the Apple Intelligence language model. However, this feature will be released with another update to iOS 18—possibly next year. Once available, Siri will understand context and act based on content on the screen—all processing is done on the device.

Source: 9to5mac

In addition to ChatGPT integration, iOS 18.2 also includes Image Playground, Genmoji, and a redesigned Mail app.

Currently, the iOS 18.2 beta is only available for devices compatible with Apple Intelligence, meaning iPhone 15 Pro or higher and M1 iPad or higher. This update will eventually be available for other devices as well.

Apple previously hinted that iOS 18.2 will be released to the public in December. iOS 18.1, which includes Apple's intelligent writing tools as well as notifications and text summaries, will be released next week.

ChatGPT-4o Major Upgrade: Creative Writing Skills Comparable to Eminem More Aligned with Needs

OpenAI has launched a major upgrade to its ChatGPT Plus subscription users with the latest large language model (LLM) upgrade - the 'Creative Writing' feature of ChatGPT-4o. According to a brief statement from OpenAI on X (formerly Twitter), the upgraded model's writing capabilities are more natural and engaging, and can be tailored to user needs, enhancing the relevance and readability of the content. With this upgrade, ChatGPT-4o is considered to have reached new heights in the field of creative writing.

Spanish BBVA Bank Successfully Deploys ChatGPT Enterprise Edition, Creates 2900 GPTs in Five Months

Spanish BBVA Bank (Banco Bilbao Vizcaya Argentaria) recently announced the successful global deployment of the ChatGPT Enterprise Edition to enhance employee efficiency and innovation. In just five months, the bank has created over 2900 customized GPT applications by closely collaborating with legal, compliance, and IT security teams to ensure the safe and responsible use of AI. BBVA Bank has always been at the forefront of technological innovation.

Stronger Writing Skills! OpenAI's Major Upgrade to GPT-4o Reclaims the Title of Best AI Model

OpenAI has announced a significant upgrade to its flagship model GPT-4o, further enhancing its performance in reasoning, cross-media, and conversational abilities. This update is referred to as ChatGPT-4o (20241120) and aims to provide users with a more natural and engaging text generation experience. Prior to the upgrade, OpenAI conducted rigorous performance testing on GPT-4o in a large language model (LLM) environment known as the 'Chatbot Arena'.

Apple is Developing a More Conversational Siri Voice Assistant Set to Launch in 2026

Recently, according to Bloomberg, Apple is developing a new voice assistant, Siri, which will utilize advanced large language models (LLMs) technology to achieve a more natural conversational experience. This move aims to close the gap with competitors, as products like Google's Gemini Live have demonstrated higher naturalness in conversational capabilities in recent years. According to sources, the new Siri assistant will completely replace the current Siri interface used by customers, with a planned release in 2026.