Recently, Apple has encountered some challenges in rolling out new AI features. Many large websites have blocked Apple's AI crawler, forcing Apple to negotiate licensing agreements with these sites. This situation starkly contrasts with Google, which, due to its strong market influence, can exert pressure to compel publishers to allow its AI access to content.
According to a report by WIRED, well-known websites such as Facebook, Instagram, Craigslist, Tumblr, The New York Times, Financial Times, The Atlantic, Vox Media, USA Today Network, and Condé Nast are all blocking access by Apple's AI crawler, Applebot-Extended. Some of these publishers have already reached agreements with OpenAI, allowing its AI to use their content.
To address this situation, Apple recently introduced a new crawler tool, Applebot-Extended. Website operators can block this crawler via the robots.txt file. According to WIRED analysis, about 7% of the 1,000 analyzed websites have chosen to block this AI crawler. Another analysis by data journalist Ben Welsh shows that out of 1,167 primarily US-based English publications, 294 have blocked access by Applebot-Extended.
In addition to Apple, companies like Anthropic and OpenAI also face similar challenges, with OpenAI's crawler being blocked at even higher rates.
In this process, Google seems to hold more initiative. Due to its dominant position in the search engine market, Google can pressure publishers to allow its AI access to content, or risk reduced search visibility. Google strengthens this by combining its AI overview with traditional web scraping. Moreover, Google has ceased early discussions with publishers about content usage licensing agreements. Given Google's enormous daily search volume, such agreements could be very costly, further squeezing search profits generated by AI.
In contrast, while Apple also uses crawlers for its search products, publishers can block the AI crawler without affecting visibility in Apple's search results. Although this approach appears more equitable, search is not Apple's core business. As for how OpenAI's ChatGPT will be used in Apple products in the future, and how licensing agreements will operate, remains to be seen.
Key Points:
🌐 Many large websites block Apple's AI crawler, forcing Apple to negotiate licensing agreements.
🤖 Apple's newly introduced crawler tool, Applebot-Extended, has been blocked by 7% of websites.
📉 Google leverages its market dominance to pressure publishers for AI content access rights.