Search AI Products and News

Recently, an investigation revealed that hundreds of websites attempted to block the AI company Anthropic from scraping their content, but due to outdated directives, they ended up blocking the wrong bots. This phenomenon highlights the challenges website owners face in coping with the ever-evolving ecosystem of AI crawlers.

Anthropic's ClaudeBot web crawler frequently accessed the iFixit website within 24 hours, allegedly violating terms of use by unauthorized content usage and consuming development resources. iFixit has added a crawl-delay extension in robots.txt to limit crawler access. This incident also affected other websites, such as Read the Docs and Freelancer.com. AI companies like OpenAI refuse crawler access through robots.txt.