Recently, an investigation revealed that hundreds of websites attempted to block the AI company Anthropic from scraping their content, but due to outdated directives, they ended up blocking the wrong bots. This phenomenon highlights the challenges website owners face in coping with the ever-evolving ecosystem of AI crawlers.

Cloudflare provided website hosting customers with a method on Wednesday to block AI robots from scraping website content and using data to train machine learning models without permission. It is based on the customers' dislike of AI robots and the company's statement that it aims to help protect content creators' safety on the internet.Image Source Note: The image was generated by AI, and the image is authorized by the service provider Midjourney "We have clearly heard that customers do not wan

A study found that nearly half of popular news websites block OpenAI crawlers. Traditional print media websites are more likely to block OpenAI crawlers. New AI models may experience degradation when trained using previous models. AI crawlers are used to collect data for training language models. News organizations in Northern Hemisphere countries are more inclined to block AI crawlers.

A study by Reuters found that 48% of popular news websites block OpenAI crawlers. The issue of using copyrighted materials by artificial intelligence divides media into traditional print publications, television and radio broadcasters, and digital media. Some websites block Google and OpenAI crawlers, with northern hemisphere news media more likely to stop AI crawlers.

An increasing number of media publishers are starting to block AI crawlers from accessing content. Direqt sees an opportunity in developing AI chatbots for instant messaging applications. These chatbots help publishers directly engage with readers and enhance content participation. Direqt is collaborating with companies like OpenAI to provide generative AI experiences for publishers. These chatbots can generate an average click-through rate of 24.16%, bringing direct traffic to publishers.

Reddit may block Google's and Bing's search crawlers, potentially severing ties with Google. Reddit requires users to log in to read content. The Washington Post corrected its report about Reddit. Over 535 news organizations have chosen to block crawling of their content.

🛡️ Artists use the Kudurru tool to fight against AI web crawlers by sending false images to protect their works. 🌐 Kudurru is web-based and blocks web crawlers or sends misleading images. 🔒 Solutions are limited, calling for government intervention to ensure that AI training does not misuse artists' works.