Wikimedia Foundation Warns of Bandwidth Strain from AI Crawlers

AIbase基地

Published inAI News · 5 min read · Apr 3, 2025

In recent years, the rapid development of artificial intelligence (AI) technology has led to a significant increase in bandwidth pressure on Wikimedia projects due to web crawlers. Representatives from the Wikimedia Foundation have pointed out that since January 2024, bandwidth consumption for serving multimedia files has increased by 50%. This growth primarily stems from automated programs constantly scraping content from Wikimedia's openly licensed image library to train AI models.

Wikipedia

In an open letter, Wikimedia Foundation staff members Birgit Mueller, Chris Danis, and Giuseppe Lavagetto stated that this bandwidth increase isn't due to human users, but rather the intense demands of robotic programs. They emphasized: "Our infrastructure is designed to withstand bursts of traffic from human users during high-interest events, but the traffic generated by crawlers is unprecedented, posing increasing risks and costs to us."

According to Wikimedia's statistics, approximately 65% of high-cost content traffic is generated by these crawlers, despite crawlers only accounting for 35% of page views. This is because Wikimedia's caching scheme distributes popular content to data centers worldwide to improve performance, while crawlers don't consider content popularity when accessing pages. Therefore, they request less popular content, forcing retrieval from core data centers, consuming more computing resources.

Over the past year, the issue of excessive scraping by web crawlers has garnered attention from several open-source projects. For example, Git hosting service Sourcehut, Diaspora developer Dennis Schubert, repair website iFixit, and ReadTheDocs have all expressed concerns, reflecting the excessive demands of AI crawlers in content scraping.

The Wikimedia Foundation's 2025/2026 annual plan includes the goal of "reducing crawler-generated traffic," aiming to reduce the request rate by 20% and bandwidth usage by 30%. They hope to prioritize the user experience for human users and support Wikimedia projects and contributors.

While many websites recognize that providing bandwidth for crawlers is part of doing business, the proliferation of generative AI like ChatGPT has made crawler scraping increasingly aggressive, potentially threatening the existence of source websites. The Wikimedia Foundation acknowledges that while Wikipedia and Wikimedia Commons are crucial for training machine learning models, they must prioritize the needs of human users.

To address this challenge, tools have emerged to combat excessive crawler scraping, such as the data poisoning projects Glaze, Nightshade, and ArtShield, and web tools Kudurru and Nepenthes. However, existing robots.txt protocols are not entirely effective in limiting the behavior of these crawlers, especially as they may disguise themselves as other crawlers to circumvent blocks.

Key Points:
🌐 Crawlers have increased Wikimedia bandwidth consumption by 50%, primarily due to AI model content scraping.
🤖 Approximately 65% of high-cost content traffic is generated by crawlers, although crawlers only account for 35% of page views.
📉 The Wikimedia Foundation plans to reduce crawler-generated traffic in 2025/2026, prioritizing the needs of human users.

Hugging Face Adds Handy Feature: One-Click Check for Compatible Models

Hugging Face, a leading open-source AI community platform, has launched a highly anticipated new feature: users can quickly see which machine learning models their computer hardware can run via platform settings. Users simply add their hardware information, such as GPU model, to their Hugging Face profile settings (located at the top right corner: Profile Icon > Settings > Local Apps and Hardware).

Study Claims OpenAI May Have Used O'Reilly Paid Books to Train AI Models Without Authorization

A new study has raised concerns that OpenAI may have used copyrighted O'Reilly Media books to train its latest AI models without permission. The study was released by the AI Disclosures Project, a non-profit organization founded in 2024 by media mogul Tim O'Reilly and economist Ilan Strauss. AI models can be viewed as sophisticated prediction engines that...

Meta's Use of Unpublished Books to Train AI Models Raises Concerns

Last week, The Atlantic launched a new tool to search LibGen, a database allegedly used by Meta to train its AI models. This has raised significant concerns due to the inclusion of numerous unpublished works. Writer Maris Kreizman, in an article for Literary Hub, revealed that her forthcoming essay collection was found in this database. Kreizman stated that her collection is...

High School Student Creates AI Model Evaluation Website Using Minecraft

In today's rapidly advancing AI landscape, effectively evaluating and comparing the capabilities of different generative AI models is a significant challenge. Traditional AI benchmarking methods are increasingly showing their limitations, prompting AI developers to explore more innovative evaluation approaches. Recently, a website called "Minecraft Benchmark" (MC-Bench) has emerged, uniquely leveraging Microsoft's sandbox game Minecraft to facilitate model assessment.

OpenAI Research Lead: Certain Forms of Reasoning AI Could Have Existed 20 Years Sooner

Noam Brown, Head of AI Reasoning Research at OpenAI, recently stated in a panel discussion at NVIDIA's GTC conference that certain forms of reasoning AI models "could have existed 20 years earlier if researchers had known the right approach and algorithms." He pointed out several reasons why this research direction was overlooked. Brown recounted his experience in game AI research at Carnegie Mellon University, including Pluribus, which defeated top human professional poker players. He stated that he helped create...

Microsoft CEO Says AI Models Are Becoming Commoditized; System Integration and Product Development Are Key

In a recent podcast, Microsoft CEO Satya Nadella stated that AI models are increasingly becoming standardized and commoditized as language models mature. He noted this shift is driving a strategic change in enterprise AI development, moving the focus from model research to system integration and product development. Nadella emphasized that models alone are insufficient; companies need to focus on complete system architecture and successful product creation. He believes the current AI industry is experiencing a "wave-like activity" since November 2022.

The 150,000 RMB 'Intelligent Driving King' is Here?! Leapmotor B10 Pre-sale Starts March 10th: Featuring First-Ever Laser Radar + DeepSeek and Tongyi Qianwen Dual AI Models!

The automotive market is experiencing another wave of intelligent upgrades! Leapmotor, a leading domestic new energy vehicle brand, today announced that its new model, the Leapmotor B10, will officially begin pre-sales on March 10th. This new car, upon its unveiling, immediately ignited market anticipation with its hard-core configuration of "150,000 RMB class first-ever laser radar urban intelligent driving" and cutting-edge technology of "built-in DeepSeek and Tongyi Qianwen dual AI models." It has been hailed by the industry as the 'Intelligent Driving King' of the 150,000 RMB automotive market! According to the latest pre-release information from Leapmotor, the Leapmotor B10...