Cloudflare’s Position on Google’s Crawler Bundling

Matthew Prince, chief executive of Cloudflare, traveled to London to discuss with the U.K.’s Competition and Markets Authority (CMA) the need for tighter regulation of Google’s web crawler. Prince says the CMA’s recent designation of Google as a “special status” player in search and advertising markets opens the door for broader oversight, including Google’s AI Overviews, AI Mode, Discover feed, Top Stories, and News tab.

Cloudflare is not an AI company itself, but it provides infrastructure to a large share of AI firms—Prince notes that “80% of the AI companies are our customers.” Because of that position, Cloudflare believes it can offer the regulator concrete data on how Google’s crawler operates and why it is difficult for competitors to replicate its reach.

The Problem With Bundled Crawlers

Prince contends that Google’s practice of using the same crawler for both its search engine and its AI products creates an “unfair advantage.” He explains that Google claims a “God‑given right” to access all web content without paying, leveraging the crawler it has built over “27 years.” This bundling forces website owners to opt out of both search and AI content at the same time, a scenario that is “not feasible for most,” especially media companies that rely on search traffic for a significant portion of revenue.

Blocking Google’s crawler also disables the company’s ad‑safety mechanisms, meaning that advertisements across platforms would stop working. Prince argues that this creates a “non‑starter” for publishers who might consider refusing Google’s crawler.

Calls for Competition and Payment for Content

Prince proposes that a competitive market would allow thousands of AI companies to purchase content from media outlets, rather than relying on a single free source. He praises the CMA’s move to flag Google as a potential regulatory target, describing it as a “thoughtful move” that acknowledges Google’s unique advantage.

Other industry leaders echo Prince’s concerns. Neil Vogel, CEO of People, Inc., the largest digital and print publisher in the United States, recently labeled Google a “bad actor” and highlighted that media firms feel compelled to let Google crawl their sites for AI content because of the bundled approach. Vogel’s company has adopted Cloudflare’s solution to block unpaid AI crawlers and reports that discussions are underway with several large language‑model providers.

Potential Outcomes

If the CMA adopts stricter rules, Google may be required to separate its search crawler from its AI data collection, leveling the playing field for other AI developers. Cloudflare’s data and advocacy aim to support that regulatory shift, encouraging an environment where AI firms must pay for the content they use, thereby fostering broader competition and protecting the interests of publishers.

Used: News Factory APP - news discovery and automation - ChatGPT for Business