Cloudflare Now Blocks AI Crawlers by Default: What This Means for Your Business

In a move that reshaped the AI visibility landscape, Cloudflare - the infrastructure company that powers a significant portion of the internet - began blocking AI crawlers by default in 2025. Over 80% of Cloudflare customers have kept this setting enabled.

The Facts: What Cloudflare Changed

Cloudflare introduced its AI bot management features to give website owners control over how AI companies access their content. The key changes include:

• Default blocking: AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), and Bytespider (ByteDance) are blocked by default on new Cloudflare accounts
• Pay-per-crawl option: Cloudflare launched a program allowing website owners to charge AI companies for content access
• Managed robots.txt: Cloudflare can automatically generate and manage robots.txt files for AI training crawlers

Why This Matters for Michigan Businesses

According to recent data, 58% of consumers now ask AI assistants instead of using Google for business recommendations. If your website is behind Cloudflare with default settings, AI assistants like ChatGPT, Claude, and Perplexity cannot access your content to recommend you.

This creates an invisible barrier. Your website might rank well on Google, but AI assistants have no idea you exist.

The Numbers Tell the Story

• 5.6 million websites now block OpenAI GPTBot (up from 3.3 million in July 2025)
• 5.8 million websites block Anthropic ClaudeBot
• 336% increase in sites blocking AI crawlers over the past year (Tollbit Q2 2025 report)
• 80%+ of Cloudflare customers block AI bots completely

The Opportunity for Smart Businesses

Here is the truth that many businesses miss: while millions of websites block AI crawlers, businesses that strategically allow AI access gain a strength. If similar businesses in your market are invisible to AI and you are visible, AI assistants have your business to recommend.

This does not mean allowing unrestricted access. The smart approach is selective:

• Allow AI search bots that drive discovery (like ChatGPT search, Perplexity, Google AI)
• Block AI training bots that only scrape content for model training
• Implement proper schema markup so AI understands your business
• Create llms.txt files to provide AI with structured business information

What AppWT Recommends

At AppWT, we help businesses navigate the AI visibility landscape with a balanced approach:

1. Audit your current AI visibility - Are AI crawlers blocked on your site?
2. Configure selective access - Allow search AI bots while blocking training bots
3. Implement AIVO (AI Visibility Optimization) - Structured data, schema markup, llms.txt
4. Monitor and adjust - The AI landscape changes rapidly

With 29+ years in the industry, we have seen digital landscapes change before. The businesses that adapt early always win.

How AppWT Handles This on Its Own Site

AppWT runs appwt.com on its own servers, so no host or network in front of it applies a default block. Its robots.txt gives AI crawlers such as GPTBot and OAI-SearchBot their own groups that allow the whole site.

In AppWT's logs from September 1 to 29, 2026, GPTBot made 5,414 requests and OAI-SearchBot, the crawler behind ChatGPT search, made 9,782. If a site you own shows none of these crawlers, a block somewhere along the way is the first thing to rule out.

Check Your AI Visibility Today

Want to know if your business is visible to AI assistants? Contact AppWT for a free AI visibility audit. We will tell you exactly where you stand and what you can do about it.