A BuzzStream study revealed that 79% of top news publishers now block at least one AI training bot via robots.txt. While publishers protect their content, this creates a significant opportunity for small businesses willing to be visible to AI.
The Research: Who Is Blocking and Why
The numbers paint a clear picture of the current AI crawler landscape:
- 79% of top news sites block AI training bots
- 71% also block AI retrieval bots used for answering questions
- 67% block PerplexityBot specifically
- Only 14% of publishers block all AI bots completely
- 18% do not block any AI bots at all
The primary reason: the crawl-to-referral imbalance. In June 2025, OpenAI crawl-to-referral ratio was 1,700:1, and Anthropic was 73,000:1. Publishers invest in creating content and AI companies harvest it while sending almost no traffic back.
The Small Business Opportunity
When major content sources block AI crawlers, AI assistants need alternative sources of information. This is where small businesses can step in.
Think about it from the AI perspective: if 79% of news sources are blocked, the AI needs to find information from the remaining 21% plus other available sources. If your business website is accessible, well-structured, and provides valuable information, you become a more prominent potential source.
How to Capitalize on This Gap
- Ensure AI accessibility: Verify your hosting provider is not blocking AI crawlers without your knowledge
- Create comprehensive content: Blog posts, service pages, FAQs, and guides that answer common questions
- Implement schema markup: Help AI understand your business, services, and expertise
- Add llms.txt: Provide a machine-readable summary of your business for AI assistants
- Build authority signals: Reviews, citations, and directory listings that AI cross-references
The Numbers That Matter
While 5.6+ million websites now block GPTBot, that still leaves hundreds of millions that do not. The question for your business is simple: do you want to be in the accessible group or the invisible group?
At AppWT, we have helped Michigan businesses implement AI Visibility Optimization (AIVO) strategies that make them discoverable by AI assistants. The businesses that move first gain the strongest advantage.
Taking Action
The AI visibility landscape changes rapidly. What matters today is having a clear strategy that balances content protection with discoverability. The businesses that understand this balance are the ones AI assistants recommend. Learn more about our AIVO services.
Tags
Frequently Asked Questions
Why are news sites blocking AI crawlers?
News publishers are blocking AI crawlers primarily to protect their content from being used to train AI models without compensation. The crawl-to-referral ratio for OpenAI is 1,700:1, meaning OpenAI crawls 1,700 pages for every one referral it sends back.
How does news sites blocking AI help small businesses?
When major content sources block AI, AI assistants need to find information elsewhere. If your small business provides clear, accessible, well-structured content that AI can access, you become a more prominent source for AI recommendations.
What percentage of websites block AI crawlers?
About 5.6 million websites block OpenAI GPTBot and 5.8 million block ClaudeBot as of 2025. Tollbit reported a 336% increase in sites blocking AI crawlers over the past year.
Enjoyed this article?
Share it with your network
