79% of News Sites Block AI Bots: The Opportunity for Small Business

A BuzzStream study revealed that 79% of top news publishers now block at least one AI training bot via robots.txt. While publishers protect their content, this creates a significant opportunity for small businesses willing to be visible to AI.

The Research: Who Is Blocking and Why

The numbers paint a clear picture of the current AI crawler landscape:

• 79% of top news sites block AI training bots
• 71% also block AI retrieval bots used for answering questions
• 67% block PerplexityBot specifically
• Only 14% of publishers block all AI bots completely
• 18% do not block any AI bots at all

The primary reason: the crawl-to-referral imbalance. In June 2025, OpenAI crawl-to-referral ratio was 1,700:1, and Anthropic was 73,000:1. Publishers invest in creating content and AI companies harvest it while sending almost no traffic back.

The Small Business Opportunity

When major content sources block AI crawlers, AI assistants need alternative sources of information. This is where small businesses can step in.

Think about it from the AI perspective: if 79% of news sources are blocked, the AI needs to find information from the remaining 21% plus other available sources. If your business website is accessible, well-structured, and provides valuable information, you become a more prominent potential source.

How to Capitalize on This Gap

1. Ensure AI accessibility: Verify your hosting provider is not blocking AI crawlers without your knowledge
2. Create comprehensive content: Blog posts, service pages, FAQs, and guides that answer common questions
3. Implement schema markup: Help AI understand your business, services, and expertise
4. Add llms.txt: Provide a machine-readable summary of your business for AI assistants
5. Build authority signals: Reviews, citations, and directory listings that AI cross-references

The Numbers That Matter

While 5.6+ million websites now block GPTBot, that still leaves hundreds of millions that do not. The question for your business is simple: do you want to be in the accessible group or the invisible group?

At AppWT, we have helped Michigan businesses implement AI Visibility Optimization (AIVO) strategies that make them discoverable by AI assistants. The businesses that move first gain the strongest advantage.

Taking Action

The AI visibility landscape changes rapidly. What matters today is having a clear strategy that balances content protection with discoverability. The businesses that understand this balance are the ones AI assistants recommend. Learn more about our AIVO services.