Cloudflare now sorts AI bots into Search, Agent and Training. Here is what changed, what robots.txt can do, and what to check this week.
Cloudflare now lets website owners control AI bots by what the bots do, not with one on or off switch. If your site uses Cloudflare, check your settings this week. Cloudflare's own pages give different answers about the defaults for new sites, and your robots.txt file should say what you really mean.
What changed at Cloudflare
A crawler is a program that visits web pages and reads them on its own. Cloudflare launched a one-click "Block AI Bots" option and a Pay-Per-Crawl marketplace. In July 2026, Cloudflare wrote on its blog that owners wanted more choices than blocking all automation every time.
The Cloudflare changelog says all customers can now manage AI crawlers by behavior, instead of a single Block AI bots toggle. The three behaviors are:
- Search: bots that build search indexes, which are the lists search tools use to find pages.
- Agent: bots that act as an agent, doing tasks for a person.
- Training: bots that collect data to train AI models. Training means feeding content to an AI so it can learn.
The feature is now called AI Crawler and Agent access policies, according to Cloudflare's developer docs. It was once called Configure AI bot policies. Each policy covers verified bots with that behavior and unverified bots that match. You can also set access for the top Personal Agents. These follow your Agents setting by default, and Cloudflare recommends Allow for them.
Why this matters for a small business
Cloudflare's blog says the real problem for a small site is often that nobody can find it. In the past, owners faced a hard choice. They could show up in search and let AI train on their content, or they could risk being harder to find.
The blog also says Google search now answers many questions right on the results page. Some crawlers do more than one job. Cloudflare encourages companies to split their bots into separate Search, Agent and Training crawlers. The docs say your Training setting also applies to mixed-purpose crawlers used for both Search and Training.
The defaults: the sources disagree
Cloudflare's pages do not agree on the default settings for new domains. Here is what each says:
- The changelog says new defaults take effect for new domains on September 15, 2026. Under those defaults, Training and Agent are blocked on pages that show ads, and Search stays allowed.
- The developer docs say Search, Agents, and Training are allowed by default for new domains.
Because of this, do not guess. Open your own dashboard and look at what is set for your site.
What robots.txt can and cannot do
A robots.txt file is a plain text file at the root of your site, like example.com/robots.txt. It gives crawlers rules about which parts of your site they may visit. Here are key points from Google:
- Google's robots.txt guide says it is not a way to keep a page out of Google. For that, use noindex or a password.
- A blocked page can still show up in search results, but without a description.
- Google's rules page says a robots.txt file only covers the host, protocol and port where it lives. A file on a subdomain only covers that subdomain.
- Google usually keeps a copy of your file for up to 24 hours.
- Most 4xx errors, except 429, are treated as if no file exists. That means no limits.
Cloudflare's docs add that robots.txt rules are voluntary, so crawlers can ignore them. When Bot Preference Sync is on, a Disallow rule also blocks AI training crawlers at Cloudflare's edge. Mixed-purpose crawlers are left out of that edge block. The changelog also says Cloudflare puts its managed robots.txt before your own, in one combined file.
Steps to take this week
- If you use Cloudflare, go to Security, then Settings. Filter by Bot traffic and select AI Crawler and Agent access policies.
- Choose an action for Search, Agent and Training. Think about whether you want to be found in AI search tools.
- Open a private browser window and go to yoursite.com/robots.txt. Make sure the rules there match your goals.
- If you edit the file, Google's how-to page says to use a plain text editor, not a word processor. Save it with UTF-8 encoding if asked.
- If you use a hosted builder like Wix or Blogger, you may not be able to edit the file. Look for a search settings page instead.
- For any page you truly want hidden, use noindex or a password, not robots.txt.
How I can help
AppWT is based in Livonia, Michigan. AppWT has built 600+ websites, by its own count. If you want help with these choices, my service is AI Visibility Optimization, starting price $2,497/mo.
Have a question about anything here? Email me at sales@appwt.com or call (888) 565-0171, and I will answer it.
Sources
- Your site, your rules: new AI traffic options for all customers, Cloudflare, 2026-07-01
- Changelog, Cloudflare Developers, 2026-04-15
- AI Crawler and Agent access policies, Cloudflare Developers, 2026-11-02
- Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers, Google Search Central, 2026-10-03
- How Google Interprets the robots.txt Specification | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers, Google Search Central, 2026-10-26
- Create and Submit a robots.txt File | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers, Google Search Central, 2026-10-03
- AI Visibility Optimization at AppWT, AppWT
Tags
Frequently Asked Questions
Does robots.txt stop all AI bots?
No. Cloudflare says robots.txt rules are voluntary, and crawlers can ignore them. With Bot Preference Sync on, a Disallow rule also blocks AI training crawlers at Cloudflare's edge, but not mixed-purpose crawlers.
What are the three AI behaviors in Cloudflare?
They are Search, Agent and Training. Search bots build search indexes, Agent bots act for a person, and Training bots collect data to train AI models.
Are AI bots blocked by default on new Cloudflare sites?
The sources disagree. The changelog says Training and Agent are blocked on pages with ads for new domains from September 15, 2026, while the developer docs say all three are allowed by default. Check your own dashboard.
Can robots.txt keep a page out of Google?
No. Google says to use noindex or a password for that. A page blocked by robots.txt can still appear in results without a description.
How do I check my robots.txt file?
Open a private browser window and go to your site address followed by /robots.txt. If you see the file's contents, it is publicly available.
Enjoyed this article?
Share it with your network
