Weebly, the popular website builder now owned by Square, has taken a specific approach to AI crawlers by blocking ClaudeBot (Anthropic) and Bytespider (ByteDance) at the server level.
What Weebly Blocks
Weebly specifically blocks requests from two AI crawler user agents:
- ClaudeBot: Used by Anthropic to index content for Claude AI
- Bytespider: Used by ByteDance for various AI products
What This Means in Practice
When someone asks Claude "Who is the best plumber near me?" or "Can you recommend a web designer in Michigan?", Claude draws from content it has been able to access. If your site is on Weebly, Claude may not have your business information available.
Options for Weebly Users
- Supplemental content: Create presence on platforms AI can access (Google Business Profile, directory listings, social media)
- Review your platform needs: Evaluate whether Weebly still meets your business requirements in 2026
- Build authority elsewhere: Earn mentions on sites that AI can crawl
- Consider migration: Move to a platform that gives you control over AI crawler access
What These Two Crawlers Did on AppWT's Own Site
AppWT's own site runs on its own servers and allows both crawlers in its robots.txt. From September 1 to 29, 2026, requests identifying as ClaudeBot reached appwt.com 74,878 times and Bytespider 8,839 times, not counting AppWT's own traffic.
Crawling and answering are separate jobs: the same logs show Claude-User, the fetcher that loads a page while someone is chatting with Claude, at 1,278 requests. Counting both shows how much of a bot's traffic is indexing and how much comes from people asking questions.
Making the Right Decision
There is no universal right answer. Some businesses prioritize simplicity and low cost (Weebly strengths). Others prioritize maximum online visibility (where Weebly has limitations). The key is making an informed decision based on your specific business goals. Contact AppWT if you need help evaluating your options.
Tags
Frequently Asked Questions
Does Weebly block AI crawlers?
Yes. Weebly specifically blocks requests from ClaudeBot (Anthropic) and Bytespider (ByteDance) user agents. This means websites hosted on Weebly are partially invisible to some AI assistants.
Can I allow AI crawlers on my Weebly site?
Weebly has limited customization options for robots.txt and server-level bot management. Business owners may need to explore alternative hosting if AI visibility is a priority.
Is Weebly still a good option for business websites in 2026?
Weebly serves basic website needs, but its limited AI visibility controls put it behind platforms that offer more AI-era capabilities.
Enjoyed this article?
Share it with your network
