Which website builders name AI crawlers in robots.txt? We read 9 files, and only 2 name them
Your blog's robots.txt decides which crawlers may read it, and on a hosted builder you often can't edit it. On 8 November 2026 we requested /robots.txt on the main domain of nine builders (curl, following redirects) and looked for the well-known AI crawlers. These are the builders' own marketing sites, not customer sites, so this shows how each company writes the file, not what your site serves.
What we found
| Platform | HTTP | AI crawlers in the file |
|---|---|---|
| Squarespace | 200 | 30 user agents listed (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider…) in one group with `*`; the rules are the same for all: /config, /search, /account, /api/… |
| Jimdo | 200 | 35 user agents listed (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…); the only difference from `*` is that they may open /ai/, which `*` may not |
| Webflow | 200 | No named bots; `Content-Signal: ai-train=yes, search=yes, ai-input=yes` |
| Framer | 200 | No named bots; the same `Content-Signal: ai-train=yes, search=yes, ai-input=yes` |
| Wix | 200 | Only `User-agent: *` (92 lines of paths), no named bots |
| Ghost | 200 | Only `*`, a handful of /explore paths |
| WordPress.com | 200 | Only `*`, one disallow; a comment points to the firehose docs |
| GoDaddy | 403 | Access Denied page instead of the file |
| Weebly | 403 | Cloudflare challenge page instead of the file |
Three things to take from it
- Naming a bot is not blocking it. Squarespace and Jimdo list dozens of AI user agents, but in both files those agents fall under the same path rules as everyone else (Squarespace puts them in one group with
*). A quick scan for “GPTBot” would suggest a ban that the file does not contain. - Two builders publish a preference instead. Webflow and Framer use a
Content-Signalline that says training, search and AI input are all allowed. It is a statement of preference, not something a crawler has to obey. - A 403 hides the answer. GoDaddy and Weebly refused a plain request, so from a script their policy is unknown. Open the file in a browser before concluding anything.
Why this matters when you leave a builder
When the new site is a WordPress, Ghost or static host, robots.txt becomes yours to edit. Check what the old site said, write down what you want for AI crawlers, and keep the sitemap line. Moving the content is the part Decamp does; the policy is a decision for you.
Limits
One request per file, from one machine, on one day. We read the files as text and did not test how any crawler behaves. The Content-Signal lines are as published on that date.
Related
See also what the sitemaps of the same builders look like and which of them publish a feed. Or scan your own site for free before you pay for anything.