Notes · 8 November 2026

Which website builders name AI crawlers in robots.txt? We read 9 files, and only 2 name them

Your blog's robots.txt decides which crawlers may read it, and on a hosted builder you often can't edit it. On 8 November 2026 we requested /robots.txt on the main domain of nine builders (curl, following redirects) and looked for the well-known AI crawlers. These are the builders' own marketing sites, not customer sites, so this shows how each company writes the file, not what your site serves.

What we found

PlatformHTTPAI crawlers in the file
Squarespace20030 user agents listed (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider…) in one group with `*`; the rules are the same for all: /config, /search, /account, /api/…
Jimdo20035 user agents listed (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…); the only difference from `*` is that they may open /ai/, which `*` may not
Webflow200No named bots; `Content-Signal: ai-train=yes, search=yes, ai-input=yes`
Framer200No named bots; the same `Content-Signal: ai-train=yes, search=yes, ai-input=yes`
Wix200Only `User-agent: *` (92 lines of paths), no named bots
Ghost200Only `*`, a handful of /explore paths
WordPress.com200Only `*`, one disallow; a comment points to the firehose docs
GoDaddy403Access Denied page instead of the file
Weebly403Cloudflare challenge page instead of the file

Three things to take from it

Why this matters when you leave a builder

When the new site is a WordPress, Ghost or static host, robots.txt becomes yours to edit. Check what the old site said, write down what you want for AI crawlers, and keep the sitemap line. Moving the content is the part Decamp does; the policy is a decision for you.

Limits

One request per file, from one machine, on one day. We read the files as text and did not test how any crawler behaves. The Content-Signal lines are as published on that date.

Related

See also what the sitemaps of the same builders look like and which of them publish a feed. Or scan your own site for free before you pay for anything.