1. Home
  2. Tools
  3. AI robots.txt generator

Let AI answer engines in, keep training bots out

Tick the crawlers you want to allow or block, set your Content-Signal, and copy the result. Pick Shopify to get a robots.txt.liquid template that keeps Shopify's default rules for cart, checkout and account pages.

Some links may earn us a commission at no cost to you. Drafted with AI, checked by a human. How we earn.

  • Your policy, line by line
  • Shopify robots.txt.liquid option
  • Runs in your browser
Illustration of a support desk sorting incoming messages

Your crawler policy

Nothing leaves your browser
1Where it goes
2Allow: AI answer and user-fetch bots

Untick a bot to keep it out. These feed AI answers and open pages when a shopper asks, so blocking them can remove you from those answers.

3Block: AI training crawlers

Ticked bots get Disallow: /. This is a policy choice; it does not remove you from AI answers or search.

4Content-Signal
5Extras (optional)

Your file

How the file is built

The plain version starts with a User-agent: * group carrying your Content-Signal line and any paths you listed. Allowed AI bots get their own group with Allow: /; your paths are repeated there, because a bot with its own group ignores the * group. Blocked bots get Disallow: /.

The Shopify version keeps the robots.default_groups loop from Shopify's documentation, adds your Content-Signal and paths to the * group, and appends groups only for the bots you block. Shopify notes that adding robots.txt.liquid replaces its generated file, so save a copy of your current /robots.txt first and compare.

Bot names come from each operator's own documentation: OpenAI lists GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic lists ClaudeBot, Claude-SearchBot and Claude-User. robots.txt is a request, not a lock, so use firewall rules for anything that must be enforced.

FAQ

Which AI bots should an online store allow?

The ones that put you in answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, plus the user-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) that open a page when a shopper asks. Blocking them can remove the store from those answers.

Does blocking GPTBot or ClaudeBot hurt my visibility?

OpenAI and Anthropic both document separate crawlers for training and for search, so blocking the training crawler is meant to keep content out of model training without removing the site from their search results.

What is Content-Signal?

A robots.txt line that states your preferences in three parts: search, ai-input (using content in AI answers) and ai-train (training models). It sits inside the User-agent: * group. It states a preference; it does not technically block anyone, which is why the generator also writes Disallow groups.

How do I edit robots.txt on Shopify?

In the theme code editor, add a robots.txt.liquid template. Adding it replaces Shopify's generated file, so the template here keeps Shopify's default rules loop and adds your lines to it. Save a copy of your current /robots.txt before you publish.

Is robots.txt enforced?

No. Reputable crawlers follow it; others may not. Use your firewall or bot-protection settings for anything that must be enforced.