How Shopify stores block AI crawlers without ever deciding to

Ask a store owner whether they want their products to show up when ChatGPT recommends things to buy, and the answer is obviously yes. Yet a large share of stores publish rules that tell AI crawlers exactly the opposite. Almost none of those owners made that choice on purpose.

Where the blocks come from

• Theme or app defaults — some themes and SEO apps ship robots templates written before AI crawlers existed, with broad disallow rules that catch them.
• Copy-pasted robots.txt edits — advice from old forum posts, applied once and forgotten.
• Bot-protection and firewall layers that classify AI crawlers as scrapers and silently block them at the network level, so even a clean robots file doesn't help.
• Deliberate blocks of one bot that accidentally use patterns matching many bots.

Why it matters more every month

The crawlers in question — OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended — are what feed AI shopping answers. When a shopper asks an assistant for "a good vitamin C serum for sensitive skin," the assistant can only draw on stores it has been allowed to read. Blocked stores don't rank lower; they don't exist.

Check your own store in two minutes

Open yourstore.com/robots.txt in a browser and look for the AI user agents by name: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. A 'Disallow: /' under any of them means that bot is shut out. No mention at all usually means the bot follows your general rules — which may or may not be what you want.

The robots file is only the visible half, though. Network-level blocks don't show up there — the only way to catch those is to request your pages the way the bots do and see what comes back. Our free scan does exactly that as part of its crawlability check, bot by bot, alongside the structured-data and agentic-readiness audits.

The fix is usually trivial

Once found, most blocks are a few lines to fix — which is what makes this such an unusual problem: enormous downside, near-zero cost to repair. The hard part is knowing it's happening to you.