How Shopify stores block AI crawlers without ever deciding to
Most store owners never wrote a robots rule in their life — yet their store may be telling ChatGPT and Perplexity to go away. Here's how that happens and how to check yours in two minutes.
Ask a store owner whether they want their products to show up when ChatGPT recommends things to buy, and the answer is obviously yes. Yet a large share of stores publish rules that tell AI crawlers exactly the opposite. Almost none of those owners made that choice on purpose.
Where the blocks come from
- Theme or app defaults — some themes and SEO apps ship robots templates written before AI crawlers existed, with broad disallow rules that catch them.
- Copy-pasted robots.txt edits — advice from old forum posts, applied once and forgotten.
- Bot-protection and firewall layers that classify AI crawlers as scrapers and silently block them at the network level, so even a clean robots file doesn't help.
- Deliberate blocks of one bot that accidentally use patterns matching many bots.
Why it matters more every month
The crawlers in question — OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended — are what feed AI shopping answers. When a shopper asks an assistant for "a good vitamin C serum for sensitive skin," the assistant can only draw on stores it has been allowed to read. Blocked stores don't rank lower; they don't exist.
Check your own store in two minutes
Open yourstore.com/robots.txt in a browser and look for the AI user agents by name: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. A 'Disallow: /' under any of them means that bot is shut out. No mention at all usually means the bot follows your general rules — which may or may not be what you want.
The robots file is only the visible half, though. Network-level blocks don't show up there — the only way to catch those is to request your pages the way the bots do and see what comes back. Our free scan does exactly that as part of its crawlability check, bot by bot, alongside its structured-data and agentic-readiness checks.
The fix is usually trivial
Once found, most blocks are a few lines to fix — which is what makes this such an unusual problem: enormous downside, near-zero cost to repair. The hard part is knowing it's happening to you.
Wondering where your store stands?
The free scan checks a sample of your actual store for everything this article describes. The full-store audit checks every page.
Get your free AI-readiness scan