Stop Scrapers Stealing Your Shopify Product Catalog
Every Shopify store exposes its full catalog by default. Here is how scrapers take it, why robots.txt fails, and the detection layers that actually stop them.
On this page
TL;DR: Every Shopify store ships with its full catalog exposed through
/products.jsonand the sitemap, and robots.txt does nothing to stop a scraper that ignores it. Stopping catalog theft takes layered detection: user-agent fingerprinting, headless browser markers, request-velocity limits, spoofed-crawler verification, and datacenter ASN blocking. Cordon runs all five layers while always allowing verified search engines and good bots, so SEO is never at risk.
If a competitor's store looks suspiciously like yours, same products, same descriptions, prices always one euro under, you are probably not imagining it. Catalog scraping on Shopify is cheap, automated, and common, because the platform makes catalogs unusually easy to extract. This post explains how it works and which defenses hold up.
How catalog scraping works on Shopify
Shopify stores are the easiest e-commerce platform to scrape, and it is worth understanding why.
Every store exposes structured catalog endpoints by default. Append /products.json to almost any Shopify store URL and you get a clean, paginated JSON feed of products: titles, descriptions, variants, prices, image URLs. The sitemap (/sitemap.xml) lists every product page. Neither requires a login or an API key, so a scraper does not need to parse your HTML at all. Shopify's own developer documentation (shopify.dev) treats these as public surface area, and you cannot turn them off.
The tooling is commodity. The typical scraping stack, roughly in order of sophistication:
- python-requests or curl hitting
/products.jsonin a loop. Trivial to write, no browser involved, identifiable by its default user agent unless the operator fakes one. - Scrapy, the industrial framework, which handles pagination, retries, and rotation and can crawl your whole sitemap in minutes.
- Puppeteer, Selenium, and headless Chrome for cases where simple scripts are already blocked. These present as real browsers at first glance.
The infrastructure is predictable. Most scraping runs from datacenter networks: AWS, Alibaba Cloud, Tencent, Huawei Cloud, and the long tail of VPS providers. Real shoppers do not browse from an AWS IP range. Careful operators route through residential proxy networks to look like home connections, which defeats network-based checks but not behavioral ones.
What scrapers do with your data
Scraped catalogs feed three concrete businesses:
- Price undercutting. Automated repricers watch your catalog and set their prices just below yours, permanently. You are competing with your own price feed minus a margin.
- Catalog cloning. Dropshipping copycats lift your titles, descriptions, and images wholesale, spin up a store selling the same products, and run ads against your keywords.
- Review and content theft. Product reviews, size guides, and buying guides get scraped and republished, sometimes outranking the original.
Your content took time and money to produce, and scraping transfers it for free. The full breakdown is on our stop content scraping page.
Why robots.txt does not stop them
Robots.txt is a text file that politely asks crawlers to stay out of certain paths. Googlebot honors it. Bingbot honors it. A scraper copying your catalog reads it, if at all, as a map of what you consider valuable. There is no enforcement: the server serves the page whether the client honored the file or not. Anything that claims to stop scrapers via robots.txt, meta tags, or terms-of-service pages is decoration. Stopping scrapers means detecting and refusing requests.
Detection layers that actually work
No single check catches everything, which is why serious scraper protection is layered. Five layers, what each catches, and what Cordon does at each:
| Layer | Catches | Misses |
|---|---|---|
| User-agent fingerprinting | Lazy scripts, default UAs, malformed headers | Anyone who spoofs a browser UA |
| Headless browser markers | Puppeteer, Selenium, headless Chrome | Plain HTTP scripts (caught by layer 1) |
| Request-velocity detection | Anything crawling faster than a human | Slow, patient scrapers |
| Spoofed-crawler verification | Fake Googlebots and fake Bingbots | Scrapers not pretending to be crawlers |
| Datacenter ASN blocking | Cloud-hosted scrapers (the majority) | Residential proxy traffic |
A scraper has to beat all five simultaneously, and each evasion makes them more expensive to run and easier to catch on another layer.
1. User-agent fingerprinting
The cheapest scrapers do not bother disguising themselves. python-requests/2.31, Scrapy's default agent, empty user agents, and header combinations no real browser sends are all dead giveaways. Cordon fingerprints the user agent and the surrounding request headers and blocks the obvious automation outright. This layer alone removes a surprising share of scraping volume, because most scraping is lazy.
2. Headless browser markers
Puppeteer and Selenium drive real Chrome, so the user agent looks legitimate. But headless browsers leak: automation flags, missing browser features, and property mismatches that real user-driven browsers do not exhibit. Cordon checks for these markers and blocks confirmed automation only. An in-app webview or an unusual but human browser never trips this layer, because Cordon acts on definitive automation signals rather than heuristic guesses.
3. Request-velocity detection
Behavior is the hardest thing to fake. A real shopper browses a few product pages a minute, lingers, backtracks, adds something to cart. A scraper loads 40 product pages in 30 seconds in perfect sequential order. No user agent spoofing and no residential proxy changes that pattern, because the pattern is the job: a scraper that crawls at human speed takes days per store and stops being economical. Cordon tracks request velocity per visitor and blocks clients whose crawl rate no human shopper could produce. This is the layer that catches the careful operators who beat every other check.
4. Spoofed-crawler verification
The oldest trick in scraping: set your user agent to Googlebot and watch naive defenses wave you through, since nobody wants to block Google. The defense is verification. The real Googlebot comes from Google's own network, and that is checkable. Cordon verifies crawler claims against the actual network: a "Googlebot" arriving from an Alibaba Cloud IP is a scraper wearing a costume, and it gets blocked. The real Googlebot passes verification and is always allowed, no configuration needed.
5. Datacenter ASN blocking
Every network on the internet belongs to an autonomous system with an ASN, and cloud providers' ASNs are public knowledge. AWS, Alibaba Cloud, Tencent, Huawei Cloud: these networks host servers, not shoppers. Cordon blocks entire datacenter networks, and whole ASNs, with a single rule, removing the scraping industry's default habitat in one move. Real customers browse from residential ISPs and mobile carriers, so the false-positive cost is near zero. Deeper explainer here: what is an ASN and why block one. And if the scraper origin clusters geographically, country blocking stacks on top with a one-click flag picker.
The bots you must not block
Scraper protection that hurts your SEO is worse than no protection. Cordon hard-codes the safeguard: verified Google and Bing crawlers are always allowed and cannot be blocked, by any rule, preset, or misconfiguration. The same allowlist covers the good bots that now drive real discovery: GPTBot, ClaudeBot, PerplexityBot (AI search visibility) and Ahrefs (your own SEO tooling). Blocking scrapers and staying fully visible to search and AI answers are not in tension when verification is done properly. If you want the full argument, read do bot blockers hurt SEO?
Setting up scraper protection with Cordon
- Install Cordon from the Shopify App Store. It is a theme app extension with an embedded admin: no script tags, no theme edits, Lighthouse-friendly.
- Apply the "Anti-scraper" preset. It enables user-agent fingerprinting, headless markers, velocity detection, and spoofed-crawler verification in one click. (There is also a "Strict mode" preset if you want the aggressive version.)
- Turn on datacenter network blocking to cut off AWS, Alibaba, Tencent, Huawei Cloud and the rest of the hosting world, or block specific ASNs you see in your logs.
- Watch the live visitor log. Every decision shows up in real time, with IPs SHA-256 hashed under a daily-rotating salt so the log stays GDPR-safe. This is where you confirm scrapers are dying and shoppers are not.
- Add allowlist entries for any partner, feed service, or monitoring tool of your own that legitimately needs automated access.
Every decision completes in under 50 milliseconds and never slows the storefront. If the service ever hiccups, it fails open: real customers shop normally. Datacenter and scraper detection ship on the Pro plan ($49/mo, 5M visitors); basic bot blocking is on the Free plan, and every paid plan has a 7-day trial. The pricing page has the full matrix, and features covers the detection engine in full.
Frequently asked questions
How do scrapers get my Shopify product catalog?
Through the public surface every Shopify store exposes by default: /products.json returns a clean JSON feed of your products, variants, prices, and image URLs, and the sitemap lists every product page. No login or API key is needed, and these endpoints cannot be turned off, which is why the defense has to happen at the traffic level.
Does robots.txt stop scrapers on Shopify?
No. Robots.txt is a request, not an enforcement mechanism. Legitimate crawlers like Googlebot respect it, but a scraper copying your catalog ignores it, or uses it as a map of what you consider valuable. Stopping scrapers requires detecting and refusing their requests, which robots.txt cannot do.
Will scraper protection block Google or hurt my SEO?
Not with verified-crawler allowlisting. Cordon checks that a crawler claiming to be Googlebot actually comes from Google's network and always allows verified search engines, with no way to block them even by misconfiguration. Good bots like GPTBot, ClaudeBot, PerplexityBot, and Ahrefs stay allowed too, so SEO and AI search visibility are never hurt.
Can scrapers using headless Chrome or residential proxies still be caught?
Yes. Headless Chrome driven by Puppeteer or Selenium leaves detectable automation markers even with a perfect user agent. And residential proxies only disguise the network, not the behavior: request-velocity detection catches any client loading product pages far faster than a human shopper, regardless of where the requests appear to come from.
What is datacenter ASN blocking and why does it matter for scrapers?
Every network belongs to an autonomous system identified by an ASN, and cloud providers like AWS, Alibaba, Tencent, and Huawei Cloud have well-known ones. Blocking the ASN blocks the entire network with one rule. Since scrapers overwhelmingly run on cloud servers and real shoppers browse from residential and mobile networks, it is one of the highest-value, lowest-collateral rules you can set.
Wrapping up
Your catalog is exposed by default and robots.txt will not save it. But the scrapers taking it are identifiable: by their tooling, their behavior, their fake crawler costumes, and the datacenter networks they run from. Install Cordon from the Shopify App Store, apply the Anti-scraper preset, and watch the visitor log do its work. Free plan available, 7-day trial on every paid plan.