Crawl Budget Consumed by Junk URLs
Facets, filters, parameters, and pagination eat the budget. Signs: thousands crawled, low product index count, high “Discovered – not indexed”.
Your store has 4,000 products. Google indexed 800. The other 3,200 are invisible in organic search — a revenue emergency, not housekeeping. Learn why product pages fail indexing and how to fix it at scale with Bulk Index Request.
Standard indexing advice is often insufficient — and sometimes actively harmful — for product catalogs.
A 200-product store can generate 10,000–50,000 crawlable URLs — variants, filters, pagination, search, cart. Googlebot must navigate that landscape to find 200 real product pages.
Manufacturer copy, color/size variants, parameter URLs, and multi-path breadcrumbs create near-identical pages. Google indexes one and ignores the rest — sometimes the wrong one.
Products launch, sell out, and discontinue. Every change is an indexing event. Without a system, indexed count drifts as the catalog evolves.
An unindexed product page doesn’t exist in organic search. Multiply by thousands of SKUs and it’s a revenue emergency, not SEO housekeeping.
Diagnose before you submit. Submitting blocked pages wastes quota.
Facets, filters, parameters, and pagination eat the budget. Signs: thousands crawled, low product index count, high “Discovered – not indexed”.
Variants too similar, manufacturer copy shared across retailers, parameter duplicates. Signs: “Crawled – not indexed,” one variant indexed only.
Name + image + price + 2-sentence manufacturer blurb. Google treats it as insufficient. Fix: unique 150–300+ word descriptions, specs, FAQs, reviews.
Filter/noindex rules applied too broadly to real product pages. Check GSC URL Inspection for “Excluded by noindex” or robots blocks.
Variants canonicalizing to the wrong parent or to category pages. Wrong version indexed — or none.
Title, price, availability loaded after initial HTML. Google may index an empty shell → thin content de-index.
Heavy product images cause crawl abandonment. Fix WebP/AVIF, lazy-load, CDN, TTFB under 200ms.
Deep products only reachable from low-priority categories. Ensure ≤3 clicks from homepage; add hubs like bestsellers & new arrivals.
Technically clean, still “Discovered – not indexed” for weeks. Exact case for Bulk Index Request — skip the queue.
All checked → submit via Bulk Index Request. Any unchecked → fix first, then submit.
If you run more than a few hundred products, this is the most important concept in ecommerce SEO.
Small (<1,000 pages)
100–500 pages/day
Medium (1k–10k pages)
500–5,000 pages/day
Large (10k+ pages)
5,000–50,000+ pages/day
2,000 products + 50 categories can explode into tens of thousands of crawlable URLs:
| URL Type | Est. Count | Should Index? |
|---|---|---|
| Product pages | 2,000 | Yes |
| Category / subcategory | 250 | Usually yes |
| Paginated category pages | 500 | No |
| Filtered / facet URLs | 8,000–50,000 | No |
| Product variant URLs | 6,000 | Sometimes |
| Search / cart / account | Unlimited+ | No |
Five filter dimensions × 10 options each = 100,000 theoretical URLs from one category. Even “only” 50,000 facets steal budget from 2,000 real products.
Block parameterized URLs (e.g. Disallow: /*?*). Effective but blunt — ensure no important pages use parameters.
Point all filtered URLs back to the base category URL to concentrate signals.
Allow link discovery without indexing the filter page itself.
robots.txt for sort/session params · canonicals on facets · noindex pagination beyond page 1 · index only products, base categories, and content.
Pagination tip: keep pages crawlable for discovery, but noindex beyond page 1. Prefer URL fragments for infinite scroll — fragments aren’t crawled as separate URLs.
After crawl budget waste, duplication is the #2 reason product pages fail indexing.
Same supplier copy on every retailer → Google indexes one (usually manufacturer or highest-authority store) and treats the rest as duplicates.
Rewrite uniquely. Prioritize high-margin, exclusive, and bestseller SKUs. Even 100 unique words (use cases, Q&A) helps.
Same price/description variants → single URL or canonical to base. Different price/use case → separate indexed URLs with unique content. Different models → fully unique pages.
Tracking params and sort URLs split signals. Canonical to clean URL, strip params at CDN/server, sitemap = clean URLs only.
Same product under /mens/…, /sale/…, /new-arrivals/…. Pick one primary hierarchy and canonicalize all alternatives.
Crawl → export canonicals → verify self-referencing / correct base / HTTPS / trailing-slash consistency → fix mismatches → submit via Bulk Index Request.
The minimum viable product page for indexing readiness.
| Element | Requirement |
|---|---|
| HTTP Status | 200 OK |
| Canonical | Self-referencing, correct URL |
| Title / Meta / H1 | Unique; product name + key attribute |
| Description | Unique, minimum 150–300 words |
| Images | 2–3+ with descriptive alt text |
| Price & Availability | Visible in HTML (not JS-only) |
| Product Schema | Valid Product + offers fields |
| Internal Links | At least 2–3 inbound links |
| Page Speed | LCP under 2.5s on mobile |
| Mobile | Content identical to desktop |
| No noindex / robots block | Product paths fully crawlable |
Doesn’t force indexing — but signals a legitimate product page. Required fields: name, image, description, sku, brand, offers (price, currency, availability), aggregateRating when available.
Add Review, BreadcrumbList, and FAQPage schema. Validate with Google’s Rich Results Test before submission.
Wave 1 may fetch an empty shell; Wave 2 JS render is delayed and not guaranteed. Empty shells trigger thin-content de-indexing.
Test: GSC URL Inspection → View Crawled Page. Fix with SSR/SSG. Shopify is mostly server-rendered by default (watch app JS). WooCommerce: verify product data in View Source before JS.
Homepage → Category → Product. Deeper than 3 clicks = lower crawl priority. Use featured products, Top Sellers, New Arrivals hubs.
Sitemap index by type (products, categories, content). Only canonical, 200, non-noindex URLs. Accurate lastmod. Priority: bestsellers 0.9, products 0.8, categories 0.7.
Supplement with Bulk Index Request for launches.
Breadcrumbs + BreadcrumbList schema · Related products in HTML (not JS) · Buying guides linking to SKUs · Homepage featured rotation.
New, out-of-stock, and discontinued — each stage needs a different decision.
Organic timeline: 1–7 days (high authority) · 1–4 weeks (medium) · 1–3 months (new/low). Commercially unacceptable for launches.
Submit every new product via Bulk Index Request on launch day. Pre-flight: live permanent URL, unique copy, images, schema, self-canonical, sitemap, internal links, no blockers.
| Scenario | Action | Indexing |
|---|---|---|
| Temporarily OOS | Keep live + OOS messaging | Keep indexed |
| Seasonal OOS | Keep live + notify me | Keep indexed |
| Discontinued + replacement | 301 to replacement | De-index original |
| Discontinued + category | 301 to category | De-index original |
| Discontinued, no replacement | 404 or 410 | De-index (410 faster) |
| May return, uncertain | Live + OutOfStock schema | Keep indexed; revisit |
Don’t 404 ranked OOS pages · Don’t noindex temporary OOS · Don’t show “in stock” when unavailable · Update Offer availability schema to OutOfStock.
Duplicate URLs: /products/… vs /collections/…/products/…. Verify canonicals → /products/. Submit only /products/ URLs to Bulk Index Request.
Locked robots.txt: Facet filters remain crawlable by default. Disallow filter params or canonical filtered collections.
App JS: Badges/reviews/pricing apps can break render. Check GSC crawled screenshot.
Variant params: ?variant=… must canonical to base product.
Yoast/RankMath: Confirm products & categories indexable; often noindex thin tags.
Excess archives: Noindex tags, attribute archives (if thin), cart/checkout/account, pagination beyond page 1.
Caching: Don’t serve stale stock/price to Googlebot.
Speed: Plugin bloat kills crawl rate — CDN + performance plugin + CWV on product pages.
5,000 products ÷ ~12 GSC submissions/day = 417 days. That’s not a workflow.
Platform export, product sitemap, or Screaming Frog crawl.
200 only · no noindex · self-canonical · base URLs · unique content.
Know what’s indexed vs submission targets before spending quota.
Tier 1 bestsellers/launches now · Tier 2 in 24h · Tier 3 in 72h.
Upload batches; concurrent Indexing API with live confirmation.
Re-check in 48–72h. Fix “Crawled – not indexed” content; resubmit clean stragglers.
Or automate with Scheduled Index Check.
Indexed ÷ live products × 100. Target 90%+.
Filter /products/ or /product/ patterns monthly.
Launch → indexed. Measure Bulk Index Request ROI vs organic.
GA4 segment on product URLs — should rise with index rate.
GSC Product rich results — only for indexed + valid schema.
Outdoor equipment retailer · 3,800 WooCommerce SKUs · ~$180k/month organic revenue.
Bulk Index Checker audit: 836 of 3,800 indexed (22%). Estimated lost revenue: $85k–$120k/month.
| Root Cause | Pages |
|---|---|
| Faceted navigation crawl waste | ~15,000 filter URLs |
| Thin product tag archives indexed | 450 pages |
| Manufacturer duplicate descriptions | 1,800 products |
| Variant canonical errors | 620 pages |
| Paginated categories indexed | 280 pages |
| Clean but not crawled yet | 544 pages |
Indexed Pages
836 → 3,572
22% → 94%
Weekly Sessions
12.4k → 47.8k
Product organic
Weekly Revenue
$41.5k → $138.6k
+~$97k/month
Rich Impressions
8.2k → 61k
Product results
Cleanup → content rewrites → Bulk Index Request batches → verify/resubmit. Remaining 6% were genuine thin-content flags.
Ecommerce product page indexing — crawl budget, duplicates, OOS pages, Shopify & WooCommerce.
Most often: (1) crawl budget wasted on facets/filters, (2) duplicate manufacturer or variant content, (3) thin content, (4) canonical errors, (5) Google hasn't crawled them yet. Use Bulk Index Checker to see what’s indexed and Bulk Index Request to submit the rest.
Use the Google Indexing API via Bulk Index Request for concurrent submission, paired with crawl budget cleanup, canonical fixes, and unique product content.
Organically: 1 day to 3 months. Via Bulk Index Request: typically 24–72 hours; high-authority domains sometimes hours.
Yes for temporary/seasonal OOS — preserves rankings. Use OutOfStock schema. Permanently discontinued: 301 to replacement or category.
Yes. Identical copy across retailers = duplicate content. Unique descriptions are the highest-ROI content fix for ecommerce indexing.
Filter menus create unique URLs per combination that can number thousands–millions, stealing crawl budget from real products. Fix with canonicals, robots.txt, or noindex on filters.
Indexing ≠ ranking. Intent mismatch, stronger competitors, weak backlinks, or poor mobile CWV. Indexing gets you in the game — ranking needs quality and authority.
Well-optimized stores: 85–95%+. Below 50% = emergency (budget waste, duplicates, or blockers). Below 70% needs immediate audit.
Both can hit excellent rates. Shopify defaults (variant canonicals) are cleaner; WooCommerce needs more manual config. Configuration, content, and crawl budget matter more than platform.
Yes. Bulk Index Checker uploads your full product list and returns live indexed/not-indexed status in minutes. For post-migration catalogs, also see Google Indexing After Migration.
Join 300+ ecommerce SEOs and store owners who trust Bulk Index Checker to keep product catalogs indexed, visible, and generating organic revenue.