Your Ecommerce Site Has a URL Factory Before It Has a Crawl Budget Problem

Ecommerce crawl-budget work should begin with the component creating public addresses, not the report counting crawler requests.

A filter, sort control, campaign system or product-state rule can manufacture thousands of routes before anybody decides whether those routes deserve to exist, be linked, be crawled or appear in search. A canonical tag or cleaned crawl export treats the inventory after the platform has already produced it.

This is why “Google is wasting crawl budget” is often an overconfident diagnosis. The observable problem may be repeated requests to low-value parameter combinations while important products appear slow to update. The controllable problem is that the website has no rule governing which interface states may become persistent, discoverable URLs. My first question is not “How do we make Google crawl more?” It is:

Prove there is a crawl-budget constraint

Crawl budget is useful language only when scale, change frequency, serving capacity, discovery and observed request demand point towards the same constraint.

Google describes crawl budget as the URLs its systems can and want to crawl. Its crawl-budget guidance is aimed mainly at very large or rapidly changing sites and sites with a large share of URLs reported as discovered but not indexed. Google presents its volume figures as rough guidance, not universal thresholds.

That boundary matters. A catalogue can contain many URLs without exhausting crawl capacity. A new product can remain unindexed because of weak discovery, duplication, rendering, canonicalisation, page value or processing after crawling. Discovered – currently not indexed is an investigation lead, not a confession from Google. Before changing controls, establish:

• catalogue size and meaningful change rate;
• whether important new or changed pages are actually slow to be discovered or refreshed;
• verified request demand across unwanted URL families;
• host availability, latency and capacity during the observed period;
• sitemap, navigation and product-discovery paths;
• duplication, rendering and canonical conditions;
• alternative page-value or indexability explanations.

If the evidence does not align, call the problem what it is: crawlability, indexability, duplication, serving reliability, discovery or page value. I will not manufacture a crawl-budget emergency for a modest shop because a dashboard looks busy.

One useful filter can create an address factory

Filters help customers narrow a catalogue. Size, colour, material, stock, brand and price can turn an impossible product shelf into a useful buying experience. The problem begins when every interaction is allowed to create, expose and preserve another public route.

/walking-boots?size=8&colour=black&availability=in-stock&sort=price-asc

Change the parameter order and another address may appear. Remove a filter while a campaign parameter remains and another state survives. Add pagination, repeated values, internal-search terms or session data and the inventory multiplies again.

Google’s faceted-navigation guidance warns that parameter-based filters can create effectively infinite URL spaces which consume crawling resources before their value is understood. That does not make filters the enemy. The error is allowing one legitimate user-experience decision to answer five different questions:

“Customers need this filter” does not mean “search engines need every permutation.” Web design creates the interaction. URL governance decides which states become part of the public information architecture.

Group by generated job, not shared punctuation

URL audits commonly group everything containing the same parameter into one family. That is tidy and sometimes unsafe. One maintained colour collection may have a durable organic search purpose. A colour-plus-price-sort state may exist only for browsing. A repeated-value combination may be invalid. A tracking parameter may be unrelated to the underlying product state. Shared syntax does not give those routes the same job. Split a family when it contains materially different states:

This creates four boundaries: crawl frequency does not grant search value; revenue does not excuse an invalid state; buyer utility does not automatically grant index eligibility; and shared parameter names do not justify one family-wide treatment.

One broad rule can make the report quieter while suppressing a collection the ecommerce team genuinely needs. Split the jobs before selecting controls.

Give every family a decision record

Use a Crawl Value Ledger with one versioned row per closed decision family.

Unknown is a valid state. It is not permission to ship. The ledger should return the first honest state:

• Wrong diagnosis no evidenced family-scale crawl constraint; route to crawlability, indexability or another problem.
• Family evidence hold pattern, generator, scope or observation is incomplete.
• Mixed family materially different jobs remain bundled and must be split.
• Specialist review the closed family needs a search-asset, buyer-state, alternate-representation, sequence, lifecycle or temporary-state decision.
• Generator repair required the approved state differs from what the component produces or exposes.
• Verification pending a release occurred but live representative evidence is incomplete.
• Family controlled intended state, ownership, generator and observed behaviour agree within scope.

This is deliberately not a score. Several positive fields cannot compensate for an unbounded generator.

Do not prescribe the control before identifying the object

Technical SEO reaches for familiar tools: robots.txt, canonical tags, noindex, redirects and sitemaps. Each performs a different job. A robots rule governs fetching. A noindex instruction must be fetched before a crawler can read it. A canonical is a representative signal rather than a command that stops alternate URLs being generated or requested. A redirect changes the route. Sitemap inclusion can support discovery without proving that a page should be indexed.

Google’s canonicalisation guidance explains that canonical signals help consolidate duplicate or similar URLs but do not act as production shut-offs. Its crawl-budget guidance also warns that capacity freed from one family is not automatically reassigned elsewhere unless capacity was genuinely constrained.

I will not accept “we added canonicals” as evidence that the URL factory stopped. The generator, links, sitemaps, parameter patterns and observed requests still need to match the approved family decision. Likewise, do not block a family before deciding whether it contains a useful search collection. Control selection belongs after family classification.

Product discovery needs deliberate durable routes

Google recommends persistent ecommerce URLs and consistency across preferred links, sitemaps and canonical annotations in its ecommerce URL guidance. Google also explains that crawlable category-to-product navigation supports discovery and that Googlebot generally does not submit internal-search forms (ecommerce site structure).

This creates a useful design principle: valuable products should not depend on a crawler reproducing a shopper’s filter session. The ecommerce information architecture needs stable routes through categories, collections, product links and appropriate sitemaps. Filter states may enhance the customer journey without becoming the only path through which products can be found.

Product lifecycle complicates this. Out-of-stock, discontinued, replaced and seasonal items need business decisions, not a universal status-code rule. A temporarily unavailable product may retain demand and useful information. A permanently retired product may have an equivalent successor or none. The crawl ledger routes these cases to lifecycle ownership rather than letting a URL-count report decide.

SEO cannot design the catalogue alone. Merchandising knows the product relationships. Development knows the generator. Brand and content owners know what a maintained collection can credibly promise. The ledger makes those dependencies visible.

Logs matter only after routes have names

Crawl reports and server logs become useful when requests are classified against the same families the platform can change.

Search Console’s Crawl Stats report includes request totals, response types, host status, file types, crawl purpose and Googlebot type. Google describes example URLs as representative rather than comprehensive. Absence from an example list does not prove a route was never requested.

Verified server or CDN logs can provide more granular request evidence within their retention and infrastructure scope. A user-agent string alone is not requester verification. Edge logs, origin logs and crawler runs may observe different network layers and times. For each family, ask whether it was discovered or requested during the observed period; which component or release introduced it; whether important product and category routes remained discoverable; whether latency, errors or capacity affected serving; whether the unwanted pattern recurred after repair; and what sampling and log-retention limits remain.

Do not pour everything into a flat “crawl waste” total. Not present in this sample does not mean never crawled. Falling request volume does not prove reallocation, indexing, rankings or sales. The evidence should support a family decision, not write a success story for the agency.

Repair the generator instead of deleting examples

The durable repair usually belongs upstream at the component or process manufacturing and exposing the family. Possible generators include filter forms and navigation templates; sort and variant controls; campaign-link builders; internal-search routes; pagination components; sitemap jobs; product-status rules; and feeds and integration layers.

The specialist owner selects the appropriate treatment. The platform owner changes the generator or exposure source. The ledger preserves representative URLs, baseline evidence, release receipt and verification result.

That is also the commercial trade-off. Some buyer controls are worth keeping even when they create technical complexity. The aim is not to flatten the shopping experience for a cleaner crawl chart. It is to separate valuable interaction from unnecessary public inventory.

Make the release prove control

A family is not controlled because example URLs disappeared from one crawl. Before closing the release, confirm:

• The family definition and search/human jobs are approved
• The generator or discovery source changed as intended
• Valuable routes remain available and linked
• Representative unwanted patterns behave as approved
• Sitemap, canonical, robots and indexation signals agree where relevant
• Request observations are bounded by time and source
• No mixed family was hidden by a broad rule
• An owner and reopening trigger exist

If the filter feature can still generate new unowned combinations, the work is not complete. If the report is quieter because useful collections vanished, the work is not successful.

Start with one proliferating pattern. Reproduce it through the interface. Identify the component that creates it. Split the different jobs inside the family. Then decide which public addresses the ecommerce business is willing to own.

The cursor does not belong on “request indexing” while the address bar is still printing inventory nobody approved.