#What crawl budget is
Crawl budget has two components. Crawl rate limit is how much crawling a server can handle without degrading for real visitors; a slow or overloaded server gets crawled more cautiously. Crawl demand is how much a search engine wants to crawl a given site, driven by the site's perceived value, freshness, and popularity. Together they set a rough ceiling on how many URLs get fetched in a given window.
For the overwhelming majority of sites this is a non-issue: they simply have fewer URLs than any plausible crawl budget would ever constrain. The constraint becomes live and practical specifically on large sites with combinatorial URL generation, and a dealer's own inventory search is the textbook example.
TABLE
What eats a dealer site's crawl budget, and the usual fix
The pattern repeats across dealer sites: filterable inventory generates far more URLs than the crawl budget can usefully cover.
| Source of the problem | Why it happens | Usual fix |
|---|---|---|
| Faceted inventory filters | Make × model × trim × color × price × mileage combinations multiply into thousands of URLs | Canonical to the unfiltered category page; noindex low-value filter combinations |
| Sort-order parameter URLs | Every sort option (price asc/desc, newest, mileage) creates a distinct crawlable URL for the same inventory | Parameter handling or canonical tags pointing to one canonical sort |
| Session or tracking parameters in URLs | Marketing or analytics parameters appended to otherwise identical pages | Canonical tags stripping the parameter; avoid linking internally with tracked URLs |
| Sold-vehicle VDPs left live indefinitely | A vehicle detail page for a car sold months ago still exists and gets recrawled | 301 to the relevant category, or a clear sold-status page, once inventory turns over |
Compiled 2026-09-13 from the well-documented pattern of faceted-navigation crawl-budget issues on large e-commerce and inventory-driven sites, applied to dealer inventory specifically.
#Why it matters for a dealer site specifically
When a crawler's available budget on a domain is consumed by thousands of low-value filter permutations, the pages that matter for a customer's decision, a new vehicle's detail page, an updated financing page, a freshly published service special, get crawled less often and discovered more slowly. That shows up as real listings taking longer to appear in search, and updates to important pages taking longer to be reflected.
The problem is invisible to a human visitor, who never notices that a crawler spent its budget on a filter combination nobody would ever type into a search bar. The effect only shows up in aggregate, in how quickly the site's valuable pages get indexed and refreshed, which is exactly why it is easy to overlook until someone specifically goes looking for it.
#What fixes it
The fix is nearly always exclusion and consolidation, not more content. Canonical tags pointing filtered variations back at the unfiltered category page, robots directives excluding low-value parameter combinations, and a routine process for retiring vehicle detail pages once a unit sells, so the crawler is not repeatedly revisiting pages describing inventory that no longer exists. See VDP and SRP indexation for the deeper, page-type-specific version of this same discipline.