Crawl efficiency in SEO is a metric that SEOs use to describe the ability for a search bot to crawl and index your site’s pages and products. The easier it is for bots to crawl, the better the efficiency. It might help to understand how bots crawl a website. Simply put, they’ll land on a page and crawl all the corresponding links on that page, indexing those pages and connections as it goes. A crawl is completed once it’s crawled all links on your site or has run out of crawl bandwidth.
By having an efficient site for search engines, you’ll get your pages indexed quicker, meaning you’ll rank for a wider range of search terms, or bolster existing rankings with content depth.
Here are some characteristics of site’s that have great crawl efficiency:
- Utilizing an XML sitemap to easily guide bots through all the pages on your site (Volusion generates a sitemap automatically, so no need to worry here)
- Clear page hierarchy. You’ll want to make sure your pages have a clear and logical hierarchy to them. For Volusion store it’s common to have Homepage > Category Page > Product Page hierarchy.
- No Content Bottlenecks. Bottlenecks happen when a search engine can’t efficiently crawl a page or pages without tediously combing through individual links on single or orphaned pages.
- Minimal 404s on their site. Search engines read 404s as a “dead end.” If they find an abundance of 404s on your site, you’ll get less crawl depth and might even lose some authority
- Minimize unnecessary 301s. Abundant 301s can send search engines on a wild goose chase to find out where your site actually is. Domain consistency and URL canonicalization is important to make sure your site is crawled as efficiently as possible.
- Exclude unnecessary pages. Search engines don’t need to crawl ALL the pages on your site. Sometimes, it’s a good idea to exclude pages that don’t add any SEO value. For instance, admin pages, login pages, and more can be excluded from search engine indexing.
When it comes to crawl efficiency, simplicity is key. The ultimate goal should be that your site is easy to understand from a search engine perspective. By improving your site's crawl efficiency you'll not only be getting good with Google, but you'll also be making the customer experience better for shoppers.
Have any questions about crawl efficiency? Ask them in the comments and we'll get back to you!