Ari Kurniawan
OverSimplified SEO

OverSimplified SEO

SEO Crawl Budget (Oversimplified)

SEO Crawl Budget (Oversimplified)

An oversimplified introduction to Crawl Budget.

Ari Kurniawan

Published on May 10, 2021

2 min read

The term "Crawl Budget" is quite new in the SEO industry. Backlinko describes it the best as:

The number of pages Googlebot crawls and indexes on a website within a given timeframe.

In the more technical description, the crawl budget represents the number of simultaneous connections Googlebot uses to crawl the site and the time it has to wait between fetches.

Why you should care about this "Crawl Budget"?

If you wasting your crawl budget, search engines won't be able to crawl your website efficiently, it would end up hurting your SEO performance.

For example, with an optimum crawl budget, Googlebot can crawl all the pages on your website every time. But, if you wasting the crawl budget, maybe Googlebot only crawls 50% of your website, which means 50% of it may be not indexed or not updated on Google Search Result.

This problem becomes more severe for a big website such as e-commerce with more than 10k pages. 50% of 10K pages is a lot! 🙀🙀🙀

What's waste your crawl budget?

Slow Website

A slow website will slow the crawl process, make Googlebot spent too much time to load and crawl one page and leave your website before all the pages crawled.

Even Google stated it on their official website :

Making a site faster improves the users' experience while also increasing crawl rate.

Wait, what's this "crawl rate"?

Crawl rate means how many requests per second Googlebot makes to your website when it is crawling it. You cannot change how often Googlebot crawls your site, BUT we can request a recrawl after update content and add new ones.

You also can limit the crawl rate or adjust it between 0.002 to 2 requests per second. You can learn more about crawl rate and how to limit it here .

Bad Internal Linking / Orphan Pages

You want to get backlinks pointing to all your website pages, don't you? That's not realistic. But, you can set up so all your pages get links from each other, internal linking!

Just make sure there are no Orphan Pages on your website.

What are these "orphan pages" again?

Pages with no internal or external links pointing to them. You maybe can't control the external links, but you can control the internal link. Just make sure every page has at least one link point to it.

Duplicate Content

Just don't. It just wastes your crawl budget.

 
Share this