A new article on
Marketing Land reports that Googlebot is now crawling with HTTP/2 protocol where it's available. This change begins this month, and Google Developer pages just got updated to reflect that.
This update got announced in September, and the change is now live. Google says:
|
"Generally, Googlebot crawls over HTTP/1.1. However, starting November 2020, Googlebot may crawl sites that may benefit from it over HTTP/2 if the site supports it."
|
Why Use the HTTP/2 Network Protocol
HTTP/2 is the latest network protocol and enables Googlebot to perform faster data transfer between servers and browsers. That reduces the time it takes to crawl multiple pages, which has to be a great thing for publishers everywhere.
This is what an official IETF FAQ page on Github has to say about the development:
|
"HTTP/1.x has a problem called "head-of-line blocking," where effectively only one request can be outstanding on a connection at a time.... Multiplexing addresses these problems by allowing multiple request and response messages to be in flight at the same time; it's even possible to intermingle parts of one message with another on the wire. This, in turn, allows a client to use just one connection per origin to load a page."
|
More on this from Google:
|
"...starting November 2020, Googlebot may crawl sites that may benefit from it over HTTP/2 if it's supported by the site. This may save computing resources (for example, CPU, RAM) for the site and Googlebot, but otherwise it doesn't affect indexing or ranking of your site."
|
If you're a publisher, you can opt out of HTTP/2 crawling by configuring your server to send a 421 server response code. That code is described by the Internet Engineering Task Force (IETF.org) as a Misdirected Request. That means that a request for HTTP/2 gets misdirected if it's not available. The IETF says:
|
"The 421 (Misdirected Request) status code indicates that the request was directed at a server that is not able to produce a response. This can be sent by a server that is not configured to produce responses for the combination of scheme and authority that are included in the request URI."
|
These are the recommendations on Google's developer page:
|
"To opt out from crawling over HTTP/2, instruct the server that's hosting your site to respond with a 421 HTTP status code when Googlebot attempts to crawl your site over HTTP/2. If that's not feasible, you -can send a message to the Googlebot team- (however, this solution is temporary)."
|
