[3.3.x] Support Forum • GoogleOther misbehaving? Hundreds of board guest sessions active

This afternoon I have noticed a sharp increase in guests on a couple of boards that I admin.

Digging deeper I'm seeing many instances from IP addresses 66.249.xxx.xxx. with the user-agent Checking Google Crawler (User Agent) Overview | Google Search Central  |  Documentation  |  Google for Developers

GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.

The page also says

Google's common crawlers are used for building Google's search indices, perform other product specific crawls, and for analysis. They always obey robots.txt rules and generally crawl from the IP ranges published in the googlebot.json object.

The IP addresses I am seeing definitely are in the googlebot.json object. So I'm a bit confused. Perhaps someone at Google has programmed something wrong and triggered this storm versus expected one-off crawls for internal research.

Added: More information about this new bot -- Google launches a new crawler named GoogleOther

Are others seeing this also on their boards?

Statistics: Posted by P_I — Tue Apr 16, 2024 9:05 pm

---