Listing of web crawlers that do not support compression

If you are the author of any of these spiders, then please add support for content compression when you crawl the web. This will save you bandwidth on your crawling system, and it saves bandwidth on the servers that you crawl.

Adding compression support can be very simple -- if your spider is coded in Perl using LWP::UserAgent, then the addition of a single line of code will enable compression support. 

$ua->default_header('Accept-Encoding' => 'gzip');
and then you need to make sure that you always refer to 'decoded_content' when dealing with the response object.

For other languages, all you need to do is to add 

Accept-encoding: gzip
to the HTTP request that you send, and then be prepared to deal with a 'content-encoding: gzip' in the response.

Happily, some of the large spiders do support compression -- the googlebot and Yahoo Slurp do (to name but two). Since I started prodding crawler implementors, a couple have implemented compression (one within hours), and another reported that it was a bug that it didn't work -- which would be fixed shortly.

Crawlers which do more than 5% of the total (uncompressed) crawling activity are marked in bold below.

CrawlerLast IP used
DF Bot 1.0" "gladstonefamily.net192.29.97.49
DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)" "gladstonefamily.net136.243.59.237
ia_archiver" "pond1.gladstonefamily.net18.207.120.96
masscan/1.0 (https://github.com/robertdavidgraham/masscan)" "-80.82.77.240
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "blog1.gladstonefamily.net216.244.66.194
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "charon.gladstonefamily.net216.244.66.194
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "gladstone.name216.244.66.228
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "pond.gladstonefamily.net216.244.66.194
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "pond1.gladstonefamily.net216.244.66.194
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "www.gladstone.name216.244.66.228
Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)185.143.223.210
Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)" "gladstonefamily.net217.8.117.47
Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)" "gladstonefamily.net:8080217.8.117.47
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www.xforce-security.com/crawler/)" "blog.gladstonefamily.net194.153.113.35
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www.xforce-security.com/crawler/)" "gladstonefamily.net194.153.113.35
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www.xforce-security.com/crawler/)" "pond.gladstonefamily.net194.153.113.35
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "73.253.74.102206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "blog.gladstonefamily.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "blog1.gladstonefamily.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "c-73-253-74-102.hsd1.ma.comcast.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "charon.gladstonefamily.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "gladstonefamily.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "pond.gladstonefamily.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "pond1.gladstonefamily.net206.253.224.75
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)" "www.gladstonefamily.net206.253.224.75
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0" "gladstonefamily.net45.10.234.5
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.105 Safari/537.36" "73.253.74.10245.40.196.167
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" "pond1.gladstonefamily.net116.203.251.246
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" "73.253.74.102:808080.82.68.115
Comments, problems etc to
Philip Gladstone

Last modified Sunday, 19 November 2006