Adding compression support can be very simple -- if your spider is coded in Perl using LWP::UserAgent, then the
addition of a single line of code will enable compression support.
$ua->default_header('Accept-Encoding' => 'gzip');
and then you need to make sure that you always refer to 'decoded_content' when dealing with the response object.
For other languages, all
you need to do is to add
Accept-encoding: gzipto the HTTP request that you send, and then be prepared to deal with a 'content-encoding: gzip' in the response.
Happily, some of the large spiders do support compression -- the googlebot and Yahoo Slurp do (to name but two). Since I started prodding crawler implementors, a couple have implemented compression (one within hours), and another reported that it was a bug that it didn't work -- which would be fixed shortly.
Crawlers which do more than 5% of the total (uncompressed) crawling activity are marked in bold below.
| Crawler | Last IP used |
|---|---|
| (info@domaincrawler.com; http://www.domaincrawler.com/gladstonefamily.net)" "gladstonefamily.net | 185.6.9.216 |
| curl/7.47.0" "www.gladstonefamily.net | 61.204.24.144 |
| ia_archiver" "pond.gladstonefamily.net | 54.173.35.129 |
| ia_archiver" "pond1.gladstonefamily.net | 54.173.35.129 |
| masscan/1.0 (https://github.com/robertdavidgraham/masscan)" "- | 80.82.70.118 |
| Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)" "pond1.gladstonefamily.net | 207.241.229.227 |
| Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)" "blog.gladstonefamily.net | 203.133.169.114 |
| Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)" "www.gladstonefamily.net | 203.133.168.171 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "blog1.gladstonefamily.net | 216.244.66.194 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "charon.gladstonefamily.net | 216.244.66.194 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "gladstone.name | 216.244.66.228 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "pond.gladstonefamily.net | 216.244.66.194 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "pond1.gladstonefamily.net | 216.244.66.194 |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" "www.gladstone.name | 216.244.66.228 |
| Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)" "gladstonefamily.net | 91.242.162.133 |
| Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)" "gladstonefamily.net | 213.174.152.181 |
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36" "pond.gladstonefamily.net | 46.105.99.163 |
| Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:28.0) Gecko/20100101 Firefox/28.0" "73.253.74.102 | 203.195.189.185 |
| Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" "73.253.74.102 | 103.68.224.1 |
| Mozilla/5.0 (X11; Linux x86_64; rv:109)" "blog.gladstonefamily.net | 23.81.27.114 |