Export my brain

Load high, who's hitting us?...

Mark Servilla -

Operating a fairly large open environmental data repository seems to be a good target for scrapers. When they hit us, along with Google, our system loads begin to peak above our alarm threshold. When this happens, I am always curious about the source of the scraping, including count statistics to see if there is a cluster of sources. Here is a simple one-liner to get a quick peek of those statistics:

grep -E -o "^([0-9]{1,3}[\.]){3}[0-9]{1,3}" access.log | sort | uniq -c | sort -n

It's a combination of an extended grep regex to get a list of only the IP addresses, a sort of the raw address data to provide fodder for uniq, in which I specify the count option, and then an ascending numerical sort of the count values, along with the corresponding IP addresses.

The final outcome looks like this:

1    104.208.25.229
1    104.208.31.220
1    104.208.32.116
1    104.208.32.158
1    104.208.32.245
1    104.208.32.248
1    104.208.33.161
1    104.208.33.58
1    104.208.34.140
1    104.208.34.172
.
.
.
51   44.206.229.210
55   13.58.19.146
55   34.231.43.114
61   54.200.250.148
78   52.205.195.10
83   66.249.69.42
85   34.236.26.31
199  108.0.0.0
780  66.249.69.40
1915 105.0.0.0
8999 66.249.69.38

The biggest hit is Google (66.249.69.38), followed by a host in South Africa (105.0.0.0).*

*How do I know the origin of these addresses, you ask? I use a Python tool I wrote called ip-lookup. Google's address looks like this:

ip-lookup 66.249.69.38
ip: 66.249.69.38
hostname: crawl-66-249-69-38.googlebot.com
type: ipv4
continent_code: NA
continent_name: North America
country_code: US
country_name: United States
region_code: FL
region_name: Florida
city: Valrico
zip: 33587
latitude: 27.963499069213867
longitude: -82.206298828125
location: {'geoname_id': 4176318, 'capital': 'Washington D.C.', 'languages': [{'code': 'en', 'name': 'English', 'native': 'English'}], 'country_flag': 'https://assets.ipstack.com/flags/us.svg', 'country_flag_emoji': '🇺🇸', 'country_flag_emoji_unicode': 'U+1F1FA U+1F1F8', 'calling_code': '1', 'is_eu': False}

The South African address looks like this:

ip-lookup 105.0.0.0
ip: 105.0.0.0
hostname: 105.0.0.0
type: ipv4
continent_code: AF
continent_name: Africa
country_code: ZA
country_name: South Africa
region_code: GT
region_name: Gauteng
city: Johannesburg
zip: 2000
latitude: -26.199169158935547
longitude: 28.0563907623291
location: {'geoname_id': 993800, 'capital': 'Pretoria', 'languages': [{'code': 'af', 'name': 'Afrikaans', 'native': 'Afrikaans'}, {'code': 'en', 'name': 'English', 'native': 'English'}, {'code': 'nr', 'name': 'South Ndebele', 'native': 'isiNdebele'}, {'code': 'st', 'name': 'Southern Sotho', 'native': 'Sesotho'}, {'code': 'ss', 'name': 'Swati', 'native': 'SiSwati'}, {'code': 'tn', 'name': 'Tswana', 'native': 'Setswana'}, {'code': 'ts', 'name': 'Tsonga', 'native': 'Xitsonga'}, {'code': 've', 'name': 'Venda', 'native': 'Tshivenḓa'}, {'code': 'xh', 'name': 'Xhosa', 'native': 'isiXhosa'}, {'code': 'zu', 'name': 'Zulu', 'native': 'isiZulu'}], 'country_flag': 'https://assets.ipstack.com/flags/za.svg', 'country_flag_emoji': '🇿🇦', 'country_flag_emoji_unicode': 'U+1F1FF U+1F1E6', 'calling_code': '27', 'is_eu': False}