Scan de CCbot
-
Common Crawl - CCBot
CCBot is Common Crawl's Nutch-based web crawler that makes use of the Apache Hadoop project. We use Map-Reduce to process and extract crawl candidates from our crawl database.
.png)
(commoncrawl.org)
# grep "CCBot" /var/log/apache2/access.*.log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | wc -l 23442344 Ips différentes … misère.
-
# apt-get install geoip-bin # # grep "CCBot" /var/log/apache2/access.*.log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g' 1 BZ, Belize 1 CN, China 1 GA, Gabon 1 GT, Guatemala 1 JM, Jamaica 1 JO, Jordan 1 KG, Kyrgyzstan 1 KR, Korea, Republic of 1 LK, Sri Lanka 1 LT, Lithuania 1 NO, Norway 1 OM, Oman 1 PS, Palestinian Territory 1 SI, Slovenia 1 SV, El Salvador 1 VE, Venezuela 2 AL, Albania 2 AU, Australia 2 AZ, Azerbaijan 2 BA, Bosnia and Herzegovina 2 BH, Bahrain 2 DE, Germany 2 KE, Kenya 2 KW, Kuwait 2 KZ, Kazakhstan 2 NP, Nepal 2 PA, Panama 2 TN, Tunisia 3 BG, Bulgaria 3 BO, Bolivia 3 CZ, Czech Republic 3 JP, Japan 3 PK, Pakistan 3 SE, Sweden 4 UZ, Uzbekistan 5 DO, Dominican Republic 5 HN, Honduras 5 IQ, Iraq 6 IN, India 6 MA, Morocco 6 PE, Peru 7 UY, Uruguay 8 UA, Ukraine 9 BD, Bangladesh 9 EG, Egypt 11 PY, Paraguay 13 CL, Chile 17 CR, Costa Rica 18 ZA, South Africa 22 RU, Russian Federation 26 MX, Mexico 29 CO, Colombia 32 US, United States 38 ID, Indonesia 55 EC, Ecuador 57 IP Address not found 143 AR, Argentina 581 VN, Vietnam 1174 BR, Brazil -
Le retour …
# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | wc -l 1337 # grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g' 1 AL, Albania 1 AM, Armenia 1 AU, Australia 1 BH, Bahrain 1 BN, Brunei Darussalam 1 BO, Bolivia 1 BW, Botswana 1 BZ, Belize 1 CY, Cyprus 1 EE, Estonia 1 GB, United Kingdom 1 GE, Georgia 1 GT, Guatemala 1 HN, Honduras 1 HR, Croatia 1 HU, Hungary 1 KE, Kenya 1 KG, Kyrgyzstan 1 KZ, Kazakhstan 1 LK, Sri Lanka 1 NO, Norway 1 NP, Nepal 1 NR, Nauru 1 PR, Puerto Rico 1 RO, Romania 2 AZ, Azerbaijan 2 DO, Dominican Republic 2 IL, Israel 2 JO, Jordan 2 KW, Kuwait 3 DZ, Algeria 3 HK, Hong Kong 3 PA, Panama 3 PK, Pakistan 4 TN, Tunisia 5 EG, Egypt 5 IN, India 5 IQ, Iraq 5 UZ, Uzbekistan 6 VE, Venezuela 7 MA, Morocco 7 PE, Peru 7 UA, Ukraine 7 ZA, South Africa 9 UY, Uruguay 10 PY, Paraguay 11 BD, Bangladesh 11 MX, Mexico 13 CO, Colombia 13 CR, Costa Rica 14 RU, Russian Federation 15 CL, Chile 25 EC, Ecuador 30 US, United States 33 ID, Indonesia 43 IP Address not found 74 AR, Argentina 379 VN, Vietnam 607 BR, Brazil -
Ajout de 1337 IP … dans le Firewall.
# wc -l /etc/pve/firewall/cluster.fw 7448 /etc/pve/firewall/cluster.fwOn va bien arriver à 10.000 … Misère.
-
Encore des nouveaux :
# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | awk '{print "IN DROP -source " $1 " -p tcp -log notice # CCBot"}' | wc -l 1308 -
Je pense que je devrais bloqué Brésil et Vietnam …
# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g' 1 AL, Albania 1 AU, Australia 1 AZ, Azerbaijan 1 BA, Bosnia and Herzegovina 1 BG, Bulgaria 1 BO, Bolivia 1 BW, Botswana 1 BZ, Belize 1 CN, China 1 DZ, Algeria 1 GT, Guatemala 1 HK, Hong Kong 1 IE, Ireland 1 KW, Kuwait 1 MK, Macedonia 1 MU, Mauritius 1 NO, Norway 1 NP, Nepal 1 PA, Panama 1 PK, Pakistan 1 PL, Poland 1 PS, Palestinian Territory 1 RO, Romania 1 SE, Sweden 1 TW, Taiwan 1 VU, Vanuatu 2 HN, Honduras 2 IQ, Iraq 2 KE, Kenya 2 KZ, Kazakhstan 2 MA, Morocco 2 SG, Singapore 2 UY, Uruguay 2 VE, Venezuela 3 JP, Japan 4 DO, Dominican Republic 4 PE, Peru 4 TN, Tunisia 4 UZ, Uzbekistan 6 PY, Paraguay 6 UA, Ukraine 7 EG, Egypt 8 CR, Costa Rica 9 BD, Bangladesh 9 CL, Chile 12 ZA, South Africa 18 RU, Russian Federation 19 CO, Colombia 25 EC, Ecuador 25 ID, Indonesia 27 US, United States 29 MX, Mexico 37 IP Address not found 81 AR, Argentina 317 VN, Vietnam 655 BR, Brazil -
Cela devient gros … je vais devoir faire de la compression.
# wc -l /etc/pve/firewall/cluster.fw 10014 /etc/pve/firewall/cluster.fw -
Je vais compresser via ses IPs:
# cat /etc/pve/firewall/cluster.fw | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g' | awk '{print $1 "." $2 "." $3 ".0/24"}' | sort -n | uniq -c | sort -n | tail -20 5 14.191.123.0/24 5 14.191.137.0/24 5 14.191.210.0/24 5 177.52.82.0/24 5 177.86.20.0/24 5 177.87.33.0/24 5 187.19.233.0/24 5 187.73.24.0/24 5 190.102.47.0/24 5 195.178.110.0/24 5 201.162.73.0/24 5 216.98.214.0/24 5 78.153.140.0/24 6 14.191.95.0/24 6 18.97.9.0/24 7 14.191.196.0/24 7 185.177.72.0/24 7 189.84.180.0/24 9 23.178.112.0/24 11 186.158.200.0/24 -
Et voila :
# cat /etc/pve/firewall/cluster.fw | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g' | awk '{print $1 "." $2 "." $3 ".0/24"}' | sort -n | uniq -c | sort -n | awk '{print "IN DROP -source " $2 " -p tcp -log notice # CCBot compress"}' | tail -20 IN DROP -source 14.191.123.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.137.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.210.0/24 -p tcp -log notice # CCBot compress IN DROP -source 177.52.82.0/24 -p tcp -log notice # CCBot compress IN DROP -source 177.86.20.0/24 -p tcp -log notice # CCBot compress IN DROP -source 177.87.33.0/24 -p tcp -log notice # CCBot compress IN DROP -source 187.19.233.0/24 -p tcp -log notice # CCBot compress IN DROP -source 187.73.24.0/24 -p tcp -log notice # CCBot compress IN DROP -source 190.102.47.0/24 -p tcp -log notice # CCBot compress IN DROP -source 195.178.110.0/24 -p tcp -log notice # CCBot compress IN DROP -source 201.162.73.0/24 -p tcp -log notice # CCBot compress IN DROP -source 216.98.214.0/24 -p tcp -log notice # CCBot compress IN DROP -source 78.153.140.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.95.0/24 -p tcp -log notice # CCBot compress IN DROP -source 18.97.9.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.196.0/24 -p tcp -log notice # CCBot compress IN DROP -source 185.177.72.0/24 -p tcp -log notice # CCBot compress IN DROP -source 189.84.180.0/24 -p tcp -log notice # CCBot compress IN DROP -source 186.158.200.0/24 -p tcp -log notice # CCBot compress -
La compression n’est pas énorme :
# wc -l /etc/pve/firewall/cluster.fw 8785 /etc/pve/firewall/cluster.fw -
-
Nouveau ajout de CCBot :
# wc -l /etc/pve/firewall/cluster.fw 9251 /etc/pve/firewall/cluster.fw # wc -l /etc/pve/firewall/cluster.fw 9952 /etc/pve/firewall/cluster.fw -
Je vais devoir faire encore une compression :
# grep "# CCBot" /etc/pve/firewall/cluster.fw | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g' | awk '{print $1 "." $2 "." $3 ".0/24"}' | sort -n | uniq -c | sort -n | awk '{print "IN DROP -source " $2 " -p tcp -log notice # CCBot compress"}' | tail -10 IN DROP -source 177.131.178.0/24 -p tcp -log notice # CCBot compress IN DROP -source 177.152.87.0/24 -p tcp -log notice # CCBot compress IN DROP -source 177.54.199.0/24 -p tcp -log notice # CCBot compress IN DROP -source 181.91.86.0/24 -p tcp -log notice # CCBot compress IN DROP -source 187.180.212.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.161.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.163.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.230.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.25.0/24 -p tcp -log notice # CCBot compress IN DROP -source 14.191.92.0/24 -p tcp -log notice # CCBot compress -
Je suis tellement gonflé que je vais mettre :
IN DROP -source 14.191.0.0/16 -p tcp -log notice # CCBot compress Vietnam -
En gros :
# grep "# CCBot" /etc/pve/firewall/cluster.fw | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g' | awk '{print $1 "." $2 ".0.0"}' | sort -n | uniq -c | sort -n | tail 41 179.125.0.0 42 177.37.0.0 54 113.172.0.0 55 123.21.0.0 67 14.186.0.0 67 14.187.0.0 73 113.173.0.0 73 14.169.0.0 83 123.20.0.0 221 14.191.0.0 -
Finalement je vais mettre :
IN DROP -source 14.169.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 123.20.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 14.191.0.0/16 -p tcp -log notice # CCBot compress Vietnam -
Je pense qu’il faut bloquer tous le Vietnam :
IN DROP -source 113.172.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 123.21.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 14.186.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 14.187.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 113.173.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 14.169.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 123.20.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 14.191.0.0/16 -p tcp -log notice # CCBot compress Vietnam -
Première fois que j’ai une liste aussi courte …
IN DROP -source 14.244.113.213 -p tcp -log notice # CCBot IN DROP -source 36.76.141.212 -p tcp -log notice # CCBot IN DROP -source 39.194.5.127 -p tcp -log notice # CCBot IN DROP -source 45.182.243.249 -p tcp -log notice # CCBot IN DROP -source 45.239.229.164 -p tcp -log notice # CCBot IN DROP -source 117.5.147.154 -p tcp -log notice # CCBot IN DROP -source 123.16.246.87 -p tcp -log notice # CCBot IN DROP -source 125.167.51.134 -p tcp -log notice # CCBot IN DROP -source 138.117.55.41 -p tcp -log notice # CCBot IN DROP -source 138.59.239.70 -p tcp -log notice # CCBot IN DROP -source 152.174.97.241 -p tcp -log notice # CCBot IN DROP -source 170.150.132.229 -p tcp -log notice # CCBot IN DROP -source 170.246.81.226 -p tcp -log notice # CCBot IN DROP -source 177.129.25.77 -p tcp -log notice # CCBot IN DROP -source 177.184.101.224 -p tcp -log notice # CCBot IN DROP -source 177.189.108.89 -p tcp -log notice # CCBot IN DROP -source 177.220.186.160 -p tcp -log notice # CCBot IN DROP -source 179.125.149.52 -p tcp -log notice # CCBot IN DROP -source 181.209.78.10 -p tcp -log notice # CCBot IN DROP -source 181.46.185.101 -p tcp -log notice # CCBot IN DROP -source 186.248.207.130 -p tcp -log notice # CCBot IN DROP -source 200.100.17.102 -p tcp -log notice # CCBot IN DROP -source 200.236.234.166 -p tcp -log notice # CCBot IN DROP -source 201.13.60.69 -p tcp -log notice # CCBot IN DROP -source 201.50.138.239 -p tcp -log notice # CCBot IN DROP -source 202.59.194.99 -p tcp -log notice # CCBot -
Je viens de l’ajouter à ma liste …
je vais aussi ajouter :
IN DROP -source 14.188.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 1.54.0.0/16 -p tcp -log notice # CCBot compress Vietnam IN DROP -source 36.76.0.0/16 -p tcp -log notice # CCBot compress Indonesie IN DROP -source 45.182.0.0/16 -p tcp -log notice # CCBot compress Bresil -
Aie … uniquement 3 aujourd’hui :
# grep '"CCBot"' /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | awk '{print "IN REJECT -source " $1 " -p tcp -log notice # CCBot"}' IN REJECT -source 114.130.186.201 -p tcp -log notice # CCBot IN REJECT -source 177.234.217.135 -p tcp -log notice # CCBot IN REJECT -source 190.97.224.56 -p tcp -log notice # CCBot
Bonjour ! Vous semblez intéressé par cette conversation, mais vous n’avez pas encore de compte.
Marre de refaire défiler les mêmes messages ? Créez un compte pour retrouver votre position, recevoir des notifications des nouvelles réponses, sauvegarder vos favoris et voter pour les messages que vous appréciez.
Grâce à votre participation, ce message peut devenir encore meilleur 💗
S'inscrire Se connecter