<![CDATA[Scan de CCbot]]>
Link Preview Image
Common Crawl - CCBot

CCBot is Common Crawl's Nutch-based web crawler that makes use of the Apache Hadoop project. We use Map-Reduce to process and extract crawl candidates from our crawl database.

favicon

(commoncrawl.org)

 

# grep "CCBot"  /var/log/apache2/access.*.log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | wc -l
2344

2344 Ips différentes … misère.

]]>https://lemmy.cyber-neurones.org/topic/194/scan-de-ccbotRSS for NodeThu, 30 Apr 2026 11:36:06 GMTTue, 14 Oct 2025 19:15:26 GMT60<![CDATA[Reply to Scan de CCbot on Sat, 08 Nov 2025 14:23:58 GMT]]>Le prix sur les architectures de CCbot est énorme :

image.png

]]>https://lemmy.cyber-neurones.org/post/539https://lemmy.cyber-neurones.org/post/539Sat, 08 Nov 2025 14:23:58 GMT<![CDATA[Reply to Scan de CCbot on Sat, 08 Nov 2025 14:22:38 GMT]]>On dirait que le blocage de 3% des IPs à porté ses fruits :

image.png

]]>https://lemmy.cyber-neurones.org/post/538https://lemmy.cyber-neurones.org/post/538Sat, 08 Nov 2025 14:22:38 GMT<![CDATA[Reply to Scan de CCbot on Sat, 08 Nov 2025 14:21:47 GMT]]>Aie … uniquement 3 aujourd’hui :

# grep '"CCBot"' /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | awk '{print "IN REJECT -source " $1 " -p tcp -log notice # CCBot"}' 
IN REJECT -source 114.130.186.201 -p tcp -log notice # CCBot
IN REJECT -source 177.234.217.135 -p tcp -log notice # CCBot
IN REJECT -source 190.97.224.56 -p tcp -log notice # CCBot
]]>https://lemmy.cyber-neurones.org/post/537https://lemmy.cyber-neurones.org/post/537Sat, 08 Nov 2025 14:21:47 GMT<![CDATA[Reply to Scan de CCbot on Wed, 29 Oct 2025 15:46:12 GMT]]>Je viens de l’ajouter à ma liste …

je vais aussi ajouter :

IN DROP -source 14.188.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 1.54.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 36.76.0.0/16 -p tcp -log notice # CCBot compress Indonesie
IN DROP -source 45.182.0.0/16 -p tcp -log notice # CCBot compress Bresil
]]>https://lemmy.cyber-neurones.org/post/472https://lemmy.cyber-neurones.org/post/472Wed, 29 Oct 2025 15:46:12 GMT<![CDATA[Reply to Scan de CCbot on Wed, 29 Oct 2025 15:40:54 GMT]]>Première fois que j’ai une liste aussi courte …

IN DROP -source 14.244.113.213 -p tcp -log notice # CCBot
IN DROP -source 36.76.141.212 -p tcp -log notice # CCBot
IN DROP -source 39.194.5.127 -p tcp -log notice # CCBot
IN DROP -source 45.182.243.249 -p tcp -log notice # CCBot
IN DROP -source 45.239.229.164 -p tcp -log notice # CCBot
IN DROP -source 117.5.147.154 -p tcp -log notice # CCBot
IN DROP -source 123.16.246.87 -p tcp -log notice # CCBot
IN DROP -source 125.167.51.134 -p tcp -log notice # CCBot
IN DROP -source 138.117.55.41 -p tcp -log notice # CCBot
IN DROP -source 138.59.239.70 -p tcp -log notice # CCBot
IN DROP -source 152.174.97.241 -p tcp -log notice # CCBot
IN DROP -source 170.150.132.229 -p tcp -log notice # CCBot
IN DROP -source 170.246.81.226 -p tcp -log notice # CCBot
IN DROP -source 177.129.25.77 -p tcp -log notice # CCBot
IN DROP -source 177.184.101.224 -p tcp -log notice # CCBot
IN DROP -source 177.189.108.89 -p tcp -log notice # CCBot
IN DROP -source 177.220.186.160 -p tcp -log notice # CCBot
IN DROP -source 179.125.149.52 -p tcp -log notice # CCBot
IN DROP -source 181.209.78.10 -p tcp -log notice # CCBot
IN DROP -source 181.46.185.101 -p tcp -log notice # CCBot
IN DROP -source 186.248.207.130 -p tcp -log notice # CCBot
IN DROP -source 200.100.17.102 -p tcp -log notice # CCBot
IN DROP -source 200.236.234.166 -p tcp -log notice # CCBot
IN DROP -source 201.13.60.69 -p tcp -log notice # CCBot
IN DROP -source 201.50.138.239 -p tcp -log notice # CCBot
IN DROP -source 202.59.194.99 -p tcp -log notice # CCBot
]]>https://lemmy.cyber-neurones.org/post/471https://lemmy.cyber-neurones.org/post/471Wed, 29 Oct 2025 15:40:54 GMT<![CDATA[Reply to Scan de CCbot on Tue, 28 Oct 2025 19:44:42 GMT]]>Je pense qu’il faut bloquer tous le Vietnam :

IN DROP -source 113.172.0.0/16  -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 123.21.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 14.186.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 14.187.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 113.173.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 14.169.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 123.20.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 14.191.0.0/16 -p tcp -log notice # CCBot compress Vietnam
]]>https://lemmy.cyber-neurones.org/post/468https://lemmy.cyber-neurones.org/post/468Tue, 28 Oct 2025 19:44:42 GMT<![CDATA[Reply to Scan de CCbot on Tue, 28 Oct 2025 19:40:07 GMT]]>Finalement je vais mettre :

IN DROP -source 14.169.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 123.20.0.0/16 -p tcp -log notice # CCBot compress Vietnam
IN DROP -source 14.191.0.0/16 -p tcp -log notice # CCBot compress Vietnam
]]>https://lemmy.cyber-neurones.org/post/467https://lemmy.cyber-neurones.org/post/467Tue, 28 Oct 2025 19:40:07 GMT<![CDATA[Reply to Scan de CCbot on Tue, 28 Oct 2025 19:38:07 GMT]]>En gros :

# grep "# CCBot" /etc/pve/firewall/cluster.fw  | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g'  | awk '{print $1 "." $2 ".0.0"}' | sort -n | uniq -c  | sort -n | tail
     41 179.125.0.0
     42 177.37.0.0
     54 113.172.0.0
     55 123.21.0.0
     67 14.186.0.0
     67 14.187.0.0
     73 113.173.0.0
     73 14.169.0.0
     83 123.20.0.0
    221 14.191.0.0
]]>https://lemmy.cyber-neurones.org/post/466https://lemmy.cyber-neurones.org/post/466Tue, 28 Oct 2025 19:38:07 GMT<![CDATA[Reply to Scan de CCbot on Tue, 28 Oct 2025 19:33:15 GMT]]>Je suis tellement gonflé que je vais mettre :

IN DROP -source 14.191.0.0/16 -p tcp -log notice # CCBot compress Vietnam
]]>https://lemmy.cyber-neurones.org/post/465https://lemmy.cyber-neurones.org/post/465Tue, 28 Oct 2025 19:33:15 GMT<![CDATA[Reply to Scan de CCbot on Tue, 28 Oct 2025 19:28:00 GMT]]>Je vais devoir faire encore une compression :

# grep "# CCBot" /etc/pve/firewall/cluster.fw  | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g'  | awk '{print $1 "." $2 "." $3 ".0/24"}' | sort -n | uniq -c | sort -n | awk '{print "IN DROP -source " $2 " -p tcp -log notice # CCBot compress"}' | tail -10
IN DROP -source 177.131.178.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 177.152.87.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 177.54.199.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 181.91.86.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 187.180.212.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.161.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.163.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.230.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.25.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.92.0/24 -p tcp -log notice # CCBot compress
]]>https://lemmy.cyber-neurones.org/post/464https://lemmy.cyber-neurones.org/post/464Tue, 28 Oct 2025 19:28:00 GMT<![CDATA[Reply to Scan de CCbot on Tue, 28 Oct 2025 19:26:43 GMT]]>Nouveau ajout de CCBot :

# wc -l /etc/pve/firewall/cluster.fw 
9251 /etc/pve/firewall/cluster.fw
# wc -l /etc/pve/firewall/cluster.fw
9952 /etc/pve/firewall/cluster.fw
]]>https://lemmy.cyber-neurones.org/post/463https://lemmy.cyber-neurones.org/post/463Tue, 28 Oct 2025 19:26:43 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 08:56:00 GMT]]>Mise à jours de https://git.cyber-neurones.org/farias/iptablesCCBot

Link Preview Image
iptablesCCBot/iptables-CCbot.txt at main

iptablesCCBot

favicon

Forgejo de cyber-neurones.org: Jusqu'ici tout va bien. (git.cyber-neurones.org)

]]>https://lemmy.cyber-neurones.org/post/462https://lemmy.cyber-neurones.org/post/462Mon, 27 Oct 2025 08:56:00 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:50:23 GMT]]>La compression n’est pas énorme :

# wc -l /etc/pve/firewall/cluster.fw 
8785 /etc/pve/firewall/cluster.fw
]]>https://lemmy.cyber-neurones.org/post/460https://lemmy.cyber-neurones.org/post/460Mon, 27 Oct 2025 07:50:23 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:48:04 GMT]]>Et voila :

# cat /etc/pve/firewall/cluster.fw  | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g'  | awk '{print $1 "." $2 "." $3 ".0/24"}' | sort -n | uniq -c | sort -n | awk '{print "IN DROP -source " $2 " -p tcp -log notice # CCBot compress"}' | tail -20
IN DROP -source 14.191.123.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.137.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.210.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 177.52.82.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 177.86.20.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 177.87.33.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 187.19.233.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 187.73.24.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 190.102.47.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 195.178.110.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 201.162.73.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 216.98.214.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 78.153.140.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.95.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 18.97.9.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 14.191.196.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 185.177.72.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 189.84.180.0/24 -p tcp -log notice # CCBot compress
IN DROP -source 186.158.200.0/24 -p tcp -log notice # CCBot compress
]]>https://lemmy.cyber-neurones.org/post/459https://lemmy.cyber-neurones.org/post/459Mon, 27 Oct 2025 07:48:04 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:31:58 GMT]]>Je vais compresser via ses IPs:

# cat /etc/pve/firewall/cluster.fw  | awk '{print $4}' | grep -v "^$" | grep -v "/" | sort -n | sed 's/\./ /g'  | awk '{print $1 "." $2 "." $3 ".0/24"}' | sort -n | uniq -c | sort -n | tail -20
      5 14.191.123.0/24
      5 14.191.137.0/24
      5 14.191.210.0/24
      5 177.52.82.0/24
      5 177.86.20.0/24
      5 177.87.33.0/24
      5 187.19.233.0/24
      5 187.73.24.0/24
      5 190.102.47.0/24
      5 195.178.110.0/24
      5 201.162.73.0/24
      5 216.98.214.0/24
      5 78.153.140.0/24
      6 14.191.95.0/24
      6 18.97.9.0/24
      7 14.191.196.0/24
      7 185.177.72.0/24
      7 189.84.180.0/24
      9 23.178.112.0/24
     11 186.158.200.0/24
]]>https://lemmy.cyber-neurones.org/post/458https://lemmy.cyber-neurones.org/post/458Mon, 27 Oct 2025 07:31:58 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:28:01 GMT]]>Cela devient gros … je vais devoir faire de la compression.

# wc -l /etc/pve/firewall/cluster.fw 
10014 /etc/pve/firewall/cluster.fw
]]>https://lemmy.cyber-neurones.org/post/457https://lemmy.cyber-neurones.org/post/457Mon, 27 Oct 2025 07:28:01 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:21:35 GMT]]>Je pense que je devrais bloqué Brésil et Vietnam …

# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g'
      1 AL, Albania
      1 AU, Australia
      1 AZ, Azerbaijan
      1 BA, Bosnia and Herzegovina
      1 BG, Bulgaria
      1 BO, Bolivia
      1 BW, Botswana
      1 BZ, Belize
      1 CN, China
      1 DZ, Algeria
      1 GT, Guatemala
      1 HK, Hong Kong
      1 IE, Ireland
      1 KW, Kuwait
      1 MK, Macedonia
      1 MU, Mauritius
      1 NO, Norway
      1 NP, Nepal
      1 PA, Panama
      1 PK, Pakistan
      1 PL, Poland
      1 PS, Palestinian Territory
      1 RO, Romania
      1 SE, Sweden
      1 TW, Taiwan
      1 VU, Vanuatu
      2 HN, Honduras
      2 IQ, Iraq
      2 KE, Kenya
      2 KZ, Kazakhstan
      2 MA, Morocco
      2 SG, Singapore
      2 UY, Uruguay
      2 VE, Venezuela
      3 JP, Japan
      4 DO, Dominican Republic
      4 PE, Peru
      4 TN, Tunisia
      4 UZ, Uzbekistan
      6 PY, Paraguay
      6 UA, Ukraine
      7 EG, Egypt
      8 CR, Costa Rica
      9 BD, Bangladesh
      9 CL, Chile
     12 ZA, South Africa
     18 RU, Russian Federation
     19 CO, Colombia
     25 EC, Ecuador
     25 ID, Indonesia
     27 US, United States
     29 MX, Mexico
     37 IP Address not found
     81 AR, Argentina
    317 VN, Vietnam
    655 BR, Brazil
]]>https://lemmy.cyber-neurones.org/post/456https://lemmy.cyber-neurones.org/post/456Mon, 27 Oct 2025 07:21:35 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:20:43 GMT]]>Encore des nouveaux :

# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | awk '{print "IN DROP -source " $1 " -p tcp -log notice # CCBot"}' | wc -l
1308
]]>https://lemmy.cyber-neurones.org/post/455https://lemmy.cyber-neurones.org/post/455Mon, 27 Oct 2025 07:20:43 GMT<![CDATA[Reply to Scan de CCbot on Mon, 27 Oct 2025 07:20:14 GMT]]>Ajout de 1337 IP … dans le Firewall.

# wc -l /etc/pve/firewall/cluster.fw
7448 /etc/pve/firewall/cluster.fw

On va bien arriver à 10.000 … Misère.

]]>https://lemmy.cyber-neurones.org/post/454https://lemmy.cyber-neurones.org/post/454Mon, 27 Oct 2025 07:20:14 GMT<![CDATA[Reply to Scan de CCbot on Fri, 24 Oct 2025 08:22:01 GMT]]>Le retour …

# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | wc -l
1337
# grep "CCBot" /var/log/apache2/acces*log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g'
      1 AL, Albania
      1 AM, Armenia
      1 AU, Australia
      1 BH, Bahrain
      1 BN, Brunei Darussalam
      1 BO, Bolivia
      1 BW, Botswana
      1 BZ, Belize
      1 CY, Cyprus
      1 EE, Estonia
      1 GB, United Kingdom
      1 GE, Georgia
      1 GT, Guatemala
      1 HN, Honduras
      1 HR, Croatia
      1 HU, Hungary
      1 KE, Kenya
      1 KG, Kyrgyzstan
      1 KZ, Kazakhstan
      1 LK, Sri Lanka
      1 NO, Norway
      1 NP, Nepal
      1 NR, Nauru
      1 PR, Puerto Rico
      1 RO, Romania
      2 AZ, Azerbaijan
      2 DO, Dominican Republic
      2 IL, Israel
      2 JO, Jordan
      2 KW, Kuwait
      3 DZ, Algeria
      3 HK, Hong Kong
      3 PA, Panama
      3 PK, Pakistan
      4 TN, Tunisia
      5 EG, Egypt
      5 IN, India
      5 IQ, Iraq
      5 UZ, Uzbekistan
      6 VE, Venezuela
      7 MA, Morocco
      7 PE, Peru
      7 UA, Ukraine
      7 ZA, South Africa
      9 UY, Uruguay
     10 PY, Paraguay
     11 BD, Bangladesh
     11 MX, Mexico
     13 CO, Colombia
     13 CR, Costa Rica
     14 RU, Russian Federation
     15 CL, Chile
     25 EC, Ecuador
     30 US, United States
     33 ID, Indonesia
     43 IP Address not found
     74 AR, Argentina
    379 VN, Vietnam
    607 BR, Brazil
]]>https://lemmy.cyber-neurones.org/post/442https://lemmy.cyber-neurones.org/post/442Fri, 24 Oct 2025 08:22:01 GMT<![CDATA[Reply to Scan de CCbot on Tue, 14 Oct 2025 21:11:06 GMT]]># apt-get install geoip-bin # # grep "CCBot" /var/log/apache2/access.*.log | sed 's/:/ /g' | awk '{print $2}' | sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g' 1 BZ, Belize 1 CN, China 1 GA, Gabon 1 GT, Guatemala 1 JM, Jamaica 1 JO, Jordan 1 KG, Kyrgyzstan 1 KR, Korea, Republic of 1 LK, Sri Lanka 1 LT, Lithuania 1 NO, Norway 1 OM, Oman 1 PS, Palestinian Territory 1 SI, Slovenia 1 SV, El Salvador 1 VE, Venezuela 2 AL, Albania 2 AU, Australia 2 AZ, Azerbaijan 2 BA, Bosnia and Herzegovina 2 BH, Bahrain 2 DE, Germany 2 KE, Kenya 2 KW, Kuwait 2 KZ, Kazakhstan 2 NP, Nepal 2 PA, Panama 2 TN, Tunisia 3 BG, Bulgaria 3 BO, Bolivia 3 CZ, Czech Republic 3 JP, Japan 3 PK, Pakistan 3 SE, Sweden 4 UZ, Uzbekistan 5 DO, Dominican Republic 5 HN, Honduras 5 IQ, Iraq 6 IN, India 6 MA, Morocco 6 PE, Peru 7 UY, Uruguay 8 UA, Ukraine 9 BD, Bangladesh 9 EG, Egypt 11 PY, Paraguay 13 CL, Chile 17 CR, Costa Rica 18 ZA, South Africa 22 RU, Russian Federation 26 MX, Mexico 29 CO, Colombia 32 US, United States 38 ID, Indonesia 55 EC, Ecuador 57 IP Address not found 143 AR, Argentina 581 VN, Vietnam 1174 BR, Brazil ]]>https://lemmy.cyber-neurones.org/post/406https://lemmy.cyber-neurones.org/post/406Tue, 14 Oct 2025 21:11:06 GMT