<![CDATA[Ajout d'un robots.txt à Git]]>Creation d’un fichier /var/www/html/robots.txt

User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

Ajout dans ma conf apache2 :

<Location "/robots.txt">
 SetHandler None
 Require all granted
</Location>
Alias /robots.txt /var/www/html/robots.txt
]]>https://lemmy.cyber-neurones.org/topic/239/ajout-d-un-robots.txt-à-gitRSS for NodeThu, 30 Apr 2026 12:53:03 GMTThu, 06 Nov 2025 17:15:37 GMT60<![CDATA[Reply to Ajout d'un robots.txt à Git on Thu, 06 Nov 2025 17:17:32 GMT]]>Misère.

# grep "robots.txt" /var/log/apache2/access*.log | awk '{print $12}' | sort -n | uniq -c
      2 "-"
      3 "caveman-hunter/0.0.0
   2367 "CCBot"
      3 "facebookexternalhit/1.1
      2 "FediDB/0.5.0;
      4 "FediIndex/1.0
      1 "Go-http-client/1.1"
      1 "Mastodon/4.6.0-nightly.2025-11-06
      1 "Minoru's
     80 "Mozilla/5.0
      6 "Mozilla/5.0"
]]>https://lemmy.cyber-neurones.org/post/528https://lemmy.cyber-neurones.org/post/528Thu, 06 Nov 2025 17:17:32 GMT