mirrors
/
gotosocial
mirror of
1
Fork 0
Code Issues Releases Activity

[chore] Update robots.txt with more AI scrapers (#3248)

This commit is contained in:
Daenney 2024-08-29 17:42:48 +02:00 committed by GitHub
parent 1f3dfbf10c
commit 277b043633
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
internal/web/robots.go
View File

@ -31,6 +31,8 @@ const (
# AI scrapers and the like.
# https://github.com/ai-robots-txt/ai.robots.txt/
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: AdsBot-Google
User-agent: Amazonbot
User-agent: anthropic-ai
@ -50,9 +52,11 @@ User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
@ -61,6 +65,7 @@ User-agent: PetalBot
User-agent: Scrapy
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: YouBot
Disallow: /