mirrors
/
gotosocial
mirror of
1
Fork 0
Code Issues Releases Activity

[feature] Block Amazonbot (#2692) 

Blocks the Amazon crawler bot.

Closes: #2686
This commit is contained in:
Daenney 2024-02-27 14:25:08 +01:00 committed by GitHub
parent 9cadc764b3
commit 6528592dd2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
internal/web/robots.go
View File

@ -64,6 +64,11 @@ Disallow: /
User-agent: WellKnownBot
Disallow: /
# Block Amazonbot, because Amazon.
# https://developer.amazon.com/amazonbot
User-agent: Amazonbot
Disallow: /
# Rules for everything else.
User-agent: *
Crawl-delay: 500