9b50151f17
* [chore] Synchronise our robots.txt with upstream * [feature] Add headers to escape AI crawlers This adds 2 headers that a number of AI crawlers respect to signal that content should not be included in their datasets. |
||
|---|---|---|
| .. | ||
| cachecontrol.go | ||
| contentsecuritypolicy.go | ||
| contentsecuritypolicy_test.go | ||
| cors.go | ||
| extraheaders.go | ||
| gzip.go | ||
| headerfilter.go | ||
| headerfilter_test.go | ||
| logger.go | ||
| ratelimit.go | ||
| ratelimit_test.go | ||
| requestid.go | ||
| session.go | ||
| session_test.go | ||
| signaturecheck.go | ||
| throttling.go | ||
| throttling_test.go | ||
| tokencheck.go | ||
| useragent.go | ||
| util.go | ||