![]() * [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers * use switch for RobotsHeaders |
||
---|---|---|
.. | ||
hostmeta.go | ||
hostmetaget.go |
![]() * [feature] Use `X-Robots-Tag` headers to instruct scrapers/crawlers * use switch for RobotsHeaders |
||
---|---|---|
.. | ||
hostmeta.go | ||
hostmetaget.go |