Nicolas/website-astro
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(robots.txt): add new bots

Browse source
This commit is contained in:
nico 2025-02-13 11:27:54 +01:00
parent d1ddca8ee4
commit 879b250725
Signed by: Nicolas
SSH key fingerprint: SHA256:ELi8eDeNLl5PTn64G+o2Kx5+XVDfHF5um2tZigfwWkM
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
public/robots.txt
View file

@ -36,7 +36,8 @@ User-agent: PanguBot
User-agent: PerplexityBot User-agent: PerplexityBot
User-agent: PetalBot User-agent: PetalBot
User-agent: Scrapy User-agent: Scrapy
User-agent: SemrushBot User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Sidetrade indexer bot User-agent: Sidetrade indexer bot
User-agent: Timpibot User-agent: Timpibot
User-agent: VelenPublicWebCrawler User-agent: VelenPublicWebCrawler