Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
Geezus_42
45 days ago
|
parent
|
context
|
favorite
| on:
So You Want to Define a Well-Known URI
> add a robots.txt
Which bots will then ignore.
Ferret7446
44 days ago
[–]
I didn't expect I'd get to use this so soon. Obligatory canned advisory:
https://www.felesatra.moe/blog/2026/06/19/robots
Geezus_42
44 days ago
|
parent
[–]
They still ignore it and scan everything even though you have told them not to scan certain paths.
Consider applying for YC's Fall 2026 batch!
Applications
are open till July 27.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Which bots will then ignore.