Robots.txt
• Steve Leroux [Feedwhip admin] • Sat, Jul 01 at 06:57 PM
No comments
Feedwhip doesn't currently look at robots.txt files. That's because we don't walk an entire website -- we only look at specific pages as requested by our users.
When we add functionality to track an entire website, then we'll also parse robots.txt files.
If there's a website you'd like to be excluded from Feedwhip, please let us know.
When we add functionality to track an entire website, then we'll also parse robots.txt files.
If there's a website you'd like to be excluded from Feedwhip, please let us know.