General
Open discussion about anything Feedwhippy.
| Obey Robots.txt • Adam Paul • Wed, Jun 28 at 05:26 AM | View comments (1) |
I am a new user and tracking a web page. One question - whether feedwhip obey the robots.txt of that domain.
Comments
| Robots.txt • Steve Leroux [Feedwhip admin] • Sat, Jul 01 at 08:57 PM | No comments |
Feedwhip doesn't currently look at robots.txt files. That's because we don't walk an entire website -- we only look at specific pages as requested by our users.
When we add functionality to track an entire website, then we'll also parse robots.txt files.
If there's a website you'd like to be excluded from Feedwhip, please let us know.
When we add functionality to track an entire website, then we'll also parse robots.txt files.
If there's a website you'd like to be excluded from Feedwhip, please let us know.
