Change_Is_Good
Bug reports, feature requests, and suggestions from Feedwhip's users.
Comments
| re: Ignore Robots • Steve Leroux [Feedwhip admin] • Sat, Feb 03 at 09:38 PM | No comments |
This is an interesting suggestion, but I don't think I'm going to make the change.
If someone doesn't want Feedwhip crawling their site, then I need to respect that. The problem is that even though Feedwhip is what I think of as a "benign" crawler, most people just block *all* web crawlers.
The only thing I can suggest is to talk to the owners of the websites you're interested in and see if they'll add an exception for Feedwhip. It's easy to do, and that'll give you a pretty direct answer as to whether they want Feedwhip looking at their website.
If someone doesn't want Feedwhip crawling their site, then I need to respect that. The problem is that even though Feedwhip is what I think of as a "benign" crawler, most people just block *all* web crawlers.
The only thing I can suggest is to talk to the owners of the websites you're interested in and see if they'll add an exception for Feedwhip. It's easy to do, and that'll give you a pretty direct answer as to whether they want Feedwhip looking at their website.
I hate getting the filled up e-mail box and would rather use RSS to see when thes pages change. So I tried feedwhip, but found that many of the sites I am intersted in prohibit robots from viewing their sites.
Could you make an option to ignore this block so I can use your very nice tool? I don't mind if it only checks very infrequently for these sites which don't allow robots.
Regards,
Paul