Recently I have been paying closer than usual attention to my apache server logs, and noticed a few new services that are visiting much more frequently than I think they need to. It’s really not necessary to come every 30 minutes, is it? No, it’s not. A spider service should use a more intelligent method of checking a site. For example, maybe only come once per day. Then if the site regularly has multiple postings on a day, increase to checking more frequently. But starting out by checking every 30 minutes is overkill, and bordering on abuse. Another annoying thing is that the spider in question checked my robots.txt file once, and is now reading my feed (and not even my Feedburner feed) every 30 minutes. I’ve emailed them, if I don’t get a response by Friday evening, I’ll simply block their IP address range.
Well, tomorrow when you have a couple more mikes writing articles here, you will need it to come every 30 minutes. I would imagine cnn or foxnews get spired every 5 minutes. 😉
Could you perhaps explain why every 30 min is to often for spiders to come calling.Does that take up bandwidth or something.I know most people do not post every 30 minutes but would have thought perhaps 6 times in a 24 hour period would be okay?
A site like CNN or Foxnews has news, which will change frequently. But 99% of the web sites out there don’t need to be spidered any more frequently than once a day.