/2008/03/18/shablastbot-spider-considered-abusive/ Tips and tricks to get technology to work for you, by Michael Boyd Clark Sat, 20 Feb 2016 14:14:51 +0000 hourly 1 /2008/03/18/shablastbot-spider-considered-abusive/comment-page-1/#comment-14797 Mon, 24 Mar 2008 10:16:19 +0000 http://www.planetmike.com/2008/03/18/shablastbot-spider-considered-abusive/#comment-14797 Also got hammered by Shablast – no bot page on their website (in fact not much there at all), so I’ve blocked the user-agent too.

]]> /2008/03/18/shablastbot-spider-considered-abusive/comment-page-1/#comment-14795 Sun, 23 Mar 2008 21:51:09 +0000 http://www.planetmike.com/2008/03/18/shablastbot-spider-considered-abusive/#comment-14795 Also shows up frequently on my servers. Eats 403s. Thankfully only does HEAD requests to anything except the robots.txt so the 403s take less bandwidth (woot!). 🙂

]]> /2008/03/18/shablastbot-spider-considered-abusive/comment-page-1/#comment-14785 Wed, 19 Mar 2008 16:30:28 +0000 http://www.planetmike.com/2008/03/18/shablastbot-spider-considered-abusive/#comment-14785 Well, it requested the robots.txt file, but no idea what it is looking for. And you’re right, my comment wasn’t posted.

]]> /2008/03/18/shablastbot-spider-considered-abusive/comment-page-1/#comment-14784 Wed, 19 Mar 2008 15:39:47 +0000 http://www.planetmike.com/2008/03/18/shablastbot-spider-considered-abusive/#comment-14784 I’ve found the same – quite a few errors being generated by wrongly parsed URLs. I’ve blocked it too. Any idea if it recognises robots.txt. Not much on their web site, and your comment doesn’t seem to have made it onto their blog.

]]>