Flashes of Panic

Have you looked in your traffic logs and found a referring site called www.webdevboard.com in the statistics? Here’s why: they’re spidering the web (ignoring the web robots standards in the process) and essentially spamming your referrers logs in the process.

If you’re like me, you look at the referring site to see who’s linking to you. If you follow the link from www.webdevboard.com, you’ll be none the wiser; there’s no link to your site. It’s only when you grep the logs themselves, not your reports, that you find a link to a thread on their forums.

You need to register to read their forums, so I’m going to save you the trouble:

If you have been sent to this URL it is likely you found our bot crawling your site, but perhaps you are wondering why? …. Myself and M0nkey are currently working on a new project to help webmasters everywhere. This project is a broken links reporter. In the future we will be offering a service to webmasters, a helping hand if you like.

Our bot (who we are still trying to name) will eventually crawl thousands of website URL’s daily sourcing out broken links, and other HTML errors across your site. Upon finding all these errors, once a month it will email the administator of this domain to inform him of all the problem areas on their site and inform them how to fix it. This bot will continue to crawl these sites monthly and give reports out to all those sites that have link problems. Of course however, you will be also be able to remove your email from our list once this bot gets moving.

…we are currently in testing stages, and we are currently storing website information (URL’s, administator emails, etc) in order to release our bot in the near future to help you with your website. We have created this information page to let webmasters know why we are visiting them, and also give them a chance to give us some feedback on our new project. Thanks guys, and goodluck with your site/s!

This post is followed by a series of responses, about 90% pointing out the following serious flaws in this plan:

First, we already get enough spam. They’re spidering the web in search of email addresses, and sending unrequested mail to those addresses. Sounds like spam to me. Sure, we’ll be able to unsubscribe “once we get the first email” but oh, please, doesn’t every spam offer us the same option?

Second, the spider doesn’t follow robots.txt. Very, very poor form.

Third, instead of reporting itself in the “UserAgent” field, like responsible robots, you have to look in “referrers” instead to find out what it is. Why? Because they want their url showing up on all the sites which list recent referrers on their pages, which the Googlebot will then see. It’s called referrer spam, and it’s remarkably un-classy.

Fourth, once you do figure out where to look for information about this bot, you have to register for their forum to even see the explanation of what the bot is. Requiring the webmaster to jump hoops like this leaves a poor taste in my mouth; why not put the robot information on a page without restricted access? They tried to explain this, but the explanation doesn’t hold water.

Don’t register for these yahoos’ site. Don’t display their phony referrer information on your website. In fact, if you’ve got that control, you might want to consider blocking requests including that referrer, or requests coming from the IP 80.202.228.57.

As one victim posted:

You wasted my time and harvested my email, I ban your IP 80.202.228.57

If you change it I will ban the new number.

A very bad start for a dubious service, I check my own links, thank you.

(By the way, I’m aware that “referer” is misspelled in the title of this post. This is a long-standing quirk in web statistics; sometime early in the dawn of the web, it was misspelled this way in the NCSA web server configuration, and the error has spread too widely to be successfully corrected. I used the single-‘r’ spelling here for the search engines.)

(Is this boring as all get-out for anyone who reads here regularly? Yes, and I’m sorry. But I’m writing for robots and I want this post found by anyone looking for these idiots in the search engines.)

Now playing: Nine to Five from Live @ Elboroom Chicago, IL, July 2002 by Patiokings