Referer Spam

August 9, 2003 -- About a week ago, I noticed my referer logs were being flooded with requests from a set of porn sites originating in Russia. My theories as to what was happening were wrong.

The problem

The porn site(s) are apparently running some sort of script which lumps together a bunch of sites and uses a simple 1 by 2px image tag to request a URL. These image tags behave almost exactly the same as a user-request would, thereby masking the site's IP behind their visitor's identity and making my server think someone clicked a referring link.

After spending a few more hours than I intended looking for a way to crash or hang Explorer though an image file, I'm giving up. There isn't anyway I could find or figure out to bring down the browser through an image.

The best solution I came up with was an htaccess-based referer blacklist. Because the IPs are so diverse blacklisting them would be irresponsible. Those people, when not surfing for Russian porn, might be customers or looking for something else on my site. I don't care in the slightest about people's pornography habits. What I do care about is a dishonest business, essentially outside any reasonable jurisdiction, using my site without my permission to promote themselves. This interferes with a web metric I use to measure site performance and costs me money in bandwidth. Additionally, this could get affected sites banned by public filtering software. I like being a resource for students and educators, I don't want my site blocked in schools and libraries. Yes this feels like an attempt at vigilante justice, but I don't see any other solution.

My research into sploits and hacks was in an effort to essentially shut down the porn site. It may still be possible, if someone can come up with a particularly destructive image, multiple-header-flood, script-insertion or whatever, post it.

The solution

Here is the section of my .htaccess file which denies requests based on HTTP referer. Between August 3, 2003 and August 9, 2003 my server denied over 13,000 requests from the blacklisted sites. I'll try to stay on top of updating that.

Microsoft deserves some credit. Despite the conventional wisdom that WinMSIE is a security catastrophe, they've been doing a very good job patching holes in their products. There are probably still some sploits out there which will crash the browser, but considering how many millions of people are using MSIE, they've done a remarkable job closing up security holes.

A few ideas that didn't work included:

Server redirects to local files

security hole patched by Microsoft

Server redirects to javascript functions

code not in page, so not executable

Bogus urls which would close the image tag

the contents of the image tag are processed as an image, not text

XML in Photoshop 7 JPEGs

apparently patched by Microsoft

JavaScript embedded in JPEG XML

nothing happened

Notepad popups via server redirect

Doesn't seem to work.

Malformed XBM Files

also patched by Microsoft

Header flooding the request

This didn't seem to do anything except slow the page load. I tried sending 5000 unique headers, about 150k, but nothing seemed to happen except to slow the page transfer. Since it's another burden on my server I didn't bother looking for a larger numbers.

Bogus Content-Length Memory Bug

This might work although it might put undo strain on my server since the porn site will be hitting it constantly. However this could be a nice thing to do if everyone affected could implement it simultaneously. Considering how many people's sites are listed there, it would probably take down their server in a few minutes. I don't want to be responsible for bringing down performance on any sites I share a server with, so I'm not going to be trying this one out.

Oversize Cookie Flood

I didn't try this.

Others were thinking the same way. Bitflux Blog is serving a Buffer Overflow in URLMON.DLL sploit which, while mostly patched, is still crashing quite a few unpatched copies of MSIE. It took down MSIE in Virtual PC after a few minutes.

Some of the other sites being affected by this are listed below. I doubt this list was assembled by hand, until recently all of them appeared to have visible referer listings and probably listed highly in a Google search for 'referers'. Quite a few big names in this list, I'm almost flattered to be included.

• glennf.weblogs.com/
• www.xmlrpc.com/
• awsd.com/
• www.hawaii-nation.org/
• www.metacites.net/
• scriptingnews.userland.com/
• jemisa.editthispage.com/
• wws.editthispage.com/
• backend.userland.com/
• afroginthevalley.weblogs.com/
• www.lights.com/
• seblogging.cognitivearchitects.com/
• dodsonbrown2.weblogger.com/
• jiz.editthispage.com/
• www.bayareawritingproject.org/
• www.bayareawritingproject.org/
• misnomer.editthispage.com/
• www.bsablogs.com/
• www.headangle.com/
• mpt.phrasewise.com/
• anndejong.weblogger.com/
• uzopia.editthispage.com/
• scriptingnews.userland.com/
• blogs.law.harvard.edu/
• www.collins-maxwell.k12.ia.us/
• sirclifford.manilasites.com/
• www.astalavista.net/
• www.helifever.com/
• www.bayareawritingproject.org/
• demo.iwebsoftware.com/
• www.upsoclose.com/
• www.joemaller.com/
• inessential.com/
• reiter.weblogger.com/
• www.lights.com/
• ranchero.com/
• blog.bitflux.ch/
• www.hartfordstreetzen.com/
• nosi.editthispage.com/
• hornygoat.org/
• feistycat.subtlehints.net/
• traumwind.editthispage.com/
• burnallgifs.org/
• www.mycgiscripts.com/
• poptech.manilasites.com/
• www.schoolblogs.com/
• www.onlyoncapitolhill.com/
• xray.bmc.uu.se/
• www.cs.rpi.edu/
• www.parkmall.co.uk/
• adamcurry.editthispage.com/
• reiter.weblogger.com/
• www.lights.com/
• ranchero.com/
• blog.bitflux.ch/
• hornygoat.org/
• feistycat.subtlehints.net/
• traumwind.editthispage.com/

More information on blocking spam with .htaccess from Mark Pilgim's site:
How to block spambots, ban spybots, and tell unwanted robots to go to hell

Update May 1, 2004

I've had a new flood of referer spam from a wide range of IP addresses, but pointing to a specific group of sites. I'm not going to link directly to the offending page, but you can go look if you want: http://www.superface.net

That page has a ton of links various sites which look like lotteries and other crap by their URLs. Anyway, thanks for the list. Here it is in one easy to block list:

apart-design
appartdesign
auktion1
autogewinne24
autospiele24
babay
euromillionen
eurowins
geldspiele24
goovle
heil-fasten
immobiliengewinne24
keywordmaster
reisegewinne24
rootfood
single66
spielepsychatrie
superface
topgewinn24
topspiele24
usa-wins
webmasterplan
wseeker
yahhhoo
yahhoo

I've added those to the .htaccess file as well.

I still hate blacklists.