Is Gmail Broken?

A couple weeks ago I temporarily switched our company email over to Google Apps Gmail. The switch only lasted 36 hours because, without explanation or recourse, Google suspended several of our users–including one of the owners. I had no administrative control over our accounts or access to their data on our Google-hosted services (Mail, Docs, Calendar etc.) Based on Google’s Gmail support forums, wrongful account suspensions are common. This effectively ended our experiment with Google Apps, which I can no longer recommend as a realistic solution for small businesses.

But that’s not what this post is about.

Immediately after switching our email, we noticed a significant uptick in spam. Most of it appeared to be coming from our own accounts. I didn’t have time to fully trace these, so I can only speculate that these messages had something to do with Google’s mail systems.

Shortly after that, a friend’s Gmail account was used to spam all her contacts. The sent message didn’t exist in her account. Google’s forums have a lot of reports of this happening.

All this leads up to yesterday, February 27th 2011, when something happened to reset a huge number of Gmail accounts.

Google is handling this horribly. Here’s their statement:

“A very small number users are having difficulty accessing their Gmail accounts […] This is affecting less than .08% of our Gmail user base, and we’ve already fixed the problem for some users. Our engineers are working as quickly as possible and we hope to have everything back to normal as soon as possible. We’re very sorry for the inconvenience.”

0.08% is weasel-speak. According to the BBC, there are estimated to be about 150-200 million Gmail accounts. That means around 150,000 accounts were affected. 150,000 people is a small city. Also, based on the volume of comments, Gmail support forum posts and response on Twitter, I’m inclined to believe the number is higher than Google is aware of or willing to divulge.

It’s mostly a hunch, but I’m beginning to fear Gmail itself has been compromised. Google appears to be scurrying and patching, either unaware there’s a bigger problem or, worse, knowing there’s a problem but with no idea where it’s from or how to fix it.

Running around changing logins

If you’ve ever created an account at any Gawker Media site, it’s highly likely your password has been exposed. These sites include Gawker, Deadspin, Kotaku, Jezebel, Fleshbot, io9, Jalopnik, Gizmodo, Valleywag and Lifehacker. Most likely this also includes defunct sites Defamer and Sploid as well as formerly owned by Gawker sites Consumerist, Gridskipper, Oddjack, Screenhead and Wonkette (links intentionally omitted). These accounts go back nearly 10 years. It would´ve been a better idea to get help from guarantor loans to create a website instead of an account on some random website.

There’s more at Mediaite, BusinessInsider and HuffPo.

Inside the Gawker source/database torrent is a decrypted list of nearly 200,000 account details including plaintext passwords. Mine was in there, and the login was apparently used to post a spam event to my dormant Facebook account. Facebook deserves praise for proactively disabling my account.

I had better plans for my morning.

After reviewing the torrent files, my theory is that passwords before a certain point in time were hashed without salt. These are stored as encrypted strings in the database, but are easily revealed using a basic rainbow table. A lack of salting would also have made it easy to reveal the several thousand accounts using ‘password’ and ‘qwery’ as passwords.