So, the dust has settled now on the Facebook “incident” that I wrote about on the 31st July.
As the day progressed I was reading more and more reports of people seeing other people’s details, as well as the problems I saw too. Facebook later made a statement saying that they hadn’t been hacked, but the problem was actually due to a proxy issue for some users, whereby some content was cached and subsequently delivered inappropriately.
When it returned, Facebook had added various parameters to the query string of all links on any page you looked at - particularly a “pwstdfy” parameter that had a lot of people pondering over the meaning of the apparent acronym. The value looked like some sort of 32 character hash which varied with each logon and people were naturally assuming that this was some sort of new security feature. However, this evening I was looking and noticed that this parameter has now disappeared from the urls.
So what actually happened in the end then? I was talking to someone about this and they suggested something that I’d not considered: perhaps what happened was a problem with Facebook’s load balancers? At their simplest level, a load balancer can just cache URLs - for example “http://www.facebook.com/home.php?” might get cached when you visit it, then I get to see your cached home page when I look at the same URL a moment later. Adding in the unique pwstdfy hash each time would make sure people aren’t seeing each other’s pages.
Even though the pwstdfy parameter has now gone, there is a still a 32 character hash in the cookie that could potentially be used by the load balancing system to map requests to the correct users. I have no idea if it was there before the outage or not, but I am guessing it was as Facebook has been around for quite some time without these problems before (as far as I know).
Presumably if this really was a load balancing problem, the pwstdfy parameter was just there temporarily whilst they fixed the issues. This doesn’t really explain why we saw the problems on the live site that we did though - maybe someone just screwed up and broke the configuration, but you’d hope that they have development systems in place where they can break things to their heart’s content. Facebook have been quite quiet on the whole issue though so I suppose we’ll never really know!