I ran snoop (a Solaris utility similar to tcpdump) on one of our production servers, and after combing through 180MB of packet traces with Wireshark, it turned out the data was being corrupted before even hitting our web servers. While it was a relief to find out our own infrastructure was not to blame, I still had to identify the culprit, e.g. whether our hosting provider’s switches, firewalls or load-balancers were to blame.

TCP has built-in checksums, so a malfunctioning switch working at layers 1–3 would not cause this problem, a corrupted packet would be dropped and resent, with a slight hit on performance but no errors. Thus the problem would need to be at a L4 or higher device such as a load balancer.

I added some extra logging and let it run over the weekend. After analyzing the data, it turns out the problem is very circumscribed (76 requests out of hundreds of millions), and all the affected IP addresses come from the same ISP, Singapore Telecom Magix (AS9506). The only plausible explanation is that SingTel is running some sort of deep packet inspection gear, and some of the DPI gateways have corrupt memory or software bugs, that are causing the data flowing through them to get corrupted,

Deep Packet Inspection is a scourge the general public is insufficiently aware of. At a high level, DPI gateways watch over your shoulder as you use the Internet. They decode the data packets passing through them, reconstruct unencrypted HTTP requests (in other words, spy on your browsing history). In their transparent proxy incarnation, they can rewrite the requests or responses. Verizon Wireless uses the technology to resize and recompress images or videos requested by smartphones. Back when I used to work for France Telecom (circa 1996-1999), vendors would regularly approach us to peddle their wares and how they would allow us to price-gouge our customers more effectively. Hardware has progressed dramatically since and a single Xeon processor is capable of inspecting at least 10 Gbps of data.

The whole premise of DPI and other snooping devices is profoundly repugnant to me as a former network engineer, on both moral and technical grounds. Any additional “bump in the wire” slows things down and is yet another potential point of failure, as shown by this incident, but the potential for abuse is the real concern. Not to mince words, the legitimate purposes for the technology, such as fighting cybercrime, are just rationalizations, it was really developed for purposes most people would consider abusive.

When I joined FT, I had to go to a Paris courthouse and swear a solemn oath to defend the privacy of our customers’ communications, and report any infringement of the same. DPI technology originates in spy agencies, and is much beloved of authoritarian governments. China uses the technology, combined with voice recognition, to drop calls at the merest mention of the word “protest”. The Ben Ali regime in Tunisia used it to snoop Facebook users’ authentication cookies. Singapore’s government has a well-demonstrated intolerance of criticism, and who knows what SingTel is doing with their defective gear? Western companies like Cisco were disgracefully eager to sell censorware to dictatorships, but those governments now have homegrown capabilities from the likes of Huawei.

For telco oligopolies, the endgame is to practice perfect price discrimination, e.g. charge you more for packets that carry a voice over IP call or a Netflix video on demand session that compete with the carriers’ own services. Telcos and cablecos cannot be permitted to use their stranglehold over public networks for what is essentially racketeering. Strowger invented the automatic telephone switch because the operator at his manual exchange would divert his calls to one of his competitors, her husband. Telcos, in their monopolistic arrogance, feel a sense of entitlement to all the value the network creates, even when they are not responsible, and want to reverse this. Letting them get away with it, as is consistently the case in the US, is a recipe for long-term economic stagnation.

What can we as the general public do to fight back? The telcos are one of the largest lobbies in Washington, and wireless spectrum auction fees are one of the crutches propping up Western budgets, so help is unlikely to come from the venal legislatures. The most practical option is to start using SSL and DNSSEC for everything. Google now offers an encrypted search option and Facebook has an option to use SSL for the entire session, not just for login.

Immediately after switching from my earlier “snake oil” self-signed test certificate to the production certificate, I saw SSL errors on Google Chrome for Mac and Safari for Mac, i.e. the two browsers that use OS X’s built-in crypto and certificate store. I suppose I should have tested the certificate on another server before going live, but I trusted GoDaddy (they are my DNS registrars, and competent, if garish).

Big mistake.

I called their tech support hotline, which is incredibly grating because of the verbose phone tree that keeps trying to push add-ons (I guess it is consistent with the monstrosity that is their home page).

After a while, I got a first-level tech. He asked whether I saw the certificate error on Google Chrome for Windows. At that point, I was irate enough to use a four-letter word. Our customers are Android mobile app developers. A significant chunk of them use Macs, and almost none (less than 5%) use IE, so know-nothing “All the world is IE” demographics are not exactly applicable.

After about half an hour of getting the run-around and escalating to level 2, with my business partner Michael getting progressively more anxious in the background, the level 1 CSR tells me the level 2 one can’t reproduce the problem (I reproduced it on three different Macs in two different locations). I gave them an ultimatum: fix it within 10 minutes or I would switch. At this point, the L1 CSR told me he had exhausted all his options, but I could call their “RA” department, and offered to switch me. Inevitably, the call transfer failed.

I dialed their SSL number, and in parallel started the certificate application process on RapidSSL. They offered a free competitive upgrade, I tried it, and within 3 minutes I had my fresh new, and functional certificate, valid for 3 years, all for free and in less time than it takes to listen to GoDaddy’s obnoxious phone tree (all bout “we pride ourselves in customer service” and other Orwellian corporate babble).

I then called GoDaddy’s billing department to get a refund. Surprisingly, the process was very fast and smooth. I guess it is well-trod.

The moral of the story: GoDaddy—bad. RapidSSL—good.

I contacted the support for the three vendors. Joyent got back to me, and said:

We have traced this back to a third-party provider that was used to distribute service notifications. We have been in contact with this service provider, and they have determined that subscriber email addresses of their clients were compromised. They have launched their own investigation, which is ongoing, and have also reached out to their local FBI office.

After some digging, I found some interesting posts. Some email marketing company called iContact, that I had never heard about before, was the source of the compromise. They claim to be SAS-70 compliant, but of course like most bureaucratic certifications, SAS-70 is mostly security theater that makes sysadmins’ life miserable for no meaningful security benefit (SAS-70 auditors, on the other hand, profit handsomely).

Just another example of how outsourcing critical functions to outside vendors can backfire spectacularly and take down your own reputation in the process.

Unfortunately far too many lame organizations (cough, Google) qualify their SPF record using a too permissive ?all or ~all clause, which means they have servers other than those listed, and thus their SPF record is useless for filtering purposes.

In the last month, I noticed the opposite problem: I did not receive emails from Eurostar and BookMooch because their SPF records did not list the mail servers they actually use. If they are not clueful enough to manage a simple list of IP addresses, or have basic change management discipline, they should do us all a favor and ditch the SPF record they clearly are incapable of maintaining.

The time has long come to stop coddling grossly inefficient and anti-competitive cellular carriers in the West. They are no longer fledgling businesses in the shadow of landlines, quite the opposite, in fact. One good place to start would be to require them to offer consumers the choice of carrier for international calls and for roaming, as is the case with landlines. Their rates are simply extortionate.

WebPass finesses around the incumbent monopoly by using newer buildings’ data-grade wiring plant to bring 100MBps Ethernet connections right into your home (all they had to do was change a wall plate and patch some cables in the closet) and use microwave links to backhaul traffic to their data center. They claim to use a mesh network for backhaul, but I think this just means a standard network of microwave links where some sites have to hop multiple microwave links to get to the transit connection, rather than a purely centralized hub and spoke model. In my case their offices are a mere two blocks away. This would allow me the pleasure of ditching the scumbags at AT&T altogether (were it not for the fact my building requires an entirely unnecessary landline for its security system).

AT&T is probably the worst telco in the US now, and is notorious for starving its infrastructure of investment to maximize short-term profits, unlike Verizon, who is investing heavily in its FiOS optical network. Unfortunately San Francisco is in AT&T territory and will not get true optical networks anytime soon. Municipalities can usually reassign the cable franchise every so many years, but there is no such provision for involuntary transfer of telcos that I know of.

The new service is $45 a month with no installation fee, vs. $70 a month for Raw Bandwidth, but it does not include a static IP address (they do offer it as part of their prohibitively expensive metered business service). Configuring my home router (a Cisco 877) to use both connections was incredibly painful, but I will run the two ISPs side by side for the next few months. If WebPass proves as reliable as Raw bandiwdth, I may just have to find a work-around for the static IP issue, or just rely on DHCP lease pinning.

If you live in San Francisco, or are moving there, definitely have a look at the buildings they have covered. The service is a glimpse of what people not in broadband backwater USA get.

The law was exceedingly stacked towards the content industry. The burden of proof was on the defendant rather than the prosecution, and an extra-judicial quango named Hadopi was to be set up to enforce these sanctions. The European Parliament, to its credit, had opposed such measures and restated that Internet access is a fundamental right that can only be curtailed by proper judicial authority. The first reading of the law led to a surprise defeat, as the majority UMP legislators were unenthusiastic about supporting a law that would alienate the young, and absenteeism was such that the minority Socialist party managed to overwhelm those few present. This is one of the exceedingly few times I actually agree with the feckless Socialists… The President brought his whip to bear and the law was put back on the agenda and voted in the second time.

Today, the Conseil Constitutionnel ruled on a challenge to the law put by Socialist parliamentarians, and gutted it in line with the European Parliament. In doing so, it affirmed that Internet access is a fundamental human right, drawing all the way back to the original Human Rights declaration of 1789, and that Internet users are innocent until proven guilty.

This is an important decision. In Roman law, judges’ discretion is much more limited than in the Anglo-Saxon Common law tradition. The US Supreme Court found in Roe vs. Wade a right to abortion in the US Constitution that is far from obvious, and such a decision by unelected judges lacked universal legitimacy. In contrast, abortion was legalized by an act of Parliament in France, which is why opposition to it is nowhere near as bitter as in the US. The Conseil Constitutionnel does not invent constitutional principles, it only censures laws or more commonly individual articles (the role of ultimate court of appeals belongs to another institution, the Cour de Cassation). The significance of it finding Internet access a fundamental right cannot be overstated.

I noticed my email was going through, however. On further inspection, it turns out they only intercept port 80 HTTP traffic, but not on other ports. Security through (very thin) obscurity, in other words.

I tried using Firefox from my home machine over X and SSH port forwarding, but it was painfully slow.

At this point, I was considering setting up a HTTP proxy on my home machine and using it over SSH port forwarding, but I remembered reading something about SSH and SOCKS. I had never used a SOCKS proxy before, but it turns out this is incredibly easy: just add the -D option to ssh with a local port number, e.g. 9999, and set up your browser to use localhost:9999 as the SOCKS proxy. It worked flawlessly with my Mac OS X SSH client and Solaris 10 stock server.

This has applications beyond routing around hotel paywalls. Many public WiFi access points are unsecured. Even if they are legit (many are peer-to-peer vs. infrastructure, and presumably used by thieves to harvest passwords), they can be snooped for passwords trivially easily. Using SSH and SOCKS provides you with security when using an untrusted Internet access point, even for non-SSL sites. My email uses IMAPS and SMTP TLS so I don’t need to reconfigure it to use SOCKS, but that would also be an important protocol to secure.

All in all, a very worthwhile addition to my toolset. I can’t believe I waited so long to try it.

Update (2009-04-12):

To its credit, New Orleans’ Louis Armstrong international airport has free WiFi throughout the terminal. Chic!