SPDY configuration: tcp_slow_start_after_idle

December 3rd, 2011

If you’re a SPDY server implementor, you’ve likely already read about the impact of CWND. Fortunately, the TCP implementors now largely agree that we can now safely increase CWND, and the standard will likely change soon. The default linux kernel implementation already has.

But, there is a second cwnd-related kernel flag which is not often mentioned. It’s important in all cases, but particularly important if you’re trying to establish long-lived connections. It’s not just important to SPDY – it’s important for HTTP keepalives or pipelines too. And many of the large web service providers are already tuning it:

    > sysctl -a | grep tcp_slow_start_after_idle
    net.ipv4.tcp_slow_start_after_idle = 1

At casual glance, you probably think “this sounds good, after a minute or so, it will go back into slow start mode”. That is fine, right?

Not quite. “Idle” in this case doesn’t mean a ‘minute or so’. In fact, it doesn’t even mean a second. This flag comes from RFC2861’s recommendation, which states that cwnd be cut in half with each RTT of idleness. That means that a persistently held open connection soon degrades back to the performance of an un-warmed connection very quickly.

So why does this matter? If you’re attempting to use a long-lived SPDY connection and think that the initial CWND won’t affect you because you’re only opening one connection anyway, you’re wrong. The slow-start-after-idle will still get you.

While there has been a tremendous amount of investigation and discussion about the initial cwnd value, I’m not aware of any recent debate about the slow-start-after-idle. I know that many websites are already disabling this flag to make HTTP keepalive connections perform more reasonably. Sadly, I can’t find any research which actually measured the effects of this behavior in the real world, so I can’t fall back on any real data. Given how aggressive TCP already is at backing off should network congestion change, I see no reason to enable this flag. Further, if you’re helping the net by dropping from N connections to 1, there is no reason you should be further penalized for your good deeds! Turn this one off.

Fixing the CEO Pay Problem

November 22nd, 2011

I’m generally a pretty free-market kind of guy. But, when it comes to CEO pay, there is no doubt in my mind that America is screwed up and that the free market is failing us. This isn’t the biggest of our problems, but it raises unnecessary doubt about “corporate greed” and about the livelihood of the American Dream.

If you don’t believe me, check out some of the compensation paid to CEOs of companies that are losing massive amounts of money:

  • Aubrey McClendon, Chesapeake Energy, paid $18.6M while the company lost $5.8B.
  • Carol Bartz, Yahoo, paid $39.0M in the same year she’s fired.
  • Timothy Armour, Janus Capital, paid $11.4M while the company lost $757.1M.
  • Rupert Murdoch, News Corp, paid $18.0M while the company lost $3.4B.
  • Robert Stevens, Lockheed Martin, paid $21.7M while the company lost $3.0B.
  • Daniel Hesse, Sprint, paid $10.3M while the company lost $2.4B.
  • Gregory Brown/Sanjay Jha, Motorola, paid $11.7M while the company lost $111M.
  • Ronald Hovsepian, Novell, paid $5.2M while the company lost $214.6M.
  • William Klesse, Valero, paid $11.3M while the company lost $353M.
  • Klaus Kleinfeld, Alcoa, paid $14.3M while the company lost $985M.
  • Ahmad Chatila, MEMC Electronic Materials, paid $16.7M while the company lost -$68.3M.
  • The list goes on and on…

Still not convinced? Why do CEOs get golden parachutes? Why did Leo Apotheker get paid $25M after getting fired 11mos into the job? Do you get one? It makes no sense to ever have a guaranteed payout even if you screw up.

Mark Cuban once again puts this in perspective by demonstrating that the risk-reward for CEOs is out of whack.

Fortunately, it is easy to fix.

The free market should remain free. If a company wants to pay a CEO $50M in advance, they are free to do so. But the Board of Directors, whose sole responsibility is to the shareholders best interests, needs to be able to prove that such a plan is good for the shareholders. If not, the Directors need to be held personally liable.

I’d like to see the SEC adopt new rules about executive pay – including any form of guaranteed pay, pay for non-performance, pay while the company is losing money, or pay for early termination. These rules should outline a very strict and narrow definition for when such compensation would be “good for shareholders”. Common sense should win out here, and the right answer is “almost never”. We all know that if an employee isn’t working out you should fire them with impunity. CEO’s are no exception.

As for the CEOs that are already beneficiaries of guaranteed payouts – if they have any character at all, they should forfeit these benefits and ask their Board of Directors to rework their compensation to something in line with what the rest of the company gets.

SPDY of the Future Might Blow Your Mind Today

November 17th, 2011

This post is definitely for protocol geeks.

SPDY has been up and running in the “basic case” at Google for some time now. But I never wrote publicly about some wicked cool possibilities for SPDY in the future. (Much to my surprise, it may be that someone is doing them today already!)

To start this discussion, lets consider how the web basically works today. In this scenario, we’ve got a browser with 3 tabs open:

As you can see, these pages use a tremendous number concurrent connections. This pattern has been measured both with Firefox and also with Chrome. Many mobile browsers today cap the connections at lower levels due to hardware constraints, but their desktop counterparts generally don’t because the only way to get true parallelism with HTTP is to open lots of connections. The HTTPArchive adds more good data into the mix, showing that an average web page today will use data from 12 different domains.

Each of these connections needs a separate handshake to the server. Each of these connections occupies a slot in your ISP’s NAT table. Each of these connections needs to warm up the TCP SlowStart algorithm independently (Slow Start is how TCP learns how much data your Internet connection can handle). Eventually, the connections feed out onto the internet and on to the sites you’re visiting. Its impressive this system works very well at all, for it is certainly not a very inefficient use of TCP. Jim Gettys, one of the authors of HTTP has observed these inefficiencies and written about the effects of HTTP’s connection management with ‘bufferbloat’.

SPDY of Today

One first step to reduce connection load is to migrate sites to SPDY. SPDY resides side by side with HTTP, so not everyone needs to move to SPDY at the same time. But for those pages that do move to SPDY, they’ll have reduced page load times and transmitted with always-on security. On top of that, these pages are much gentler on the the network too. Suddenly those 30-75 connections per page evaporate into only 7 or 8 connections per page (a little less than one per domain). For large site operators, this can have a radical effect on overall network behavior. Note that early next year, when Firefox joins Chrome implementing SPDY, more than 50% of users will be able to access your site using SPDY.

SPDY of the Future

Despite its coolness, there is an aspect of SPDY that doesn’t get much press yet (because nobody is doing it). Kudos for Amazon’s Kindle Fire for inspiring me to write about it. I spent a fair amount of time running network traces of the Kindle Fire, and I honestly don’t know quite what they’re doing yet. I hope to learn more about it soon. But based on what I’ve seen so far, it’s clear to me that they’re taking SPDY far beyond where Chrome or Firefox can.

The big drawback of the previous picture of SPDY is that it requires sites to individually switch to SPDY. This is advantageous from a migration point of view, but it means it will take a long time to roll out everywhere. But, if you’re willing to use a SPDY gateway for all of your traffic, a new door opens. Could mobile operators and carriers do this today? You bet!

Check out the next picture of a SPDY browser with a SPDY gateway. Because SPDY can multiplex many connections, the browser can now put literally EVERY request onto a single SPDY connection. Now, any time the browser needs to fetch a request, it can send the request right away, without needing to do a DNS lookup, or a TCP handshake, or even an SSL handshake. On top of that, every request is secure, not just those that go to SSL sites.

Wow! This is really incredible. They’ve just taken that massive ugly problem of ~200 connections to the device and turned it into 1! If your socks aren’t rolling up and down right now, I’m really not sure what would ever get you excited. To me, this is really exciting stuff.

Some of you might correctly observe that we still end up with a lot of connections out the other end (past the SPDY gateway). But keep in mind that the bottleneck of the network today is the “last mile” – the last mile to your house. Network bandwidth and latencies are orders of magnitude faster on the general Internet than they are during that last mile to your house. Enabling SPDY on that link is the most important of them all. And the potential network efficiency gains here are huge for the mobile operators and ISPs. Because latencies are better on the open internet, it should still yield reduced traffic on the other side – but this is purely theoretical. I haven’t seen any measure of it yet. Maybe Amazon knows :-)

More Future SPDY

Finally, as an exercise to the reader, I’ll leave it to you to imagine the possibilities of SPDY in light of multiplexing many sites, each with their own end-to-end encryption. In the diagram above, SSL is still end-to-end, so that starting a SSL conversation still requires a few round trips. But maybe we can do even better….

SPDY is not hard. Securing the Internet is.

November 12th, 2011

The F5 folks wrote a little about SPDY a few weeks ago. It’s a nice write up. But I want to challenge one particular point of it which I commonly hear:

“The most obvious impact to any infrastructure between a SPDY-enabled client and server is that it drives intermediate processing back to layer 4, to TCP”

This isn’t actually true. SPDY is not what makes load balancing or hierarchical caching difficult. SSL is what makes these hard. But even blaming SSL is a bit unfair – any protocol which introduces encryption to avoid 3rd party tampering of the data stream is going to have this problem.

In other words, it’s not deploying SPDY that is hard, it’s securing the web that is hard.

To the contrary, SPDY actually makes deployment of secure content easier. One of the common complaints against using SSL is that of performance – both in terms of client latency and also server scalability. When SSL is combined with SPDY, the performance objection is substantially lessened.

Now, don’t get me wrong, I am sympathetic to the difficulty of securing the web, and we need a lot more tools, debugging, and effort to make it simpler and cheaper for everyone. This will be especially difficult for infrastructure solutions which leverage the fact that HTTP is unsecured to do L7 packet analysis. But that doesn’t change the fact that we live in an electronic world full of bad guys. Whenever we ultimately decide to protect the web, it’s going to be hard. SPDY doesn’t create this problem at all.

Firefox SPDY Presentation

November 12th, 2011

Patrick McManus of Mozilla did a great presentation on SPDY. He’s excited about how much more efficient SPDY is than HTTP when layered over TCP. Firefox has now independently verified basically all of the claims that we made abut SPDY with Chrome.

Can’t wait for Firefox 11.

For the record, the standardization process for SPDY is coming.

What could cause all this?

October 25th, 2011

We’ve got roads blocked off with Police Mobile Command units….

We’ve got cops loitering on every street corner….

We’ve got every police officer in SF on a motorbike….

And snipers on the rooftops….

What could it be?

Obama’s back in town!

[gallery columns="2"]

AT&T Call Dropping

October 24th, 2011

I called AT&T today. Here is the approximate transcript from the call:

me: Hi, I’d like to make a change to my service.

att: Ok, what would you like to change?

me: I’d like to cancel my free Call Dropping Service.

att: Ok, I’m loading your account now, one minute.

me: Thanks.

att: I’m sorry, sir, we’ don’t have any Call Dropping Service.

me: Are you sure? Don’t use AT&T?

The operator was quite nice and helped lookup a variety of other AT&T services as well. She checked to see if they had a “Call Scrambling” service, as well as a “Automatic Call Blocking” service (where AT&T automatically doesn’t deliver a call). The last one tripped her up because they do have a service by that name, but she didn’t understand that I had never configured it and that I seemed to be describing having used it.

Eventually I got bored and hung up. Thank you to the patient operator. I still hate AT&T though.

Good Idea: Slash Government-backed Student Loans

October 16th, 2011

Why is the price of college going through the roof? They could be rising due to increased costs. Or they could be rising because that is what the market will bear.

Mark Cuban proposed this recently, and I think he’s right. The government keeps throwing more money to students. Awash with cash, those students then go to school and spend it. With plenty of eligible students, the schools simply raise rates, the government loans more, and the cycle repeats. Shall we stop the madness? Stop giving out massive government student loans and this problem could evaporate entirely.

Here’s a fun info-graphic about college loans.

And here’s Mark:

3. Limit the Size of Student Loans to $2,000 per year

Crazy ? Maybe, maybe not. What happened to the price of homes when the mortgage loan bubble popped ? They plummeted. If the size of student loans are capped at a low level, you know what will happen to the price of going to a college or university ? It will plummet. Colleges and universities will have to completely rethink what they are, what purpose they serve and who their customers will be. Will some go out of business ? Absolutely. That is real world. Will the quality of education suffer ? Given that TAs will still work for cheap, I doubt it.

Now some might argue that limiting student loans will limit the ability of lower income students to go to better schools. I say nonsense on two fronts. The only thing that allowing students to graduate with 50k , 80k or even more debt does is assure they will stay low income for a long, long time after they graduate ! The 2nd improvement will be that smart students will find the schools that adapt to the new rules and offer the best education they can afford. Just as they do now, but without loading up on debt.

The beauty of capitalism is that people like me will figure out new and better ways to create and operate for profit universities that educate as well or better as today’s state institutions, AND I have no doubt that the state colleges and universities will figure out how to adapt to the new world of limited student loans as well.

Finally, the impact on the overall economy will be ENORMOUS. There is more student loan debt than credit card debt outstanding today. By relieving this burden at graduation, students will be able to participate in the economy

Macs Are So Easy To Use

August 29th, 2011

I’m still struggling with my mac, but I am getting better.  Nonetheless, this week’s UI episode was so comical I decided to post it.

The Task:  By default, the screen saver locks me out of my Mac every 5 minutes!  Make it not timeout for 15-30 minutes and not force the immediate password.  This shouldn’t be too hard, right?

Step #1:  Go to the System Preferences, click on “Desktop and Screen Saver”, and move the slider from 5 minutes to 15 minutes.  Great!  Hey, they even let me choose from a bunch of cute screen saver options!  Nice.

Disappointment #1:  It didn’t work!  My machine still locks me out after 5 minutes.

At this point, it took me days before realizing that it was probably the Mac equivalent of power-saving mode causing the problem.  If I weren’t a techie, I never would have thought of this.  But hey, no problem, I can fix that…

Step #2: Go to the System Preferences again, and find the “Energy Saver”!  Great!  Sure enough, the defaults are too low.  Extend them out.  Nice.  Now I must be all done.

Disappointment #2:  OK – it partially worked, but shoot, it’s still prompting me for my password all the time.  How do I fix that?

I spent a lot of time combing through both the screen saver and also the energy saver panels.  But I couldn’t find where they have an option to not force password prompting.  Searching through other panels didn’t discover anything either.

Finally, Google solved the problem.

Step #3: Go to the System Preferences “Security” tab, with “Require password [immediately] after sleep or screen saver begins.”  Change that value.

Alas, 3 days, 3 control panels, and 1 Google search later, I no longer have my Mac locking me out all the time.  Score one up for Apple’s mastery of User Interface design!  Hope this helps someone else someday too.


IPv6 DNS Lookup Times

June 15th, 2011

A couple of weeks ago, I posted an article about slow IPv6 DNS query performance.  Several readers suggested in the comments that my observations were isolated to a few bad implementations and that perhaps Mac and Linux systems were not prone to this.  I hoped they were right, but I now have data to show they’re wrong.

Measuring performance is routine in Chrome, so a couple of days ago I was able to add a simple test to measure the speed of pure IPv4 lookups (A record) vs IPv6 lookups (A + AAAA records).  Today, with hundreds of millions of measurements in hand, we know the impact of IPv6 on DNS.


Windows users face a ~43% DNS latency increase and Mac users face a 146% DNS latency increase when they install an IPv6 client-side address on their machines. 

Today, there are two basic approaches to the IPv6 lookup problem:  you can issues the requests in serial or in parallel.  Obviously, issuing in serial will be more latency impacting than doing them in parallel.  But, even issuing them in parallel will be slower, as the maximum latency of two samples along a normal curve will be lower than the average latency of a single sample along the same curve.

Some readers may argue that these are dominated by older implementations.  Further investigation into the data does not confirm this.  Even comparing the fastest 10% of Mac or Windows users (which should hopefully be using the best DNS resolver algorithms available) are seeing greater than 100% DNS lookup latency increases with IPv6.  Of course, we have conflation here, as DNS server response times may be slower in addition to the DNS client double-lookup being slower.  A deeper study about why these results are so bad is warranted.


Performance-sensitive users should be cautious about assigning a global IPv6 address on their clients.  As soon as they do, the browser will switch into IPv6 lookup mode, and take on a 40-100% increase in DNS latency.