We’ve got roads blocked off with Police Mobile Command units….
We’ve got cops loitering on every street corner….
We’ve got every police officer in SF on a motorbike….
And snipers on the rooftops….
What could it be?
Obama’s back in town!
What could cause all this?
October 25th, 2011
AT&T Call Dropping
October 24th, 2011I called AT&T today. Here is the approximate transcript from the call:
me: Hi, I’d like to make a change to my service.
att: Ok, what would you like to change?
me: I’d like to cancel my free Call Dropping Service.
att: Ok, I’m loading your account now, one minute.
me: Thanks.
att: I’m sorry, sir, we’ don’t have any Call Dropping Service.
me: Are you sure? Don’t use AT&T?
The operator was quite nice and helped lookup a variety of other AT&T services as well. She checked to see if they had a “Call Scrambling” service, as well as a “Automatic Call Blocking” service (where AT&T automatically doesn’t deliver a call). The last one tripped her up because they do have a service by that name, but she didn’t understand that I had never configured it and that I seemed to be describing having used it.
Eventually I got bored and hung up. Thank you to the patient operator. I still hate AT&T though.
Good Idea: Slash Government-backed Student Loans
October 16th, 2011Why is the price of college going through the roof? They could be rising due to increased costs. Or they could be rising because that is what the market will bear.
Mark Cuban proposed this recently, and I think he’s right. The government keeps throwing more money to students. Awash with cash, those students then go to school and spend it. With plenty of eligible students, the schools simply raise rates, the government loans more, and the cycle repeats. Shall we stop the madness? Stop giving out massive government student loans and this problem could evaporate entirely.
Here’s a fun info-graphic about college loans.
3. Limit the Size of Student Loans to $2,000 per year
Crazy ? Maybe, maybe not. What happened to the price of homes when the mortgage loan bubble popped ? They plummeted. If the size of student loans are capped at a low level, you know what will happen to the price of going to a college or university ? It will plummet. Colleges and universities will have to completely rethink what they are, what purpose they serve and who their customers will be. Will some go out of business ? Absolutely. That is real world. Will the quality of education suffer ? Given that TAs will still work for cheap, I doubt it.
Now some might argue that limiting student loans will limit the ability of lower income students to go to better schools. I say nonsense on two fronts. The only thing that allowing students to graduate with 50k , 80k or even more debt does is assure they will stay low income for a long, long time after they graduate ! The 2nd improvement will be that smart students will find the schools that adapt to the new rules and offer the best education they can afford. Just as they do now, but without loading up on debt.
The beauty of capitalism is that people like me will figure out new and better ways to create and operate for profit universities that educate as well or better as today’s state institutions, AND I have no doubt that the state colleges and universities will figure out how to adapt to the new world of limited student loans as well.
Finally, the impact on the overall economy will be ENORMOUS. There is more student loan debt than credit card debt outstanding today. By relieving this burden at graduation, students will be able to participate in the economy
Macs Are So Easy To Use
August 29th, 2011I’m still struggling with my mac, but I am getting better. Nonetheless, this week’s UI episode was so comical I decided to post it.
The Task: By default, the screen saver locks me out of my Mac every 5 minutes! Make it not timeout for 15-30 minutes and not force the immediate password. This shouldn’t be too hard, right?
Step #1: Go to the System Preferences, click on “Desktop and Screen Saver”, and move the slider from 5 minutes to 15 minutes. Great! Hey, they even let me choose from a bunch of cute screen saver options! Nice.
Disappointment #1: It didn’t work! My machine still locks me out after 5 minutes.
At this point, it took me days before realizing that it was probably the Mac equivalent of power-saving mode causing the problem. If I weren’t a techie, I never would have thought of this. But hey, no problem, I can fix that…
Step #2: Go to the System Preferences again, and find the “Energy Saver”! Great! Sure enough, the defaults are too low. Extend them out. Nice. Now I must be all done.
Disappointment #2: OK – it partially worked, but shoot, it’s still prompting me for my password all the time. How do I fix that?
I spent a lot of time combing through both the screen saver and also the energy saver panels. But I couldn’t find where they have an option to not force password prompting. Searching through other panels didn’t discover anything either.
Finally, Google solved the problem.
Step #3: Go to the System Preferences “Security” tab, with “Require password [immediately] after sleep or screen saver begins.” Change that value.
Alas, 3 days, 3 control panels, and 1 Google search later, I no longer have my Mac locking me out all the time. Score one up for Apple’s mastery of User Interface design! Hope this helps someone else someday too.
IPv6 DNS Lookup Times
June 15th, 2011A couple of weeks ago, I posted an article about slow IPv6 DNS query performance. Several readers suggested in the comments that my observations were isolated to a few bad implementations and that perhaps Mac and Linux systems were not prone to this. I hoped they were right, but I now have data to show they’re wrong.
Measuring performance is routine in Chrome, so a couple of days ago I was able to add a simple test to measure the speed of pure IPv4 lookups (A record) vs IPv6 lookups (A + AAAA records). Today, with hundreds of millions of measurements in hand, we know the impact of IPv6 on DNS.
Windows users face a ~43% DNS latency increase and Mac users face a 146% DNS latency increase when they install an IPv6 client-side address on their machines.
Today, there are two basic approaches to the IPv6 lookup problem: you can issues the requests in serial or in parallel. Obviously, issuing in serial will be more latency impacting than doing them in parallel. But, even issuing them in parallel will be slower, as the maximum latency of two samples along a normal curve will be lower than the average latency of a single sample along the same curve.
Some readers may argue that these are dominated by older implementations. Further investigation into the data does not confirm this. Even comparing the fastest 10% of Mac or Windows users (which should hopefully be using the best DNS resolver algorithms available) are seeing greater than 100% DNS lookup latency increases with IPv6. Of course, we have conflation here, as DNS server response times may be slower in addition to the DNS client double-lookup being slower. A deeper study about why these results are so bad is warranted.
Summary
Performance-sensitive users should be cautious about assigning a global IPv6 address on their clients. As soon as they do, the browser will switch into IPv6 lookup mode, and take on a 40-100% increase in DNS latency.
Firefox Idle Connection Reuse
June 11th, 2011httpwatch does some anecdotal testing to conclude that Firefox’s new algorithm for selecting which idle connection to reuse has some strong benefits.
This is great stuff, and in general it should definitely help. This is part of why getting to one-connection-per-domain is an important goal. HTTP’s use of 6 or more connections per domain make it so that each connection must “warm up†independently. A similar algorithm should land in Chrome soon too.
Fortunately, there is a protocol for this stuff too 
Hopefully firefox will pick that up soon too.
SSL: It’s a Matter of Life and Death
May 28th, 2011Over a year ago, when we first announced SPDY, the most prominent criticism was the requirement for SSL. We’ve become so accustomed to our unsecure HTTP protocol, that making the leap to safety now seems daunting.
Since that time, many things have happened, and it is now more clear than ever that SSL isn’t an option – it’s a matter of life and death.
SSL was invented primarily to protect our online banking and online purchasing needs. It has served us fairly well, and most all banks and ecommerce sites use SSL today. What nobody ever expected was that SSL would eventually become the underpinnings of safety for political dissidents.
Last year, when China was caught hacking into Google, were they trying to steal money? Two months ago, when Comodo was attacked (and suspected the Iranian government), did they forge the identities of Bank of America, Wells Fargo, or Goldman Sachs? No. They went after Twitter, Gmail, and Facebook – social networking sites. Sites where you’d find information about dissidents, not cash. To say that these attacks were used to seek and destroy dissidents would be speculation at this point. But these incidents show that the potential is there and governmental intelligence agencies are using these approaches. And of course, it is well known fact that the Egyption government turned off the Internet entirely so that their citizens could not easily organize.
The Internet is now a key communication mechanism for all of us. Unfortunately, users can’t differentiate safe from unsafe on the web. They rely on computer professionals like us to make it safe. When we tell them that the entire Web is built upon an unsecured protocol, most are aghast with shock. How could we let this happen?
As we look forward, this trend will increase. What will Egypt, Libya, Iran, China, or Afghanistan do to seek out and kill those that oppose them? What does the US government do?
Fortunately, major social networking sites like Facebook and Twitter have already figured this out. They are now providing SSL-only versions of their services which should help quite a bit.
So does all this sound a little dramatic? Maybe so, and I apologize if this sounds a bit paranoid. I’m not a crypto-head, I swear. But these incidents are real, and the potential is real so long as our Internet remains unsecure. The only answer is to secure *everything* we do on the net. Even the seemingly benign communications must be encrypted, because users don’t know the difference – and for some of them, their lives are at stake.
Codeflattery (n)
May 25th, 2011
codeflattery
[kohd-flat-uh-ree] noun
1. when journalists write about your half-done, partial work as though it is news.
My employer (Google), my project (Chrome), and my team (SPDY) make such amazing products that the press even pays attention to my tiny hacks. That makes me proud. Codeflattery.
http://www.conceivablytech.com/7582/products/google-adds-possible-tcp-replacement-to-chrome
SSL FalseStart Performance Results
May 19th, 2011From the Chromium Blog:
Last year, Google’s Adam Langley, Nagendra Modadugu, and Bodo Moeller proposed SSL False Start, a client-side only change to reduce one round-trip from the SSL handshake.
We implemented SSL False Start in Chrome 9, and the results are stunning, yielding a significant decrease in overall SSL connection setup times. SSL False Start reduces the latency of a SSL handshake by 30%1. That is a big number. And reducing the cost of a SSL handshake is critical asmore and more content providers move to SSL.
Our biggest concern with implementing SSL False Start was backward compatibility. Although nothing in the SSL specification (also known as TLS) explicitly prohibits FalseStart, there was no easy way to know whether it would work with all sites. Speed is great, but if it breaks user experience for even a small fraction of users, the optimization is non-deployable.
To answer this question, we compiled a list of all known https websites from the Google index, and tested SSL FalseStart with all of them. The result of that test was encouraging: 94.6% succeeded, 5% timed out, and 0.4% failed. The sites that timed out were verified to be sites that are no longer running, so we could ignore them.
To investigate the failing sites, we implemented a more robust check to understand how the failures occurred. We disregarded those sites that failed due to certificate failures or problems unrelated to FalseStart. Finally, we discovered that the sites which didn’t support FalseStart were using only a handful of SSL vendors. We reported the problem to the vendors, and most have fixed it already, while the others have fixes in progress. The result is that today, we have a manageable, small list of domains where SSL FalseStart doesn’t work, and we’ve added them to a list within Chrome where we simply won’t use FalseStart. This list is public and posted in the chromium source code. We are actively working to shrink the list and ultimately remove it.
All of this represents a tremendous amount of work with a material gain for Chrome SSL users. We hope that the data will be confirmed by other browser vendors and adopted more widely.
1Measured as the time between the initial TCP SYN packet and the end of the TLS handshake.
IPv6 Will Slow You Down (DNS)
May 18th, 2011
When you turn on IPv6 in your operating system, the web is going to get slower for you. There are several reasons for this, but today I’m talking about DNS. Every DNS lookup is 2-3x slower with IPv6.
What is the Problem?
The problem is that the current implementations of DNS will do both an IPv4 and an IPv6 lookup in serial rather than parallel. This is operating as-per the specification.
We can see this on Windows:
TIME EVENT
0 DNS Request A www.amazon.com
39 DNS Response www.amazon.com
39 DNS Request AAAA www.amazon.com
79 DNS Response www.amazon.com
<the browser cannot continue until here>
The “A†request there was the IPv4 lookup, and it took 39ms. The “AAAA†request is the IPv6 lookup, and it took 40ms. So, prior to turning IPv6 on, your DNS resolution finished in 39ms. Thanks to your IPv6 address, it will now take 79ms, even if the server does not support IPv6! Amazon does not advertise an IPv6 result, so this is purely wasted time.
Now you might think that 40ms doesn’t seem too bad, right? But remember that this happens for every host you lookup. And of course, Amazon’s webpage uses many sub-domain hosts. In the web page above, I saw more of these shenanigans, like this:
TIME EVENT
0 DNS Request A g-ecx.images-amazon.com
43 DNS Response g-ecx.images-amazon.com
43 DNS Request AAAA g-ecx.images-amazon.com
287 DNS Response g-ecx.images-amazon.com
Ouch – that extra request cost us 244ms.
But there’s more. In this trace we also had a lookup for the OCSP server (an Amazon’s behalf, for SSL):
TIME EVENT
0 DNS Request A ocsp.versign.com
116 DNS Response ocsp.versign.com
116 DNS Request AAAA ocsp.versign.com
203 DNS Response ocsp.versign.com
Ouch – another 87ms down the drain.
The average website spans 8 domains. A few milliseconds here, and a few milliseconds there, and pretty soon we’re talking about seconds.
The point is that DNS performance is key to web performance! And in these 3 examples, we’ve slowed down DNS by 102%, 567%, and 75% respectively. I’m not picking out isolated cases. Try it yourself, this is “normal†with IPv6.
What About Linux?
Basically all of the operating systems do the same thing. The common API for doing these lookups is getaddrinfo(), and it is used by all major browsers. It does both the IPv4 and IPv6 lookups, sorts the results, and returns them to the application.
So on Linux, the behavior ends up being like this:
TIME EVENT
0 DNS Request AAAA www.calpoly.edu
75 DNS Response www.calpoly.edu
75 DNS Request A www.calpoly.edu
93 DNS Response www.calpoly.edu
In this particular case, we only wasted 75ms, when the actual request would have completed in 18ms (416% slower).
But It’s Even Worse
I wish I could say that DNS latencies were just twice as slow. But it’s actually worse than that. Because IPv6 is not commonly used, the results of IPv6 lookups are not heavily cached at the DNS servers like IPv4 addresses are. This means that it is more likely that an IPv6 lookup will need to jump through multiple DNS hops to complete a resolution.
As a result, it’s not just that we’re doing two lookups instead of one. It’s that we’re doing two lookups and the second lookup is fundamentally slower than the first.
Surely Someone Noticed This Before?
This has been noticed before. Unfortunately, with nobody using IPv6, the current slowness was an acceptable risk. Application vendors (namely browser vendors) have said, “this isn’t our problem, host resolution is the OS’s jobâ€.
The net result is that everyone knows about this flaw. But nobody fixed it. (Thank goodness for DNS Prefetching!)
Just last year, the “Happy Eyeballs†RFC was introduced which proposes a work around to this problem by racing connections against each other. This is an obvious idea, of course. I don’t know of anyone implementing this yet, but it is certainly something we’re talking about on the Chrome team.
What is The Operating System’s Job?
All browsers, be it Chrome or Firefox or IE, use the operating system to do DNS lookups. Observers often ask, “why doesn’t Chrome (or Firefox, or IE) have its own asynchronous DNS resolver?†The problem is that every operating system, from Windows to Linux to Mac has multiple name-resolution techniques, and resolving hostnames in the browser requires using them all, based on the user’s operating configuration. Examples of non-DNS resolvers include: NetBIOS/WINS, /etc/hosts files, and Yellow Pages. If the browser simply bypassed all of these an exclusively used DNS, some users would be completely broken.
If these DNS problems had been fixed at the OS layer, I wouldn’t be writing this blog post right now. But I don’t really blame Windows or Linux – nobody was turning this stuff on. Why should they shine a part of their product that nobody uses?
Lesson Learned: Only The Apps Can ‘Pull’ Protocol Changes
IPv6 deployment has been going on for over 10 years now, and there is no end in sight. The current plan (like IPv6’s break the internet day) is the same plan we’ve been doing for 10 years. When do we admit that the current plan to deploy IPv6 is simply never going to work?
A lesson learned from SPDY is that only the applications can drive protocol changes. The OS’s, bless their hearts, can only do so much and move too slowly to push new protocols. There is an inevitable chicken-and-egg problem where applications won’t use it because OS support is not there and OSes won’t optimize it because applications aren’t there.
The only solution is at the Application Layer – the browser. But that may be the best news of all, because it means that we can fix this! More to come…