The day Google Chrome disables HTTP/2 for nearly everyone: May 31st, 2016

If you've been reading this blog for a while (or have been reading my rants on Twitter), you'll probably know this was coming already. If you haven't, here's the short version.

The Chromium project (whose end result is the Chrome Browser) has switched the negotiation protocol by which it decides whether to use HTTP/1.1 or the newer HTTP/2 on May 15th, 2016 May 31st, 2016.

Update: Chrome 51 is released and should be updated everywhere in a couple of hours/days. If HTTP/2 stops working for you, it’s probably because NPN has been disabled from now on.

That in and of itself isn’t a really big deal, but the consequences unfortunately are. Previously (as in: before May 31st, 2016), a protocol named NPN was used – Next Protocol Negotiation. This wasn’t a very efficient protocol, but it got the job done.

There’s a newer negotiation protocol in town called ALPN – Application-Layer Protocol Negotiation. This is a more efficient version with more future-oriented features. It’s a good decision to switch from NPN to ALPN, there are far more benefits than there are downsides.

However, on the server side – the side which runs the webservers that in turn run HTTP/2 – there’s a rather practical issue: to support ALPN, you need at least OpenSSL 1.0.2.

So what? You’re a sysadmin, upgrade your shit already!

I know. It sounds easy, right? Well, it isn’t. Just for comparison, here’s the current (May 2016) state of OpenSSL on Linux.

<th>
  OpenSSL version
</th>

<td>
  <font color="red">0.9.8e</font>
</td>

<td>
  <font color="red">1.0.1e</font>
</td>

<td>
  <font color="red">1.0.1e</font>
</td>

<td>
  <font color="red">1.0.1f</font>
</td>

<td>
  <font color="green">1.0.2g</font>
</td>

<td>
  <font color="red">1.0.1e</font>
</td>

<td>
  <font color="red">1.0.1k</font>
</td>

As you can tell from the list, there’s a problem: out of the box, only the latest Ubuntu 16.04 LTS (out for less than a month) supports OpenSSL 1.0.2.

Upgrading OpenSSL packages isn’t a trivial task, either. Since just about every other service links against the OpenSSL libraries, they too should be re-packaged (and tested!) to work against the latest OpenSSL release.

On the other hand, it’s just a matter of time before distributions have to upgrade as support for OpenSSL 1.0.1 ends soon.

Support for version 1.0.1 will cease on 2016-12-31. No further releases of 1.0.1 will be made after that date. Security fixes only will be applied to 1.0.1 until then.

OpenSSL Release Strategy

To give you an idea of the scope of such an operation, on a typical LAMP server (the one powering the blogpost you’re now reading), the following services all make use of the OpenSSL libraries.

$ lsof | grep libssl | awk '{print $1}' | sort | uniq
anvil
fail2ban
gdbus
gmain
httpd
postfix
mysqld
NetworkManager
nginx
php-fpm
puppet
sshd
sudo
tuned
zabbix_agent

A proper OpenSSL upgrade would cause all of those packages to be recreated too. That’s a hassle, to say the least. And truth be told, it probably isn’t just repackaging but potentially changing the code of each application to be compatible to the newer or changed API’s in OpenSSL 1.0.2.

Right now, the proper simplest way to run HTTP/2 on a modern server (that isn’t Ubuntu 16.04 LTS) would be to run a Docker container, based on Ubuntu 16.04, and run your webserver inside of it.

I don’t blame Google for switching protocols and evolving the web, but I’m sad to see that as a result of it, a very large portion of Google Chrome users will have to live without HTTP/2, once again.

Before May 15th, 2016 – a Google Chrome user would see this in its network inspector:

After May 31st, it’ll be old-skool HTTP/1.1.

It used to be that enabling HTTP/2 in Nginx was a very simple operation, but in order to support Chrome it’ll be a bit more complicated from now on.

This change also didn’t come out of the blue: Chrome had disabled NPN back in 2015 but quickly undid that change when the impact was clear. We knew, since the end of 2015, that this change was coming – we were given 6 months time to get support for ALPN going, but by the current state of OpenSSL packages that was too little time.

If you want to keep track of the state of Red Hat (Fedora, RHEL & CentOS) upgrades, here’s some further reading: RFE: Need OpenSSL 1.0.2.

As I’m mostly a CentOS user, I’m unaware of the state of Debian or Ubuntu OpenSSL packages at this time.