The day Google Chrome disables HTTP/2 for nearly everyone: May 31st, 2016

Mattias Geniar, Saturday, May 14, 2016 - last modified: Thursday, May 26, 2016

If you've been reading this blog for a while (or have been reading my rants on Twitter), you'll probably know this was coming already. If you haven't, here's the short version.

The Chromium project (whose end result is the Chrome Browser) has switched the negotiation protocol by which it decides whether to use HTTP/1.1 or the newer HTTP/2 on May 15th, 2016 May 31st, 2016.

Update: Chrome 51 is released and should be updated everywhere in a couple of hours/days. If HTTP/2 stops working for you, it's probably because NPN has been disabled from now on.

That in and of itself isn't a really big deal, but the consequences unfortunately are. Previously (as in: before May 31st, 2016), a protocol named NPN was used -- Next Protocol Negotiation. This wasn't a very efficient protocol, but it got the job done.

There's a newer negotiation protocol in town called ALPN -- Application-Layer Protocol Negotiation. This is a more efficient version with more future-oriented features. It's a good decision to switch from NPN to ALPN, there are far more benefits than there are downsides.

However, on the server side -- the side which runs the webservers that in turn run HTTP/2 -- there's a rather practical issue: to support ALPN, you need at least OpenSSL 1.0.2.

So what? You're a sysadmin, upgrade your shit already!

I know. It sounds easy, right? Well, it isn't. Just for comparison, here's the current (May 2016) state of OpenSSL on Linux.

Operating System OpenSSL version
CentOS 5 0.9.8e
CentOS 6 1.0.1e
CentOS 7 1.0.1e
Ubuntu 14.04 LTS 1.0.1f
Ubuntu 16.04 LTS 1.0.2g
Debian 7 (Wheezy) 1.0.1e
Debian 8 (Jessie) 1.0.1k

As you can tell from the list, there's a problem: out of the box, only the latest Ubuntu 16.04 LTS (out for less than a month) supports OpenSSL 1.0.2.

Upgrading OpenSSL packages isn't a trivial task, either. Since just about every other service links against the OpenSSL libraries, they too should be re-packaged (and tested!) to work against the latest OpenSSL release.

On the other hand, it's just a matter of time before distributions have to upgrade as support for OpenSSL 1.0.1 ends soon.

Support for version 1.0.1 will cease on 2016-12-31. No further releases of 1.0.1 will be made after that date. Security fixes only will be applied to 1.0.1 until then.

OpenSSL Release Strategy

To give you an idea of the scope of such an operation, on a typical LAMP server (the one powering the blogpost you're now reading), the following services all make use of the OpenSSL libraries.

$ lsof | grep libssl | awk '{print $1}' | sort | uniq
anvil
fail2ban
gdbus
gmain
httpd
postfix
mysqld
NetworkManager
nginx
php-fpm
puppet
sshd
sudo
tuned
zabbix_agent

A proper OpenSSL upgrade would cause all of those packages to be recreated too. That's a hassle, to say the least. And truth be told, it probably isn't just repackaging but potentially changing the code of each application to be compatible to the newer or changed API's in OpenSSL 1.0.2.

Right now, the proper simplest way to run HTTP/2 on a modern server (that isn't Ubuntu 16.04 LTS) would be to run a Docker container, based on Ubuntu 16.04, and run your webserver inside of it.

I don't blame Google for switching protocols and evolving the web, but I'm sad to see that as a result of it, a very large portion of Google Chrome users will have to live without HTTP/2, once again.

Before May 15th, 2016 -- a Google Chrome user would see this in its network inspector:

protocol_http2_enabled

After May 31st, it'll be old-skool HTTP/1.1.

protocol_http2_disabled

It used to be that enabling HTTP/2 in Nginx was a very simple operation, but in order to support Chrome it'll be a bit more complicated from now on.

This change also didn't come out of the blue: Chrome had disabled NPN back in 2015 but quickly undid that change when the impact was clear. We knew, since the end of 2015, that this change was coming -- we were given 6 months time to get support for ALPN going, but by the current state of OpenSSL packages that was too little time.

If you want to keep track of the state of Red Hat (Fedora, RHEL & CentOS) upgrades, here's some further reading: RFE: Need OpenSSL 1.0.2.

As I'm mostly a CentOS user, I'm unaware of the state of Debian or Ubuntu OpenSSL packages at this time.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear!. Follow me on Twitter as @mattiasgeniar.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

nginx user Saturday, May 14, 2016 at 15:19 - Reply

A bit overdramatic. You can build nginx with openssl 1.0.2 on any distribution of Linux.

./configure –with-openssl=/path/to/openssl


    Pav Monday, June 13, 2016 at 12:51 - Reply

    A bit unrealistic. As an admin you do not want to re-compile every time there is a nginx mainline or an OpenSSL update, thus relying on your distro’s repositories is much more convenient, secure and stable approach.

    Also don’t forget SPDY is out of the game too…


Alan Orth Saturday, May 14, 2016 at 15:55 - Reply

Then you’re responsible for your own nginx and openssl updates.


Mark Saturday, May 14, 2016 at 16:14 - Reply

Pretty much the above. This is not Windows.


Clay Smith Saturday, May 14, 2016 at 17:54 - Reply

Had the same issue a few months ago, great post. I wrote a guide to compiling OpenSSL on Ubuntu 14.04 here: https://www.clay.fail/posts/ubuntu-http2-in-mere-hours/

I later switched to using Docker to a kid the source code compiling entirely: https://www.clay.fail/posts/hip-http2-using-docker/


WYSIWYG Saturday, May 14, 2016 at 18:52 - Reply

What I’m getting out of this topic is – don’t use Chrome, stick to Firefox.


Will Saturday, May 14, 2016 at 19:59 - Reply

Are you implying that those packages listed all link against OpenSSL statically? That is not necessarily the case and might “just work” if there is a system-wide update to the shared version of openssl. I know that’s besides the point, but it seems like lsof is not differentiating between libraries loaded dynamically and those loaded statically.

Will


MojoDwarf Saturday, May 14, 2016 at 20:04 - Reply

Has anyone looked at libressl compatibility with ALPN?


moneytoo Saturday, May 14, 2016 at 21:41 - Reply

Yeah, ideally just rebuild the source package you were using till this point. On CentOS it’s just a few lines… https://gist.github.com/moneytoo/ab3f34e4fddc2110675952f8280f49c5


Joshua McKinnon Sunday, May 15, 2016 at 00:09 - Reply

This could definitely be an issue. Speaking as someone who runs OpenSSL on Ubuntu Server LTS releases, unless LTS policies have changed, I don’t see Ubuntu jumping major versions of OpenSSL in a given LTS release. (I hope I’m proven wrong)

This was an issue for me back in Ubuntu 10.04 LTS – whatever version of OpenSSL it shipped with, probably 0.9. something, did not support TLS 1.1 and 1.2. Sure, back when it originally shipped that wasn’t an issue, but for the last year or two of its LTS support? Crazy to be stuck on TLS 1.0. I recall finding a thread where it was noted that it was too risky/stability issue to jump. I expect Ubuntu team to backport security fixes to older branch in LTS line, rather than jump to new version. That was what they did in 10.04, even though it was “supported” you either had to compile it yourself or upgrade to 12.04 to get the newer OpenSSL line.

We all know we can compile and patch it ourselves, but that is not a great solution for many servers when you need to be able to fix 0-days in a matter of a coupe of hours the next time another heartbleed level CVE comes along.


Asinglenoob Sunday, May 15, 2016 at 04:28 - Reply

Cry harder? You’re a sys admin, it’s your job to keep stuff up to date. It’s this sort of lazy half assed update policy that causes massive security breaches every year.


Mike skaloola Sunday, May 15, 2016 at 07:05 - Reply

Really? The “proper” way is to use docker? Wanna qualify that in any way shape or form?

Running inside a Docker container is only proper if your infrastructure is already based around Docker

The proper thing to do is to upgrade if that aligns with how you’re managing your infrastructure.

Enabling openssl 1.0.2 in your site specific packages is the “peoper” way if you already roll your own Debs/rpms..

And so on and so forth…

.


Jesse Sunday, May 15, 2016 at 08:37 - Reply

You sysadmin guys love your IT but sometimes you forget about pragmatic solutions. The easier solution here is to be using CloudFlare, which has HTTP/2 support for ALL connections made over SSL (HTTPS). No reason to hurry your Ubuntu upgrades if you’re not ready:

https://www.littlebizzy.com/blog/nginx-server-block-ssl

Just a basic OpenSSL Nginx block, since CloudFlare doesn’t need your origin server to enable HTTP/2 (and in fact, you should NOT enable it). Voila… free Comodo SSL and HTTP/2 ;)


    Maximillian Laumeister Monday, May 16, 2016 at 06:35 - Reply

    This is a good sentiment, but not all web applications run very well behind a caching proxy like CloudFlare (especially dynamic applications). Plus, in certain situations CloudFlare can increase latency due to the extra trip to fetch from the origin server. CloudFlare is a great solution for many types of web apps / sites, but it’s not a perfect catch-all solution.


Mattias Geniar Sunday, May 15, 2016 at 08:37 - Reply

Wrong choice of words on my part, replaced “proper” with “simplest” (as that’s probably the case right now).


Chris Sunday, May 15, 2016 at 09:23 - Reply

Static linking of libraries embed them into the executable. It would not show up in lsof (or ldd for that matter).


lzap Sunday, May 15, 2016 at 09:26 - Reply

Backporting is the answer. Red Hat will definitely backport the fetature without upgrading the version.


Lul Sunday, May 15, 2016 at 09:36 - Reply

Do ppl still run red hat? Wtf???


David Sunday, May 15, 2016 at 10:03 - Reply

In layman’s terms
What must an all Android professional do to avoid y2k like drama …..??


TWiStErRob Sunday, May 15, 2016 at 10:44 - Reply

This sounds really overdramatic, it’s not like the whole Web will be unaccessible after that update. If you want to use cutting edge technology (HTTP/2), then you have to use cutting edge technology (OpenSSL 1.0.2) to build on; otherwise everything will still work, maybe a little bit slower.

Please correct me if the above is not true.

Also you shouldn’t blame Google for advancing technology, rather try to keep up with latest releases. I would imagine as a sysadmin that’s one of the responsibilities. If I upgrade a version of a dependency on a library in any Java software I usually get new versions for libraries it depends on; that’s just the normal way of doing software development. We have to allocate time to change that single number, and it’s usually worth it in the log run.


    Nils Tuesday, May 17, 2016 at 14:47 - Reply

    I think most sysadmins aren’t comfortable with packaging and maintaining software themselves. It’s really a problem with distributions being too conservative with upgrades in their stable releases.


Rouxenator Sunday, May 15, 2016 at 11:18 - Reply

Good thing Server 2016 and Windows 10 has HTTP/2 support built in.


Jeremy Spangle Sunday, May 15, 2016 at 13:57 - Reply

Great article! Thanks for taking the time to write it. I just wanted to add that I run Fedora 23 and its version of openssl is at 1.0.2h-fips.


veloc1ty Sunday, May 15, 2016 at 18:55 - Reply

For me your mentioned lsof “query” is running very long on a busy server. I want to add another approach. I first locaed libssl:
:~$ locate libssl

The result: /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

After that I issued:
:~$ lsof /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

Which then gives me really fast the applications using libssl.


Joel Coehoorn Sunday, May 15, 2016 at 21:15 - Reply

Great and important article! I think it would be even more useful going a bit more in depth about why we should care about http/2 vs http/1.1.


Prakash Monday, May 16, 2016 at 09:06 - Reply

Debian is on 1.0.2 as well. Did you do an ‘apt-get update’?


Jan Wildeboer Wednesday, May 18, 2016 at 14:00 - Reply

There is a world outside of OpenSSL ;-) Red Hat Enterprise Linux offers GnuTLS with ALPN since RHEL 7.1 …


    Mattias Geniar Wednesday, May 18, 2016 at 14:04 - Reply

    This package is installed on most RHEL 7.x boxes by default (as a dependency for the NetworkManager), but Nginx or Apache won’t use that by default — it’s linked against OpenSSL.

    How does GnuTLS fix this particularly? (I’m genuinely asking as it sounds like it could be a good fix, but I don’t see how it fits into this, yet)


    eRadical Thursday, June 2, 2016 at 22:01 - Reply

    http://www.gnutls.org/manual/html_node/Compatibility-with-the-OpenSSL-library.html still says:

    “To ease GnuTLS’ integration with existing applications, a compatibility layer with the OpenSSL library is included in the gnutls-openssl library. This compatibility layer is not complete and it is not intended to completely re-implement the OpenSSL API with GnuTLS. It only provides limited source-level compatibility.”

    Any thoughts?


eRadical Thursday, June 2, 2016 at 21:56 - Reply

As I had to do some refactoring in puppet… I delved into enabling HTTP/2 in Nginx (1.11.1).
It’s not that much of a pain!

Download nginx & openssl alongside… like this:
[root@web2 installed]# ls -al | grep -P “(nginx-1.11|openssl-1.0.2)”
drwxr-xr-x. 9 root root 4096 Jun 2 19:28 nginx-1.11.1
-rw-r–r–. 1 root root 913417 Jun 2 13:35 nginx-1.11.1.tar.gz
drwxr-xr-x. 21 root root 4096 Jun 2 19:30 openssl-1.0.2h
-rw-r–r–. 1 root root 5274412 May 3 13:57 openssl-1.0.2h.tar.gz

Just go into nginx folder and append to your configure command to be like:
./configure –….your-other-configure-options… –with-openssl=../openssl-1.0.2h –with-openssl-opt=”-fPIC”

Then you just make && make install :)
All is well for me on CentOS 7!

Cheers,


abdussamad Thursday, June 2, 2016 at 23:53 - Reply

So Chrome users will just use http 1.1 right? It’s not like they won’t be able to access your website AT ALL? I’d appreciate an answer for this.


Paul Braren Thursday, June 16, 2016 at 21:47 - Reply

Wonderful article.
Finally, yesterday, Cloudways (which is using Digital Ocean gear) allows easy turnkey HTTP/2 for existing customers who already have https enabled:
http://feedback.cloudways.com/forums/203824-service-improvement/suggestions/7562001-implement-http2
Sure enough, I lit up HTTP/2 today temporarily, then used webpagetest.org to retest, using webpagetest.org’s handy browser selection menu. The Firefox test was about 1.5x faster than the Chrome test. More testing needed, but still, interesting…


Svenn Friday, August 19, 2016 at 14:11 - Reply

Off course they will be able to see your website at http/1.1 . But creating a website/app with http/2 in mind can be different then for http/1(.1)


Anna Jonna Ármannsdóttir Friday, August 19, 2016 at 17:23 - Reply

Ok! Im one of those lazy halfassed sysadmins. I observe that only one server version supports OpenSSL 1.0.2 and only Chrome demands it. I am observing one WordPress website that is broken in Chrome but not Firefox and not IE. This site is served by a CentOS 7 server that only supports OpenSSL 1.0.1 . My lazy halfass has only used dozens of hours compiling Apache and OpenSSL 1.0.2 only to find out that I would probably break the OS. And repositories that offer packages like EPEL, IUS, Dag Wieer, none of them offer an openssl v. 1.0.2 package.
So in order to fix this for Chrome users, I will have to disable http2.0 for everyone else.
How about Chrome users just get some sense and use Firefox for a chance?
As for Google advancing technology: Are you or are you?


Sicco Thursday, September 8, 2016 at 16:49 - Reply

You can also use the official Nginx Docker image based on Alpine Linux instead of Debian Jessie, as it comes with OpenSSL 1.0.2. See my Serverfault answer for more details and an example: http://serverfault.com/a/801881/124535


crstin Monday, May 15, 2017 at 11:11 - Reply

Still, to this day the same problem with httpd. centos7: `OpenSSL 1.0.1e-fips 11 Feb 2013`
And no, I don’t want to use nginx. I intend to keep using apache.


    Mattias Geniar Thursday, May 18, 2017 at 13:16 - Reply

    And no, I don’t want to use nginx. I intend to keep using apache.

    Since both Nginx & Apache share the samen OpenSSL on the server, both are affected here. The only reason Nginx is shown as an alternative is because there are simple-to-use Docker images available.

    I’m sure those are available for Apache too, which means you can use Apache, in Docker, with an up-to-date OpenSSL version, allowing HTTP/2.


Leave a Reply to WYSIWYG Cancel reply

Your email address will not be published. Required fields are marked *

Inbound links