The day Google Chrome disables HTTP/2 for nearly everyone: May 31st, 2016

Mattias Geniar, Saturday, May 14, 2016 - last modified: Thursday, May 26, 2016

If you've been reading this blog for a while (or have been reading my rants on Twitter), you'll probably know this was coming already. If you haven't, here's the short version.

The Chromium project (whose end result is the Chrome Browser) has switched the negotiation protocol by which it decides whether to use HTTP/1.1 or the newer HTTP/2 on May 15th, 2016 May 31st, 2016.

Update: Chrome 51 is released and should be updated everywhere in a couple of hours/days. If HTTP/2 stops working for you, it's probably because NPN has been disabled from now on.

That in and of itself isn't a really big deal, but the consequences unfortunately are. Previously (as in: before May 31st, 2016), a protocol named NPN was used -- Next Protocol Negotiation. This wasn't a very efficient protocol, but it got the job done.

There's a newer negotiation protocol in town called ALPN -- Application-Layer Protocol Negotiation. This is a more efficient version with more future-oriented features. It's a good decision to switch from NPN to ALPN, there are far more benefits than there are downsides.

However, on the server side -- the side which runs the webservers that in turn run HTTP/2 -- there's a rather practical issue: to support ALPN, you need at least OpenSSL 1.0.2.

So what? You're a sysadmin, upgrade your shit already!

I know. It sounds easy, right? Well, it isn't. Just for comparison, here's the current (May 2016) state of OpenSSL on Linux.

Operating System OpenSSL version
CentOS 5 0.9.8e
CentOS 6 1.0.1e
CentOS 7 1.0.1e
Ubuntu 14.04 LTS 1.0.1f
Ubuntu 16.04 LTS 1.0.2g
Debian 7 (Wheezy) 1.0.1e
Debian 8 (Jessie) 1.0.1k

As you can tell from the list, there's a problem: out of the box, only the latest Ubuntu 16.04 LTS (out for less than a month) supports OpenSSL 1.0.2.

Upgrading OpenSSL packages isn't a trivial task, either. Since just about every other service links against the OpenSSL libraries, they too should be re-packaged (and tested!) to work against the latest OpenSSL release.

On the other hand, it's just a matter of time before distributions have to upgrade as support for OpenSSL 1.0.1 ends soon.

Support for version 1.0.1 will cease on 2016-12-31. No further releases of 1.0.1 will be made after that date. Security fixes only will be applied to 1.0.1 until then.

OpenSSL Release Strategy

To give you an idea of the scope of such an operation, on a typical LAMP server (the one powering the blogpost you're now reading), the following services all make use of the OpenSSL libraries.

$ lsof | grep libssl | awk '{print $1}' | sort | uniq
anvil
fail2ban
gdbus
gmain
httpd
postfix
mysqld
NetworkManager
nginx
php-fpm
puppet
sshd
sudo
tuned
zabbix_agent

A proper OpenSSL upgrade would cause all of those packages to be recreated too. That's a hassle, to say the least. And truth be told, it probably isn't just repackaging but potentially changing the code of each application to be compatible to the newer or changed API's in OpenSSL 1.0.2.

Right now, the proper simplest way to run HTTP/2 on a modern server (that isn't Ubuntu 16.04 LTS) would be to run a Docker container, based on Ubuntu 16.04, and run your webserver inside of it.

I don't blame Google for switching protocols and evolving the web, but I'm sad to see that as a result of it, a very large portion of Google Chrome users will have to live without HTTP/2, once again.

Before May 15th, 2016 -- a Google Chrome user would see this in its network inspector:

protocol_http2_enabled

After May 31st, it'll be old-skool HTTP/1.1.

protocol_http2_disabled

It used to be that enabling HTTP/2 in Nginx was a very simple operation, but in order to support Chrome it'll be a bit more complicated from now on.

This change also didn't come out of the blue: Chrome had disabled NPN back in 2015 but quickly undid that change when the impact was clear. We knew, since the end of 2015, that this change was coming -- we were given 6 months time to get support for ALPN going, but by the current state of OpenSSL packages that was too little time.

If you want to keep track of the state of Red Hat (Fedora, RHEL & CentOS) upgrades, here's some further reading: RFE: Need OpenSSL 1.0.2.

As I'm mostly a CentOS user, I'm unaware of the state of Debian or Ubuntu OpenSSL packages at this time.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. Currently working on DNS Spy. Follow me on Twitter as @mattiasgeniar.

I respect your privacy and you won't get spam. Ever.
Just a weekly newsletter about Linux and open source.

SysCast podcast

In the SysCast podcast I talk about Linux & open source projects, interview sysadmins or developers and discuss web-related technologies. A show by and for geeks!

cron.weekly newsletter

A weekly newsletter - delivered every Sunday - for Linux sysadmins and open source users. It helps keeps you informed about open source projects, Linux guides & tutorials and the latest news.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

nginx user Saturday, May 14, 2016 at 15:19 (permalink)

A bit overdramatic. You can build nginx with openssl 1.0.2 on any distribution of Linux.

./configure –with-openssl=/path/to/openssl

Reply


    Pav Monday, June 13, 2016 at 12:51 (permalink)

    A bit unrealistic. As an admin you do not want to re-compile every time there is a nginx mainline or an OpenSSL update, thus relying on your distro’s repositories is much more convenient, secure and stable approach.

    Also don’t forget SPDY is out of the game too…

    Reply


Alan Orth Saturday, May 14, 2016 at 15:55 (permalink)

Then you’re responsible for your own nginx and openssl updates.

Reply


Mark Saturday, May 14, 2016 at 16:14 (permalink)

Pretty much the above. This is not Windows.

Reply


Clay Smith Saturday, May 14, 2016 at 17:54 (permalink)

Had the same issue a few months ago, great post. I wrote a guide to compiling OpenSSL on Ubuntu 14.04 here: https://www.clay.fail/posts/ubuntu-http2-in-mere-hours/

I later switched to using Docker to a kid the source code compiling entirely: https://www.clay.fail/posts/hip-http2-using-docker/

Reply


WYSIWYG Saturday, May 14, 2016 at 18:52 (permalink)

What I’m getting out of this topic is – don’t use Chrome, stick to Firefox.

Reply


Will Saturday, May 14, 2016 at 19:59 (permalink)

Are you implying that those packages listed all link against OpenSSL statically? That is not necessarily the case and might “just work” if there is a system-wide update to the shared version of openssl. I know that’s besides the point, but it seems like lsof is not differentiating between libraries loaded dynamically and those loaded statically.

Will

Reply


MojoDwarf Saturday, May 14, 2016 at 20:04 (permalink)

Has anyone looked at libressl compatibility with ALPN?

Reply


    Olivier Tuesday, May 17, 2016 at 01:51 (permalink)

    Sure, you can compile nginx with LibreSSL and get your website compatible with ALPN.

    Reply


moneytoo Saturday, May 14, 2016 at 21:41 (permalink)

Yeah, ideally just rebuild the source package you were using till this point. On CentOS it’s just a few lines… https://gist.github.com/moneytoo/ab3f34e4fddc2110675952f8280f49c5

Reply


Joshua McKinnon Sunday, May 15, 2016 at 00:09 (permalink)

This could definitely be an issue. Speaking as someone who runs OpenSSL on Ubuntu Server LTS releases, unless LTS policies have changed, I don’t see Ubuntu jumping major versions of OpenSSL in a given LTS release. (I hope I’m proven wrong)

This was an issue for me back in Ubuntu 10.04 LTS – whatever version of OpenSSL it shipped with, probably 0.9. something, did not support TLS 1.1 and 1.2. Sure, back when it originally shipped that wasn’t an issue, but for the last year or two of its LTS support? Crazy to be stuck on TLS 1.0. I recall finding a thread where it was noted that it was too risky/stability issue to jump. I expect Ubuntu team to backport security fixes to older branch in LTS line, rather than jump to new version. That was what they did in 10.04, even though it was “supported” you either had to compile it yourself or upgrade to 12.04 to get the newer OpenSSL line.

We all know we can compile and patch it ourselves, but that is not a great solution for many servers when you need to be able to fix 0-days in a matter of a coupe of hours the next time another heartbleed level CVE comes along.

Reply


Asinglenoob Sunday, May 15, 2016 at 04:28 (permalink)

Cry harder? You’re a sys admin, it’s your job to keep stuff up to date. It’s this sort of lazy half assed update policy that causes massive security breaches every year.

Reply


Mike skaloola Sunday, May 15, 2016 at 07:05 (permalink)

Really? The “proper” way is to use docker? Wanna qualify that in any way shape or form?

Running inside a Docker container is only proper if your infrastructure is already based around Docker

The proper thing to do is to upgrade if that aligns with how you’re managing your infrastructure.

Enabling openssl 1.0.2 in your site specific packages is the “peoper” way if you already roll your own Debs/rpms..

And so on and so forth…

.

Reply


Jesse Sunday, May 15, 2016 at 08:37 (permalink)

You sysadmin guys love your IT but sometimes you forget about pragmatic solutions. The easier solution here is to be using CloudFlare, which has HTTP/2 support for ALL connections made over SSL (HTTPS). No reason to hurry your Ubuntu upgrades if you’re not ready:

https://www.littlebizzy.com/blog/nginx-server-block-ssl

Just a basic OpenSSL Nginx block, since CloudFlare doesn’t need your origin server to enable HTTP/2 (and in fact, you should NOT enable it). Voila… free Comodo SSL and HTTP/2 ;)

Reply


    Maximillian Laumeister Monday, May 16, 2016 at 06:35 (permalink)

    This is a good sentiment, but not all web applications run very well behind a caching proxy like CloudFlare (especially dynamic applications). Plus, in certain situations CloudFlare can increase latency due to the extra trip to fetch from the origin server. CloudFlare is a great solution for many types of web apps / sites, but it’s not a perfect catch-all solution.

    Reply


Mattias Geniar Sunday, May 15, 2016 at 08:37 (permalink)

Wrong choice of words on my part, replaced “proper” with “simplest” (as that’s probably the case right now).

Reply


Chris Sunday, May 15, 2016 at 09:23 (permalink)

Static linking of libraries embed them into the executable. It would not show up in lsof (or ldd for that matter).

Reply


lzap Sunday, May 15, 2016 at 09:26 (permalink)

Backporting is the answer. Red Hat will definitely backport the fetature without upgrading the version.

Reply


Lul Sunday, May 15, 2016 at 09:36 (permalink)

Do ppl still run red hat? Wtf???

Reply


David Sunday, May 15, 2016 at 10:03 (permalink)

In layman’s terms
What must an all Android professional do to avoid y2k like drama …..??

Reply


TWiStErRob Sunday, May 15, 2016 at 10:44 (permalink)

This sounds really overdramatic, it’s not like the whole Web will be unaccessible after that update. If you want to use cutting edge technology (HTTP/2), then you have to use cutting edge technology (OpenSSL 1.0.2) to build on; otherwise everything will still work, maybe a little bit slower.

Please correct me if the above is not true.

Also you shouldn’t blame Google for advancing technology, rather try to keep up with latest releases. I would imagine as a sysadmin that’s one of the responsibilities. If I upgrade a version of a dependency on a library in any Java software I usually get new versions for libraries it depends on; that’s just the normal way of doing software development. We have to allocate time to change that single number, and it’s usually worth it in the log run.

Reply


    Nils Tuesday, May 17, 2016 at 14:47 (permalink)

    I think most sysadmins aren’t comfortable with packaging and maintaining software themselves. It’s really a problem with distributions being too conservative with upgrades in their stable releases.

    Reply


Rouxenator Sunday, May 15, 2016 at 11:18 (permalink)

Good thing Server 2016 and Windows 10 has HTTP/2 support built in.

Reply


Jeremy Spangle Sunday, May 15, 2016 at 13:57 (permalink)

Great article! Thanks for taking the time to write it. I just wanted to add that I run Fedora 23 and its version of openssl is at 1.0.2h-fips.

Reply


Arch Sunday, May 15, 2016 at 14:52 (permalink)

Arch is also on 1.0.2.h-1 as of today:
https://www.archlinux.org/packages/core/x86_64/openssl/

Reply


veloc1ty Sunday, May 15, 2016 at 18:55 (permalink)

For me your mentioned lsof “query” is running very long on a busy server. I want to add another approach. I first locaed libssl:
:~$ locate libssl

The result: /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

After that I issued:
:~$ lsof /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

Which then gives me really fast the applications using libssl.

Reply


Joel Coehoorn Sunday, May 15, 2016 at 21:15 (permalink)

Great and important article! I think it would be even more useful going a bit more in depth about why we should care about http/2 vs http/1.1.

Reply


Prakash Monday, May 16, 2016 at 09:06 (permalink)

Debian is on 1.0.2 as well. Did you do an ‘apt-get update’?

Reply


Jan Wildeboer Wednesday, May 18, 2016 at 14:00 (permalink)

There is a world outside of OpenSSL ;-) Red Hat Enterprise Linux offers GnuTLS with ALPN since RHEL 7.1 …

Reply


    Mattias Geniar Wednesday, May 18, 2016 at 14:04 (permalink)

    This package is installed on most RHEL 7.x boxes by default (as a dependency for the NetworkManager), but Nginx or Apache won’t use that by default — it’s linked against OpenSSL.

    How does GnuTLS fix this particularly? (I’m genuinely asking as it sounds like it could be a good fix, but I don’t see how it fits into this, yet)

    Reply


    eRadical Thursday, June 2, 2016 at 22:01 (permalink)

    http://www.gnutls.org/manual/html_node/Compatibility-with-the-OpenSSL-library.html still says:

    “To ease GnuTLS’ integration with existing applications, a compatibility layer with the OpenSSL library is included in the gnutls-openssl library. This compatibility layer is not complete and it is not intended to completely re-implement the OpenSSL API with GnuTLS. It only provides limited source-level compatibility.”

    Any thoughts?

    Reply


eRadical Thursday, June 2, 2016 at 21:56 (permalink)

As I had to do some refactoring in puppet… I delved into enabling HTTP/2 in Nginx (1.11.1).
It’s not that much of a pain!

Download nginx & openssl alongside… like this:
[root@web2 installed]# ls -al | grep -P “(nginx-1.11|openssl-1.0.2)”
drwxr-xr-x. 9 root root 4096 Jun 2 19:28 nginx-1.11.1
-rw-r–r–. 1 root root 913417 Jun 2 13:35 nginx-1.11.1.tar.gz
drwxr-xr-x. 21 root root 4096 Jun 2 19:30 openssl-1.0.2h
-rw-r–r–. 1 root root 5274412 May 3 13:57 openssl-1.0.2h.tar.gz

Just go into nginx folder and append to your configure command to be like:
./configure –….your-other-configure-options… –with-openssl=../openssl-1.0.2h –with-openssl-opt=”-fPIC”

Then you just make && make install :)
All is well for me on CentOS 7!

Cheers,

Reply


abdussamad Thursday, June 2, 2016 at 23:53 (permalink)

So Chrome users will just use http 1.1 right? It’s not like they won’t be able to access your website AT ALL? I’d appreciate an answer for this.

Reply


Paul Braren Thursday, June 16, 2016 at 21:47 (permalink)

Wonderful article.
Finally, yesterday, Cloudways (which is using Digital Ocean gear) allows easy turnkey HTTP/2 for existing customers who already have https enabled:
http://feedback.cloudways.com/forums/203824-service-improvement/suggestions/7562001-implement-http2
Sure enough, I lit up HTTP/2 today temporarily, then used webpagetest.org to retest, using webpagetest.org’s handy browser selection menu. The Firefox test was about 1.5x faster than the Chrome test. More testing needed, but still, interesting…

Reply


Svenn Friday, August 19, 2016 at 14:11 (permalink)

Off course they will be able to see your website at http/1.1 . But creating a website/app with http/2 in mind can be different then for http/1(.1)

Reply


Anna Jonna Ármannsdóttir Friday, August 19, 2016 at 17:23 (permalink)

Ok! Im one of those lazy halfassed sysadmins. I observe that only one server version supports OpenSSL 1.0.2 and only Chrome demands it. I am observing one WordPress website that is broken in Chrome but not Firefox and not IE. This site is served by a CentOS 7 server that only supports OpenSSL 1.0.1 . My lazy halfass has only used dozens of hours compiling Apache and OpenSSL 1.0.2 only to find out that I would probably break the OS. And repositories that offer packages like EPEL, IUS, Dag Wieer, none of them offer an openssl v. 1.0.2 package.
So in order to fix this for Chrome users, I will have to disable http2.0 for everyone else.
How about Chrome users just get some sense and use Firefox for a chance?
As for Google advancing technology: Are you or are you?

Reply


Sicco Thursday, September 8, 2016 at 16:49 (permalink)

You can also use the official Nginx Docker image based on Alpine Linux instead of Debian Jessie, as it comes with OpenSSL 1.0.2. See my Serverfault answer for more details and an example: http://serverfault.com/a/801881/124535

Reply


crstin Monday, May 15, 2017 at 11:11 (permalink)

Still, to this day the same problem with httpd. centos7: `OpenSSL 1.0.1e-fips 11 Feb 2013`
And no, I don’t want to use nginx. I intend to keep using apache.

Reply


    Mattias Geniar Thursday, May 18, 2017 at 13:16 (permalink)

    And no, I don’t want to use nginx. I intend to keep using apache.

    Since both Nginx & Apache share the samen OpenSSL on the server, both are affected here. The only reason Nginx is shown as an alternative is because there are simple-to-use Docker images available.

    I’m sure those are available for Apache too, which means you can use Apache, in Docker, with an up-to-date OpenSSL version, allowing HTTP/2.

    Reply


Leave a Reply to Joel Coehoorn Cancel reply

Your email address will not be published. Required fields are marked *

Inbound links