Modeling Package Manager Dependencies In Config Management

The recent glibc (CVE-2015-0235) and heartbleed OpenSSL (CVE-2014-0160 ) patches have given me and other sysadmins quite a bit of work. Which makes me wonder, what can we do to ease our pains?

Package Managers Are Smart

I’m an avid Puppet user as it’s my Config Management tool of choice. So I’ve been thinking about making these kind of CVE’s and software patches less painful. Here’s what I’m currently thinking of, but I don’t think it’s a viable strategy in the long run.

For starters, it would be too time consuming to manage and keep this in sync with reality. It’ll also make Puppet intolerably slow. But it’s a thought, and I’ll see where it takes me – perhaps it’ll spark a discussion or two and get me on the right track.

So here’s the thing: package manager have dependencies built-in on their package format (in the SPEC files for RPM and the debian build-files). Those package managers know that in order to install, say, httpd, the OS also needs glibc, apr, openssl (if mod_ssl is requested), …

(example RPM SPEC code)
...
Requires(post): openssl >= 0.9.7f-4, /bin/cat
Requires(post): systemd-units
...

The OS’ Package Manager has all these details. Our config management does not.

It would have been very easy in the case of the Heartbleed bug to simply put this down in our Puppet code.

package { 'openssl':
  ensure => latest,
}

And be done with it.

But that’s not how it works. Many services depend on that package and load those library files on start, once. If the library gets updated, the service needs to restart in order to read those new libraries.

Modeling The Package Manager in Config Management

So what if we “rebuilt” the Yum or Apt package managers into our config management? If we could import the entire dependency tree into our configs and have the above actually work?

It can be done.

In a simplistic version, it would look like this.

service { 'httpd':
  ensure    => running,
  subscribe => Package [ 'openssl', 'abr', 'glibc' ],
}

service { 'mysqld':
  ensure    => running,
  subscribe => Package [ 'openssl', 'glibc' ],
}

...

If we can model the package dependencies this way, fixing the Heartbleed bug would have been as easy as just updating OpenSSL.

# This will trigger a reload of the httpd and mysqld service,
# with the subscribes above.
package { 'openssl':
  ensure => latest,
}

But it just isn’t maintainable.

Debian has a tool to help with this, called “checkrestart” in the debian-goodies package. It can help list all services that are still using old version of upgraded files. It’s basically a prettier version of the following one-liner.

$ lsof | grep libssl | awk '{print $1}' | sort | uniq

To my knowledge, only Debian has such a tool to help find old usages of libraries. There’s no such tool for Red Hat/CentOS or Ubuntu?

Thanks to the comments, I’ve learned Red Hat / CentOS have a tool in the yum-utils package called needs-restarting that lists all services that need a restart to load the newly installed libraries.

What if …

I think there are 2 possible solutions to this – at least, that I can see now.

One, is to actually rebuild the Package Manager dependencies in config management (as in my example above, by actually writing it in config management DSL), but in an automated way. We won’t be able to correctly set all dependencies by hand, so it would have to be automated.

But how do you fit it in to your current modules? Do you let a script modify your puppet code? Seems unlikely.

Alternatively, a tool like checkrestart can be ported and built into our config management toolset. Sure, we can build hacky workarounds with tools like mcollective, writing custom agents and have them perform the task for us (before you say “that’s probably the best solution”: I consider that a hack and a limitation of the CMS).

Wouldn’t this be better suited in the config management tools themselves?

Granted, you can’t blindly update libraries and have the config management tool restart services whenever it wants to. For some servers, it’s just not an option. But the restart of those services had to happen anyway, to resolve the vulnerability – so why not make it a little easier for us?