Kernel patching with kexec: updating a CentOS 7 kernel without a full reboot

Mattias Geniar, Thursday, February 23, 2017

tl;dr: you can use kexec to stage a kernel upgrade in-memory without the need for a full reboot. Your system will reload the new kernel on the fly and activate it. There will be a service restart of every running service as the new kernel is loaded, but you skip the entire bootloader & hardware initialization.

By using kexec you can upgrade your running Linux machine's kernel without a full reboot. Keep in mind, there's still a new kernel load, but it's significantly faster than doing the whole bootloader stage and hardware initialization phase performed by the system firmware (BIOS or UEFI).

Yes, calling this kernel upgrades without reboots is a vast exaggeration. You skip parts of the reboot, though, usually the slowest parts.

Installing kexec

On a CentOS 7 machine the kexec tools should be installed by default, but just in case they aren't;

$ yum install kexec-tools

After that, the kexec binary should be available to you.

Install your new kernel

In this example I'll upgrade a rather old CentOS 7 kernel to the latest.

$ uname -r
3.10.0-229.14.1.el7

So I'm now running the 3.10.0-229.14.1.el7 kernel.

To upgrade your kernel, first install the latest kernel packages.

$ yum update kernel
...
===================================================================================
 Package                 Arch      Version                        Repository  Size
===================================================================================
Installing:
 kernel                  x86_64    3.10.0-514.6.1.el7             updates     37 M

This will install the

3.10.0-514.6.1.el7

kernel on my machine.

So a quick summary (on new lines, so you see the kernel version difference):

From: 3.10.0-229.14.1.el7
To: 3.10.0-514.6.1.el7

$ rpm -qa | grep kernel | sort
kernel-3.10.0-229.14.1.el7.x86_64
kernel-3.10.0-514.6.1.el7.x86_64

Once you installed the new kernel, it's time for the kexec in-memory upgrading magic.

In-memory kernel upgrade with kexec

As a safety command, unload any previously attempted kernels first. This is harmless and will make sure you start "cleanly" with your upgrade process.

$ kexec -u

Now, state the new kernel to be loaded. Note these are the version numbers of the latest installed kernel with yum, as shown above.

$ kexec -l /boot/vmlinuz-3.10.0-514.6.1.el7.x86_64 \
 --initrd=/boot/initramfs-3.10.0-514.6.1.el7.x86_64.img \
 --reuse-cmdline

Careful: next command will reload a new kernel and will impact running services!

Once prepared, start kexec.

$ systemctl kexec

Your system will freeze for a couple of seconds, load the new kernel and be good to go.

Some benchmarks

A very quick and unscientific benchmark of doing a yum update kernel with and without kexec.

Normal way, kernel upgrade + reboot: 28s
Kexec way, kernel upgrade + reload: 19s

So you have a couple of seconds of the new kernel load, for big physical machines with lots of RAM, this will be even more as the entire POST check can be skipped with this method.

Here's a side-by-side run of the same kernel update. On the left: the kexec flow you've read above. On the right, a classic yum update kernel && reboot.

Notice how the left VM never goes into the BIOS or POST checks.

If you're going to be automating these updates, have a look at some existing scripts to help you going: kexec-reboot, ArchWiki on kexec.



Hi! My name is Mattias Geniar. 👋 I'm an independent software developer ⌨️ & Linux sysadmin 👨‍💻, a general web geek & public speaker. Currently working on DNS Spy & Oh Dear! Follow me on Twitter as @mattiasgeniar 🐦.

🔥 If you're stuck with a technical problem, I'm available for hire to help you fix it!

Share this post

Did you like this post? Help me share it on social media! Thanks. 🤗

Have feedback?

New comments have been disabled on this blog, existing comments will remain as-is. Want to give feedback? Is there a mistake in the post?

Send me a tweet on @mattiasgeniar!

Comments

Igor Friday, March 3, 2017 at 15:23 -

Thanks for promoting kexec (which is nothing new and is able to do what you have described for more than 7 years) but please update your article/video:

Using kexec has nothing to do with “patching” a kernel. And you aren’t doing anything “live”. You are just restarting the OS (not the whole system!), i.e. your are avoiding your system’s firmware (BIOS, now EFI) initializing including PXE boot, HW RAID, or other peripheral devices. But you have to shutdown all running processes and services, unmount file systems, will lose network connectivity…

The real live patching capability of the kernel is a different thing. It allows you to keep your entire system up and running without interrupting services.


gnutux95 Friday, March 10, 2017 at 02:18 -

Thank you for your presentation of kexec.

Juste a note for best practice professionnel :

No update kernel -> NEVER -> yum update kernel-* (NEVER)

You make an NEW install not an upgrade -> yum install kernel-*


Illya Sunday, March 12, 2017 at 10:50 -

IIRC, Yum will treat a yum upgrade kernel* as a yum install operation thus giving the admin a helping hand.


Jack Thursday, August 10, 2017 at 08:00 -

I tried as you suggested as above., But after I run the command systemctl kexec server is not responding until I reboot manually.


Mats Saturday, May 26, 2018 at 15:33 -

This works perfectly, updated 5 Centos 7 hosts with
yum update
kexec -u
kexec -l /boot/vmlinuz-3.10.0-862.3.2.el7.x86_64 –initrd=/boot/initramfs-3.10.0-862.3.2.el7.x86_64.img –reuse-cmdline
systemctl kexec (lost ssh conection here and loged in again to a live upgraded machine)
I give two thumbs up!


Christian Hummel Tuesday, January 15, 2019 at 12:11 -

Hi, as alread mentioned.
This is not an Online Kernel Switch.

Your programs, processes and screen sessions, they all get killed!

It’s just a shorter reboot.


r9host.com Tuesday, January 29, 2019 at 10:43 -

Thanks , it was very helpful. But for those who just can afford, the paid KernelCare software published by CloudLinux, is the perfect way of automatic live kernel updates.


Yar Wednesday, July 3, 2019 at 15:44 -

Nice solution, if your servers are slow as a crap (like HP).
but, it is not “without reboot” in its meaning. It takes some downtime.

so, needed to get solution without any downtime, if it is possible.