Kmorph Aims To Provide Rebootless Linux Kernel Crash Recovery

Kmorph aims to provide reboot-less kernel crash recovery for Linux systems and can yield as little as ~2 second downtime for recovering the Nginx web server or other important services in the case of a kernel crash. The Kmorph open-source project was announced on Friday for this rebootless kernel crash recovery and is built around the Multikernel work that's been happening out-of-tree over the past year. So for now at least, Kmorph will not work on the mainline Linux kernel and remains to be seen if/when the multikernel code will make it to mainline. With Multikernel providing multiple independent kernel instances to co-exist on the same physical system, one of the use-cases opened up by that is rather quick recovery in the case of one kernel crashing and being able to quickly transition to an alternative kernel. With Kmorph, the system can survive a kernel crash without a reboot: "With multikernel, each kernel owns its own CPUs, memory and devices, and the kernels talk to each other over vsock. kmorph uses this to replace both reboots with a takeover. A second kernel, the successor, is booted ahead of time on a CPU or two and a slice of memory beside the running kernel, the predecessor, and watches it. When the predecessor panics or hangs, the successor fences its CPUs, adopts and resets its devices, restores the network identity, optionally saves a kdump compatible vmcore, then hot-adds the remaining CPUs and memory and becomes the machine's kernel. Detection to a running successor takes about one second, with no reboot, no firmware and no operator. The same takeover gives a live kernel update: "kmorph upgrade" hands a healthy machine over to a successor running a newer kernel. Either way the new kernel can arm a successor of its own, generation after generation. All of the novel work is in the multikernel kernel: fencing, device and IOMMU handover, and CPU and memory hot-add from a host device tree. kmorph is plain C orchestration on top of its interfaces, and integrates with distros the way kdump does, through dracut, initramfs-tools, kernel-install and a systemd service. It works today on both x86_64 and ARM64, and RISC-V support is in progress." Kmorph also supports the notion of important systemd services to run in standby on the fallback kernel so that service recovery can be very quick. With Nginx as an example, on a kernel crash it could be brought back online in as little as two seconds followed by the other less important services starting up. The mainline Linux kernel does have the Kexec Handover (KHO) and Live Update Orchestrator (LUO) functionality but currently doesn't support similar kernel crash recovery with a Kexec reboot still being needed for kernel updates and not able to sustain kernel crashes either. Kmorph is an interesting concept with a lot of potential albeit dependent upon the future of Multikernel going mainline for any widespread adoption. More details on Kmorph can be found via the Linux kernel mailing list announcement. The code is available on GitHub.