LKRG Guide

Overview¶

The Linux Kernel Runtime Guard (LKRG) is a kernel module that performs runtime integrity checking of the Linux kernel and detects security vulnerability exploits. LKRG comes pre-installed and enabled by default on CIQ’s RLC-H systems as a critical security component.

Key Features¶

Real-time Kernel Integrity Monitoring: Continuously monitors integrity of the kernel code and its most security-critical data (notably including task credentials), as well as CPU security features.
Detection and Alerting: Detects and logs alerts on unauthorized modifications to the running kernel, its data, or CPU configuration.
Active Response: Automatically mitigates detected attacks to prevent usage of rootkits and privilege escalation exploits.
Remote Logging: Sends its own and other kernel messages to a remote system directly from the kernel via an encrypted connection.

Installing LKRG¶

On RLC-H, LKRG is pre-installed and enabled.

To install `lkrg`¶

sudo dnf install lkrg
sudo systemctl start lkrg
sudo systemctl enable lkrg

To update `lkrg`¶

sudo dnf update lkrg

Uninstalling LKRG (Not Recommended)¶

To stop, disable, and remove LKRG:

sudo systemctl stop lkrg
sudo systemctl disable lkrg
sudo dnf remove lkrg

Note

LKRG supports the kernel command-line parameter nolkrg, which may be passed via the bootloader to temporarily disable LKRG, such as when troubleshooting system boot-up issues..

Configuring LKRG¶

The default configuration shipped with CIQ’s LKRG package is also the recommended configuration, so normally you don’t need to change anything (except for remote logging if required).

You can adjust settings in three ways:

Temporary (in-memory): via sysctl (affects the current in-memory configuration only, not persistent)
Persistent (on boot): /etc/sysctl.d/01-lkrg.conf (applies shortly after LKRG is loaded into the kernel on system boot-up)
Module load-time: /etc/modprobe.d/lkrg.conf (provides initial configuration for LKRG)

For available configuration options, see /usr/share/doc/lkrg/README

Monitoring LKRG¶

To view LKRG logs¶

dmesg | grep LKRG
sudo journalctl -k | grep LKRG

Real-time log monitoring¶

sudo journalctl -kf | grep LKRG

Simple Monitoring and Alerting¶

Email alerts with cron¶

Create a (pseudo-)user account in group systemd-journal, then edit its crontab:

sudo useradd -G systemd-journal lkrg-monitor
sudo crontab -e -u lkrg-monitor

Add this line (replace with your email address):

MAILTO=admin@example.org
*/5 * * * * journalctl -k --cursor-file=lkrg-monitor-cursor | egrep 'kernel: LKRG: (ALERT|ALIVE|DYING|FATAL|FAULT|STATE): '

Remote Logging¶

LKRG includes a real-time remote kernel message logging capability, which is more reliable than the simple email setup above. Notably, it doesn't give a possible intruder much time to tamper with the logs, and it is likely to capture events happening during the last second before a system crash, which systemd journal is likely to miss.

Receiving Logs¶

The corresponding userspace tools are found in the lkrg-logger sub-package. Documentation is also included in there, in /usr/share/doc/lkrg-logger/LOGGING. With our packaging, most of the setup described in that upstream documentation file had already been performed. The only steps left to perform on the receiving log server are:

sudo -s
dnf install lkrg-logger
umask 077
lkrg-keygen > /etc/lkrg-logger.conf
systemctl enable lkrg-logger
systemctl start lkrg-logger

Logs will appear under /var/log/lkrg-logger readable by users in group lkrg-logger, and indeed by root. Please use the lkrg-logctl program to display them with proper timestamps.

Sending Logs¶

On the system running LKRG, create /etc/modprobe.d/lkrg.conf with:

options lkrg net_server_addr=<SERVER_IP> net_server_pk=<SERVER_PUBLIC_KEY>

Replace <SERVER_IP> with the log server’s IP and <SERVER_PUBLIC_KEY> with the key from LKRG_LOGGER_PK= in /etc/lkrg-logger.conf on the log server.

Please refer to the "Remote logging configuration" section in /usr/share/doc/lkrg/README.

LKRG Integration into Security Operations¶

Keep updated: Regularly update both LKRG and the kernel.
Integrate logs: Forward LKRG logs into your SIEM or logging system (Splunk, ELK, Graylog).
Incident response: Define clear procedures for handling LKRG alerts in your security operations plan.