What is Journalctl and what can it do?

Journalctl is an efficient solution for managing and analysing system logs in Linux environments. This tool can be used for monitoring system activities, troubleshooting and real-time log tracking, providing crucial insights for effectively diagnosing system problems.

What is Journalctl?

journalctl is a powerful utility for querying and displaying event logs or logfiles under Linux. It’s a central component of the system and service management suite systemd, which comes with many modern Linux distributions such as Ubuntu, Fedora and Arch Linux. The name ‘journalctl’ is a mix of ‘journal’ (log) and ‘ctl’ (control), which refers to the fact that the command is used to control and analyse logs.

journalctl simplifies accessing the system log, which is managed by systemd-journald. The system log is a centralised collection of messages and events that occur during the operation of a Linux system. Unlike traditional text-based logs, journalctl provides a structured and efficient way to search, filter and display log data in real time. This can help diagnose problems or monitor system status.

For this purpose, Journalctl saves the log files in binary format, because binary log files are more compact and efficient than their text-based equivalents. Given the large amount of logged data, this enables a faster and more efficient search for specific events or information within the log files. With their structured and encrypted nature, binary log files also provide enhanced security compared to text-based logs, since their format makes manipulating log data more challenging.

How to adjust storage space for log files

journalctl can be used to limit and configure storage space that log files occupy on the hard disk. This is done via the systemd-journald service settings. The configuration settings are stored in the file /etc/systemd/journald.conf. Here, you’ll find the following entries:

• SystemMaxUse: Limits the storage space for logs in the system directory
• RuntimeMaxUse: Limits the storage space for logs in the temporary directory

To set the memory limit, add the following lines or change them as required:

[Journal] 
SystemMaxUse=50M 
RuntimeMaxUse=50M

The values from this example (50M) can be modified as required. You can also use different units such as K (kilobytes), M (megabytes), G (gigabytes) or T (terrabytes). Once you’ve edited the configuration file, systemd-journald must be restarted for the changes to take effect:

sudo systemctl restart systemd-journald

Following these steps, systemd-journald will be configured to restrict disk space for log files. Make sure the limit you choose can adequately store essential log data. At the same time, it’s important to avoid using an excessive amount of disk space. Keep in mind that older logs are automatically removed once the limit of allotted disk space has been reached.

Assess disk space usage

Before adjusting the storage space for log files, check how much storage space is currently occupied by the journal. To do this, use --disk-usage:

journalctl --disk-usage

Here’s an example of what the output looks like:

Journals take up 8.0M on disk.

Delete old log entries

If Journal takes up too much memory, you can delete old log entries. There are two ways to do so:

Use --vacuum-size to reduce the size of your journal by specifying the size. With this method, old entries are removed until the total journal storage space occupied on the hard disk has reached the desired size.

sudo journalctl --vacuum-size=1G

Alternatively, you can shrink the journal by setting a retention period using the --vacuum-time option. Entries older than the specified time will be deleted. To retain log entries from the past year, you can use the following command:

sudo journalctl --vacuum-time=1years

What can Journalctl do?

journalctl offers powerful filtering capabilities, allowing users to sift through log entries based on various criteria. This feature enables targeted searches for pertinent information, facilitating quicker problem identification. Here are some commonly used journalctl filter options:

Show logs

Executing the command journalctl will display log entries for the current system in reverse chronological order. Use journalctl -f or journalctl --follow to display the entries in real time. New entries are automatically displayed in the order in which they arrive.

Depending on how long systemd has been running on your system, you’ll probably be shown an unmanageable amount of data, which can be tens or hundreds of thousands of lines long. To find what you’re looking for more quickly, you can filter the logs further using various Linux commands.

How to filter by time

Using journalctl, logs can be filtered according to a specific point in time so they can be searched more efficiently for relevant information.

Filter by boot process

Filtering logs by the boot process proves invaluable for examining system events at precise times or during boot scenarios. This targeted approach streamlines problem diagnosis by focusing log entries on specific system statuses or configurations.

1. Current boot: With journalctl -b all log entries are displayed that have been collected since the last reboot.
2. Past boots: Use the -b option followed by a number to display the log entries for a specific previous boot. For example, if you enter journalctl -b 1, the previous boot will be displayed.
3. List of all available boot processes: The command journalctl --list-boots displays a list of available boots with their respective IDs. You can use the desired boot ID to display the logs for a specific boot.

While the saving of past boot processes is activated by default on some Linux distributions, users may need to first activate it on others. To do this, create a directory where the log will be saved by entering sudo mkdir -p /var/log/journal. Alternatively, you can edit the journal configuration file with sudo nano /etc/systemd/journald.conf. Then set the option Storage= under [Journal] to persistent to activate persistent logging:

. . . 
[Journal] 
Storage=persistent

Filter by time window

Sometimes it may be necessary to display log entries for a specific time period. journalctl offers the options --since and --until, which can be used to restrict the entries to a specific period. To do this, use the time format YYYY-MM-DD HH:MM:SS. The command to display all log entries between January 1, 2023 at 12:00 and January 2, 2023 at 12:00 looks like this:

journalctl --since "2023-01-01 12:00:00" --until "2023-01-02 12:00:00"

The combination of the two commands can also filter for a shorter period of time:

journalctl --since 09:00 --until "1 hour ago"

Alternatively, you also omit parts of the format. For instance, if you want to display all entries starting at a certain point in time:

journalctl --since "2023-11-16 15:25:00"

journalctl also recognises relative values such as yesterday, today or tomorrow. To display entries from yesterday, enter the following:

journalctl --since yesterday

How to filter by message characteristics

Using journalctl to filter log entries based on importance or contents is also extremely useful as it allows you to search for relevant information and focus on specific aspects of the system logs. This enables efficient error diagnosis, early detection of security problems and rapid performance monitoring, among other things.

Filter by priority

To filter logs with journalctl by message importance, you can use the priority categories for log entries. To do this, you can use either the priority name or its corresponding numerical value. The lower the number, the more important the message:

• 0: emerg (emergency)
• 1: alert (alarm)
• 2: crit (critical)
• 3: err (error)
• 4: warning (warning)
• 5: notice (note)
• 6: info (information)
• 7: debug (troubleshooting)

Messages with a certain priority can be filtered using the -p option. For example, the following command only displays log entries with the priority ‘err’ (error) and higher:

journalctl -p err

Filter by unit

Filtering logs by unit is useful for focusing on specific services or processes. These can be filtered using the -u option. For example, to display log entries for the Apache web server, enter the following:

journalctl -u apache2

The search can be further refined using relative time values. To determine if the service has already been performed today, you can enter the following:

journalctl -u apache2 --since today

journalctl can merge data records from different units. For instance, if your Nginx process is linked to a PHP FPM unit, their entries can be merged chronologically. The command for this is:

journalctl -u nginx.service -u php-fpm.service --since today-u apache2 --since today

Filter by process, user or group ID

Journalctl can filter log entries by process, user or group ID. If you have the exact PID of the process you want to search for, you can use the _PID option to filter entries. For example, if the PID is 8088, the command would look like this:

journalctl _PID=8088

Alternatively, you can use the _UID or _GID filters to display all entries logged by a particular user or group. For example, if your web server is named ‘www-data’, you can find the user ID by doing the following:

id -u www-data
33

The journal entries can be filtered using the ID:

journalctl _UID=33 --since today

To determine which group IDs entries have been created for, you can use the -F option. This displays all the values that have been saved for the Group ID field:

journalctl -F _GID

Here’s an example of the output:

32
99
102
133
81
84
100
0
124
87

Filter by component

Filtering by component is useful for concentrating on specific applications, services or processes. The component field is typically used by various services or software components to distinguish specific information in the logs. This filtering makes it possible to narrow log entries down to a particular component, application or service unit. For instance, to filter entries containing the executable file bash, enter the following command:

journalctl /usr/bin/bash

Display kernel messages

Filtering log entries for kernel messages with journalctl is an effective way to analyse information about kernel operation in a Linux system. Kernel messages can provide indications of hardware problems, driver conflicts or other system events.

Kernel messages that can be found in the dmesg output can also be filtered from the journal. They can be displayed with the flags -k or–dmesg `:

journalctl -k

The kernel messages of current boot process are displayed by default. You can filter for messages from an alternative boot using the previously mentioned boot selection flags. If you, for example, want to view the kernel messages of the last five boot processes, enter:

journalctl -k -b -5

Changing the journal display in Journalctl

Customising the display in journalctl lets users search through logs more precisely and extract information swiftly. Users can tailor the display to show log data over a specific period or in real time to facilitate rapid identification of system errors and issues.

Shortening or expanding the output

You can customise how journalctl displays data by shortening or expanding the output. By default, journalctl displays the entire entry in the pager and runs them on the right-hand side of the screen. The output can be shortened with the --no-full option:

journalctl --no-full

You can expand the display using the -a flag:

journalctl -a

Set journalctl to standard output

In journalctl, log output is by default displayed using a pager like less. This lets users view output incrementally, allowing them to navigate through lengthy log files more easily. However, there are occasions where displaying the standard output of logs is necessary. Here’s how to do it:

journalctl --no-pager

Set output formats

journalctl also offers options for customising the output format of logs. To do this, you can use the -o option with the respective format identifier. For instance, to output log entries in JSON format, enter the following code:

journalctl -b -u nginx -o json

Here’s the output:

{ "__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635", "__REALTIME_TIMESTAMP" : "1422990364739502", "__MONOTONIC_TIMESTAMP" : "27200938", "_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "3fffffffff", "_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee", "_HOSTNAME" : "desktop", "SYSLOG_FACILITY" : "3", "CODE_FILE" : "src/core/unit.c", "CODE_LINE" : "1402", "CODE_FUNCTION" :

The following formats can be used in Journalctl:

• cat: Displays only the message field
• export: Binary format suitable for transferring or saving
• json: Standard JSON with one entry per line
• json-pretty: JSON formatted for better readabilit
• json-sse: Wrapped JSON formatted output that allows adding events sent by the server
• short: Standard syslog style output
• short-iso: Standard format for displaying ISO-8601 wallclock timestamps
• short-monotonic: Standard format with monotonic timestamps
• short-precise: Standard format with microsecond precision
• verbose: Displays every journal field available for the respective entry

How does journalctl conduct active process monitoring?

During active process monitoring with journalctl, the command-line program tail is used to track logs in real time and display the most recent entries. This makes it easier to monitor system events in real time and react to problems quickly.

How to display current logs

The -n option can be used to display a specific number of data records. It works in exactly the same way as tail -n. To display the last 10 entries, use the following command:

journalctl -n

You can set the number of entries as well, e.g. to 20:

journalctl -n 20