Back to lessons

Hosting Operations

Summarize Journal Severity During an Incident

An alert fired and you need to know whether the recent journal is mostly warnings, errors, or critical failures.

Command

journalctl -p warning..alert --since "2 hours ago" --no-pager -o short-iso | awk '{count[$4]++} END {for (level in count) print count[level], level}' | sort -nr

What changed

Nothing changes. The command counts journal lines by severity.

Danger

safe

When to use it

Use at the start of incident triage when you need a fast severity profile.

When not to use it

Do not use severity counts as root cause; follow up by grouping by unit and reading the timeline.

Undo or recovery

No undo needed because the command is read-only.

Expected output

Counts followed by journal priority names.

demo script

Disposable terminal steps

  1. journalctl -p warning..alert --since "2 hours ago" --no-pager -o short-iso
  2. journalctl -p warning..alert --since "2 hours ago" --no-pager -o short-iso | awk '{count[$4]++} END {for (level in count) print count[level], level}' | sort -nr

simulated output

What it looks like

disposable vessel
::fixture-ready::
$ journalctl -p warning..alert --since "2 hours ago" --no-pager -o short-iso
2026-06-25T14:02:06+00:00 vps api[1842]: warning upstream_slow upstream=db latency_ms=2200
2026-06-25T14:03:08+00:00 vps api[1842]: err request_id=req-103 ERROR database timeout after 30000ms
2026-06-25T14:03:12+00:00 vps api[1842]: err request_id=req-103 ERROR retry failed upstream=db
2026-06-25T14:05:10+00:00 vps worker[2201]: crit FATAL job runner exited code=137
2026-06-25T14:05:12+00:00 vps systemd[1]: warning worker.service restarted
2026-06-25T14:06:33+00:00 vps api[1842]: err request_id=req-107 ERROR payment provider returned 500
::exit-code::0
$ journalctl -p warning..alert --since "2 hours ago" --no-pager -o short-iso | awk '{count[$4]++} END {for (level in count) print count[level], level}' | sort -nr
3 err
2 warning
1 crit
::exit-code::0

YouTube Short

Count journal severity first.

Before reading every incident line, count warning, error, and critical entries. It gives you the shape of the problem in one pass.

LinkedIn hook

Start with severity counts before opening every log line.

Question: When an alert fires, do you start with individual log lines or a severity summary?

experiments

A/B tests to run

Metric: save_rate

A: Count severity before reading logs.

B: Get the incident shape first.