← All posts

Detection

SSH alerts that were generated but invisible

Ubuntu logged every SSH attempt, but nothing reached the Wazuh dashboard. I traced the log through each hop of the SIEM pipeline and fixed four stacked causes.

Date: 14 and 20 October 2026
Systems: Ubuntu DMZ server (agent ubuntu-dmz), Wazuh 4.14.8 server
Outcome: Detection working end to end. Four stacked causes were found and fixed.

Summary

I simulated SSH login attempts with an invalid user from Kali (10.0.1.101) against the DMZ server. Ubuntu logged every attempt, but no alert appeared in the Wazuh dashboard. I traced the log through every hop of the SIEM pipeline. The final root cause was the SIEM server's clock running 1 hour ahead, which stamped every alert in the future and hid it from time-based searches.

Attack simulation

# From Kali, repeated 3 times
ssh fakeuser@10.0.2.10

Expected detection: Wazuh rule 5710, "sshd: Attempt to login using a non-existent user". Level 5, MITRE T1110.001 Password Guessing (tactic: Credential Access).

The pipeline I investigated

auth.log → agent collects → agent sends (TCP 1514) → manager rules → alerts.json
        → Filebeat → indexer (daily index) → dashboard search

Investigation timeline

# Hypothesis Test Result
1 Ubuntu 26.04 logs as sshd-session, so Wazuh's SSH decoder doesn't match Pasted the raw log line into Wazuh's Ruleset Test Disproved. The sshd decoder extracted srcip and srcuser, and rule 5710 fired. No custom rule needed.
2 The agent isn't reading the log file grep "auth.log" /var/ossec/logs/ossec.log At the time: believed confirmed, since the agent monitored journald only, so I added auth.log. Later disproved (22 Oct): journald was already delivering SSH events. See the correction below.
3 The dashboard can't reach the manager ("Check API connection") systemctl is-active wazuh-manager Confirmed. The manager had crashed at boot. Restarted it and verified the API was listening on port 55000.
4 The agent isn't connected Read wazuh-agentd.state Ruled out: status='connected', recent last_ack.
5 The manager isn't generating alerts grep fakeuser /var/ossec/logs/alerts/alerts.log and alerts.json Ruled out: alerts present (65 matching lines in alerts.json).
6 Filebeat can't ship to the indexer filebeat test output Ruled out: every stage OK.
7 The indexer isn't storing alerts Dev Tools: GET _cat/indices/wazuh-alerts-*?v Ruled out: today's index was green with 258 documents.
8 Alerts are stored under the wrong time timedatectl on the Wazuh server Root cause. The clock read 03:36 UTC when real UTC was 02:36. Every alert was stamped 1 hour in the future, so "Last 1 hour" never included them.

Root causes

  1. Collection gap Withdrawn. Journald was already collecting SSH events (see the correction below).
  2. Service failure: wazuh-manager failed at boot.
  3. Clock skew (main cause): the Wazuh VM clock was exactly 1 hour ahead. Windows keeps the hardware clock in local time (Lagos, UTC+1), and the Linux VM read it as UTC.
  4. Why it never self-corrected: NTP (UDP 123) is blocked by my own MGMT firewall rules, so the VM couldn't sync time.

Fixes

# Agent (Ubuntu): also collect auth.log
<localfile>
  <log_format>syslog</log_format>
  <location>/var/log/auth.log</location>
</localfile>

# Wazuh server: correct the clock, restart the manager
sudo date -s "1 hour ago"
sudo systemctl restart wazuh-manager

Verification: new attempts from Kali appeared as rule 5710 alerts with data.srcip: 10.0.1.101 and data.srcuser: fakeuser.

Lessons learned

  • Test hypotheses before changing anything. My first theory (a decoder gap) was wrong, and one Ruleset Test saved me from writing an unnecessary custom rule.
  • Trace the pipeline hop by hop. Each check either cleared a hop or found the fault.
  • Time is a security control. Skewed clocks break correlation, time-window rules and evidence. Kerberos fails beyond 5 minutes of skew, and PCI DSS 10.6 requires time sync.
  • UTC is not local time. The clock "matched my phone", and that was exactly the problem.
  • Read the prompt. Twice I ran server commands on the wrong machine. user@hostname tells you where you are.

Follow-up actions

  • Make pfSense the lab's NTP server; Wazuh now syncs from it (Phase 4.1). DMZ server still to do.
  • Remove duplicate alerts: the auth.log source was removed, so each SSH attempt now gives one alert from journald
  • Make wazuh-manager wait for the indexer at boot

Correction (22 October 2026)

Re-reading the evidence overturned root cause 1. On 20 October, alerts.log already contained SSH alerts in the journald format (Oct 20 02:28:48 ...) as well as the auth.log format (2026-10-20T02:28:48.658051+00:00 ...). So journald was collecting SSH events all along, and adding auth.log only created a duplicate alert for every attempt.

Proof: one SSH attempt (sshd-session[4027], source port 36664) appeared twice, with location: journald and location: /var/log/auth.log. After removing the auth.log <localfile> (with a backup of ossec.conf first), a new attempt on source port 46654 produced exactly 1 hit, from journald.

Lessons:

  • Correct a report when new evidence appears, rather than defending the first conclusion.
  • Deduplicate on a fingerprint that identifies one event: the process ID plus the source port.
  • Duplicates matter. A rule counting 8 failures in 2 minutes would have fired after only 4 real attempts.

Evidence

Screenshot Shows
Ruleset Test Rule 5710 matches the sshd-session line
Agent connected + 7-day events Agent connected; no events today
Filebeat test Filebeat reaches the indexer
Indexer indices Today's index holds 258 alerts
Clock 1 hour ahead timedatectl reading 03:36 UTC at 02:36 real UTC
Rule 5710 Rule definition, compliance and MITRE mapping