Detection
SSH alerts that were generated but invisible
Ubuntu logged every SSH attempt, but nothing reached the Wazuh dashboard. I traced the log through each hop of the SIEM pipeline and fixed four stacked causes.
Date: 14 and 20 October 2026
Systems: Ubuntu DMZ server (agent ubuntu-dmz), Wazuh 4.14.8 server
Outcome: Detection working end to end. Four stacked causes were found and fixed.
Summary
I simulated SSH login attempts with an invalid user from Kali (10.0.1.101) against the DMZ server. Ubuntu logged every attempt, but no alert appeared in the Wazuh dashboard. I traced the log through every hop of the SIEM pipeline. The final root cause was the SIEM server's clock running 1 hour ahead, which stamped every alert in the future and hid it from time-based searches.
Attack simulation
# From Kali, repeated 3 times
ssh fakeuser@10.0.2.10Expected detection: Wazuh rule 5710, "sshd: Attempt to login using a non-existent user". Level 5, MITRE T1110.001 Password Guessing (tactic: Credential Access).
The pipeline I investigated
auth.log → agent collects → agent sends (TCP 1514) → manager rules → alerts.json
→ Filebeat → indexer (daily index) → dashboard searchInvestigation timeline
| # | Hypothesis | Test | Result |
|---|---|---|---|
| 1 | Ubuntu 26.04 logs as sshd-session, so Wazuh's SSH decoder doesn't match |
Pasted the raw log line into Wazuh's Ruleset Test | Disproved. The sshd decoder extracted srcip and srcuser, and rule 5710 fired. No custom rule needed. |
| 2 | The agent isn't reading the log file | grep "auth.log" /var/ossec/logs/ossec.log |
At the time: believed confirmed, since the agent monitored journald only, so I added auth.log. Later disproved (22 Oct): journald was already delivering SSH events. See the correction below. |
| 3 | The dashboard can't reach the manager ("Check API connection") | systemctl is-active wazuh-manager |
Confirmed. The manager had crashed at boot. Restarted it and verified the API was listening on port 55000. |
| 4 | The agent isn't connected | Read wazuh-agentd.state |
Ruled out: status='connected', recent last_ack. |
| 5 | The manager isn't generating alerts | grep fakeuser /var/ossec/logs/alerts/alerts.log and alerts.json |
Ruled out: alerts present (65 matching lines in alerts.json). |
| 6 | Filebeat can't ship to the indexer | filebeat test output |
Ruled out: every stage OK. |
| 7 | The indexer isn't storing alerts | Dev Tools: GET _cat/indices/wazuh-alerts-*?v |
Ruled out: today's index was green with 258 documents. |
| 8 | Alerts are stored under the wrong time | timedatectl on the Wazuh server |
Root cause. The clock read 03:36 UTC when real UTC was 02:36. Every alert was stamped 1 hour in the future, so "Last 1 hour" never included them. |
Root causes
Collection gapWithdrawn. Journald was already collecting SSH events (see the correction below).- Service failure: wazuh-manager failed at boot.
- Clock skew (main cause): the Wazuh VM clock was exactly 1 hour ahead. Windows keeps the hardware clock in local time (Lagos, UTC+1), and the Linux VM read it as UTC.
- Why it never self-corrected: NTP (UDP 123) is blocked by my own MGMT firewall rules, so the VM couldn't sync time.
Fixes
# Agent (Ubuntu): also collect auth.log
<localfile>
<log_format>syslog</log_format>
<location>/var/log/auth.log</location>
</localfile>
# Wazuh server: correct the clock, restart the manager
sudo date -s "1 hour ago"
sudo systemctl restart wazuh-managerVerification: new attempts from Kali appeared as rule 5710 alerts with data.srcip: 10.0.1.101 and data.srcuser: fakeuser.
Lessons learned
- Test hypotheses before changing anything. My first theory (a decoder gap) was wrong, and one Ruleset Test saved me from writing an unnecessary custom rule.
- Trace the pipeline hop by hop. Each check either cleared a hop or found the fault.
- Time is a security control. Skewed clocks break correlation, time-window rules and evidence. Kerberos fails beyond 5 minutes of skew, and PCI DSS 10.6 requires time sync.
- UTC is not local time. The clock "matched my phone", and that was exactly the problem.
- Read the prompt. Twice I ran server commands on the wrong machine.
user@hostnametells you where you are.
Follow-up actions
- Make pfSense the lab's NTP server; Wazuh now syncs from it (Phase 4.1). DMZ server still to do.
- Remove duplicate alerts: the auth.log source was removed, so each SSH attempt now gives one alert from journald
- Make wazuh-manager wait for the indexer at boot
Correction (22 October 2026)
Re-reading the evidence overturned root cause 1. On 20 October, alerts.log already contained SSH alerts in the journald format (Oct 20 02:28:48 ...) as well as the auth.log format (2026-10-20T02:28:48.658051+00:00 ...). So journald was collecting SSH events all along, and adding auth.log only created a duplicate alert for every attempt.
Proof: one SSH attempt (sshd-session[4027], source port 36664) appeared twice, with location: journald and location: /var/log/auth.log. After removing the auth.log <localfile> (with a backup of ossec.conf first), a new attempt on source port 46654 produced exactly 1 hit, from journald.
Lessons:
- Correct a report when new evidence appears, rather than defending the first conclusion.
- Deduplicate on a fingerprint that identifies one event: the process ID plus the source port.
- Duplicates matter. A rule counting 8 failures in 2 minutes would have fired after only 4 real attempts.
Evidence
| Screenshot | Shows |
|---|---|
| Ruleset Test | Rule 5710 matches the sshd-session line |
| Agent connected + 7-day events | Agent connected; no events today |
| Filebeat test | Filebeat reaches the indexer |
| Indexer indices | Today's index holds 258 alerts |
| Clock 1 hour ahead | timedatectl reading 03:36 UTC at 02:36 real UTC |
| Rule 5710 | Rule definition, compliance and MITRE mapping |