Security Monitoring and Dashboards

Where Log Data Is Lost Between the Source and the Index

A source that never appears, or that stops appearing entirely, shows up in an inventory check. The losses that are harder to find happen after the source is configured correctly and sending: they are partial, intermittent, and leave the source looking healthy while the data behind an investigation is quietly incomplete.

The diagnostic that separates these from a genuine absence of activity is a synthetic event. Emit a known record from each source class on a fixed interval and alert when it does not turn up as searchable in the index within an expected time. Volume-based health checks miss the parsing and quota cases entirely, because in those cases the events do arrive; only confirming that a specific expected record can be found catches them.

• Transport with no acknowledgment. Classic syslog over UDP on port 514 has no delivery confirmation and no retransmission, so a dropped datagram produces no error at either end. Syslog over TCP, or over TLS on 6514, gives you a connection that fails visibly, which is the behavior you want.
• Agent buffers that wrap. Most forwarders spool to local disk when the destination is unreachable and then overwrite the oldest entries once the spool is full. An outage longer than the buffer produces a hole covering exactly the period you most want to read. Buffer size and the queue-full behavior, block or drop, are both configurable and both worth setting deliberately rather than inheriting.
• Quota and rate limits. Ingest licenses and cloud API limits enforce a ceiling per day or per second. When it is reached, events are throttled or discarded, and the notice usually lands in the platform's own internal log rather than in anyone's alert path.
• Parsing failures after a format change. A software update changes a log line, the event still arrives, but it lands unparsed with no fields extracted. It is present in a raw text search and absent from every rule and dashboard built on those fields.
• Multi-line events split into fragments. Stack traces and Windows event XML span many lines. A collector configured for one line per event turns a single record into twenty pieces, none of which matches a rule.
• Certificates expiring on the collector. Forwarders stop shipping the moment the TLS handshake fails. Whether that is loud or silent depends entirely on whether anything is watching forwarder connection counts.

Two Timestamps, and Why Both Belong on Every Record

Every record should carry two times: the time the source says the event occurred, and the time the platform received it. Most platforms store both, but rules and dashboards are often built on whichever one the search language defaults to, which is not always the one the question needs. Search on event time when reconstructing what happened. Search on ingest time when checking whether collection is working, because a source that stopped sending an hour ago still has yesterday's events sitting at yesterday's event time, and a coverage check built on event time will call that healthy.

The gap between the two is a signal in its own right. A steady lag of a few seconds is normal. A lag that jumps to hours points at a queue backing up or a buffer draining after an outage. A negative lag, events arriving stamped in the future, points at a host whose clock is ahead. Alerting on the distribution of that gap per source catches problems that no volume check will see.

Format is part of the problem. The older BSD syslog format described in RFC 3164 carries a timestamp of the form Mmm dd hh:mm:ss with no year and no time zone, so the collector supplies both by assumption; an event forwarded across a zone boundary, or held in a buffer across a year end, is placed wrongly and nothing reports an error. RFC 5424 replaced that with a full RFC 3339 timestamp including offset and optional fractional seconds. Where the source lets you choose, choose that, store UTC, and render local time in the interface rather than baking it into the data.

Drift also breaks correlation silently. A rule that looks for a failed authentication followed by a success within thirty seconds returns nothing when the two sources are a minute apart, and an empty result looks exactly like no relationship. Before writing any time-bounded correlation, measure the real spread between the sources involved rather than assuming the window.

Detection Rules Belong in Version Control

Because tuning never finishes, rules need the handling that any other continuously edited logic gets. A rule edited in a console has no history, no reviewer and no author. Six months later nobody can say why a condition excludes one subnet, so it stays, and the exclusion outlives the reason for it. Keeping rules as text in a repository, changed through review and deployed from that repository rather than by hand, is what makes the tuning record readable later.

Writing rules in a portable format such as Sigma and converting them to the platform's query language keeps the investment separable from the platform. It also makes test fixtures possible: each rule ships with a handful of sample events that must trigger it and near-miss events that must not, and continuous integration runs them on every change, so a regression is caught before deployment instead of during an incident.

• Response steps in the same file as the query. Held in a separate wiki, the query and the runbook drift apart, and the one found at three in the morning is the stale one.
• Two delivery tiers. Every rule still carries its documented response; what differs is whether it pages a person immediately or lands in a queue worked through each shift. Separating delivery urgency from the response requirement keeps detections that are real but not worth waking someone for out of the paging path, without letting anything fire that nobody knows how to handle.
• A named owner, a team rather than an individual. A rule whose author has left is a rule whose alerts have no destination.
• The data the rule depends on, listed explicitly: which sources and which fields. A pipeline change then breaks the rule visibly instead of quietly turning it into a rule that can never fire.
• A review date. Rules written for a specific migration, a specific vulnerability or a specific vendor incident should be reconsidered and removed rather than accumulating.

Making Identity Consistent Across Systems

The fix for the same person appearing under several identifiers is to pick one canonical identifier and map to it during ingest, not at query time. The identity provider's immutable subject identifier is usually the right anchor, because email addresses change with name changes and usernames get reassigned. Doing the mapping in the pipeline costs complexity there, but it means every rule, dashboard and investigation downstream sees one field instead of each analyst reinventing the join.

Some sources cannot be mapped that way, because the identity simply is not in the record. Those need a deliberate decision rather than a best effort:

• AWS role assumption. CloudTrail attributes an action to the assumed role session, not the person behind it. If everyone assumes the same role and no session name is set, every event names the same actor. Passing a per-user role session name, or sourceIdentity, at AssumeRole time is what preserves attribution downstream.
• Kubernetes. The audit log records the service account or certificate subject presented to the API server. When access runs through a shared kubeconfig or a proxy, the human identity exists only in the proxy's log, and the two have to be joined on time and source address.
• Databases behind connection pools. The application connects with one account for every user it serves. Without an application-level audit event carrying the end user, the database log can only tell you that the application did something.
• Shared and machine accounts. Service accounts, CI runners and API keys are identities too, and each needs a named human or team recorded as owner. Without that mapping there is no basis for judging whether an action should have happened.

When Monitoring Is Not the Next Thing to Build

A monitoring build assumes several things already exist, and when they do not, the honest sequencing is to fix those first. You cannot measure coverage gaps without an asset inventory, because gaps are a fraction and you have no denominator. Alert routing has nowhere to route to without an on-call arrangement, and a rota that exists on paper but has never been tested is the same thing as no rota. Neither is a tooling problem and neither is solved by ingesting more data.

The third prerequisite is the application itself. If your software emits only HTTP status codes and stack traces, no amount of pipeline work will produce security-relevant events, because the events were never generated. Authentication outcomes with reason codes, authorization denials, privilege and role changes, credential and MFA changes, export volumes and any crossing of a tenant boundary all have to be added in the code. That is application engineering work, which is why it sits with secure development rather than with collection.

There is also a timing question worth naming. If what prompted the conversation is something that already happened and you need to know what it was, that is investigative work on the evidence that exists today. Standing up a monitoring program does not answer it, takes months, and competes for the same people. The monitoring build is what stops the next one being investigated with the same limited evidence.