The short answer

Events describe what happened; alerts demand attention. Promote only events with actionable severity, ownership, and response expectations.

What matters in practice

  • Deduplication keys define incident grouping.
  • Acknowledgement is not resolution.
  • Escalation needs clocks and ownership.
  1. Separate detection from notification.
  2. Attach evidence and recommended next checks.
  3. Capture resolution codes for learning.

Boundaries and failure modes

  • Sending one message per threshold sample.
  • Using severity without response meaning.
  • Closing alerts when notifications are delivered.

Scope and model boundary

Events are immutable evidence that something happened; alerts are stateful decisions that the evidence requires attention. Keep event source, type, occurrence and receive time, entity, schema, quality, and correlation IDs. An alert needs severity, deduplication identity, owner, response expectation, current state, triggering evidence, and a closure condition tied to a verified operational outcome. Safety shutdowns and local interlocks remain independent protections.

Promotion and correlation criteria

Promote only conditions with an action that can improve an outcome. Define the harm of inaction, responsible team, response time, incident identity, and proof of recovery. Severity reflects impact and urgency, not novelty. One redundant sensor and a common gateway failure require different cases even when their event type matches.

Choose stable deduplication keys from affected service or asset, condition, and scope. Repeated observations update one alert. Correlate child symptoms to a gateway, region, broker, or identity dependency only when topology supports it. Preserve every source event. Add hysteresis, minimum duration, and separate trigger and clear rules for noisy measurements. Maintenance suppression needs scope, owner, start, expiry, and audit; it must not delete evidence.

Operational workflow

  1. Validate event freshness and quality.
  2. Enrich it with ownership, topology, maintenance, and recent changes.
  3. Apply deduplication and correlation.
  4. Create or update the alert with severity and response target.
  5. Notify the supported owner channel and record delivery.
  6. Separate acknowledgement from mitigation and resolution.
  7. Execute a runbook or authorized automated action.
  8. Verify the physical or business outcome independently.
  9. Close with cause, action, evidence, and follow-up.
  10. Reopen if the condition returns inside the defined window.

Failure modes

Alerting every threshold creates fatigue. Deduplicating only by device merges unrelated faults. Auto-closing when telemetry disappears marks severe outages resolved. Clearing on one good sample causes flapping. A chat notification does not prove ownership. A command acknowledgement does not prove recovery. Automated remediation without recorded authority and outcome breaks auditability.

Implementation checklist

  • Events are immutable, versioned, time-qualified evidence.
  • Alert promotion requires an action, owner, and outcome.
  • Deduplication and correlation rules are tested.
  • Trigger, clear, reopen, and suppression behavior is explicit.
  • Severity reflects current impact and urgency.
  • Acknowledgement, mitigation, resolution, and closure differ.
  • Dashboards expose age, repeats, ownership, and unresolved impact.

Policy governance and test evidence

Version the promotion policy, deduplication key, correlation rules, severity mapping, and notification routing. An alert should retain the exact policy version that created it so a later rule change does not erase the explanation. Policy deployment needs a preview against representative historical events: show which alerts would open, merge, change severity, or disappear. Historical replay is useful evidence, but it cannot predict a new failure mode, so retain a bounded canary and rollback.

Test a noisy sensor, a delayed event, an out-of-order clear, one device behind a failed gateway, a platform-wide dependency outage, planned maintenance, and recurrence after closure. Confirm that source events remain queryable and that correlation can be reversed when its hypothesis is wrong. Verify escalation when the named owner does not acknowledge. Test notification failure separately from alert creation.

Measure events-to-alert ratio by rule, duplicate suppression, flapping, time unowned, acknowledgement time, time to verified outcome, reopen rate, stale open alerts, and suppressions beyond expiry. Do not optimize for fewer alerts in isolation; a quiet system that misses important work is worse. Review false positives and missed incidents with operators who performed the response. Their evidence should change thresholds, context, runbooks, ownership, or product design rather than merely adding another dashboard.

Define retention separately for events and alert workflow history. Raw high-volume observations may aggregate, while the evidence supporting consequential decisions needs a durable audit period. Protect personal or sensitive operational context through field minimization and access control. Exporting an alert without its source references, policy version, and timestamps creates a ticket archive, not trustworthy incident evidence.

Assign a periodic owner review for every alert rule. Rules without an active service owner, tested response, and current clear condition should be disabled or redesigned; leaving them active transfers undocumented risk to whoever happens to be on call.

Primary sources

The CloudEvents specification provides a portable event envelope. OpenTelemetry semantic conventions provide maintained telemetry vocabulary. NISTIR 8259A covers relevant device cybersecurity-state capabilities. Product severity and response policy must come from the real operating model.