Direct answer

Treat offline as an expected device state, not an exception to hide. Define how absence is detected, how confidence decays, which data or commands may queue, when queued work expires, and how recovery is paced. Show operators the difference between disconnected, late, unreachable, powered down, and unknown. A safe system limits backlog and reconciles current state after reconnect instead of replaying everything blindly.

Scope and non-scope

This guide covers connectivity loss between devices, gateways, brokers, and platform services. It does not diagnose the underlying radio, power, firmware, or account problem in every case. “Offline” is an observation made by a particular layer; it is not proof that the physical asset stopped operating.

Define offline precisely

Use more than one signal where consequences matter. MQTT keepalive and session disconnect indicate transport state, but an unclean network path may take time to detect. Application heartbeats show that firmware is scheduling work. Telemetry freshness shows whether a specific measurement stream is current. Gateway inventory can distinguish a field device failure from gateway uplink loss.

Choose thresholds from expected reporting cadence and network variance. A device that reports hourly should not be declared offline after two minutes. Represent states such as online, delayed, offline, intentionally sleeping, maintenance, retired, and unknown. Include the observation timestamp and source so users can judge confidence.

Queue policy by message type

Do not apply one offline queue to every message. Replaceable telemetry can be sampled, aggregated, or discarded when storage is full. Durable events may require local persistence with sequence numbers. Desired state can often collapse to the newest version. Commands need explicit expiry, idempotency, preconditions, and an acknowledgement; a stale command must not execute merely because connectivity returned.

Bound queues by bytes, count, and age. Define what is dropped first and expose the loss. On the broker, align persistent-session expiry and message expiry with product expectations. On the device or gateway, protect flash endurance and reserve capacity for critical events.

Recovery sequence

  1. Reconnect with randomized exponential backoff and a server-directed ceiling where supported.
  2. Authenticate and restore only the session state that remains useful.
  3. Report boot, firmware, clock, and current state before processing old work.
  4. Reconcile desired and reported versions.
  5. Drain durable events at a controlled rate with original timestamps and identifiers.
  6. Reject expired commands and acknowledge their terminal status.
  7. Resume normal telemetry gradually so recovery traffic does not starve control or authentication.
  8. Mark the device healthy only after required streams and outcomes recover, not at socket connection.

Operator experience

Show last successful contact, expected cadence, affected gateway or region, queued work, and current recovery phase. Avoid a binary red icon that creates thousands of identical tickets during a shared outage. Group correlated failures and identify the likely common dependency. Ownership and escalation should depend on impact: an offline demo sensor and an unreachable safety monitor are not the same incident.

Failure modes

A fixed reconnect interval synchronizes a fleet against the broker. An unlimited broker queue turns an outage into a storage incident. Replaying old commands can reverse a more recent local decision. Accepting a new connection as “healthy” hides failed subscriptions or downstream processing. Device clocks may jump after a long outage, making event order unreliable unless sequence and receive time are preserved. A gateway can buffer data successfully while the platform incorrectly labels every child device offline.

Implementation checklist

  • Offline states, signals, thresholds, and confidence are documented.
  • Sleeping, maintenance, retired, and unknown devices are distinct.
  • Queue limits and drop order are defined per message class.
  • Commands have IDs, preconditions, expiry, and outcome acknowledgement.
  • Reconnect uses jitter and respects admission control.
  • Recovery sends current state before obsolete backlog.
  • Operators can see common-cause scope and recovery progress.
  • Tests cover broker, network, gateway, device, clock, and storage failure.

Evidence to retain

Keep traces for clean disconnect, silent network loss, broker restart, gateway outage, long sleep, queue overflow, and expired command recovery. Record actual detection delay, backlog age, reconnect distribution, and time until application health returns. Re-run the scenarios when reporting cadence, session policy, or broker limits change. A dashboard screenshot is not sufficient evidence if it cannot show which signal produced the state.

Service-level decisions

State the product promise in user terms: how quickly absence is detected, what functionality continues locally, which data survives, and when operators are notified. Tie internal thresholds to that promise rather than copying keepalive defaults. Critical devices may need an independent path or local alarm; cloud detection cannot compensate for a disconnected safety channel.

Review storage and recovery together. A longer offline retention period increases flash use, broker disk, drain time, and the window in which old context can mislead consumers. Document the chosen trade-off per device class. When changing queue or session limits, test the oldest supported firmware as well as the newest. Legacy clients often have different retry, timestamp, and acknowledgement behavior and can dominate an incident even when they are a minority of the fleet.

Primary sources

The OASIS MQTT 5.0 specification defines keepalive, sessions, wills, message expiry, and session expiry. For constrained-device lifecycle and state expectations, use NISTIR 8259A. Broker and client persistence behavior must be verified against their official documentation and failure-tested.