The short answer

The fleet did not exceed average capacity; correlated recovery work exceeded several independent rate limits at once.

Case status

This is a composite failure-mode review, not a benchmark, customer incident, or claim that a particular fleet contained one million devices. The slug names the scale problem engineers commonly search for. No throughput, duration, or recovery number is asserted because those values depend on broker, authentication, session, subscription, payload, network, and downstream architecture.

Background and goal

An IoT platform normally carries a large connected fleet with acceptable latency. The engineering goal is to preserve safe, bounded recovery after a regional network interruption or platform event causes many clients to reconnect in a correlated window.

Steady-state capacity appears comfortable. Average connection churn is low, message throughput stays below planned limits, and broker nodes have headroom. The missing question is how the system behaves when recovery work arrives together.

Constraints

Devices run different firmware and reconnect algorithms. Some retain sessions and expect queued messages. Authentication depends on certificate validation and an external identity service. Connection status is written to a database and consumed by other services. Cellular and site networks recover in waves rather than uniformly.

The fleet cannot be updated instantly to a new backoff policy. Product semantics do not allow simply dropping every persistent session or offline message.

Original approach

Clients reconnect immediately after transport loss, with a short fixed retry delay. A load balancer spreads TCP connections across broker nodes. The broker restores sessions, subscriptions, and inflight state, then downstream services record presence changes. Capacity planning multiplies average message rate by device count and adds broker nodes for headroom.

The test plan measures steady-state connections and publishes. It does not model TLS handshakes, certificate checks, session lookup, subscription restoration, retained delivery, queued messages, presence writes, and consumer restart together.

What goes wrong

Correlated recovery creates several workloads at once. TLS and authentication consume CPU and external-service capacity before ordinary MQTT traffic stabilizes. Session restoration reads stored state and rebuilds subscriptions. Clients publish boot, status, and buffered telemetry. Subscribers resume and may request retained or queued data. Presence services write a disconnect followed by a reconnect for many principals.

Adding broker nodes may move the bottleneck to authentication, load balancing, DNS, session storage, or the database. Aggressive clients time out and retry while the first attempt is still being processed, amplifying load. A platform can remain below its average message target while recovery latency grows and queues become stale.

Decision

Treat recovery rate as a first-class capacity dimension. Require exponential backoff with randomized jitter in the client contract. Apply server-side admission control so the platform rejects or delays work predictably rather than failing every dependency at once.

Separate stages of recovery: establish and authenticate connections, restore essential session state, resume critical subscriptions, and drain non-urgent backlog under quotas. Define which messages expire instead of replaying them after their business value has passed.

Revised architecture

Regional endpoints and failure domains limit correlated scope. Load balancers and brokers expose connection-attempt, handshake, acceptance, rejection, and establishment metrics. Authentication services have caches and explicit protection limits without bypassing revocation requirements.

Session storage is partitioned and observable. Broker policy caps inflight work, offline queue size, expiry, and per-principal rates. Downstream presence processing coalesces transitions and avoids one synchronous database dependency in the connection path.

Clients use randomized backoff, cap retries, and distinguish authentication rejection from transient unavailability. Critical device behavior degrades locally instead of depending on an immediate cloud reconnect.

Trade-offs

Slower admission extends the time until the entire fleet is online, but protects useful partial service and downstream systems. Queue expiry may discard data, so product teams must identify what can be replaced and what requires durable delivery. Regional isolation adds operating cost and makes cross-region state more complex.

Reducing persistent-session use can lower recovery state, but changes offline behavior and must not be treated as a free infrastructure optimization.

Result to validate

The required outcome is bounded, observable recovery under a production-shaped test. The team should know the accepted connection rate, rejected or delayed work, queue age, dependency saturation, time to restore critical service, and time to drain non-critical backlog. No universal target applies; each product defines objectives from its operational need.

The test should include clients that never receive the first response and retry, because ideal clients can hide the amplification the production network will create.

Reusable lessons

Benchmark correlated failure, not only steady state. Put jitter in clients and admission control on servers. Measure authentication, session, subscriptions, and downstream writes separately. Expire stale work. Practice recovery with realistic firmware and network waves.

What not to copy blindly

Do not choose a backoff range, queue limit, or regional design from this composite. Do not disable authentication or certificate checks to make a load test pass. Do not add brokers before locating the actual bottleneck. And do not equate “all sockets connected” with recovered service; verify message flow and business health.