The short answer

The unsafe design gave a probabilistic component broad write authority without independent interlocks or bounded actions.

Case status

This is a composite architecture safety review, not an incident report and not a claim that a named facility deployed this design. It illustrates a class of unsafe authority collapse: a probabilistic agent receives credentials that can write arbitrary PLC values. No injury, outage, or performance number is asserted.

Background and goal

An industrial operations team wants an agent to reduce diagnostic time. The intended workflow is reasonable: collect alarms and recent process data, find the applicable procedure, explain likely causes, and help the operator coordinate a response.

During prototyping, the team adds a generic PLC tool so the demo can both diagnose and “fix” a condition. Natural-language instructions tell the model to request confirmation before important changes.

Constraints

The process has operating limits, equipment modes, interlocks, maintenance states, and safety functions distributed across control systems. Telemetry may be stale or incomplete. The language model is non-deterministic and can misunderstand context or tool descriptions. Operators may approve quickly under pressure.

The existing PLC credential was created for engineering maintenance and can access far more tags than the agent needs. Network connectivity and the model service are not safety-rated.

Original approach

The agent retrieves alarms and manuals, then calls a tool with a tag address and value. The tool forwards the write using the broad maintenance credential. A prompt says to stay within safe limits and ask a human for approval. Logs record the conversation and final write.

The architecture combines observation, diagnosis, authorization, and execution inside one model-mediated flow. Safety rules exist as prose context rather than deterministic enforcement.

What goes wrong

Natural language is not a complete representation of process state or safety policy. The model may select the wrong asset, use stale evidence, confuse engineering units, or choose a plausible but unsafe setpoint. The generic tool accepts parameters outside the intended procedure.

Human approval does not repair the design. The interface may omit critical context, the reviewer may assume the tool already enforces limits, and repeated low-quality confirmations create habituation. Conversation logs help investigation after an event but do not prevent the event.

Most importantly, the credential and tool expose a larger action space than the task requires. A prompt injection, account compromise, model error, or integration bug reaches the same powerful write path.

Decision

Keep the agent read-only for diagnosis until evidence quality and operator use are measured. Replace arbitrary tag writes with a small catalog of typed operational procedures. Each procedure defines eligible assets, preconditions, parameter bounds, required process state, authorization, expiry, verification, and recovery.

Enforce those rules outside the model. High-impact actions require the appropriate human role and remain subject to PLC or independent safety-system interlocks. The agent cannot override an interlock or emergency path.

Revised architecture

An asset service resolves durable identity and current configuration. A telemetry service returns timestamped values with quality. The agent can retrieve approved procedures and propose one with cited evidence.

A policy service evaluates the authenticated requester, asset, procedure, process state, and risk. An execution service exposes only allowlisted commands with typed parameters. Local control validates bounds and interlocks again. The workflow records approval, dispatch, device acknowledgement, and physical outcome as separate states.

If evidence is stale, systems disagree, policy is unavailable, or verification fails, the workflow stops and escalates. A deterministic emergency and manual-control path remains available without the agent.

Trade-offs

The revised design delivers less spectacular autonomy. Building procedure-specific tools and policy takes more engineering than a generic write API. Operators still need training and ownership.

In return, errors are contained to a reviewed action space, tests can cover every procedure, and the organization can expand authority gradually. Read-only value arrives before control risk.

Result to validate

Success is demonstrated by safe workflow evidence, not an invented productivity claim. The system should show that the agent cites current inputs, recommends an eligible procedure, policy rejects unauthorized or out-of-range requests, local interlocks remain effective, and the recorded outcome distinguishes request, acceptance, execution, and physical verification.

The review should also demonstrate that revoking the agent or its service credential immediately removes execution capability without disabling independent manual and emergency control. That revocation path must be tested.

Reusable lessons

Separate observation, recommendation, approval, execution, and verification. Give tools the least authority needed. Enforce policy deterministically. Keep independent safety controls. Treat stale or conflicting evidence as a stop condition. Test negative cases and degraded operation.

What not to copy blindly

Do not assume every industrial action needs human approval; some deterministic automation is safer than manual work. Do not assume approval alone makes a model-selected action safe. Do not transfer procedure bounds from one machine to another. And do not present this composite as evidence that agents have caused a documented PLC incident.