Guardrails Before Autonomy: What Utility AI Needs Beyond a Good Model

Most conversations about AI in power utilities start with the model: how accurate it is, which vendor supplies it, how fast it can be deployed. Three recent developments point to a different question. The decisive piece of a utility AI system may be the layer around the model that decides what it is allowed to do, what it must show, and who must approve its actions.

A governance layer, measured

A preprint posted to arXiv on 18 September 2026 by Mylonas, Foti and Varvarigos, A Governance-Aware Large Language Model Orchestrated Agentic Digital Twin for Transmission System Operator Control Room Decision Support, puts numbers on this idea. In their design, a language model chooses and parameterises analysis tools for a digital twin of the Greek transmission network, but a governance layer that the model cannot bypass checks every proposed action. The layer enforces four rules: only whitelisted tools run, each run stays within a step budget, any action with side effects needs explicit operator approval, and every number in an answer is produced by the layer from backend results, with its unit, variable and time.

On a 118-task benchmark, the primary model reached 96.5% tool-selection accuracy and 93.7% task success across 590 runs, and all four rules held on every run. A wider study of four language models over 1,416 runs also saw the rules hold every time. The comparison without the layer is the striking part: the same model executed all 45 runs that required approval without any authorisation, and only 39.2% of its answers contained numbers supported by backend data. Enforcement added 12 to 16 milliseconds per request.

This is a research prototype on one network, so it shows what is possible rather than what utilities have deployed. Even so, it frames the key design choice: permissions and number-checking sit outside the model, where they can be tested.

Nuclear plants show the same pattern in practice

IEEE Spectrum reported on 8 October that AI assistants are spreading through the US nuclear fleet. According to the article, nearly all of the 94 US reactors have been offered AI integration and most have accepted, and one vendor, Nuclearn, reports more than 65 US partners. Typical tasks are document retrieval, preparing regulatory paperwork, finding how past problems were solved, and synthesising incident reports into guidance for components such as pumps.

The guardrails are as notable as the use cases. One vendor's products run at a single site so plant data never leaves the facility. Another assistant searches only databases that already require plant-level security clearances. Recommendations trace back to source documents, and in a pilot on weld inspection data, a human reviews every recommendation, with no plant letting AI take over inspections. The article notes that AI used for documentation and data search falls outside the strictest rules, which apply to systems that interact with plant control components. It gives no figures on time saved, so the benefit remains reported rather than measured.

The sequence matters: start with search and documents, where errors are recoverable and every answer can be checked against its source, and keep the control path separate.

India's cyber rules arrive without a word on AI

In India, the Central Electricity Authority notified its Cyber Security in Power Sector Regulations on 31 July 2026, and SolarQuarter's summary says mandatory provisions take full effect on 1 April 2027. They cover entities that own, operate or manage operational technology and connected IT systems in the power sector, including generating companies, captive plants and storage systems of 50 MW or more, as well as power exchanges, OTC platforms and technology vendors. Cyber incidents must be reported to CSIRT-Power within six hours, procurement must follow trusted-source requirements, and critical operational data must stay on systems in India, with encrypted sensitive data and backups also stored in India.

The summary mentions no AI-specific provisions. The rules still shape any AI deployment, because an assistant that reads grid telemetry, SCADA events or consumer data becomes part of the regulated estate. A model hosted outside India, or a vendor whose tooling cannot meet the incident-reporting window, would need to be checked against these obligations before go-live.

At the IEEE PES T&D Conference in June, utility leaders from Dominion Energy, Duquesne Light and Southern Company made a related point from the field: repeatable performance builds more confidence than perfect accuracy, explainability matters before crews are dispatched on an AI flag, and humans should validate outputs.

What it means for utilities

Four practical steps follow for utilities planning AI beyond pilots.

First, put permissions outside the model. List the tools an assistant may call, cap how many steps it can take, and require operator approval for anything that changes a system state. These rules should be enforced in code that can be tested, not left to prompts.

Second, make every answer checkable. Numbers should come from backend systems with units and timestamps, and recommendations should link to source documents, as the nuclear assistants do.

Third, keep the data boundary clear. Decide where models run and where operational data goes, and check that against the CEA data-localisation and incident-reporting provisions before procurement, not after.

Fourth, sequence use cases by consequence. Start with search, summarisation and documentation, move to decision support with human approval, and treat anything that touches control as a separate project with its own testing. Report how often the guardrails held, not only how accurate the model was.

Key takeaways

• A 2026 control-room study found that, without a governance layer, a model ran all 45 approval-requiring actions without authorisation; with it, all four rules held across 1,416 runs.
• US nuclear plants are adopting AI assistants for documents and search, with source traceability, data kept on site and humans reviewing recommendations.
• India's CEA cyber regulations take full effect on 1 April 2027 and, in the summary reviewed, say nothing specific on AI, but they apply to the systems AI will touch.
• Guardrails belong outside the model, and they should be measured as carefully as accuracy.
• Start with low-consequence uses and keep the control path separate.

Sources

• A Governance-Aware Large Language Model Orchestrated Agentic Digital Twin for Transmission System Operator Control Room Decision Support (arXiv)
• The U.S. Nuclear Plant Fleet Is Leaning Into AI (IEEE Spectrum)
• CEA Cyber Security Regulations 2026 Set New Power Sector Protection Standards (SolarQuarter)
• Utility Leaders Say Trust, Integration and Human Oversight Will Determine AI Adoption (T&D World)