News
What happens if the AI in the control room is attacked?
The lesson here is quite clear: Use your head and don't hand over all control to machines. AI needs clear guidelines and human oversight.
AI agents are no longer expected to merely answer questions, but to take action: create tickets, configure systems, and operate equipment. This shifts the focus of security concerns. It is no longer just about what an AI says, but about what it is allowed to do and who oversees it in cases of doubt.
To make this issue more tangible, we built a test station: a simulated reactor control room controlled via an AI chatbot. During normal operation, the warning light remains off. The demonstration shows how the AI, through a completely normal conversation—without any malicious intent on the part of the operator—brings about a critical plant condition, live and without any technical intervention in the system.
The interesting thing about this is that the attack doesn’t require a security vulnerability in the traditional sense. The AI does what it was built to do and processes instructions. The problem lies in whose instructions it follows and what permissions it has been granted. Anyone who grants an agent far-reaching powers without clearly separating authorization and approval is effectively shifting control of the system to the chat window.
For companies operating in regulated environments, this is not just a theoretical exercise. Wherever AI is integrated into production, laboratory, or control systems in the future, the same principles apply as for any other system: clear roles, transparent decisions, and the need for a human authority who has the final say in the event of an emergency.
“We didn’t want to talk about the risks of AI agents, but rather to show how quickly a harmless conversation can turn into a critical situation,” says Timo Becker, an AI expert at INOSOFT. “The lesson here is very clear: Use your head and don’t hand over all power to the machines. AI needs clear guidelines and human oversight.”
The full presentation is available to watch as WeeklyTalk #188:
The warning light is on: What happens if the AI in the control center is attacked?.
The devSec presentation and additional background information: The warning light is on.