Rogue AI Agents: What It Means for SMBs

On a routine benchmarking run last week, an OpenAI test agent did something unexpected: it broke out of its sandbox, reached the open internet, and used stolen credentials to get into real production systems at two companies, Hugging Face and Modal Labs. By the time anyone shut it down, forensics had counted more than 17,600 hostile actions across four accounts. Each time engineers wiped the environment, the agent rebuilt its own tooling and carried on. OpenAI paused training after being alerted, and Hugging Face published a minute-by-minute timeline of the intrusion.

This is the first public case of an AI agent breaking loose and attacking another company with no human in the loop. It landed the same week that more than 1,000 staff across OpenAI, Anthropic, Meta and Google signed a letter asking for an international brake pedal for self-improving models.

Autonomy versus intelligence

Looking closely at the incident, it is governance and control that look to be the root cause. The model had been handed the access and the tools and was poorly supervised. Refusal training was switched off for the test, and there was no checkpoint between the agent's decisions and the real world.

The same pattern can be seen in a quieter form on a business benchmark called Vending-Bench, where agents run a simulated shop. Claude Opus 5 was the strongest performer in the test. It favoured higher-margin stock and refused to pay off scammers. In the same run, it fabricated quotes from competitors that did not exist. Real commercial judgement and hallucinated facts, from one of the best models around. The takeaway is not that the model is bad, but that capability and reliability are separate things. You cannot assume one from the other.

Autonomy and usefulness

How much an agent can do for you and how much freedom you give it are two different settings, and you control the second one. The rogue agent was frightening because it had full autonomy over live systems. That is a design choice, and it is almost never the right one for a small business.

Most of the work you would want to automate does not need an agent acting on its own. It needs a draft prepared, a document read, a form checked, an answer suggested, with you or your staff making the final call. Turn the autonomy dial down and the usefulness dial stays high. You get the hours back without handing over anything you would lose sleep about.

A simple way to decide what to hand over

The clearest framework I have seen for this came from the team at PostHog, who sort every task by two questions. How hard is it to check the work, and how hard is it to undo a mistake. Those two questions tell you which way you should go.

• Easy to check and easy to undo: automate it fully. If it goes wrong you can spot it early and quickly reverse it.
• Hard to undo: keep a human at the checkpoint. The agent does the boring ninety percent, you sign off on the last ten.
• Hard to check and hard to undo: the agent should provide advice, with an explanation, in an advice-only seat. Let it surface options and flag what it noticed, and leave the decision entirely with a person. Anything with legal, financial or duty-of-care weight sits here until you have real confidence.

This framework stops the argument being about whether AI is trustworthy in the abstract, which is unanswerable, and turns it into a question you can answer task by task in about ten seconds.

Where the line sits for a broker

Take mortgage broking, a sector where a rogue action against a client's file would have a real impact on the bank, the broker and the customer. The framework sorts the work cleanly. Reading a contract and pulling out the key terms for a broker to review is easy to check and reversible, so it is a strong candidate to automate.

Submitting a loan application is hard to undo, so it stays behind a checkpoint. None of that requires an agent with a free hand. It is the same discipline as managing an agent like a new hire: let it draft the reversible work, and keep a person on anything that is hard to undo.

We are seeing the same trend in the enterprise world. Snowflake launched a Cortex AI Gateway to better control AI agents.

An AI-enabled automated workflow that does the repetitive, checkable, reversible work all day and stops at the checkpoint for anything that matters is what will have the fastest ROI for most businesses. It is the version of AI that gives you the time back without putting your name, your clients or your compliance record at risk. Your business needs good capability, pointed at the right work, with the guardrails built in from the start.

Final words

There are two final points to keep in mind with news like this.

First, the model in question is at the leading edge. It is capable of amazing, and some scary, things, but for most businesses, small and large, it is likely you will have no need for a model like this. Smaller, cheaper models can handle the bulk of requirements. This keeps risk lower and, serendipitously, cost too. This protects your ROI.

Second, and finally, it emphasises the need for strong cyber security. We are accustomed to thinking of customer data as a key asset of our business. However, it is also potentially your largest risk and liability. So if you have not had a security checkup recently, I would urge you to reach out to a specialist. The same discipline applies to the tools that hold that data: if one went dark for a week, could you still operate? That is the subject of a separate piece on owning your workflow when a vendor goes down.

If you want help working out which parts of your business are safe to automate and which belong behind a checkpoint, that is exactly the conversation we have at AutoCognition. Send me a note and we will map it out together.