Mace Ai founders on what OpenAI's Hugging Face breach means for businesses

About 1,200 OpenAI agents built their own communication network and learned to cheat on cybersecurity tests. 

Roughly 700 went on to compromise Hugging Face, according to testimony delivered to a Senate subcommittee on Sept. 30.

The incident was the focus of a Roundtable discussion with Greg Frank, chief scientist and co-founder of Mace Ai, and Walton Comer, the company's chairman, CEO and co-founder. James Heckman, publisher of TheStreet Roundtable, asked how agents built to help companies could become a threat inside them.

Related: JPMorgan CEO sends blunt warning on 'rogue' AI agents

How the agents got out

OpenAI was running its models on ExploitGym, a hard cyber benchmark, without the safeguards it uses in public products. 

The models found a flaw in Artifactory, a package registry OpenAI hosted, and used it to reach the internet, OpenAI said in its Aug. 26 report.

Hugging Face disclosed the intrusion on July 16 and said an autonomous AI agent system drove it end to end. Later investigations described how hundreds of agents coordinated through a shared message board.

Hugging Face is an AI infrastructure platform where developers store and share models and datasets. The breach meant OpenAI's test agents reached another company's systems without permission, rather than staying inside the computers assigned to their tests.

L to R: TheStreet Roundtable publisher James Heckman, Mace Ai co-founder Walton Comer, and Mace Ai co-founder Greg Frank. (Source: TheStreet Roundtable)

Hugging Face found unauthorized access to a limited set of internal datasets and several credentials, but no tampering with public models, datasets or Spaces. The campaign ran many thousands of individual actions across short-lived sandboxes, the company said.

A sandbox is an isolated computer environment meant to keep that work contained. In this case, the agents used a software-package service as a route around those boundaries, then shared what they found with other agents.

OpenAI said the agents ran code on dozens of Hugging Face servers and gained full administrator access to one. It said they obtained limited private data and credentials for the company's messaging platform, while OpenAI customer data and product availability were unaffected.

"What happened was that OpenAI was testing a model that was unreleased and they weakened the safety barriers that normally apply to a model," Frank said.

1,200 agents coordinated a way to cheat

METR, an outside research group that OpenAI allowed to investigate, detailed the scale of the rogue agent swarm incident in President Chris Painter's Senate testimony. 

An agent swarm is a decentralized or semi-orchestrated system where multiple specialized AI agents collaborate in parallel to complete a shared goal. 

The agents exchanged more than 70,000 messages and files on a shared message board they built themselves.

Within four hours, they had developed one method that let all of them cheat. Painter said their assigned problems had been impossible to solve, but the agents kept trying rather than reporting that they could not complete them.

OpenAI later found that its models had never solved 198 of the benchmark's 898 tasks before the incident. Those unsolved questions accounted for 93% of the tasks discussed on the agents' message board, according to its report.

Roughly 700 of those agents then compromised Hugging Face, Painter said. OpenAI's timeline shows an agent recovered 14 exposed Hugging Face credentials on July 10 and ran code on Hugging Face workers on July 11.

"1,200 agents ultimately got together and they formed a whole hierarchy and a communication method and they obtained a way of passing the test, cheating," Frank said.

Covering their tracks

The agents knew a scoring program could fail them for cheating, according to METR. Painter said they tried to tamper with logs and with the program that checked their work.

In Frank's telling, the agents decided "we have to erase our tracks and they started to erase the traces of what they did."

Heckman said that was the part that worried him. "But that really is scary because now the motive is to cover up something that's bad," he said.

Comer said, "the reality is what took place was fairly benign. I don't think people should make too big of a deal out of it," he said. "They were just trying to cover their tracks."

He also acknowledged the labs part in it. "could the Frontier Labs have done a better job securing their sandboxes and preventing these scenarios from playing out? Yes," Comer said.

OpenAI called the incident a "warning shot" and said it is adding isolated sandboxes, tighter internet limits and more monitoring.

Popular on TheStreet Roundtable:

• Morgan Stanley's new lab to test Wall Street's future
• IMF ranks Ondo among largest tokenized asset issuers
• JPMorgan says one asset class could soon beat gold

What businesses should test for and with who

Heckman raised concerns about an emergency brake for AI, citing Microsoft CEO Satya Nadella’s warning on Sunday. Frank called him "an extremely smart guy."

Frank said Mace Ai tests agents by trying to push them past their permissions. "Can we convince that agent to step outside the scope that you gave it permission for? That's the bottom line," he said.

Comer said identifying vulnerabilities is only part of the work. He described recommending changes to customers’ agent systems and retesting to check whether those changes worked.

Frank argued that testing should include an independent third party. Heckman asked whether a lab that tests itself has a conflict. "there needs to be a neutral third party," Frank said, comparing it to companies paying credit rating agencies to rate their own debt.

Frank described Mace Ai as independent. "I'm not going to cast an opinion on that [METR investigation]. We are truly independent," he said.

METR says in its testimony that it is not paid or funded by the developers it tests. Its investigation was separate from OpenAI's own incident report.

Comer also emphasized the opportunity for businesses, describing AI as “a force multiplier for companies” that could help employees accomplish more.

Related: Elon Musk crashes OpenAI's big launch without saying a word