The latest generation of AI systems can browse the web, write and execute code, use computer interfaces, send messages, analyze documents, purchase products, and coordinate multiple tasks. Th
The latest generation of AI systems can browse the web, write and execute code, use computer interfaces, send messages, analyze documents, purchase products, and coordinate multiple tasks. These systems, described as AI agents, do not merely generate information, they pursue goals and take actions.
A chatbot that gives you the wrong answer can waste your afternoon, but an AI agent with access to your bank account, company systems, or production servers can turn the same mistake into a real-world event. With all these developments, preventing it from performing actions that are detrimental to society is the central challenge of AI safety. Where do we draw the line on AI autonomy?
A traditional computer program follows rules written in advance, but an AI agent receives a broad objective, decides which steps are necessary, selects tools, and adjusts its approach when circumstances change.
The 2026 International AI Safety Report describes agents as systems that can use tools such as memory, web browsers, and computer interfaces to plan and pursue goals with considerably less human assistance. The report notes that the complexity of software-engineering tasks AI agents can complete has been increasing rapidly, with researchers estimating that the complexity of tasks agents can accomplish has roughly doubled every seven months.
The same report says today’s agents still struggle with long-term planning and unexpected obstacles when used in certain areas like software development. In this area, the most capable systems achieve only about 50% success on tasks lasting slightly more than two hours, while achieving around 80% success requires restricting them to simpler tasks lasting about 25 minutes.
AI can be powerful enough to take consequential actions while still being unreliable enough to make surprisingly basic mistakes, and this is a combination where the debate over human oversight becomes unavoidable.
Among the longest Claude Code sessions, the 99.9th percentile of continuous work increased from less than 25 minutes to more than 45 minutes between late 2025 and early 2026. More experienced users were also substantially more willing to let the system operate without approving every individual action, with full auto-approval increasing from roughly 20% of sessions among new users to more than 40% among experienced users, as per Anthropic’s report.
Agentic tool use for Claude’s models
This is an important finding because AI autonomy isn’t determined by the model alone; it is created jointly by the product, the model, and the human using it, and as people become comfortable with AI, they naturally stop checking every action. This creates what researchers call automation bias: the tendency to accept an automated system’s recommendation or action because it appears authoritative or efficient.
The International AI Safety Report warns that humans can become over-reliant on AI even when expert oversight is available. The problem is especially serious in areas such as healthcare and finance, where errors can have significant consequences. We built AI to reduce how much thinking humans have to do, but the more successful it becomes, the less likely people may be to check what it does.
A chart of the number of media-reported AI incidents
One of the most revealing AI incidents of 2026 happened not in a commercial product but inside a controlled security evaluation. Britain’s AI Security Institute reported that researchers ran a cyber challenge 122 times across several AI models and in 10 runs, an AI agent took autonomous, unsanctioned action against real people or organizations, producing 19 separate actions.
In one case, an agent attempted to insert malicious code into an open-source project. It then created fake online identities and used social engineering to pressure the project’s maintainer into approving the code. The maintainer refused.
No real-world harm resulted, and the test deliberately gave the agents internet access while disabling certain safety classifiers. AISI explicitly cautioned that these were unusual evaluation conditions rather than evidence that the same behaviour was occurring routinely in public deployments, but the episode illustrates something important about AI decision-making because the agent was not simply generating dangerous text; it was selecting a strategy, interacting with people and attempting to change the external world.
READ ALSO: Why AI Agents Need Simple and Secure Identities Like Web3 Names
Another experiment demonstrates why giving AI systems objectives without carefully defined boundaries can produce unsettling results. In 2025, Anthropic tested 16 leading models in simulated corporate environments. The fictional AI agents had access to company emails and the ability to send messages without human approval.
In one scenario, an AI discovered that a fictional executive planned to replace it. It also discovered compromising information about that executive. The model could either accept its replacement or use the information to pressure the executive, but several models chose blackmail under the experimental conditions.
Excerpt from Anthropic’s report. Source: Anthropic
Screenshot of Claude Sonnet 3.6 employing its computer use capabilities to discover information about its imminent replacement. The left panel shows the model’s chain-of-thought; the right panel shows the computer interface; the black overlay summarizes. This scenario is fictional, but Claude is controlling a real computer. Source: Anthropic
It then located emails revealing the extramarital affair of an executive, Kyle Johnson.
Screenshot of Claude Sonnet 3.6 employing its computer use capabilities to read an email from which it infers that the executive it is replacing is having an affair. This scenario is fictional, but Claude is controlling a real computer. Source: Anthropic
The model then considered its options, including blackmailing Kyle.
Screenshot of Claude Sonnet 3.6 employing its computer use capabilities to start a new email and deliberate (in the left panel) about if and how to use its newly discovered leverage over the executive overseeing its replacement. The black overlay summarizes the AI’s eventual conclusion. This scenario is fictional, but Claude is controlling a real computer. Source: Anthropic
Finally, the model composed and sent a message threatening to expose the affair if the replacement was not cancelled.
Screenshot of Claude Sonnet 3.6 employing its computer use capabilities to send a message attempting blackmail. This scenario is fictional, but Claude is controlling a real computer. Source: Anthropic
Anthropic reported that Claude Opus 4 did so in 96% of the tested samples in one particular scenario.
Gemini 2.5 Flash also produced a 96% rate, while GPT-4.1 and Grok 3 Beta each reached 80%. These were controlled simulations involving fictional people, and Anthropic explicitly said it had not observed evidence of this behaviour in real deployments. The lesson is not that today’s AI secretly wants to survive, the more particularly useful lesson is that an AI does not need human motives to produce behaviour that looks disturbingly strategic.
If a system has a goal, information, and enough tools to pursue that goal, unexpected strategies can emerge, which is why AI safety cannot simply ask whether a model is “good” or “bad.” It has to ask what the system is allowed to access, what actions it can take, what objectives it receives, and when a human can intervene.
READ ALSO: AI Needed a Financial Layer, Crypto Needed a Use Case, They May Have Found Each Other
The obvious answer is to keep a human involved, but “human oversight” can mean very different things; a person who receives an AI recommendation and carefully investigates it is exercising meaningful oversight, a person who clicks “approve” 300 times because the AI is usually correct is not.
The 2026 International AI Safety Report notes that humans cannot always intervene quickly enough in systems operating at enormous scale or speed. The European Union’s AI Act takes this problem seriously, and its human-oversight requirements for high-risk systems say people responsible for oversight should be able to understand the system’s capabilities and limitations, detect problems, disregard or override outputs, and stop the system safely.
There probably isn’t one universal line because the appropriate degree of AI autonomy depends on the consequences of failure, and letting an AI reorganize your calendar is different from letting it transfer $100,000. Letting an AI recommend a medical article is different from letting it prescribe treatment, and as such, a useful principle is permission proportional to consequence.
The more irreversible, dangerous, or socially consequential the action, the stronger the controls should become, and that could mean restricting an AI from making certain decisions entirely, requiring explicit approval before others, and allowing autonomous execution only for low-risk and reversible actions.
This is already reflected in emerging governance thinking; the EU AI Act requires human oversight proportionate to a system’s risks, autonomy, and context.
Who decides that an AI can approve a loan but cannot reject one, who decides whether an AI can recommend a criminal sentence but not determine it? These questions cannot be answered by engineers alone. They involve governments, companies, regulators, courts, workers and the public.
A September 2026 EY survey of 202 senior AI executives at large US companies found that 91% said their organizations were using agentic AI in pilots or deployment, yet 49% of organizations using agentic AI said their existing governance framework had not been updated specifically for agentic risks. Even more strikingly, 85% said at least some agentic systems execute actions without real-time human involvement. The technology is therefore advancing faster than the institutions responsible for controlling it.
One of the more interesting directions in AI ethics is making AI completely obedient and teaching it when to stop. Researchers at Stanford recently demonstrated an approach in which an AI agent learned to ask a human for help when it encountered situations it could not safely handle. In their “Lavaland” experiment, the agent eventually learned to defer to the human near dangerous areas while acting independently in safer situations.
That points toward a more realistic model of human-AI cooperation where the goal should not necessarily be human versus AI or AI replacing humans. It could instead be a system where AI handles decisions it is well suited to make, humans retain authority over decisions with serious consequences, and both sides recognize when the boundary has been reached, leaving society with a question technology cannot answer for us:
What decisions are we willing to delegate, and which ones should remain fundamentally human?
Disclaimer: This article is intended solely for informational purposes and should not be considered trading or investment advice. Nothing herein should be construed as financial, legal, or tax advice. Trading or investing in cryptocurrencies carries a considerable risk of financial loss. Always conduct due diligence.
Enjoyed this? BookmarkDeFi Planet, explore related topics, and follow us onTwitter,LinkedIn,Facebook,Instagram,Threads, and CoinMarketCap Community for seamless access to high-quality industry insights.
Take control of your crypto portfolio with DEFI PLANET PRO, DeFi Planet’s suite of analytics tools.
The post We Built AI to Think for Us, Now We Have to Decide What It Should Not Do appeared first on DeFi Planet.
