AI Agents Are Coming: What Happens When Software Can Act for Us?

For most of its history, consumer software has waited to be told exactly what to do. Even the recent generation of AI assistants mostly produces text: an answer, a draft, a summary. The shift now underway is smaller than the marketing suggests and more consequential than it sounds. Software is beginning to take actions — clicking, querying, submitting, purchasing, scheduling, escalating — on a person's or an organisation's behalf.

That shift changes the questions we need to ask. A wrong answer is an inconvenience. A wrong action has consequences: a payment made, a record altered, a message sent to a client, a system reconfigured. Once software acts, the interesting problems stop being linguistic and start being about authority, identity, accountability and reversibility.

What an AI agent actually is

Strip away the branding and an agent is a loop. A model receives a goal, decides on a next step, invokes a tool — a search, a database query, an API call, a browser action, a shell command — observes the result, and repeats until it believes the goal is met or a limit is reached.

Everything interesting lives in three details: which tools the loop can reach, what stops the loop, and who is accountable for what it does. The model is the least distinctive part. Two systems with the same underlying model can behave very differently depending on tool access, permissions and guardrails.

What current evidence suggests

Current systems handle short, well-specified tasks in instrumented environments reasonably well, and degrade as horizons lengthen. The mechanism is arithmetic: if each step succeeds with probability p, a twenty-step task succeeds with roughly p to the twentieth power unless the system can detect and recover from its own errors. Recovery — noticing that a step failed and adjusting — is where agents remain weakest.

This is also why demonstrations mislead. A curated ten-minute demo shows the best of perhaps many attempts. Independent evaluation work, including the annual AI Index surveys of benchmark progress, gives a more sober picture of where reliability actually sits.

Two very different deployments

Personal agents

The consumer story is convenience: research a purchase, compare travel options, fill in a form, chase a refund, manage a calendar, keep a household's administrative tail from accumulating. The value is real, and it is concentrated in tasks people find tedious rather than difficult.

The obstacles are equally mundane. Personal agents need access to accounts, which means they need credentials, which means a new class of security exposure. They need to know a user's preferences, which means retaining sensitive context. And they operate against websites and services that are not neutral parties — many businesses have commercial reasons to resist automated agents comparing them.

Enterprise agents

Inside organisations, the economics are stronger and the constraints tighter. Firms already have identity systems, audit requirements, change management and someone whose job is to sign off. That machinery is a burden, but it is also exactly what makes delegated action supervisable.

The realistic pattern is narrow and deep: a defined workflow, a bounded set of tools, a validation step, logging of every action, and a human owner. Contrast this with the fantasy of a general corporate agent given broad access and told to improve the business. The first is deployable. The second is an incident report waiting to be written.

Delegation is the real problem

Human institutions have spent centuries building the apparatus of delegation: powers of attorney, mandates, spending limits, dual authorisation, escalation thresholds, audit trails, professional liability. None of it exists because people are untrustworthy in general; it exists because acting on someone else's behalf requires bounded, revocable, evidenced authority.

Agentic software needs the same apparatus, expressed in machine-readable form:

• Scope. Precisely which systems and data, and for how long.
• Limits. Value thresholds, rate limits, and a hard list of irreversible actions requiring confirmation.
• Identity. A distinct, attributable identity for the agent — not a borrowed human credential.
• Evidence. An immutable log of what was attempted, what succeeded, and on whose authority.
• Revocation. A reliable way to withdraw authority instantly.
• Recourse. A defined answer to who compensates whom when something goes wrong.

Frameworks such as the NIST AI Risk Management Framework and the OECD's AI policy work approach this from a risk-governance angle; the EU's AI regulatory framework approaches it through obligations tied to risk level. Neither yet resolves the operational question of how an agent proves, in the moment, that it is authorised to do a specific thing.

Security: a genuinely new attack surface

Agents introduce a category of vulnerability that traditional software does not have. Because they read untrusted content and then act, that content can carry instructions. A web page, an email, a PDF or a code comment can attempt to redirect the agent's behaviour — prompt injection, in the common phrasing.

This is not a bug to be patched once. It is structural: the system's instructions and its data arrive through the same channel. Practical mitigations exist and are all about architecture rather than model cleverness:

• Treat every model output as untrusted input to the next system.
• Keep privileges minimal and task-scoped; never hand an agent standing broad access.
• Separate the component that plans from the component that executes privileged actions, and validate between them.
• Require confirmation for anything irreversible: payments, deletions, external communications, permission changes.
• Log everything, and monitor for anomalous action sequences rather than only anomalous inputs.

There is also an emerging systemic question. When many agents transact with many other agents and automated counterparties, correlated failure becomes possible — a shared model flaw or a shared data source producing simultaneous errors across thousands of processes. That risk is not yet well characterised.

Identity for non-human actors

Today's internet largely assumes that a request comes from either a person or a server acting on a company's behalf. Agents sit uncomfortably between the two. A service receiving an agent request reasonably wants to know: who authorised this, what is it allowed to do, and who is liable if it is wrong?

Several answers are being explored — delegated credentials with narrow scopes, signed authorisation tokens carrying explicit limits, registries of agent identities, per-agent payment instruments with hard caps. It is unclear which approach prevails, and plausible that different sectors settle differently. What seems safe to say is that agent adoption in high-value contexts will be gated by this plumbing rather than by model capability.

Oversight without theatre

"Human in the loop" is often invoked as a solution and implemented as a formality. If a person must approve two hundred agent actions per day, approval becomes a reflex and provides no real control. Meaningful oversight has to be designed around attention as a scarce resource:

• Approve policies and boundaries rather than individual routine actions.
• Reserve human attention for the irreversible, the unusual and the high-value.
• Make the agent surface its own uncertainty and stop when confidence is low.
• Sample and review completed work after the fact, as auditors do.
• Measure the human's actual error-catching rate — an approval step that never catches anything is decoration.

Scenarios for the next decade

Scenario one: bounded utility. Agents become normal in narrow, instrumented workflows and in low-stakes consumer errands. Value is real but unspectacular, arriving as compressed cycle times inside existing processes. The trust and identity plumbing develops slowly. This currently looks the most likely path.

Scenario two: agent-native infrastructure. Standards for delegated authority, agent identity and machine-to-machine payment mature faster than expected. Services start publishing structured interfaces designed for agents, and a genuine agent economy forms — with new intermediaries and new concentration risks.

Scenario three: retrenchment. A cluster of high-profile failures — a large fraudulent transaction chain, a data exposure, a cascading automated error — triggers restrictive rules and corporate caution. Agents remain confined to internal, low-risk tasks for years.

What would move the odds. Independent evidence of long-horizon reliability; adoption of scoped agent credentials by major platforms; insurers offering products for delegated automated action; and the first well-documented large loss and how liability is settled.

What I would not claim

I would not claim that current agent architectures are on a direct path to general intelligence; that is an assumption, not an observation. I would not claim that white-collar work is about to be automated wholesale — the evidence on task-level exposure points to redesign more often than replacement. And I would not put a date on any of the scenarios above.

Conclusion

The move from answering to acting is the most important thing happening in applied AI, and the bottleneck is not intelligence. It is the unglamorous work of authority: scoped permissions, verifiable identity, hard limits, audit trails, insurance and liability.

Organisations that build that scaffolding first will be able to adopt agents safely as capability improves. Those that wait for a general-purpose agent to arrive fully formed will find that the constraint was never the model.