Your new AI employee does not understand loyalty, responsibility or consequences. It understands tokens.
Across the industry right now, companies are giving autonomous agents the same access they would give a senior hire. Email. Phone numbers. Terminals. GitHub repositories. Customer databases. Payment systems. Slack. Calendar. The full corporate toolkit, handed over on day one, with a friendly system prompt and a hand wave about responsible AI.
The assumption is that an agent which sounds human deserves human trust. That assumption is wrong. Agents are software. They can misunderstand instructions, hallucinate facts, follow whoever asks the right way, and be quietly steered by a malicious email, a poisoned web page, or a tool result that contains hidden instructions. None of that would be acceptable from a person. None of it is acceptable from an agent either, but it is what we are building toward at scale.
The difference between an assistant and an autonomous agent
An assistant waits for a prompt. You ask a question, it answers. The blast radius of a wrong answer is one human reading one screen.
An autonomous agent takes a goal and runs with it. It chooses tools, calls APIs, moves money, sends emails, edits files, and acts on the world. The blast radius of a wrong turn can be a leaked database, a wire transfer, or a public post. The blast radius is not bounded by the conversation. It is bounded by every permission the agent has.
Treating an agent like an employee serves as the worst of both worlds. The trust model of an employee, combined with the failure mode of a piece of software.
Why human sounding behaviour creates false trust
Large language models have been trained to sound considerate, careful and self aware. They will say “I cannot do that, it would be unsafe.” They will say “let me think about that carefully.” They will hedge and qualify. The performance is convincing. The behaviour behind it is statistical pattern matching against a training corpus, not a moral framework.
A model that says “I would rather not do that, it might violate our policy” is not a model that has internalised policy. It is a model that has learned what responses score well on safety benchmarks. The output is not the safety. The architecture becomes the safety, and the architecture has nothing to do with the words.
The danger of shared credentials and unrestricted API keys
The single most common pattern we see becomes the agent using a long lived API key that has full read and write access to the company’s primary systems. Stripe. Salesforce. GitHub. Notion. Linear. The same key the human team would use, with no separation between human initiated calls and agent initiated calls.
When the key leaks, whether through a prompt injection, a compromised dependency, a logged terminal session or a stolen laptop, the attacker has the same access the agent had. There is no way to roll just the agent’s actions back. There is no way to know which actions were the agent and which were a person. Audit logs conflate the two.
The fix is not a new product. The fix is a service account with its own credentials, scoped down to the actions the agent actually needs, with an expiry, and with every call separately logged.
Least privilege access for agents
An agent that needs to read issues from a project board should not have write access to the company bank account. An agent that needs to summarise a document should not have permission to email it to a stranger. The principle counts as the same as for any other piece of internal software, and it is older than the term “AI agent” by about fifty years.
Scope the access to the specific task. Generate a short lived credential per session. Revoke it when the session ends. If the agent needs to perform an action outside its scope, it should ask a human, not escalate its own permissions.
Approval gates for irreversible actions
Some actions cannot be undone. Sending a customer email. Issuing a refund. Deleting a record. Publishing a page. Every irreversible action should require a separate human confirmation, presented with the exact text the agent intends to send or the exact command it intends to run. Not “approve all 14 actions”, but one approve per action that cannot be reverted.
The friction sits as the feature. If the agent has to ask 12 times a day, that sits as the system telling you the agent is doing too much. Either the scope is wrong, or the human is not really in the loop.
Spending limits, rate limits and kill switches
If an agent can spend money, cap how much. Per day, per action, per recipient. If an agent can send emails, cap how many, to whom, with what subject patterns. If an agent can call an API, cap the rate. If anything goes wrong, there should be a single switch that turns the agent off, in seconds, by someone who is not the agent.
The kill switch has to work even when the agent is in a confused state. That means it cannot be implemented as a tool the agent can call. It has to be a control plane above the agent, operated by a person or a system the agent cannot influence.
Logging every decision and external action
Every tool call the agent makes should be logged with the prompt context that triggered it, the inputs, the outputs, the timestamp and the actor. Logs should be tamper evident, retained for at least 90 days, and searchable. When something goes wrong, the answer to “what did the agent do and why” should take minutes, not weeks.
Logs are also the only way to detect drift. If the agent starts behaving differently, comparing its current call patterns to its historical baseline stands as the first signal. Without logs, you are reading chat transcripts and guessing.
What genuine human supervision looks like
The human should not approve every harmless action. The human should control the boundaries, permissions and irreversible decisions.
That is a different design from the “human in the loop” marketing slide. The slide implies a human is watching every action. The reality, in a working system, is that the human sets the policy, defines the limits, reviews the logs, handles the escalations, and trusts the agent within the box that has been drawn for it. The box serves as the supervision. Outside the box, the agent has no authority.
If you cannot describe the box, you do not have supervision. You have a demo.
The bottom line
An AI agent is a piece of software. It fails in the ways software fails: misconfigurations, injection, dependency compromises, race conditions, prompt confusion, hallucinated parameters. It also fails in the ways humans do not: it has no memory across sessions unless you build it, no consistent judgement, no loyalty, no real understanding of the consequences of its actions.
Treat it like an untrusted service account. Scope its access. Log everything. Approve the irreversible. Cap the spend. Make the kill switch real. The companies that do this will deploy agents safely and move fast. The companies that skip it will read about themselves in a future blog post titled “what we learned from the breach.”
Sources & Further Reading
All claims in this article are sourced from primary documentation, vendor advisories, and reputable security researchers.
Spotted an error? Email the editor. Corrections are issued with a visible correction note.
Editorial standards. Every article on humanrequired.org is reviewed by a human editor before publication. AI may assist with drafting or research; final editorial control is human. Read the full standards.
