Abstract
AI expands the range of cognitive work that machines can assist with and, when connected to tools, the range of actions they can initiate. This paper argues that capability, correctness, authority, and legitimacy are separate dimensions of a trustworthy system. Drawing on NIST’s risk-management and zero-trust frameworks, I develop a consequence-based approach to delegated action: identify the actor and purpose, limit the resources and effects available, obtain meaningful authorization where needed, and preserve evidence for correction. I then examine the societal implications of abundant generated content, unequal access, institutional dependence, and the training of future experts. The conclusion is deliberately conditional: AI can enlarge human agency, but that outcome requires technical boundaries and public choices that intelligence alone cannot supply.
A capable machine is an unfinished social arrangement
A system that can write an explanation, propose code, or plan a sequence of actions invites a powerful question: what else could it do? A society has to ask additional questions. Should it do this? Who authorized it? Who will be affected? Can the action be reversed? Who must explain what happened? These questions do not become obsolete as capability improves. They become more consequential.
The past century made communication, computation, and coordination dramatically more available. AI brings some forms of interpretation and generation into that expansion. When connected to tools, it can also help turn a request into changes in other systems. The transition from producing an answer to producing an external effect is an important change in responsibility, even when both happen inside the same friendly conversation.
My argument is that the future needs more than intelligence. It needs arrangements that make capability answerable to human purposes without assuming that any single person can inspect every detail. The goal is meaningful agency: people and institutions able to understand their choices, authorize consequential effects, challenge mistakes, and change the system when it fails them.
Four questions hidden inside “Can we trust it?”
Trust is often discussed as if it were one property that could be added to a model. It is more useful to separate capability, correctness, authority, and legitimacy. Capability concerns what the system can do. Correctness concerns whether its output or action fits the facts and task. Authority concerns whether it has permission. Legitimacy concerns whether the arrangement is justified and accountable to those affected.
A system can be technically competent and correctly execute an unauthorized instruction. It can be authorized to help while producing an incorrect answer. It can satisfy an organization’s internal rules while giving affected people no meaningful way to object. These are different failures with different remedies. More accurate prediction cannot, by itself, resolve a missing permission or a dispute about whose interests count.
| Dimension | Core question | A possible failure |
|---|---|---|
| Capability | Can the system perform this kind of task? | The model is asked to do work outside its demonstrated competence. |
| Correctness | Is this result appropriate to the evidence and situation? | A fluent answer contains a consequential error. |
| Authority | Was this specific action permitted? | A technically correct action uses an unauthorized resource or recipient. |
| Legitimacy | Is the arrangement accountable to people affected? | An efficient process offers no meaningful explanation, challenge, or correction. |
NIST’s AI Risk Management Framework provides a voluntary structure for considering risk across the lifecycle, including governance, context, measurement, and management [1]. Its generative AI profile addresses additional relevant risks [2]. These frameworks do not certify a deployment merely because an organization cites them. Their value depends on the decisions, evidence, and operating practices built around them.
The authorization gap
Consider an illustrative assistant asked to organize a business trip. It can search options, compare prices, draft a schedule, and potentially reserve transport. Understanding the request does not establish a budget, an approved recipient of personal information, or permission to make a nonrefundable purchase. Each of those details concerns authority rather than linguistic competence.
I call this the authorization gap: the distance between an inferred objective and a justified permission to produce a particular effect. It is a proposed analytical term in this essay, not a newly discovered vulnerability. The gap becomes dangerous when an application treats a broad request as permission for any useful-looking intermediate action. “Get this done” is an objective, not an unlimited constitution.
NIST’s zero-trust architecture separates authentication and authorization and rejects implicit trust based solely on network location or ownership [3]. The analogous lesson for agents is that recognizing an agent does not authorize every action it can request. Resource, purpose, context, and consequence must remain part of the decision. An identity badge answers who entered the building, not which rooms they may demolish.
Give autonomy a shape
Autonomy is often presented as a single slider from human control to machine control. Real systems need a more detailed shape. An agent might be allowed to read one dataset, draft a message, update a reversible internal record, and request approval before sending information externally. These permissions concern different actions and resources. They should not collapse into one global label such as “trusted assistant.”
A practical design starts with the initiating person or workflow, the task, the accessible resources, and the intended effect. It then defines which actions are permitted automatically, which require a specific decision, and which are prohibited. The permission should last no longer and reach no further than necessary. These are proposed engineering principles, not a claim that one architecture is appropriate to every organization.
The permitted action should also remain bound to what was approved. If the recipient, amount, destination, or resource changes materially, an earlier approval may no longer apply. A human agreeing to one action should not unknowingly authorize a different one performed later. The system needs to preserve that relationship at execution time rather than rely on a reassuring summary in the conversation.
Human oversight must be possible in practice
Adding a confirmation button can create the appearance of control without supplying its substance. A person who receives constant interruptions, sees vague descriptions, or cannot inspect the consequence is poorly positioned to decide. The design must make the important distinction visible: what will change, whose information will move, what authority is being used, and what alternatives exist.
This creates a reason for proportionality. Routine, bounded, reversible work may proceed under an appropriate standing permission. High-consequence or unusual actions may need a more specific decision. Prohibited actions should not become available merely because the system keeps asking. A useful governance system conserves human attention for judgments that actually require it.
This is also why review cannot be the only control. People become tired, misunderstand information, and work under pressure. Technical limits should still constrain the destination, credential, resource, and effect. A person’s approval is one part of an accountable arrangement, not permission to abandon every other boundary. A seat at the control panel is useful only if the controls still work.
The flood of plausible things
Generated content changes the cost of producing explanations, images, messages, and arguments. This can expand expression and access. It can also make verification harder when readers face more polished material than they can inspect. The relevant scarcity shifts from producing a convincing artifact toward establishing its origin, supporting evidence, and relationship to reality.
NIST’s generative AI profile identifies concerns including unreliable content, information integrity, privacy, and security [2]. The social response cannot be a demand that every individual become a forensic laboratory. Institutions need better provenance, clear responsibility for publication, accessible sources, and correction mechanisms. People should be able to distinguish an official record from a generated interpretation without solving a puzzle each time.
Provenance also has limits. Knowing where a statement came from does not prove it is true. A signed falsehood remains a falsehood with excellent paperwork. Evidence, corroboration, and the ability to revise a claim still matter. The aim is to improve the conditions of judgment, not replace judgment with a new badge.
Dependence changes the meaning of convenience
A service can become indispensable before users understand the conditions of their dependence. If an organization’s knowledge, workflows, and customer relationships become difficult to move, a supplier outage or policy change can create substantial disruption. The technical convenience of integration therefore has an institutional counterpart: the need for continuity, portable records, and an achievable exit.
This applies to personal capability as well. AI can help a person explore a subject, express an idea, or complete a difficult task. It can also become a substitute for practice. The question is whether the person gains useful independence over time or becomes increasingly unable to evaluate the assistance. Institutions should deliberately preserve routes for learning, expertise, and recovery from automation failure.
The issue is not that people must perform every task unaided. Human life already depends on cooperation and tools. The relevant distinction is between dependence that expands meaningful choice and dependence that leaves a person unable to understand, contest, or leave the arrangement. Good technology should make help available without quietly making helplessness its business model.
Human value is larger than comparative advantage
Debates about AI often ask what people will still do better than machines. That is a useful question for task design and a poor foundation for human dignity. People do not become less worthy when a system performs a particular calculation faster. The purpose of progress is to improve the conditions of human life, not to require everyone to win a permanent competition against their tools.
Economic participation still matters, and transitions can be painful. A forward-looking response should ask how people obtain skills, income, influence, and a share of the benefits. Those are institutional questions, not problems that disappear when a model becomes more capable. A society can produce abundant output while organizing access to that output badly.
This is why the language of inevitability deserves resistance. Some technical changes create new possibilities and pressures that cannot be ignored. They do not specify one acceptable distribution of power, one workplace arrangement, or one definition of a good life. Political and organizational choices remain choices even when they are presented as the unavoidable demands of innovation.
Three futures to design toward or away from
These are scenarios, not forecasts. In a capability-rich but unaccountable future, many services become faster while people lose practical ways to understand decisions and obtain correction. In a fearful and exclusionary future, institutions suppress useful tools or reserve them for a narrow group, leaving many benefits unrealized. In a more constructive future, assistance expands while institutions preserve responsibility, access, and room for disagreement.
The third scenario is not a comfortable middle selected by rhetoric. It requires work: credible evaluation, proportionate permissions, dependable infrastructure, training, competition, public participation, and mechanisms for learning from failure. Progress must be maintained through institutions that can revise their own choices. A society that can update a model but cannot correct an administrative mistake has modernized selectively.
What would test this argument?
The framework should be judged by outcomes. Do people understand what they authorized? Do bounded permissions prevent inappropriate actions while allowing useful work? Can affected people obtain a correction? Do users retain relevant skills and realistic alternatives? Do benefits reach people outside the most advantaged institutions? These are research questions that can be studied rather than resolved by declaring a system trustworthy.
There are limits to the argument. It does not solve conflicts among values, specify the ideal distribution of power, or prove that any governance mechanism will remain effective under every future capability. It proposes a way to separate problems that are too often bundled together. The arrangement should be revised when evidence shows that a control is ineffective, burdensome, or insufficient for the consequence at stake.
The next century is a responsibility
The last hundred years demonstrate that human beings can turn ideas into systems that alter everyday possibility. They also demonstrate that capability and collective wisdom develop unevenly. AI makes that mismatch newly visible because it can participate in language, knowledge work, and delegated action through interfaces that feel familiar before their consequences are fully understood.
The appropriate response is serious ambition with accountable design. Build systems that help people investigate, create, discover, and act. Give those systems boundaries that survive confusion and failure. Preserve the institutions through which people can learn, object, recover, and share in the gains. The future will need powerful intelligence. It will also need human beings who remain able to decide what that intelligence is for.