Designing AI Agents: The Constraints That Make Autonomy Useful

If you are investing in AI agents, one of your most consequential responsibilities is to understand whether the architecture your teams are building can support the autonomy your business expects. The promise is substantial: systems that can pursue an outcome, adapt as circumstances change and carry more work through to completion. Delivering that promise depends on the conditions in which those systems operate.

Those conditions include what agents can access, which decisions they can make, where their authority ends and how the organisation learns from their execution. Together, they shape how much work can be delegated responsibly, what supervision will cost and whether a successful use case can expand across the enterprise. These are architectural choices with consequences for investment and the operating model.

The central argument of this article is that explicit, well-designed constraints make greater autonomy possible. They give agents room to exercise judgment while keeping consequential actions within boundaries the organisation can uphold. Understanding that relationship gives executives a basis for assessing whether their teams are building a credible path towards the promised transformation.

Stay with the architectural detail that follows. It will help you ask more revealing questions about the choices your teams are making, the foundations worth funding and the changes the business must be prepared to make. The article closes with a checklist for those conversations with your AI and operational leaders.

Consider a request to resolve a customer complaint. An agent can read the history, interpret the policy and propose a remedy. Once it has access to operational tools, however, the same request raises further questions. Can it issue a credit? Change the customer’s contract? Make an exception to the policy? The outcome sounds clear while the authority needed to pursue it remains open.

A useful design closes that gap in three connected ways: it makes the mandate and evidence explicit, enforces consequential boundaries where actions occur, and expands autonomy only when results justify it. These three choices provide the structure for the discussion.

Make explicit what human delegation leaves unstated

When we delegate work to a colleague, we rely on much more than the words in the request. We assume some shared understanding of the business, a sense of professional responsibility and a willingness to ask when authority is unclear. The organisation also places practical limits on what the person can discover and do.

Attention and working memory are finite. Finding information takes effort, and expertise is distributed across people and systems. Herbert Simon’s work on bounded rationality explains why decision-making has to accommodate limits on our capacity to understand and compute under uncertainty. Social interactions add another influence: a colleague challenges an assumption, a manager clarifies a decision, or a conversation reveals that a technically possible action would be inappropriate.

These conditions are imperfect. They can prevent errors and create accountability, but they can also delay good decisions or preserve an unhelpful way of working. Either way, they shape the work before we write a procedure for it.

An agent changes that arrangement. With broad tool access, it can cross information boundaries and repeat operations without encountering many of the conversations and delays that normally interrupt human work. Its own limits remain real: incomplete information, finite context and fallible reasoning. We are combining those limitations with the reach and repetition of software.

That combination calls for more explicit constraints. Asking an agent to behave like an experienced colleague leaves too much of the operating model implicit. The system needs a clear account of what the agent is trying to achieve, which evidence it can use and what authority accompanies the task.

Constraints can create room to act

Complex systems thinking gives us a useful way to understand this relationship. Enabling constraints establish conditions within which different responses can emerge. Dave Snowden, drawing on Alicia Juarrero’s work, makes the idea tangible in an observation about structure:

The internal skeleton of human provides a coherent structure, but also allows variety.

Dave Snowden - Freedom through constraints.

A team with a clear customer problem, an agreed spending boundary and access to relevant expertise has room to find a solution. It can change its approach as it learns because it understands the outcome and the limits within which it is working. The boundary supports the exercise of judgment.

The same idea is useful in agent design. An agent investigating a service issue can compare explanations, inspect dependencies and propose tests while having no permission to alter production. That separation can make a wider range of investigation acceptable because the consequences remain contained.

Some of the controls involved are rigid, such as a mandatory approval or a permission boundary. The enabling effect comes from the arrangement as a whole: firm controls around consequential actions allow useful discretion elsewhere. The leadership question is how to create enough room for the work to improve while keeping its consequences within bounds the organisation is prepared to accept.

Put each constraint where it can actually work

Once the mandate is clear, architecture determines whether its boundaries hold. An instruction can explain that approval is required, but a service that rejects an unapproved operation gives that rule a different force. The distinction becomes important whenever an agent misinterprets a request, encounters misleading content or simply makes a mistake.

Six layers help organise this design: mandate, context, tools, action contracts, execution and assurance. They describe responsibilities that may be distributed across the agent application, gateways, business services and operating environment. The diagram makes that distribution visible, including where each control is commonly implemented.

The locations are typical choices and can overlap. Instructions guide the agent; services and the environment enforce consequential limits. Execution budgets and assurance span the workflow.

Start with the task and the evidence it needs

For the customer complaint, the mandate might be to prepare an eligible remedy using the current policy and account history, then submit any exception for review. That gives the agent a basis for choosing its next step. It also makes a useful incomplete result possible: a case can end with missing evidence or an unresolved exception instead of an unsupported answer.

The mandate belongs in the agent’s instructions and task configuration. Completion checks belong in the workflow that accepts its output. The business owner defines what a satisfactory result requires, while the application checks that the required evidence and deliverables are present. For predictable work, much of that sequence may be a fixed workflow; agent discretion is most useful where interpretation or adaptation contributes to the outcome. This is consistent with Anthropic’s guidance to begin with the simplest effective design.

Planning is also an active area of constraint research. Neurosymbolic planning combines a model's interpretation of a request with an explicit representation of permitted actions and their prerequisites. In robotics, SayPlan uses scene representations and simulation to check plans; the more recent Any House Any Task (AHAT) turns instructions into subgoals for a symbolic planner. The architectural lesson is useful beyond robotics: where a task has dependencies that can be specified, give the planning layer a way to check them before execution. Applying that pattern to a business process requires the domain team to model those dependencies accurately, and the runtime to recheck conditions as the world changes.

The context layer then determines what information reaches the agent. A complaint requires the relevant customer records and policy, with their source and version preserved. Retrieval and data services should enforce access permissions before returning material, while the agent application keeps external content distinct from instructions. A sentence inside a customer email cannot acquire authority to change the task.

Selecting relevant context can reduce distraction and processing cost, but excluding a decisive record creates another failure. The design therefore needs to test whether the agent can find enough evidence as well as whether it avoids irrelevant material. Context engineering continues throughout the task because the information needed for the next decision changes as the investigation develops.

Carry authority through to the receiving system

Suppose the agent concludes that a credit is appropriate. Its tool set should expose an operation suited to that task, with permissions scoped to the relevant resources. A gateway can restrict which tools may be called and check the caller’s authority; the operating environment can restrict network destinations, file access and available credentials. Those protections matter particularly when a general execution tool could reach around a narrower interface.

The business service still has its own decision to make. It must verify that this credit is allowed for this account under the current policy. A gateway may understand identity and permitted operations without holding the current transaction history needed to establish entitlement. The final check belongs close to the state it protects, and every route to that state must respect the boundary. OWASP’s agent security guidance supports scoped tools, independent validation and controls around consequential actions.

This is where an action contract becomes concrete. Grammar-constrained decoding, implemented by tools such as XGrammar, restricts generation to an allowed output structure. It belongs in the model-serving or structured-output layer and can prevent malformed requests before they reach an API. The business service must still check the account, amount, authority and approval: a correctly formed request can carry an incorrect decision.

Runtime enforcement addresses that next boundary. AgentSpec is a research approach for expressing rules as triggers, conditions and interventions around agent execution. ShieldAgent explores a complementary approach: deriving verifiable rules from policy documents and checking an agent's action trajectory against them. These approaches point towards a platform capability that evaluates proposed actions independently of the agent proposing them. The domain still owns the policy, and any translation from policy prose into executable checks needs validation for omissions and ambiguity.

The details reveal the quality of the design. If two requests arrive together, the service must protect the cumulative limit while checking and applying each credit. If a proposal changes after review, its previous approval should not silently carry over. Appropriate transaction isolation or version checks help protect the gap between inspecting a record and changing it.

These are familiar engineering responsibilities, but an agent makes omissions easier to encounter through repeated and adaptive action. The model can propose a sensible operation while the system remains responsible for deciding whether that operation is valid now.

Follow the action through failure and recovery

Even a permitted operation may leave the workflow uncertain. The service might apply the credit and then lose the response before it reaches the agent. Repeating the call without checking could issue the credit twice; abandoning the task could leave the customer and the reviewer with an inaccurate account of what happened.

A durable operation identifier lets the receiving service recognise the same request and return its result without repeating the effect. The runtime can use that identifier to reconcile an uncertain outcome before deciding whether another attempt is appropriate. This follows established practices for safe API retries. Saving the agent’s progress does not provide the same protection, since resuming a workflow can rerun earlier operations.

Execution limits belong alongside that recovery design. The runtime bounds steps, retries and elapsed time; infrastructure and gateways can enforce resource quotas. Reaching a limit should lead to a defined outcome, such as gathering a missing fact, selecting a permitted alternative or returning the case for review. It should not be recorded as successful completion merely because the agent stopped.

Research is extending monitoring from the next action to the path an agent is taking. Predictive runtime monitoring, explored in ProbGuard, learns from execution traces to estimate risk and intervene before a violation. Its evaluations concern driving and household agents; applying it to enterprise workflows would require relevant traces and separate validation. For platform design, it suggests a reason to retain sequences of events as well as individual errors: an emerging pattern may warrant intervention before any single operation breaches a rule. The intervention threshold also becomes a design choice, because earlier stopping can interrupt legitimate work.

Assurance completes the arrangement. A reviewer needs the evidence used, the proposed action, the applicable policy and the observed result. Audit records and approval workflows provide that account, while a named owner decides exceptions. The design has to include what happens when that owner is unavailable, because an unattended queue can consume the value created earlier in the process.

What the teams actually need to implement

Return to the customer-credit example. The agent may investigate a complaint and propose a remedy, but a credit may be issued only for the authenticated customer, within the applicable entitlement and with any required approval. Turning that sentence into a working constraint requires capabilities in the agent platform and rules in the customer-service domain. The following is an illustrative implementation, using the six layers above.

The first deliverable is a task configuration that carries the case identifier, permitted operations, evidence requirements and execution budget. The platform provides the mechanism for passing and enforcing that configuration. The domain team defines which records count as sufficient evidence, which remedies are available and what makes the case complete. Those definitions need versions so that a result can be traced to the rules in effect when it was produced.

Retrieval must then preserve the customer's identity and access scope as it reaches account records and policy documents. The platform can provide connectors, identity propagation and source metadata, while the source services enforce their own access rules. The domain team identifies the authoritative policy and determines which account relationships matter. Filtering the prompt after an unrestricted query would leave the access boundary in the wrong place.

The next deliverable is a narrow action interface. The agent submits a proposed credit through a defined tool rather than receiving a general-purpose account-write capability. The platform validates the request structure, checks the permitted tool scope and carries the caller's identity to the receiving service. The domain service decides whether the customer is eligible and whether the proposed amount fits the remaining entitlement. Shared infrastructure makes the check unavoidable; domain logic supplies its business meaning.

The same division applies to approval. The platform needs a review mechanism that records who approved which proposed action and prevents execution before that approval is present. The domain specifies when approval is required, who is authorised to give it and which changes invalidate it. The implementation must bind approval to the customer, amount and proposal version so that an approved request cannot be altered on the way to execution.

Some controls must remain inside the receiving service. Only the credit service can reliably check the current balance, account for concurrent requests and apply the credit within the same protected operation. The platform should pass a stable operation identifier and reconcile uncertain outcomes; the service must recognise a repeated request and avoid applying it twice. A retry limit in the agent runtime cannot substitute for this transaction behaviour.

This gives the teams a concrete integration contract to build and test: a scoped task, a proposal tied to evidence, an approval tied to that proposal, an authorised service call and a verifiable result. The platform supplies reusable enforcement and records the sequence. The domain supplies entitlement rules and authoritative business state. If the existing service cannot enforce those rules or recognise duplicate requests, that service work belongs in the delivery scope before autonomous execution is enabled.

The final deliverable is feedback that both teams can use. The platform records tool calls, policy decisions, approvals, retries and outcomes under a common case reference. The domain defines whether the remedy was correct and the complaint was resolved. Evaluation can then distinguish a reasoning problem from a missing business rule, a weak interface or a recovery failure, and direct the improvement to the component that owns it.

The design review becomes specific: where does each rule live, which component rejects a violation, and how is the result demonstrated? Those answers translate the constraint model into a platform backlog, a domain backlog and tests of the boundary between them. They also expose which apparently small use cases depend on capabilities the enterprise has yet to build.

Expand autonomy when the whole workflow earns it

The purpose of constraints is to make useful delegation sustainable. A system that blocks every difficult case may avoid some errors while placing an unacceptable burden on reviewers. A narrow retrieval policy may reduce cost while excluding the evidence needed for a correct decision. Constraints therefore need evaluation against both failure and successful work.

Lessons from the trenches

Five practices have been helping me turn constraints into something agents can work with—and I can evaluate.

1. Make design principles explicit early.

I write down the choices that should guide the solution. If completing a task appears to require breaking a principle, the agent should surface the conflict and analyse the alternatives. Changing the principle then becomes a conscious decision.

For example, keeping business rules in the domain service creates a clear boundary. Duplicating a rule in a client may complete a feature, but it also changes the architecture. That tradeoff deserves a decision.

2. Give “done” measurable conditions.

For development agents, I include performance targets, execution limits and, where relevant, binary size in the specification. Whenever the agent reports completion, I expect evidence against those requirements.

A useful constraint names the threshold, measurement method and test conditions. I am willing to revise a target, but the tradeoff and authority to change it need to be explicit.

3. Build end-to-end validation from the start.

In my experience, agents focus on the task at hand and can miss upstream dependencies or downstream effects. I now create a test suite early that checks functional outcomes and non-functional requirements across the workflow.

I choose the triggers according to the project—for example, after a shared-interface change or before delivery. For keeping objectives and scope intact, I have found these checks more valuable than unit tests alone: they reveal whether individually successful changes still work together as intended.

4. Keep the review questions consistent.

Across planning, delivery and test scope, I return to the same questions: does the work cover the requirement, are assumptions visible, and do the tests address the relevant failures?

I automate checks that code can answer. Where an LLM helps assess judgment or coverage, I keep the criteria stable. Its judgment remains fallible, but a consistent review process gives me a basis for comparing and improving the agent over time.

5. Turn execution logs into a design backlog.

Much of my current project backlog comes from reviewing execution logs. I sort the improvements into three groups:

- Make work deterministic: turn recurring checks into rules or validators.

- Improve reasoning: help the agent use evidence and choose its next action.

- Add constraints: prevent wasted effort or an unproductive path.

Logs show actions, tool results and retries; they do not reveal the model's internal reasoning. They help me identify a specific change, then test whether it improves the outcome.

Together, these practices keep the agent's work connected to the project's intent. They also make changes to that intent visible, so autonomy can grow through deliberate choices and evidence.

Test the benefit of the constraint

Research into format restrictions has found that some restrictions reduce reasoning performance in the tested settings. Draft-Conditioned Constrained Decoding (DCCD) explores a two-stage approach: generate a draft, then condition the constrained output on that draft. This is a choice in the generation pipeline, with additional processing to weigh against output quality and fewer failed attempts. A structurally valid result still needs the business checks described earlier.

Across planning, generation and runtime monitoring, these studies show how much work is going into making constraints effective. They also give teams specific alternatives to evaluate: which decisions can be checked before execution, which boundaries need enforcement at the action, and where an extra checking step earns its cost. Established service controls and emerging research techniques can contribute to the same architecture, with each evaluated against the failure it is intended to prevent.

For leadership, the relevant unit is the completed workflow. Compare correct completion, policy violations, unnecessary blocking, elapsed time and the effort needed for review and recovery. Cost per correctly completed task should include failed attempts and human correction. Otherwise, an inexpensive agent can appear productive while creating expensive work elsewhere.

The evaluation should follow the same path as the architecture. Start with ordinary cases, then introduce stale records, conflicting evidence, missing approval, duplicated requests and timeouts after an action has succeeded. These cases show whether the boundary survives conditions that a demonstration may never encounter. They also reveal whether the exception route is usable by the people expected to operate it.

That evidence provides a basis for expanding authority selectively. Reliable investigation may justify automated preparation of a proposal. A narrow class of reversible actions may later qualify for execution, provided the outcome can be verified and failures contained. Access to one class of action should not automatically extend to others with different consequences.

The executive conversation: are we creating the conditions for enterprise autonomy?

For executives, the central question is whether today's investments and architectural choices are creating an enterprise capable of delegating meaningful work to agents. That means judging the direction of the system: which business outcomes it serves, how authority is organised, what foundations can be reused and whether the operating model can sustain wider autonomy.

The AI team and operational leaders need to answer that question together. Technical capability creates an opportunity; process ownership, access, decision rights and investment determine how much of that opportunity the enterprise can use. The executive conversation should connect those choices across the portfolio.

The checklist brings those choices into one conversation. Its value lies in the connections between the answers. A team may have a sound technical approach but no authority to change the process around it. Another may have a compelling use case whose economics depend on human review growing much more slowly than adoption. Those dependencies should shape the investment decision before expansion makes them expensive to resolve.

Use the discussion to establish what the next stage requires from each part of the organisation. The AI and platform teams may need to strengthen shared foundations or change how authority is enforced. Operational leaders may need to redesign approvals, assign ownership across handoffs or decide how released capacity will be used. Leadership must resolve the priorities and funding that neither team can settle alone.

The outcome should be a small set of explicit commitments: which capability will be developed, which operating arrangement will change, who owns each decision and what would justify continuing. Revisit those commitments as evidence accumulates. A constraint that once made delegation acceptable may later become an unnecessary bottleneck; removing it should be a considered change to the operating model, supported by what the organisation has learned.

These conversations should produce decisions about architecture, funding, ownership and organisational change. A roadmap is credible when it explains how each investment makes a further degree of useful autonomy achievable, including the changes required from the business. A growing collection of demonstrations does not establish that connection by itself.

Enabling constraints provide a way to make that ambition practical. They give teams room to develop new capabilities while making the limits of delegation explicit and sustainable. The executive responsibility is to ensure that technology, authority and the organisation's capacity to learn advance together. Those are the conditions under which agents can change how the enterprise operates.