AI Agents as Users: A Journey Validation Framework

Consider a hypothetical subscription change. A customer asks an AI agent to move a team from a monthly plan to an annual plan if the change saves money and preserves every feature. The agent finds the pricing page, signs in, selects an annual option, accepts the confirmation, and reports success.

The transaction completed. That does not tell the product team whether the journey worked.

The annual price may exclude an add-on that was bundled with the monthly plan. The confirmation may contain a renewal condition that the agent left out of its summary. The account may now be committed for longer than the customer intended. If the service records only the conversion, its analytics will show a clean success while the customer is left to discover the consequences later.

This example is hypothetical, but the research problem is immediate. An external agent can now search, compare, navigate, complete forms, and attempt transactions on services that were designed around direct human interaction. The person still owns the goal, the account, the money, and the consequences. The agent performs some or all of the work. The service decides what information and actions are available to both.

Treating that arrangement as an ordinary automated session leaves important questions unanswered. What did the person authorize? Which constraints did the agent infer? What could it actually read? When did preparation become execution? Could the person understand the result and reverse it?

Those questions change the object of validation. Teams still need to study the interface, but they also need evidence about the relationship among the person, the agent, and the service over the whole journey.

A completed task can still be a failed journey

Most product analytics attach success to an observable end state: a booking exists, a form was submitted, a plan changed, or a payment cleared. Traditional usability work adds richer evidence by observing how a person reaches that state, where they hesitate, what they misunderstand, and whether they can recover from mistakes.

Agent-mediated journeys loosen the connection between those two kinds of evidence. The actor generating events may not be the person who set the goal. The person may see only a short summary after the service has changed state. An agent can reach the expected endpoint while losing a preference, exceeding its authority, or hiding a material condition in the process.

This is adjacent to, but distinct from, evaluating an agent product. Fred’s guide to usability testing AI agents covers issues such as variable outputs, transparency, and handoffs when the agent itself is the product under study. Here, the product could be a bank, marketplace, booking site, healthcare portal, or SaaS account. The research question is how that service behaves when an outside agent becomes one of its operators.

Nielsen Norman Group describes agents as functional users because they pursue goals, perceive an environment, act, and encounter success or failure. That framing is useful as long as it does not erase the human principal. A person and an agent can experience the same journey differently. The agent may find a machine-readable control easy to operate while the person has no useful preview of its consequences. Conversely, a service may work well for a person and expose too little structure for reliable agent use.

The most serious failures often appear in the gap between those experiences.

A vague request can become a narrower objective during interpretation. “Find the cheapest suitable flight” can turn into “select the lowest displayed fare,” dropping baggage, arrival time, or accessibility requirements. An authenticated agent may also possess technical permission to submit a change that the person meant it only to prepare. Later, the service can produce an accurate confirmation while the agent compresses it into a misleading reassurance.

Recovery exposes another gap. Suppose, again hypothetically, that an agent updates three account settings and times out on the fourth. A generic error page does not reveal which changes persisted. The person cannot make an informed decision until the service exposes the actual state, and the agent cannot safely resume without the same evidence.

None of these failures is visible in a binary completion rate. Validation needs to examine correctness, authorization, comprehension, and recovery separately. A completed task should count as acceptable only when all four have adequate evidence for the level of consequence involved.

What an agent actually encounters

A website does not present one stable representation to every agent. Current guidance describes several interaction paths: visual interpretation of screenshots, inspection of HTML or the Document Object Model, use of the accessibility tree, and direct calls through APIs or tool definitions. An agent may combine them during one journey.

Each path reveals some information and hides something else.

A screenshot preserves visual grouping, prominence, and proximity. It also includes banners, overlays, responsive rearrangements, and decorative elements that can make a target ambiguous. Text embedded in an image may be visible to a person while remaining difficult to extract reliably. A visually prominent warning may have no programmatic relationship to the button it qualifies.

HTML and the DOM expose more structure, although structure only helps when it carries meaning. A native button, a correctly associated field label, an explicit heading hierarchy, and a programmatically exposed state give an agent stronger cues than generic containers with click handlers. Google web.dev’s guidance on agent-friendly sites discusses these practical interaction modes and the value of semantic elements, names, labels, and visible state.

The accessibility tree can remove visual noise and present roles, accessible names, states, and relationships. That creates a useful overlap between accessible implementation and agent legibility. The overlap needs careful wording. WCAG is established guidance for making web content accessible to people with disabilities. An agent completing a task through an accessibility tree does not establish WCAG conformance, and it does not replace research with disabled people.

Machine interfaces can reduce ambiguity further. An API or tool definition can expose a price, policy, available action, and expected parameters in a structured form. Reliability at the execution layer still leaves experience questions open. Which capabilities should the agent be allowed to invoke? Does a returned price include taxes? How is freshness communicated? What evidence reaches the person before and after the call? What happens when the interface succeeds technically but the requested action exceeds the delegated scope?

This is why “agent-ready” is too broad to function as a product requirement. Readiness depends on a particular task, account state, interface, model, permission set, and risk level. A service may support product discovery well and fail at authentication. It may make a transaction easy to execute while leaving cancellation terms hard to retrieve. A single successful demonstration says little about how the same journey behaves after a wording change, a stale session, or an unexpected account configuration.

The surrounding technical work is also still developing. The W3C AI Agent Protocol Community Group is exploring areas such as discovery, identity, communication, security, and metadata. Community Group activity is evidence of active work, not a completed W3C standard. A July 2026 preprint on agent-ready websites reports controlled experimental results, but its authors’ work remains preliminary and limited in scope. Product teams can learn from it without treating it as certification.

The practical research move is to inspect every representation used in the journey. When an agent misses a cancellation restriction, the team should find out whether the restriction was absent, visually isolated, poorly structured, stale, or dropped during summarisation. “The agent failed” is an observation. The representation and decision path explain what the product team can change.

Following authority, evidence, and responsibility

Before selecting a study method, map the journey from three perspectives. The Three-Party Journey Map is the only framework needed for this exercise. It keeps the discussion close to an actual task by asking what each participant is trying to achieve, what authority they hold, what evidence they need, and what responsibility they carry when the journey goes wrong.

The map works best when filled with concrete language. “The agent may manage the subscription” is difficult to test because “manage” could include viewing, comparing, preparing, buying, cancelling, or renewing. “The agent may compare annual plans and prepare a change, but the customer must approve any loss of features or commitment beyond twelve months” gives the team observable boundaries.

Return to the hypothetical subscription change. The original request contains three constraints: reduce cost, preserve features, and avoid a commitment longer than one year. The agent needs enough current product data to compare total cost rather than headline price. The service needs to distinguish a prepared change from an executed one. Before approval, the person needs to see the current plan, proposed plan, feature differences, amount due, renewal behaviour, and cancellation conditions in a form they can understand.

At that point, a comprehension check is more useful than asking whether the confirmation “looks clear.” Ask the participant to explain what will change, what will be charged, when renewal occurs, which features are affected, and how the decision can be reversed. A legally complete block of text can still produce poor comprehension. The study should record the participant’s understanding, not the team’s opinion of the copy.

The evidence chain continues after approval. For a consequential action, preserve the request, the constraints the agent used, the alternative selected, the person’s authorization, the service response, the resulting account state, and the available recovery route. This record supports several audiences. The customer can inspect and challenge the result. Support can diagnose a dispute. Researchers can distinguish interpretation failures from service failures. Product and engineering teams can decide where to add a guardrail.

Responsibility deserves the same precision. If an agent omits a condition that the service never exposed in a usable form, improving the model alone will not solve the journey. If the service provides the condition clearly and the agent discards it, the failure sits elsewhere. If the person gave contradictory constraints and the agent proceeded without asking, the handoff policy needs attention. Mapping responsibility keeps findings from collapsing into a generic request to “make the AI better.”

Test the handoffs, variability, and recovery

A credible validation plan combines human research, interface inspection, repeated agent runs, and deliberate disruption. The proportions depend on the task. A low-consequence information lookup may need lightweight checks. A journey involving money, sensitive data, health, legal terms, or difficult-to-reverse state changes needs stronger evidence and more direct human involvement.

Begin with the people who delegate the task. Interviews can uncover the shortcuts hidden inside apparently simple requests. “Best,” “affordable,” “nearby,” and “safe” often contain personal thresholds that an agent cannot derive reliably from the words alone. Ask participants which decisions they are comfortable delegating, which require a preview, what would make them take over, and what outcome they would consider unacceptable.

Moderation is particularly useful when the team needs to probe an unexpected interpretation or observe how someone responds to uncertainty. Stable, lower-risk flows may also benefit from unmoderated tasks at greater scale. Fred’s comparison of moderated and unmoderated research can help teams choose based on the research question rather than convenience.

Next, inspect the service at the steps that carry meaning or consequence. Look at the rendered page, DOM, accessibility tree, API response, tool schema, or whichever representation the agent uses. Check whether price components, time limits, prerequisites, state changes, and error conditions are attached to the correct object. Confirm that preparation and execution have different states. Verify what remains observable after navigation or session expiry.

Then repeat the journey. Change the wording of the request while keeping the intent stable. Vary account state, product availability, data freshness, permission, and constraint complexity. A run should capture more than the final answer: record the path taken, evidence consulted, assumptions made, correction attempts, service responses, and final state. Repetition reveals whether a capability is dependable or merely possible.

The failure taxonomy can remain plain. Did the agent fail to observe relevant information? Did it connect the information to the wrong option? Did it choose an action outside the request? Did execution affect the wrong object? Did it lose track of partial completion? Could the person understand and correct the result? These labels are more actionable than one aggregate success score.

Handoffs need their own tasks. Ask a participant to delegate, review a proposed action, interrupt it, change a constraint, reject a recommendation, take over midway, and inspect the record afterward. Observe whether the agent carries the revised constraint forward and whether the service reflects the real state. A handoff that works only when everything is going well offers little protection when it is actually needed.

Introduce disruptions on purpose. A price changes between comparison and approval. A permission expires after two of four changes. A required field is missing. Two customer constraints conflict. The service returns a success response but its account page shows the old value. These are test conditions, not claims about a specific live product. Their purpose is to expose whether the journey contains enough state information to prevent a confident but wrong continuation.

Recovery should be treated as a task with its own completion criteria. Can the agent and person determine what happened? Can they cancel, roll back, correct, or escalate? How much time and effort does recovery require? Does the service preserve the previous state or a trustworthy record? A reversible action still creates harm if reversal is hidden or expensive.

A team can start from Fred’s research template library for the human study, then add the agent runs, authority probes, representation inspection, and recovery conditions relevant to the journey. The template provides scaffolding. Risk analysis still has to come from the specific service and action.

Accessibility evaluation remains a parallel human obligation. Semantic markup may improve both assistive-technology support and agent operation, but the evidence serves different claims. Test accessibility with appropriate standards, expertise, and disabled participants. Report agent legibility separately so a technical success cannot be mistaken for proof of accessibility.

Turn the evidence into release conditions

The end of the study should produce a product decision, not a collection of interesting failures.

For each critical step, state the required human outcome, required agent behaviour, acceptable service state, and recovery expectation. Define which failures block release. An agent selecting the wrong comparison filter may be tolerable in an exploratory information task if the uncertainty is visible. The same pattern should block autonomous execution when it can create a financial commitment.

Intentional friction belongs where it protects authority. A preview, explicit approval, reauthentication, or forced handoff adds effort, but speed is not the only quality of an agent-mediated experience. Consequences, reversibility, confidence, and user preference should determine the level of autonomy.

Findings should point to specific owners. If the agent cannot identify the current price, improve the source data, semantics, or machine interface. If it sees the price and ignores a mandatory fee, investigate interpretation and comparison logic. If authority is ambiguous, change the delegation and confirmation flow. If a partial update leaves an unknown state, engineering work should expose transaction status and rollback. If a task cannot be supported safely, block or clearly constrain agent execution while preserving a direct human route.

Analytics can support monitoring after release, provided the team is precise about what it knows. Where lawful and technically feasible, distinguish apparent agent-mediated activity, the interface used, the action attempted, confirmation, later correction, and recovery outcome. Automated traffic also includes crawlers, scripted workflows, monitoring, and fraud. Labelling every automated request as a customer agent would create a misleading dataset.

The service owner still has a strategic choice. Supporting agent access may improve reach and convenience. Restricting some actions may be appropriate because of security, regulation, intellectual property, or business policy. Either position should be explicit. An ambiguous service can still receive agent traffic while offering neither reliable support nor effective protection.

A sound release decision might permit information retrieval, allow recommendations with visible sources, require human approval for prepared changes, and block autonomous execution for selected consequences. Another service may choose different boundaries. The important part is that the boundary follows evidence from a real journey rather than a general claim that the site is ready for agents.

If your team is preparing a consequential journey for agent-mediated use, bring one actual task into Fred’s decision-intelligence workflow. Connect the human research, agent runs, service evidence, contradictions, and release conditions in one reviewable case, then decide where autonomy is justified and where human control must remain explicit.

Source notes

• Nielsen Norman Group, “AI Agents as Users,” April 10, 2026: source. Used for the functional-user framing, interaction modes, accessibility overlap, intentional friction, and strategic choice to support or restrict agents.
• Google web.dev, “Build agent-friendly websites,” updated April 1, 2026: source. Used for engineering guidance on screenshots, HTML or DOM, accessibility trees, semantic elements, labels, and state visibility.
• UX Magazine, “Why AI Agent Sprawl Is a UX Problem,” July 3, 2026: source. Used for fragmented agent experiences, governance, continuity, and cross-system failure.
• W3C Web Accessibility Initiative, “WCAG 2 Overview”: source. Used to distinguish established accessibility guidance from agent-readiness claims.
• W3C AI Agent Protocol Community Group: source. Used as evidence of active exploration of interoperable agent discovery, identity, communication, security, and metadata. This Community Group work is not a finished W3C standard.
• Elnaffar and Rashidi, “Designing Agent-Ready Websites for AI Web Agents,” arXiv preprint v1, July 13, 2026: source. Used as preliminary controlled evidence. It is a preprint with limited experimental scope, not a certification basis.
• Nielsen Norman Group, “Designing AI Agents: 4 Lessons from China’s Qwen Agent,” May 8, 2026: source. Used as additional context for clarity and control.
• The Three-Party Experience Model, delegated task success formula, Dual-User Acceptance Matrix, Agent-Legibility Scorecard, Delegated Action Risk Ladder, and mixed-method protocol are original Fred editorial tools. They are not validated scientific instruments or compliance standards.