I’ve spent the final a number of years watching enterprise collaboration instruments get smarter. Be part of a video name right now, and there’s a great probability 5 – 6 AI brokers are working concurrently: transcription, speaker identification, captions, summarization, job extraction. On the product aspect of it, every agent will get evaluated in isolation. Separate dashboards, separate metrics. Transcription accuracy? Test. Response latency? Test. Error charges? All inexperienced.
However here’s what I constantly observe as a UX Researcher: customers are annoyed, adoption stalls, and groups try to establish the basis trigger. Per the metrics, the dashboards look fantastic. Each particular person element passes its assessments. So, the place are customers actually struggling?
The reply, nearly each time, is orchestration. The brokers work fantastic alone. They crumble collectively. And the one manner I’ve discovered to catch these failures is thru person expertise analysis strategies that engineering dashboards have been by no means designed to seize.
The Orchestration Visibility Hole
Right here’s an instance of gaps that want a deeper understanding via person analysis: a transcription agent studies 94% accuracy and 200-millisecond response occasions. However what the dashboard doesn’t present is that customers are abandoning the characteristic as a result of two brokers gave them conflicting details about who stated what in a gathering. The transcription agent and the speaker identification agent disagreed, and the person misplaced belief in the entire system.
This drawback is about to get a lot greater. Proper now, fewer than 5% of enterprise apps have task-specific AI brokers inbuilt. Gartner thinks that’ll leap to 40% by the top of 2026. We’re headed towards a world the place a number of brokers coordinate on nearly the whole lot. If we can not determine how you can consider orchestration high quality now, we will probably be scaling damaged experiences.
UX Analysis Strategies Tailored for Agent Analysis
Normal UX strategies want some tweaking if you end up coping with AI that behaves in another way every time. I’ve landed on three approaches that truly work for catching orchestration issues.
1. Suppose-Aloud Protocols for Agent Handoffs
In conventional think-aloud research, you ask individuals to relate what they’re doing. For AI orchestration, I layer in what I name system attribution probes at key handoff factors. I pause and ask contributors to explain what they consider simply occurred behind the scenes, then map their responses towards the precise agent structure. Most customers are unaware that separate brokers deal with transcription, summarization, and job extraction. When one thing goes mistaken: a transcription error, as an example, they blame “the AI” as a monolith, even when the summarization and routing labored completely. Consumer suggestions alone received’t get you there. What I’ve discovered works is mapping what individuals suppose the system simply did towards what really occurred. The place these two diverge, that’s the place orchestration is failing. That’s the place the design work must occur
2. Journey Mapping Throughout Agent Touchpoints
Contemplate a single video name. The person clicks to hitch, and a calendar agent handles authentication. A speech-to-text agent transcribes, a show agent renders captions, and when the decision ends, a summarization agent writes up the assembly whereas a job extraction agent pulls out motion objects. A scheduling agent may then ebook follow-ups. That’s six brokers in a single workflow and 6 potential failure factors.
I construct dual-layer journey maps: the person’s expertise on high, the accountable agent beneath. When these layers fall out of sync – when customers anticipate continuity however the system has handed off to a brand new agent; that’s the place confusion units in, and the place I focus my analysis to unpack deeper points.
3. Heuristic Analysis for Agent Transparency
Nielsen Norman’s traditional heuristics stay foundational, however multi-agent techniques require us to increase them. “Visibility of system standing” has a distinct which means when six brokers are working concurrently; not as a result of customers want to know the underlying structure, however as a result of they want sufficient readability to get well when one thing goes mistaken. The purpose isn’t architectural transparency; it’s actionable transparency. Can customers inform what the system simply did? Can they appropriate or undo it? Do they know the place the system’s limitations are? These standards reframe orchestration as a UX drawback, not simply an infrastructure concern.
I’ve run heuristic evaluations the place the interface was polished and interplay patterns felt acquainted, but customers nonetheless struggled. The floor design handed each conventional test, however when the system failed, customers had no solution to diagnose what went mistaken or how you can repair it. They didn’t must know which agent triggered the problem. They wanted a transparent path to restoration.
Case Examine: Enterprise Calling AI
Right here’s an actual scenario I labored on that illustrates why orchestration high quality can matter as a lot as particular person agent efficiency.
An enterprise calling platform had deployed AI for transcription, speaker identification, translation, summarization, and job extraction. Each element hit its efficiency targets. Transcription accuracy was above 95%. Speaker identification ran at 89% precision. Job extraction caught motion objects in 78% of conferences. Nonetheless, person satisfaction was at 3.2 out of 5, and solely 34% of eligible customers had adopted the AI options. The product crew’s intuition was to enhance the fashions. I suspected the issue was in how the brokers labored collectively.
We ran think-aloud periods and found one thing the dashboards by no means confirmed: customers assumed that edits they made to dwell captions would carry over to the ultimate transcript. They didn’t. The techniques have been fully separate. Once I constructed out the journey map, plotting person actions on one layer and agent accountability on one other, I seen the timing misalignment instantly. Motion objects have been arriving in customers’ job lists earlier than the assembly abstract was even prepared. On the person layer, this seemed like duties showing out of nowhere. On the agent layer, it was merely the duty extraction agent ending earlier than the summarization agent. Each have been performing accurately in isolation. The orchestration made them really feel damaged.
Heuristic analysis surfaced a subtler challenge: when the interpretation and transcription brokers disagreed about speaker id, the system silently picked one. No indication, no confidence sign, no manner for customers to intervene.
This pointed us towards a design speculation: the issue wasn’t agent accuracy, it was coordination and recoverability. Reasonably than foyer for mannequin enhancements, we targeted on three orchestration-level modifications. First, we synchronized timing so summaries and duties arrived collectively, restoring context. Second, we constructed unified suggestions mechanisms that permit customers appropriate outputs as soon as somewhat than per-agent. Third, we added standing indicators displaying when handoffs have been occurring.
Three months later, adoption had jumped from 34% to 58%. Satisfaction scores considerably improved with scores of 4.1 out of 5. Assist tickets about AI options dropped by 41%. We hadn’t improved a single mannequin. The engineering crew didn’t suppose UX modifications alone might transfer these numbers. Truthful sufficient, truthfully. However three months of knowledge made it arduous to argue. Agent coordination isn’t simply an infrastructure drawback. It’s a UX drawback, and it deserves that stage of consideration.
A Three-Layer Analysis Framework
Based mostly on what I’ve seen throughout a number of deployments, I now suggest evaluating orchestration on three ranges. Layer one is technical metrics: latency, accuracy, and error charges for every agent. You continue to want these. They catch component-level failures. However they can not see coordination issues.
Layer two is behavioral indicators. Monitor the place customers abandon workflows, how usually they revise AI-generated outputs, and whether or not they come again after their first expertise. These patterns trace at orchestration points with out requiring direct person suggestions.
Layer three is qualitative analysis. Do customers perceive what the brokers are doing and why are they doing it? Do they belief the outputs? Does the entire system really feel coherent and accessible or disjointed? McKinsey’s 2025 AI survey discovered that 88% of organizations use AI someplace, however most haven’t moved previous pilots with restricted enterprise influence (McKinsey, 2025). I think an enormous a part of that hole comes from orchestration high quality that no one is measuring correctly.
What This Means for Product Groups
In most organizations I’ve labored with, UX researchers and AI engineers have restricted collaboration. Engineers tune particular person brokers towards benchmarks. UX researchers check interfaces. No person owns the house between brokers the place coordination occurs. That hole is precisely the place these failures dwell.
Deloitte estimates {that a} quarter of corporations utilizing generative AI will launch agentic pilots this 12 months, with that quantity doubling by 2027 (Deloitte, 2025). Groups that implement orchestration analysis early could have an actual benefit. Groups that don’t will preserve questioning why their AI options are usually not touchdown with customers. The funding required shouldn’t be huge. It consists of UX researchers in orchestration design discussions, constructing telemetry that captures agent transitions, and working common research targeted particularly on multi-agent workflows.
Conclusion
As AI merchandise evolve from single assistants to coordinated agent techniques, the definition of “working” has to evolve with them. A set of brokers that every cross their particular person benchmarks can nonetheless ship a damaged person expertise. Efficiency dashboards received’t catch it as a result of they’re measuring the mistaken layer. Consumer complaints received’t make clear it as a result of individuals blame “the AI” with out understanding which element failed or why.
That is precisely the place UX analysis earns its seat on the desk. Not as a closing test earlier than launch, however as a self-discipline woven all through the product lifecycle. UXR helps groups reply the earliest questions: Are we fixing the correct drawback? Who’re we fixing it for? It shapes success metrics that mirror actual person outcomes, not simply mannequin efficiency. It evaluates how brokers behave collectively, not simply in isolation.
UX analysis reveals you what earns belief and what chips away at it. It makes certain accessibility will get inbuilt from the beginning, not bolted on later when the system is just too tangled to repair correctly. None of that is separate work. It’s all linked, every layer feeding into the following. And as AI techniques get extra autonomous, extra opaque, this sort of rigor isn’t elective. The issue is, when groups are transferring quick, analysis looks like a pace bump. One thing to circle again to after launch.
However the price of skipping it compounds shortly. The orchestration issues I’ve described don’t floor in QA. They floor when actual customers encounter actual complexity, and by then, belief is already broken.
AI techniques are solely getting extra advanced, extra autonomous, and extra embedded in how individuals work. UX analysis is how we preserve these techniques accountable to the individuals they’re meant to serve.
Continuously Requested Questions
This is without doubt one of the commonest frustrations I see in enterprise AI. Particular person brokers cross their benchmarks in isolation, however the true issues present up when a number of brokers must work collectively. Orchestration failures occur on the handoffs, like when a transcription agent and speaker identification agent disagree about who stated what, or when job extraction finishes earlier than summarization, and customers obtain motion objects with no context.
These coordination points by no means seem on component-level dashboards as a result of every agent is technically doing its job. That’s exactly why person analysis strategies are important. They floor the place the expertise really breaks down in ways in which engineering metrics weren’t designed to catch.
Acquainted strategies like think-aloud protocols and journey mapping nonetheless work, however they want some changes for AI techniques. In think-aloud research, I’ve discovered it invaluable to incorporate what I name system attribution probes, moments the place you pause and ask customers to explain what they consider simply occurred behind the scenes. Journey maps profit from a dual-layer strategy: the person expertise on high and the accountable agent beneath.
Orchestration issues lie the place these layers are out of sync, and analysis ought to deal with figuring out and evaluating these points.
Longitudinal and ethnographic analysis are essential to know AI agent efficiency over time. Strategies like diary research and ethnography allow researchers to judge how customers work together with the AI and shift their utilization patterns throughout days or perhaps weeks, how that impacts belief, and establish new points which will emerge.
Preliminary impressions of an AI system usually differ from a person’s expertise after steady utilization. Longitudinal research reveal behaviors and workarounds that customers develop, and touchpoints that contribute to customers abandoning the characteristic solely.
Based mostly on what I’ve noticed throughout a number of deployments, I like to recommend evaluating orchestration on three ranges. Layer one covers the technical metrics comparable to latency, accuracy, and error charges for every agent.
Layer two focuses on behavioral indicators comparable to workflow abandonment charges, how usually customers revise AI-generated outputs, and if they’re returning customers. These patterns trace at orchestration points with out requiring direct person suggestions.
Layer three is qualitative analysis that evaluates if customers really belief the outputs, perceive what the brokers are doing, and understand the system as coherent somewhat than disjointed. All three layers working collectively reveal issues that any single layer would miss.
Actionable transparency shouldn’t be about educating customers the underlying structure of each agent. Customers want readability and the flexibility to know what the system simply did, appropriate or get well from errors when one thing appears incorrect, and perceive the place the system’s limitations are.
Actionable transparency offers customers clear paths to get well from errors.
When errors happen, customers have to be knowledgeable about what their choices are for resolving the problem and how you can transfer ahead. In apply, this might be unified suggestions mechanisms to let customers appropriate outputs as soon as, somewhat than individually for every agent. It may be standing indicators that floor when handoffs are occurring, or undo performance that works throughout your complete system. The purpose is to design for recoverability. When orchestration breaks down, customers can regain management and belief.
Crucial shift is recognizing that the house between brokers, the place coordination occurs, wants an proprietor. In most organizations I’ve labored with, engineers tune particular person brokers towards benchmarks whereas UX researchers check interfaces. No person owns that hole, and that’s precisely the place orchestration failures are inclined to dwell.
To shut this hole, groups ought to deliver UX researchers into orchestration design discussions early, not simply on the finish for interface testing. They need to construct telemetry that captures agent transitions and handoff factors, not simply particular person agent efficiency. They need to run common research targeted particularly on multi-agent workflows somewhat than treating AI as a single monolithic characteristic. This does require intentional cross-functional collaboration to construct higher AI-products.
Priyanka Kuvalekar is a Senior UX Researcher at Microsoft, main mixed-method analysis for Microsoft Groups Calling and agentic AI collaboration experiences
Login to proceed studying and luxuriate in expert-curated content material.
