The Measurement Gap 

Much of the measurement gap comes from different teams all trying to answer different questions. Those building and operating an AI system need to have immediate evidence that it is accurate, reliable, secure, and being used as intended. Business leaders are working on a longer horizon, asking whether the same system is improving performance, reducing costs, or producing better outcomes for customers. 

Between these perspectives sits the operational layer: the workflow in which the AI is being used. This is where technical performance is translated into organisational change. A model may respond more quickly, for example, but does that reduce the time that is required to complete the entire process? An assistant may produce more content or code, but how much of that output is accepted, corrected, or discarded? 

The measurement gap is therefore not an absence of data. It is actually the distance between what an AI system is doing and what the organisation is achieving because of it. Technical measures establish whether the system works. Operational measures show whether the surrounding process is improving. Business measures determine whether that improvement is creating something the organisation values. 

The difficulty is that the first category is typically the easiest to capture. Usage can be observed immediately, while changes in customer retention, decision quality, or employee capacity may take longer to establish. Demonstrating those outcomes also requires a credible baseline and agreement over what success should like. Without them, activity can become a practical proxy for value, even when the relationship between the two remains uncertain. 

This distinction appeared repeatedly at Tech Show London 2026. Krish Ramineni, Founder and CEO at Fireflies.ai, observed that enterprise AI pilots can struggle to progress when they are not clearly connected to revenue, productivity, or important organisational KPIs. Michail Vamvakaris, Head of Data Science and MLOps at William Hill, similarly argued that measures should be agreed before development begins, with customer-facing applications assessed differently from internal tools. 

Closing the gap requires technology, product, operations, and business teams to build a shared view of what success looks like. Accuracy, adoption, and speed can tell an organisation a great deal about its AI systems. The difficulty begins when those measures are expected to demonstrate value on their own. 

Why Familiar Metrics Are Incomplete 

The measures found on many AI dashboards have become familiar because they provide useful information. Adoption shows whether a tool is reaching its intended users. Accuracy indicates how consistently a model performs. Speed, output volume, and cost help technical teams understand whether a system can operate efficiently at scale. 

Each measure answers a valid question. The limitation appears when it is interpreted without the wider context in which the AI is being used. 

Adoption Shows Participation, Not Impact 

Usage is often the first visible sign that an AI investment is actually gaining traction. It can tell an organisation how many people have activated a tool, how frequently they use it, and which features attract the most engagement. 

What it cannot show on its own is whether that activity is improving the work. High usage may reflect genuine utility, but it can also result from curiosity, internal targets, or absence of an alternative. Understanding its value requires organisations to look at where adoption is taking place, whether it is sustained, and what happens to the resulting output. 

A more useful question is not whether AI has been adopted, but where it is being used and what has changed as a result. 

Accuracy Needs to Reflect Consequence 

Accuracy is essential when evaluating any AI system, but an overall score can hide important differences between use cases and types of error. An incorrect meeting summary does not carry the same consequence as an incorrect credit, clinical, or security decision. 

Speaking at Tech Show London 2026, Rohit Dhawan, Global Head of AI at Lloyds Banking Group, highlighted that acceptable accuracy depends on the purpose and stakes of the application. Some use cases can accommodate a degree of uncertainty. Others require much greater assurance because they affect customers, money, or critical decisions. 

Headline accuracy should therefore be accompanied by measures such as error severity, false positives, false negatives, human overrides, and the groups affected. What needs to be addressed isnot only how often the system is correct, but what happens when it is wrong. 

Speed and Output Do Not Capture the Whole Process 

Faster responses and higher output can indicate that AI is improving a particular task. They do not necessarily show that the complete workflow has become more effective. 

A customer-service system might reduce handling time while creating more repeat contacts because issues were not fully resolved. A coding assistant might increase the amount of code produced while adding further review, testing, and security work. In both cases, the AI has accelerated one stage while moving effort elsewhere. 

This distinction also appeared in the Tech Show London discussions. One Lloyds Banking Group presentation warned that optimising customer service around time to completion could overlook the more important objective of retaining the customer. The efficiency measure was not incorrect; it was simply too narrow to represent the desired outcome. 

Automation and Cost Need the Right Denominator 

Automation rates can show how much work an AI system handles, but there is a difference between touching a task and completing it successfully. A high automation rate becomes less valuable if large numbers of cases still require escalation, correction, or human intervention. 

The same applies to cost. Cost per token, or inference, helps teams manage technical consumption, but it does not account for integration, infrastructure, monitoring, human review, or remediation. For business leaders, a more meaningful measure may be cost per successful outcome. 

That outcome will then differ between use cases. It could be a resolved customer enquiry, an accepted decision, a completed workflow, or a detected risk. The principle remains the same: the calculation should reflect the result the organisation is trying to achieve. 

Familiar Metric	What it tells the orgnisation	What it should be connected to
Adoption	Whether people are using the tool	Use within priority workflows and resulting performance
Accuracy	How frequently outputs meet a defined standard	The severity and business consequence of errors
Speed	How quickly the system responds	End-to-end process time and outcome quality
Output Volume	How much work the system produces	Acceptance, correction and rework
Automation Rate	How much activity AI handles	Successful completion and escalation rates
Unit Cost	The technical cost of each interaction	Total cost per successful outcome

 

Total cost per successful outcome 

These measures do not need to be removed from AI dashboards. They need to be connected to the effects they create further along the process. To make that connection, organisations must look beyond the performance of the model and examine the workflow in which its outputs are used. 

Measure the Workflow, Not Just the Model 

An AI model rarely operates in isolation. It sits within a workflow involving data, employees, existing systems, and operational decisions. Its output may be technically sound, but its value ultimately depends on what happens next. This makes the complete workflow, rather than the model alone, the more useful unit of measurement. 

Measurement should begin with the business need, then follow the process from the AI intervention through human review and operational action to the final outcome. This reveals whether AI has removed work, improved it, or simply moved effort into another part of the organisation. 

The comparison also needs to cover the workflow before and after implementation. A faster model response is valuable when it reduces end-to-end completion time. Increased output matters when it can be accepted without additional correction. A higher automation rate becomes meaningful when fewer cases require escalation. Relevant measures could therefore include first-time acceptance, human review time, rework, error severity, and cost per completed outcome. 

Defining those measures requires shared ownership. Technical teams can explain how the model performs, but product, operations, and business owners must establish what improvement looks like. Risk and governance teams may also need to determine which outcomes are acceptable, particularly where AI influences important decisions. 

At Tech Show London 2026, Edward Steele of the Met Office described a useful framework for connecting technology with real-world outcomes. Organisations should define the overall claim they are making for a project, establish a strategy for achieving it, identify the specific properties the solution must demonstrate, and collect evidence showing whether those properties have been delivered. 

Applied to AI, that means moving beyond a general ambition to “improve productivity” and defining the change more precisely. Which part of the workflow should improve? By how much? What evidence would demonstrate it, and what result would suggest the system needs to be reconsidered? 

Once the complete workflow is visible, the dashboard can begin to connect technical performance with operational change, human experience, and business value. 

A Four-layer Scorecard for AI Value 

Once the complete workflow is visible, organisations need a way to bring its different measures together. No single metric can show whether an AI system is technically reliable, operationally effective, and valuable to the business. A four-layer scorecard offers a more balanced view by connecting the intended outcome with the process, people, and technology required to deliver it. 

Layer	Question it answers	Example Measures
Business Outcome	Is the organisation achieving something it values?	Revenue, retention, cost-to-serve, risk reduction, decision quality
Workflow Performance	Is the underlying process improving?	Completion time, throughput, first-time resolution, rework, escalation
Human Outcome	How are employees or customers affected?	Workload, time redeployed, output acceptance, satisfaction, trust
System and Control Health	Is the AI operating reliably and responsibly?	Accuracy, drift, latency, availability, security, compliance, infrastructure cost

 

The scorecard should be designed from the business outcome backwards. This requires the organisation to define what the use case is expected to improve before deciding which technical measures belong on the dashboard. That outcome does not always need to be financial. Improving decision quality, reducing exposure to risk, or making a service more accessible may be equally valuable, provided the intended result is clearly defined. 

The remaining layers show how that value is being created. Workflow measures establish whether the process is becoming more effective. Human measures reveal how employees and customers experience the change. System and control measures confirm that the AI remains accurate, secure, available, and economically sustainable. 

Going Layer by Layer 

Consider an AI system supporting a contact centre. At the system layer, the organisation might monitor accuracy, response time, and availability. At the workflow layer, it could examine first-time resolution, escalation, and the time required to complete an enquiry. The human layer would consider satisfaction, employee workload, and the amount of output requiring correction. The business layer would then establish whether these changes have improved retention, reduced cost-to-serve, or achieved another agreed objective. 

Viewed separately, each layer provides only a part of the picture. Faster responses mean less if more enquiries are escalated. Higher automation may be less valuable if employee review time increases. Lower operating costs may not represent success if customer trust declines. The scorecard makes those relationships visible and helps organisations identify where an apparently positive result is being offset elsewhere. 

To remain useful, it should contain a small number of measures at each layer, with a named owner and agreed thresholds for scaling, redesigning, or stopping the use case. It should be applied at use-case level. An enterprise-wide AI score can obscure important differences between applications with distinct purposes, risks, and measures of success. 

Even a well-designed scorecard, however, can only show what is happening now. Demonstrating that AI has produced an improvement requires a credible point of comparison. 

Start With the Baseline 

Even a carefully designed scorecard cannot demonstrate improvement without a credible point of comparison. Current performance only becomes meaningful when an organisation understands what the same workflow looked like before AI was introduced. 

That baseline does not need to capture every aspect of the process. It should focus on the measures most closely connected to the intended outcome, such as completion time, error and rework rates, employee effort, escalation, cost per outcome, or customer satisfaction. These measures should be agreed before implementation so the organisation knows what change it expects to see and how that change will be assessed. 

Where possible, organisations can strengthen the comparison through a phased rollout, control group, or testing AI-assisted and existing processes alongside one another. They should also account for the other factors that could affect performance, including changes in demand, staffing, or the wider operating model. The aim is not to create a perfect experiment, but to build enough evidence to make a credible judgement. 

Together, the baseline and four-layer scorecard provide a clearer way to evaluate enterprise AI. Technical measures establish whether the system is reliable. Workflow measures show whether the process is improving. Human measures reveal how employees and customers are affected. Business measures determine whether the organisation is achieving the outcome that justified the investment. 

This gives leaders something more useful than a collection of performance indicators. It gives them a basis for deciding whether to scale, redesign, or stop a use case altogether. 

Your AI dashboard may not be measuring the wrong things. It may simply be stopping before it reaches the outcome that matters.