How Should DA Teams Attribute Operational Improvements to AI?

Turnaround time falls after an AI-supported workflow goes live. It is tempting to report the whole improvement as AI value.

During the same period, however, volumes may have changed, staff may have gained experience, a coverholder may have improved its submission, or a separate process redesign may have removed delay. Measuring an outcome shows that something changed. Attribution asks how much confidence the organisation has that AI contributed to that change.

A useful answer does not always require an elaborate experiment. It does require a comparison that reflects business as usual, evidence about other causes, and language that matches the strength of the evaluation.

Improvement can have several causes

A simple before-and-after comparison is a valuable starting point. It shows whether processing time, correction rates, queue age or another agreed outcome moved after implementation. It cannot, by itself, explain why.

Delegated authority workflows rarely remain static. Reporting volumes vary through the month and year. The mix of coverholders, products and file complexity changes. New staff join, experienced reviewers learn the process, and upstream or downstream systems are updated. Any of these factors can influence the same measure as the AI intervention.

Start by writing a short causal pathway. For example: AI interprets variable field names, which reduces manual mapping effort, which shortens processing time without increasing corrections. This makes the claim testable. It also identifies evidence needed at each step and competing explanations, such as a simpler mix of bordereaux or an additional reviewer.

A credible comparison approximates business as usual

The central question is what would probably have happened without the AI-supported change. This alternative is the counterfactual. It cannot be observed directly, so the team needs a reasonable proxy.

One option is a comparable workflow that has not yet changed. A staggered rollout can compare similar coverholder groups during the same reporting cycles before extending the service. Another option is a longer time series that shows whether the result departs from established seasonal and volume patterns. Where rollout timing permits, alternating suitable batches can provide an even closer comparison.

Comparability matters more than superficial similarity. Two groups should be assessed for volume, record count, source quality, product mix, exception complexity and staffing. The same definitions and measurement windows must apply to both. If one group contains larger or more difficult bordereaux, an unadjusted average can attribute case mix to the technology.

Contribution analysis works when clean controls do not

A controlled comparison may be impractical where the service changes across the whole portfolio at once or only a few workflows exist. The team can still build an evidenced contribution claim.

Contribution analysis begins with the causal pathway and tests whether its expected steps occurred. Did AI handle the intended tasks? Did manual mapping activity fall? Did the released effort appear in the workflow stage expected? Did quality remain within tolerance? The team then examines evidence for alternative causes, including staffing, demand, training, source changes and other releases.

Operational data can be combined with reviewer observations, change records and representative case review. AI may help group logs or summarise evidence, but it should not judge its own contribution. Analysts, service owners and DA specialists need to challenge the claim and record uncertainty.

Confidence should match the evaluation design

The strength of language should follow the strength of evidence. A well-matched concurrent comparison may support a quantified impact estimate. A before-and-after analysis with several uncontrolled changes may support only a statement that AI probably contributed to the outcome.

Reports should state the period, population, comparison, adjustments, data limitations and other material changes. They should also distinguish the measured result from interpretation. False precision weakens trust, while a transparent range or confidence statement helps decision-makers understand the evidence.

Evaluation should be proportionate. A small reversible use case may justify a modest comparison. A large investment or high-impact workflow merits stronger analytical design, ideally planned before rollout. The objective is a defensible decision about continuing, changing or scaling the service, rather than proving that AI deserves all credit.

Example

A hypothetical managing agent introduces AI-assisted mapping to half of a group of broadly comparable coverholders. The remaining workflows continue under the established process for two reporting cycles before their planned rollout.

The DA operations lead and finance analyst compare turnaround time, correction rate and review effort using the same definitions. They adjust for record volume and identify one staffing change that affected the comparison group.

The report concludes that the AI-supported path made a material contribution to lower mapping effort, but presents the turnaround estimate as a range because the coverholder groups were not identical.

FAQs

• Is a before-and-after comparison enough to attribute value to AI?

  It establishes whether an outcome changed, but it cannot rule out changes in volume, complexity, staffing or process. Additional comparison or contribution evidence strengthens attribution.

• What if no comparable control workflow exists?

  Use repeated measurements, test the expected causal pathway and examine alternative explanations. Report the result as a contribution claim with appropriate uncertainty rather than definitive causation.

• Does attribution require a randomised trial?

  No. Randomisation can provide strong evidence but is often impractical. A matched comparison, staggered rollout, time-series design or contribution analysis may be proportionate to the decision and available data.