What controls are needed before going live with AI?

Delegated authority organisations are increasingly piloting AI tools to process bordereaux and other delegated data. Many, however, lack a clear framework for what needs to be in place before switching such tools on for live production use.

Without defined controls, organisations risk processing inaccurate data at scale, losing audit trails, or removing human oversight from decisions that still require it.

Going live with AI is not simply a technical milestone. It is an operational and governance decision, and it deserves the same discipline that any change to how delegated data is processed would require.

Why go-live controls matter for AI in delegated authority

Delegated authority data carries real financial and regulatory weight. Bordereaux inform premium accounting, claims reserving, regulatory reporting and oversight of coverholders and MGAs. Errors that flow through unnoticed can affect all of these downstream processes.

When an organisation introduces AI to process this data, the stakes are similar to any other change affecting a core operational process. The difference is that AI tools can appear deceptively simple to switch on, particularly once a pilot has performed well on sample data. That apparent simplicity can tempt teams to skip the structured readiness work they would normally apply to a new system.

The purpose of go-live controls is to ensure that confidence in the tool is earned through evidence, not assumed because a demonstration went well.

How organisations traditionally control new system go-lives

Delegated authority operations have long-established practices for introducing new systems or automated processes safely.

Common traditional controls include:

• Parallel running, where the new process operates alongside the existing manual process for a defined period, and outputs are compared.
• User acceptance testing (UAT), where representative users confirm the system behaves as expected against known scenarios.
• Manual sample checking, where a proportion of processed records are reviewed by an experienced team member before the process is trusted at scale.
• Formal sign-off, where a named individual confirms the system is fit for live use.

These practices exist because any new system, automated or not, can behave differently from what was expected once it meets the full variety of real-world data. The same logic applies directly to AI-based bordereaux processing.

What changes when the system involved is AI

Traditional rules-based automation behaves consistently: given the same input, it produces the same output every time, and its logic can be inspected directly.

AI-based processing introduces some different considerations.

First, AI output can vary in confidence depending on how closely a given bordereau resembles the data the tool has seen before. A control framework needs to account for this variability rather than assume uniform accuracy across all submissions.

Second, exception thresholds become more important. Rather than a fixed rule failing outright, an AI tool may produce a low-confidence transformation that looks plausible but requires review. Controls need to define what confidence level triggers human review.

Third, explainability matters. When the AI maps a field or flags a value, the team overseeing the process should be able to understand why, particularly when justifying decisions to internal audit or Lloyd's oversight functions.

None of this means AI requires more control rigour than traditional automation. It means the same underlying control principles, testing before trust, defined ownership, and ongoing review, need to be adapted to how AI actually behaves.

Core controls to have in place before go-live

Four categories of control should be in place before any AI-based bordereaux or data processing tool goes live.

Data quality controls. Define validation checks on both the data going into the AI tool and the data coming out of it. Do not assume that because the AI transformed the data, the output is automatically correct. Set expected ranges, mandatory field checks and consistency rules that flag anomalies for review.

Governance and sign-off. Assign sign-off responsibility to a named individual within the DA oversight function, not a general team. This person should understand both the underlying data and the operational risk involved, and should formally confirm the tool is ready to move from testing to live use.

Exception handling procedures. Define what happens when the AI cannot confidently process a record. This includes the confidence threshold that triggers review, who reviews flagged exceptions, and how those exceptions are resolved and fed back into the process.

Ongoing monitoring. Establish a monitoring routine that continues after go-live, comparing AI output accuracy against expectations or against the previous manual process. Monitoring intensity can reduce over time as confidence builds, but it should never stop entirely.

Taken together, these four categories give an organisation a defensible, evidence-based answer to the question of whether it was ready to go live, rather than a judgement made on optimism alone.

Example

A Lloyd's managing agent is preparing to go live with an AI tool that transforms monthly bordereaux from a marine cargo coverholder into the agent's internal data schema.

Before switching from manual processing to the AI tool, the agent's DA oversight team defines a set of go-live controls: a data quality gate that flags any transformed record falling outside expected ranges, a named sign-off owner in the oversight team who reviews the first three live cycles, an escalation path for bordereaux the tool cannot confidently map, and a monthly monitoring report comparing AI output accuracy against the previous manual process.

The managing agent goes live with confidence because clear controls are in place to catch errors early, assign accountability and demonstrate to internal audit and Lloyd's oversight requirements that the transition to AI processing was properly governed.

FAQs

• Do we need to fully trust the AI before going live?

  No. Trust should be built incrementally through controls such as parallel running and sampled review, rather than assumed at the outset. Most organisations start with closer oversight and reduce it gradually as the tool demonstrates consistent, reliable performance on their own data.

• Who should be accountable for sign-off before go-live?

  Accountability should sit with a named individual within the delegated authority oversight function, rather than being described generally as "the team." That person should understand both the underlying data and the operational risk involved, so they can make an informed judgement about readiness.

• How long should enhanced monitoring continue after go-live?

  Monitoring intensity typically reduces over time as confidence in the tool builds. Some baseline level of monitoring, however, should remain in place permanently, rather than stopping once the initial transition period ends.