How Do We Test Board-Level AI Understanding?

Boards are increasingly expected to oversee AI-related strategy, risk and investment decisions.

Yet many directors have no structured way of demonstrating, or improving, their understanding of AI.

Regulators and stakeholders are starting to ask not just whether AI is being used responsibly within a firm, but whether the board itself is equipped to govern it.

Without some form of assessment, firms are relying on assumption rather than evidence.

That gap is why board-level AI literacy needs to be actively tested, not simply taken on trust.

Why assumption is not good enough

Most boards already believe, informally, that they understand AI well enough to oversee it.

Directors read press coverage, attend briefings and sit through vendor presentations. That builds familiarity, but familiarity is not the same as competence.

Governance codes and supervisory expectations have long required boards to evidence collective competence across financial, risk and conduct matters. AI is increasingly treated the same way, particularly as firms adopt AI in underwriting, trading, claims and customer-facing decisions.

The UK government's AI Skills Compact reinforces this shift by framing workforce AI capability, including at senior leadership level, as a shared national priority rather than a discretionary training topic.

Against that backdrop, self-reported confidence is a weak signal. A director may feel comfortable discussing AI in general terms while lacking the specific understanding needed to ask the right oversight questions about a particular model, use case or third-party tool.

Assessment closes that gap between perceived and actual understanding.

Adapting tools boards already use

Boards do not need a new discipline to assess AI literacy. They need to extend tools already used for board effectiveness reviews.

Common adaptations include:

• Skills matrices. Many boards already maintain a matrix mapping director skills against strategic needs. Adding AI literacy as a tracked competency, alongside finance, risk and conduct, makes gaps visible at a glance.
• External board evaluations. Firms already commission periodic independent reviews of board effectiveness, often every one to three years. These can be extended to include structured questions on AI governance, rather than treated as a separate exercise.
• Structured interviews. One-to-one interviews conducted by the chair, senior independent director or an external facilitator can probe how directors think about AI risk and oversight, not just what they know.
• Scenario-based discussion. Presenting the board with a realistic AI use case, such as an AI-assisted credit decision or fraud detection tool, and observing the questions directors ask reveals far more than a written questionnaire.

These methods work because they test applied judgement rather than recall. A director does not need to explain how a model works technically. They need to demonstrate they can identify where oversight, challenge or independent assurance is required.

Where AI genuinely helps

AI tools have a supporting role in this process, not a decision-making one.

AI can help design scenario-based assessments by generating realistic, sector-relevant use cases quickly, drawing on patterns across many organisations rather than relying on a single facilitator's experience.

It can also help structure and benchmark responses, for example by identifying common gaps across a cohort of directors, or tracking how understanding develops over successive assessment cycles.

For organisations running assessment across multiple boards or committees, AI can reduce the manual effort involved in collating responses and identifying themes, freeing facilitators to focus on the conversations that matter.

What AI should not do is make the final judgement on whether a director is competent to govern AI risk. That determination involves context, seniority, the specific risks the firm faces and the collective balance of the board. It remains a human governance decision, typically owned by the chair and supported by independent advisers.

Making assessment part of governance, not compliance theatre

Assessment only works if it is treated as a development tool rather than a test to pass or fail.

A few practical principles help:

• Keep the process proportionate and non-punitive, so directors disclose gaps honestly rather than performing confidence.
• Feed results into individual and collective board development plans, not just a compliance file.
• Refresh the assessment periodically. AI capability, use cases and regulatory expectations are moving quickly, and a one-off assessment becomes outdated within a year or two.
• Align timing with the existing board effectiveness review cycle where possible, rather than creating a separate, competing process.

Handled this way, testing board-level AI understanding becomes a natural extension of existing governance practice, rather than an additional burden layered on top of it.

Example

A London-based specialty insurer preparing for its annual board effectiveness review decides to include an AI literacy component for the first time.

The chair commissions an external facilitator to run scenario-based discussions with the board, covering an AI-assisted underwriting decision tool.

Rather than asking directors to explain how the model works, the facilitator tests their ability to ask the right oversight questions: how the tool's decisions are monitored, what happens when it disagrees with an underwriter, and how model drift would be identified and escalated.

The exercise reveals uneven understanding of AI risk concepts among directors.

The board agrees a targeted development plan and commits to repeating the assessment annually alongside its standard effectiveness review.

FAQs

• Is testing board AI understanding a regulatory requirement?

  It is not always an explicit, named requirement, but it is increasingly expected as part of broader governance and competence obligations. The UK government's AI Skills Compact and evolving supervisory focus on AI oversight mean boards are more likely to be asked to evidence their understanding, even where no single rule mandates a specific test.

• How often should board AI literacy be assessed?

  Most firms align formal assessment with their existing board effectiveness review cycle, typically annually or every one to three years for independent evaluations. Given how quickly AI capability and risk evolve, lighter-touch checks between formal cycles are also worthwhile.

• Can AI tools assess the board's understanding of AI?

  AI tools can help design questions, build realistic scenarios and benchmark responses across a board or cohort. However, the final assessment of an individual director's or board's competence should remain a human governance judgement, usually supported by the chair and independent facilitators.