I
internationalbanker
Guest
- Thread Author
- #1
By Supriya Manchanda, Regulatory Partner, Darshan Chandarana, Tech Advisory Partner, and Danny Chamings, Technology Risk Partner, Grant Thornton UK
Financial-services boards are increasingly accountable for artificial intelligence (AI) systems that may operate with significant degrees of autonomy and that they may never fully understand—nor, in many cases, control.
As AI assumes more decision-making across financial services, directors do not need to become technologists. But they do need to know what these systems are doing, where the risks sit and when to intervene.
It’s a tough ask, one that is complicated when the technology in question belongs to someone else. Firms already rely on third-party models, cloud infrastructures and external providers, putting many of the systems shaping customer and business outcomes beyond their direct control.
So how does a board challenge a system it does not own, cannot fully interrogate and may have limited power to change?
This challenge is particularly acute because AI is not simply another technology-procurement decision. As systems become more autonomous, they could increasingly move from supporting human judgement to taking action within defined parameters. In many cases, people may increasingly oversee the parameters and outcomes of AI-driven processes rather than approve individual decisions.
For a financial-services board, that changes the very nature of oversight. Knowing that AI is being used is no longer enough. Directors have to understand what decisions it influences, how much autonomy it has, what controls surround it and what would cause a person to intervene.
When an external provider sits within that chain, boards need another layer of visibility. They should understand where important models originate, what infrastructure they depend on, how providers make and communicate changes, what data is being used and what happens if a service degrades, behaves unexpectedly or becomes unavailable.
The issue becomes more significant when large parts of the industry depend on the same providers, models or infrastructures. Concentration can create common points of failure. Greater use of similar underlying models could also potentially encourage correlated behaviour across firms.
The United Kingdom’s Critical Third Parties (CTP) regime already recognises the systemic risks that can arise from financial firms’ reliance on third-party providers, the disruption or failure of which could affect the wider financial system. AI introduces a much bigger concern: A regulatory regime primarily designed around the operational resilience of external providers can, in principle, extend to AI providers just as it has to cloud and infrastructure providers. But is the test sufficient for the providers that shape the substance of decisions rather than the infrastructures that enable them?
This is one of the issues behind the Financial Conduct Authority’s (FCA) Mills Review, published in July, which considers how AI could reshape UK retail financial services by 2030. The review raises the prospect of increasingly autonomous systems and recommends that the FCA secure and adapt the regulatory perimeter as the technology develops.
The answer does not necessarily have to be an entirely new AI regulatory regime. Existing frameworks, including the FCA’s Senior Managers and Certification Regime (SM&CR) and Consumer Duty, already place clear responsibilities on firms and their senior leaders. But these responsibilities become harder to discharge when part of the technology, and potentially part of the decision-making capability, is provided by organisations outside the traditional financial services’ regulatory perimeter.
Firms can outsource technology. They cannot outsource accountability.
No board can control every decision made by a global cloud, technology or AI provider. The priority is understanding where and how the firm depends on it, how significant that dependency is and what options it has if something goes wrong.
Boards should expect management to identify the firm’s most important AI dependencies, not simply provide a list of suppliers. Which business processes and customer outcomes would be affected if a provider failed or materially changed its technology? Do several important services rely on the same underlying provider? And how quickly could the firm respond?
Contracts matter here. Firms need to know what information they are entitled to if a provider makes significant changes to a model or service, what assurances they can obtain and whether they have enough visibility to manage the risks for which they remain accountable.
They also need a realistic contingency plan. If an important AI service becomes unavailable or starts producing unacceptable outcomes, is there another provider? Could the firm revert to a less automated process? Is its data portable? And how long would a switch actually take? An exit strategy that works on paper may look very different when tested.
For some providers, there may not be a straightforward substitute. Boards need to know that, too. They cannot eliminate every external dependency, but they can understand where the biggest exposures sit and make informed decisions about what they are prepared to accept.
Ultimately, AI may change how decisions are made, but it does not remove a firm’s accountability for the outcomes. As firms become more reliant on AI and the providers behind it, boards need enough understanding to know when that reliance has gone beyond the firm’s risk appetite.
The same principle applies to AI developed or controlled within the organisation. Boards do not need to understand the technical workings of every model. But they do need enough AI literacy to ask meaningful questions.
What level of autonomy has been given to the system? What could go wrong? What information would reveal that its behaviour had changed? When must a person intervene? And who is accountable for making that decision?
These questions are particularly important when AI affects customers. Whether a system is involved in credit decisioning, fraud detection, customer service or product design, firms need to demonstrate that they are delivering appropriate outcomes.
The information reaching boards, therefore, needs to evolve, too. A report showing where AI is deployed is useful, but it says little about whether those systems continue to operate safely. Boards should expect evidence on performance, customer and business outcomes, control effectiveness and changes in model behaviour over time.
Oversight should also reflect the level of risk. An internal productivity tool does not carry the same potential consequences as an autonomous system influencing a lending or insurance decision. Firms should understand where different uses sit on the autonomy spectrum and apply governance proportionately, with greater autonomy and potential impact demanding greater scrutiny.
There is a temptation to see all this governance as something that slows AI adoption. In practice, the opposite can be true. Many financial-services firms are already experimenting with AI. The harder step is moving successful use cases from pilots into important business processes.
That becomes difficult if the people ultimately accountable for the organisation lack confidence in how those systems are controlled. Clear guardrails can give boards and management the confidence to allow appropriate AI applications to operate with greater independence while concentrating scrutiny on areas where the potential consequences are higher.
This means being explicit about where human involvement remains necessary, what should trigger intervention and where AI can operate more independently.
There are significant opportunities in getting this right. Better use of AI and data could improve personalisation, make services more efficient and help address some of the information gaps that contribute to poor financial decisions. Responsible governance and innovation should therefore reinforce one another: The objective is to give firms confidence to progress, not simply another reason to say no.
The Mills Review looks towards 2030, but these are not 2030 problems. Boards should use the coming months to test whether their governance can cope with greater AI autonomy, rather than waiting for the regulatory position to be completely settled.
That starts with knowing where the firm’s most consequential AI uses and dependencies sit. Boards should understand how much autonomy those systems have, whether existing controls remain appropriate as that autonomy increases and where human judgement must remain in the process.
They should also ask management to identify the external dependencies beneath those systems and demonstrate that the firm has considered what happens when a provider changes, degrades or fails. That means looking beyond supplier due diligence towards concentration, contractual visibility, alternatives, contingency arrangements and viable exit strategies.
Risk and compliance functions should be capable of providing effective oversight of AI-driven activity, while internal audits should be equipped to provide meaningful independent assurance over the governance and control frameworks surrounding it. Crucially, those lines of defence need to evolve as quickly as the technology and the firm’s use of it. None of this requires board members to become AI experts. What is required is enough understanding to exercise the judgement and the capacity to challenge that their existing responsibilities already demand.
The next stage of AI adoption in financial services will not simply be about how quickly firms can deploy the technology. It will test whether they understand what their AI is doing, what lies underneath it, where their ability to control it ends and what they will do when something goes wrong.
For boards, approving AI is increasingly the easy part. Knowing when and how to challenge it is where the harder work begins.
Go to top
The post Approving AI Is the Easy Part. Can Financial-Services Boards Challenge It? appeared first on International Banker.
Financial-services boards are increasingly accountable for artificial intelligence (AI) systems that may operate with significant degrees of autonomy and that they may never fully understand—nor, in many cases, control.
As AI assumes more decision-making across financial services, directors do not need to become technologists. But they do need to know what these systems are doing, where the risks sit and when to intervene.
It’s a tough ask, one that is complicated when the technology in question belongs to someone else. Firms already rely on third-party models, cloud infrastructures and external providers, putting many of the systems shaping customer and business outcomes beyond their direct control.
So how does a board challenge a system it does not own, cannot fully interrogate and may have limited power to change?
Accountability despite limited control
This challenge is particularly acute because AI is not simply another technology-procurement decision. As systems become more autonomous, they could increasingly move from supporting human judgement to taking action within defined parameters. In many cases, people may increasingly oversee the parameters and outcomes of AI-driven processes rather than approve individual decisions.
Directors have to understand what decisions it influences, how much autonomy it has, what controls surround it and what would cause a person to intervene.
For a financial-services board, that changes the very nature of oversight. Knowing that AI is being used is no longer enough. Directors have to understand what decisions it influences, how much autonomy it has, what controls surround it and what would cause a person to intervene.
When an external provider sits within that chain, boards need another layer of visibility. They should understand where important models originate, what infrastructure they depend on, how providers make and communicate changes, what data is being used and what happens if a service degrades, behaves unexpectedly or becomes unavailable.
The issue becomes more significant when large parts of the industry depend on the same providers, models or infrastructures. Concentration can create common points of failure. Greater use of similar underlying models could also potentially encourage correlated behaviour across firms.
The United Kingdom’s Critical Third Parties (CTP) regime already recognises the systemic risks that can arise from financial firms’ reliance on third-party providers, the disruption or failure of which could affect the wider financial system. AI introduces a much bigger concern: A regulatory regime primarily designed around the operational resilience of external providers can, in principle, extend to AI providers just as it has to cloud and infrastructure providers. But is the test sufficient for the providers that shape the substance of decisions rather than the infrastructures that enable them?
This is one of the issues behind the Financial Conduct Authority’s (FCA) Mills Review, published in July, which considers how AI could reshape UK retail financial services by 2030. The review raises the prospect of increasingly autonomous systems and recommends that the FCA secure and adapt the regulatory perimeter as the technology develops.
The answer does not necessarily have to be an entirely new AI regulatory regime. Existing frameworks, including the FCA’s Senior Managers and Certification Regime (SM&CR) and Consumer Duty, already place clear responsibilities on firms and their senior leaders. But these responsibilities become harder to discharge when part of the technology, and potentially part of the decision-making capability, is provided by organisations outside the traditional financial services’ regulatory perimeter.
Firms can outsource technology. They cannot outsource accountability.
What can a board actually do?
The priority is understanding where and how the firm depends on it, how significant that dependency is and what options it has if something goes wrong.
No board can control every decision made by a global cloud, technology or AI provider. The priority is understanding where and how the firm depends on it, how significant that dependency is and what options it has if something goes wrong.
Boards should expect management to identify the firm’s most important AI dependencies, not simply provide a list of suppliers. Which business processes and customer outcomes would be affected if a provider failed or materially changed its technology? Do several important services rely on the same underlying provider? And how quickly could the firm respond?
Contracts matter here. Firms need to know what information they are entitled to if a provider makes significant changes to a model or service, what assurances they can obtain and whether they have enough visibility to manage the risks for which they remain accountable.
They also need a realistic contingency plan. If an important AI service becomes unavailable or starts producing unacceptable outcomes, is there another provider? Could the firm revert to a less automated process? Is its data portable? And how long would a switch actually take? An exit strategy that works on paper may look very different when tested.
For some providers, there may not be a straightforward substitute. Boards need to know that, too. They cannot eliminate every external dependency, but they can understand where the biggest exposures sit and make informed decisions about what they are prepared to accept.
Ultimately, AI may change how decisions are made, but it does not remove a firm’s accountability for the outcomes. As firms become more reliant on AI and the providers behind it, boards need enough understanding to know when that reliance has gone beyond the firm’s risk appetite.
From approving AI to challenging it
The same principle applies to AI developed or controlled within the organisation. Boards do not need to understand the technical workings of every model. But they do need enough AI literacy to ask meaningful questions.
What level of autonomy has been given to the system? What could go wrong? What information would reveal that its behaviour had changed? When must a person intervene? And who is accountable for making that decision?
These questions are particularly important when AI affects customers. Whether a system is involved in credit decisioning, fraud detection, customer service or product design, firms need to demonstrate that they are delivering appropriate outcomes.
The information reaching boards, therefore, needs to evolve, too. A report showing where AI is deployed is useful, but it says little about whether those systems continue to operate safely. Boards should expect evidence on performance, customer and business outcomes, control effectiveness and changes in model behaviour over time.
Oversight should also reflect the level of risk. An internal productivity tool does not carry the same potential consequences as an autonomous system influencing a lending or insurance decision. Firms should understand where different uses sit on the autonomy spectrum and apply governance proportionately, with greater autonomy and potential impact demanding greater scrutiny.
Governance should enable adoption, not prevent it
There is a temptation to see all this governance as something that slows AI adoption. In practice, the opposite can be true. Many financial-services firms are already experimenting with AI. The harder step is moving successful use cases from pilots into important business processes.
That becomes difficult if the people ultimately accountable for the organisation lack confidence in how those systems are controlled. Clear guardrails can give boards and management the confidence to allow appropriate AI applications to operate with greater independence while concentrating scrutiny on areas where the potential consequences are higher.
This means being explicit about where human involvement remains necessary, what should trigger intervention and where AI can operate more independently.
There are significant opportunities in getting this right. Better use of AI and data could improve personalisation, make services more efficient and help address some of the information gaps that contribute to poor financial decisions. Responsible governance and innovation should therefore reinforce one another: The objective is to give firms confidence to progress, not simply another reason to say no.
The questions that boards should be asking now
The Mills Review looks towards 2030, but these are not 2030 problems. Boards should use the coming months to test whether their governance can cope with greater AI autonomy, rather than waiting for the regulatory position to be completely settled.
That starts with knowing where the firm’s most consequential AI uses and dependencies sit. Boards should understand how much autonomy those systems have, whether existing controls remain appropriate as that autonomy increases and where human judgement must remain in the process.
They should also ask management to identify the external dependencies beneath those systems and demonstrate that the firm has considered what happens when a provider changes, degrades or fails. That means looking beyond supplier due diligence towards concentration, contractual visibility, alternatives, contingency arrangements and viable exit strategies.
Risk and compliance functions should be capable of providing effective oversight of AI-driven activity, while internal audits should be equipped to provide meaningful independent assurance over the governance and control frameworks surrounding it. Crucially, those lines of defence need to evolve as quickly as the technology and the firm’s use of it. None of this requires board members to become AI experts. What is required is enough understanding to exercise the judgement and the capacity to challenge that their existing responsibilities already demand.
The next stage of AI adoption in financial services will not simply be about how quickly firms can deploy the technology. It will test whether they understand what their AI is doing, what lies underneath it, where their ability to control it ends and what they will do when something goes wrong.
For boards, approving AI is increasingly the easy part. Knowing when and how to challenge it is where the harder work begins.
Go to top
The post Approving AI Is the Easy Part. Can Financial-Services Boards Challenge It? appeared first on International Banker.
Last edited: