Model risk-management has gone up the regulatory agenda over the last few years, as an integral element of Basel 3.1 and internal ratings based (IRB) approaches. But models are inherently complex, needing specialist skill sets for building, validating, and maintaining them. As a result, firms are increasingly choosing to use third-party vendor models – but they require significant oversight, in line with current regulatory expectations outlined in SS2/21 ‘Outsourcing and third-party risk management’.
The PRA published SS1/23 outlining its key expectations to help firms strengthen their policies, procedures, and practices to identify, manage, and control the risks associated with the use of externally developed models, third-party and vendor products. This upholds the key third-party risk principles in SS2/21, and reiterates that ultimate regulatory responsibility remains with your organisation and doesn't pass to the third-party. That includes oversight of the underlying data, model validation, accuracy, and correct application.
SS1/23 sets clear expectations for third-party vendor models. To meet them, you must:
With a third-party vendor or outsourced provider, the people with modelling expertise are essentially outside your organisation. That presents unique challenges for risk management, which are compounded by the proprietary nature of some vendor products. As such, you need close co-operation with your vendor to make sure you receive the necessary support to place trust in the model outputs and effectively manage the risks.
Under SS1/23, there are five key principles for firms to follow, which will inform the model risk practices expected from your third-party vendor.
Every organisation has a different definition of the term ‘model’, so you need to establish if your vendor’s product falls into that category. If so, your model risk management framework will apply.
You will need to review your vendor’s model risk governance arrangements across every stage of the model lifecycle. This includes model development, approval, review, and validation. Models must be supported with appropriate standards and documentation, with mechanisms to monitor performance and initiate change as needed. You need to be up to date on any model changes and exceptions, and consequent remedial actions. Requesting written descriptions of these activities, and the associated governance arrangements, can help you demonstrate regulatory compliance and give senior leadership greater assurance over model risk.
Perhaps the biggest challenge for firms is the proprietary element of many vendor models. This may inherently reduce transparency around model methodology, and present challenges for the model risk-management team. However, you do need assurance that the methodology is sound and fit for purpose. That includes sufficiently granular model documentation covering key components, design, intended use, model operation, and dependencies. You also need details of the key assumptions, limitations, the data sources used and the data dictionary. The vendor should also be undertaking model testing (including back testing and stress testing) to ensure accuracy and assess sensitivity. Model development testing is also crucial to make sure the product works as expected.
Once implemented, you need evidence of quality control tests – including testing inputs, outputs, and limitations. You can conduct these tests yourself, if the model is implemented on your system. If not, your vendor should give you appropriate documentation, including processes to initiate change throughout the model lifecycle.
Vendors need to undertake ongoing monitoring as an essential element of the model risk-management framework, to make sure the model continues to perform as intended. You need access to regular model-performance monitoring results, including results of performance tests and standards, and threshold indicators that trigger when performance has materially changed.
While vendors do typically perform independent reviews to give you assurance over the model performance, they don’t necessarily share these with third parties. You should ask to see the independent validation report, alongside details of how independence, challenge and oversight is maintained. This includes details of the frequency of independent reviews, any model changes, review of the model code data or language, and alignment of model data to your vendor’s quality standards.
Your vendor needs to actively mitigate model risks, and you are free to request information on how performance monitoring outcomes are analysed. This includes monitoring the frequency of performance measure breaches, and action plans for any subsequent modifications or updates.
Documentation should also include governance arrangements for any planned or unplanned model changes, and any limitations and adjustments. If expert judgement is a key part of the model, and used to override model outputs, the documentation should explain the process involved and the supporting governance arrangements.
Model risk-management is a key regulatory priority, and its essential to gain the necessary assurance over your vendor models. You need to build your internal understanding and knowledge of the model, so you can effectively evaluate assurance activity and documentation supplied by the third party. It’s also important to undertake effective due diligence, with ongoing oversight to ensure the vendor continues to meet its contractual obligations, and the associated activity continues to meet regulatory expectations.
There’s also business continuity and resilience to consider, and you should think about contingency plans if the vendor’s product stops being available. This includes transferal to another vendor, or following an exit plan, as per any other outsourcing or third-party risk management approach.
For more insight and guidance, get in touch with Vivian Lagan.
