On November 27, 2019 the Canadian Regulatory Technology Association (CRTA) (Association canadienne de la technologie réglementaire (ACTR)) and KPMG co-hosted A Look into the Future – Emerging Trends and the Use of AI for Model Risk Management (MRM). Many thanks to KPMG for the ideal space, refreshments and hospitality.
The morning opened with a presentation by Mike McCausland of KPMG’s recent Global Model Risk Survey of domestic and global banks, which uncovered some very interesting (and perhaps concerning) trends, including that:
Carl Barrelet of KPMG followed with a presentation on the essential elements of a governance framework for the use and validation of AI models, pointing out that OSFI’s primary concern – which is shared by Canadian banks – is legal and reputational risk.
We then enjoyed case-study presentations from two vendors with products and services that are relevant to and useful for MRM. First, Paul Finlay presented Xanadu.ai’s “Silver Hammer” – a toolkit for actioning model validation that is modeled after Scikit Learn. Paul pointed out that model development is often completely decoupled from validation, in large part because developers are not aware of the validation standards. This undoubtedly results in tremendous inefficiencies. Silver Hammer enables firms to manage the development and validation processes, with clear accountability and traceability.
Next, Tony Bethell presented Cluster Seven’s services, which help firms identify, manage and monitor “shadow” IT – applications implemented and managed by end-users rather than the corporate IT function. Tony shared a “spider diagram” of one firm’s data sources for a particular model. While the firm thought there were a few dozen input files, Cluster Seven uncovered several hundred! Shadow IT is one of the biggest threats to a firm’s reputation, and a major contributor to legal risk.
The morning ended with a thought-provoking discussion, moderated by Craig Davis of KPMG, among industry experts from various parts of the industry: Manuel Morales (National Bank and the University of Montreal), Carl Barrelet (KPMG), Paul Finlay (Xanadu.ai) and Dina Duhon (BMO Financial Group). There was consensus among the panelists that the greatest risks of MRM are data quality and bias, privacy and governance. Bias is inherent in any data set; understanding it from the outset is critical to the development of low-risk models. They also concurred on the importance of explainability; not all models need to be explained, but for most, a firm’s inability to explain to users or customers why a model produced its results can lead to serious consequences. Paul Finlay commented that the hype around AI leads to overengineering of solutions beyond the needs of a problem and causes explainability issues.
To bring the discussion back to practical examples, the panel cited some interesting use cases for AI, such as:
By 10:30, attendees – representing financial institutions, RegTech firms, regulators and others – were back at their desks with plenty of food for thought.
 The survey was conducted between June 2019 and August 2019 among model risk management executives from 48 significant banks representing 16 countries/regions/jurisdictions globally, including 5 North American banks.
Written by: Wendy Rudd, Member of the Board, CRTA