Fundamentally, an economic capital model seeks to consistently use exposure information and limited actual experience to estimate the impact of remote events that may or may not have ever happened in the past. Those who seek to make use of the outputs of such a model need assurance that the model is fit for its intended purpose.
In essence, economic model validation is the process of determining that the degree of model risk—not the risk associated with the processes described by the model, but the risk inherent in using the model itself for business purposes—is acceptable.
For example: Users want to have a reasonable degree of confidence that the economic capital model adheres consistently to its own guiding conceptual principles, aligns logically with prior editions of the same model, and conforms to standards and tests imposed by the regulator.
But there are few commonly accepted standards for model validation. Typically, the validating party must either contend with scant documentation, or sift through an overwhelming amount of documentation not created with validation in mind.
The complexity of today’s economic capital models requires a more structured validation process, which can readily be communicated to executives not directly involved in modeling.
Framework for Model Validation
A recent white paper developed by Willis Re Analytics team members working together with Dr. Markus Stricker, a scholar sponsored by the Willis Research Network, proposes a framework for model validation based on a clear definition of model risk. Components of model risk:
- Conceptual risk: the risk that the modeling concepts are not suitable or inappropriate for the purpose of the application.
- Implementation risk: includes
- The risk that the modeling concepts are not implemented appropriately, i.e. the wrong algorithms were chosen to implement the specified modeling concepts.
- The risk that the implementation contains errors, i.e. appropriate algorithms were chosen, but they contain coding errors, bugs.
- Input risk: the risk that the input parameters are inappropriate, incomplete or inaccurate
- Output risk: the risk that the key figures and statistics which can be produced by the model are too sensitive with respect to the provided input parameters or do not support the business purpose.
- Reporting risk: the risk that the representation of the output for the business users is incomplete or misleading. This is related to what is called the “use test” under Solvency II. In a use test the firm has to prove to the regulator that reports obtained from the model are used in a business process. Some people consider this to be outside the scope of a model validation, but we believe it is a vital step of the process.
With this clear definition of model risk as a starting point, it is possible to design a standardized framework for the process of determining the level of model risk—that is, the validation process. The definition also yields a natural sequence for assessing the model risks.
In order to assess conceptual risk, it’s critical to understand the purpose of the model, the applications for which it will be used, and the people who will be using it. Clear documentation of the limitations of the model concepts is also very important.
Most of our guidance on evaluating implementation risk comes from best practices for software engineering. For complex software, the realistic question is not whether it contains errors, but rather whether the errors it contains are substantial. Using best practices in development helps to minimize the risk of material errors; and a range of well-developed test techniques can be applied to the finished software.
When it comes to input risk, one must consider the differences between “raw” inputs and “pre-processed” inputs. These two different types of input require different types of validation. While this category of risk is familiar to most practitioners, there is often too much emphasis on extracting all possible information from internal data and not enough attention to peer group benchmarks. Appropriate benchmarking can be very useful in reducing the level of input risk.
Only if conceptual, implementation and input risk have been assessed positively does it make sense to assess output risk and reporting risk. The former deals with the full data set of outputs, and checks whether the model yields reasonable values; the latter deals with the manner in which selected outputs are presented to users. The outputs must be calculated correctly, but this alone is not sufficient – even correct outputs may be highly sensitive to input parameters, which is an undesirable model feature.
Reporting risk, on the other hand, deals with the communication of model results. A report, normally containing only a small fraction of the model outputs is provided to users of the ECM. In the evaluation of reporting risk, we assume that the output reflects the company’s risk situation well; we focus on the selection and presentation of key figures. As these can have significant influence, they must be assessed in the light of the intended use and the users. This is by far the most difficult part of ECM validation, because the validation team members themselves are influenced by the report content and format. We recommend that this assessment is done by the most senior validation team members.
Finally, we observe that larger ECMs typically have several sub-models; the guidance of assessing model risk subcategories applies to each of these. But a positive assessment of each sub-model is insufficient—their aggregation needs to be assessed as well. Our hope is that a more standardized validation process will:
- Create a more objective and less people-dependent result.
- Allow more stakeholders to understand the model’s capabilities and restrictions.
- Improve the efficiency of the validation process.
- Lead to more concise and useful documentation.
We presented the white paper at the GIRO40 meeting of the UK Actuarial Profession in Edinburgh, Scotland this October, to a positive reception. A full copy of the white paper may be downloaded here.
This article was authored with David Simmons.