I’ve recently been asked a number of times as to the differences between various sources of risks one is exposed to when creating a model, in specific, for the difference between parameter and model risk. I’ve used the following allegory a number of times to explain how I see the differences between process, parameter, model, and unknown risk. Let’s start with a seemingly random sequence of 0s and 1s which you are told represents the heads or tails of a fair coin. Now you need to predict the next flip or 12. In front of you is a bright, shiny coin. So, being an empiricist, you take the coin, and flip it a dozen or so times.

Process Risk

While in the long run—the very, very long run—you should see an approximately equal number of heads and tails, in any one sub-sequence of flips, you can see just about anything. If you flip long enough—a countably infinite number of flips to be precise—you are guaranteed to see every possible sub-sequence. The fact that you may get 8 heads and 4 tails in one sequence of 12 flips is just a manifestation of process risk. Similarly, in all modeling of random processes, we are relatively comfortable with the “random” element of the outcome—process risk.

Parameter Risk

Finite Sample Size

This coin that you are flipping, what makes you think it is fair? You tested it, didn’t you? How many times did you test it? As long as the “testing” of the coin occurred in finite time, you cannot be certain that the coin is fair. Of course, there are levels of certainty. For example, depending on the outcomes of your testing, you can be more certain that the coin is fair or not. There are both frequentist and Bayesian methods for getting a feel for just how good the parameter estimates from the data are. The most well known is probably the estimates that come from maximum likelihood estimation. Under the principles of maximum likelihood, the estimators are asymptotically multivariate normal, and the Fisher information at the point of convergence can be inverted to create a variance-covariance matrix for the parameters. When you see statements like “the distribution is gamma with = 1.5, SD = .4; = 100,000, SD = 50,000; cor() = 37%,” these statistics are obtained most often from the Hessian at the point of convergence under maximum likelihood.

Changing Parameters over Time

Even if we are absolutely certain that the coin is fair when we start, we have no guarantee that it will remain so. Are we flipping the coin over a trampoline made of spider-silk or a concrete sidewalk? Over time, one of the faces of the coin, or the edge, may abrade due to the constant landing on the ground. So even if the coin was fair to start, it may not be over time. Similarly with any of the parameters in a model. Parameters may change over time, and allowing for some amount of drift in a stochastic model helps reflect this form of parameter risk.

Model Risk

Why are we flipping a coin? Because we have a sequence of 0s and 1s which we believed were generated by a coin, right? Perhaps we were wrong, though. Maybe these 0s and 1s did not come from a coin at all, but a spinner with two equal halves. In this case, our modeling may be incorrect. We may have built a really sophisticated model taking into account the coin’s thickness, diameter, and metal content; the height, handedness, and thumb musculature of the coin flipper; and the wind direction, absolute and relative humidities, and ambient temperature of the location of the flip. In reality, most of these statistics are meaningless. What we really need is a model of the material makeup of the spinner, the post, and the surface, and the co-efficients of kinetic and static friction between the spinner and the other two components; the musculature of the spinner flicker’s index finger; and the shape and aerodynamics of the spinner itself. Yes, both give 0s and 1s, but they are two different processes and require two different models.

Unknown Risk

For this spinner that actually was the generator for the 0s and 1s, we’ve assumed that the area of the spinner backing is evenly split into two semi-circles: one 0 and the other 1. Perhaps that isn’t the case, but that 49.9985% of one area reads 0, 49.9985% of the other area reads 1, but the remaining 0.03% reads 100. If this spinner represented a game where you paid a multiple of the shown value, it would be quite a shock to see a 100 when all that were seen until now were 0s and 1s. This is unknown risk—the risk of something happening which has not been seen before. Whether you call this the “black swan” event or the “unknown unknown,” this may be one of the most important, yet hardest to handle, element of risk that needs to be considered. It cannot be modeled well, as all of our experience perforce has happened. We can brainstorm, but it is very hard to really jump outside the comfort zone of our experiences.