One of the things I think about a lot is the difference between precision and accuracy or from a modeling perspective the difference between verification and validation. The difference between precision and accuracy can be understood in terms of measuring tools such as a clock. A precise clock measures time very consistently. For instance, if the clock measures the time it takes a rock to drop a certain distance four times and it gets 1.541, 1.540, 1.541, and 1.543 it is very precise. A fifth measurement with a precise clock would most likely yield something close to 1.54. However, such a clock might not be accurate. For instance, if the actual time it takes the rock to drop is 1.8 seconds then the precise clock above is running slow. An accurate but imprecise clock would yield something like 1.4, 1.9, 2.1, …. but if measured sufficiently many time the average would be 1.8 which for the sake of the argument is the right value.

This difference is important because it reflects on the way that humans think about the world. The world is a highly complex place. Therefore people either consciously or subconsciously build a simplified model or story of the world and then try to understand the model or story instead. A model or story is true insofar that it provides useful predictions about the real world.

There are two things to consider now. These two things are the most important things to consider when thinking about models. The two things are verification and validation.

Verification tests whether the model is consistent and internally correct. Many models can be verified. This usually involves mathematical proof or logical reasoning. It does not even need to be stringent. When people argue philosophically they are often trying to figure out if a model is internally correct.

Many times, particularly when it comes to economics, verified models are cast in mathematics which also makes them very precise. However, just because a model is precise it does not mean it provides an accurate understanding of reality.
Validation tests whether the model is the right one! In other words, validation tests whether the model provides a good description of reality. Many many people (especially those with a college education) skip this step. The reason is that nobody has time to learn and question all assumptions. Therefore one must rely on so-called experts practically all the time. In essence many formal educations at least to the masters level teach people not to question the models but to use the models. Even experts are usually only experts within a very narrow field. Being a polymath is no longer encouraged in this overly specialized world. Thus we generally live in a world that is run by a likely inaccurate model (system) which is the aggregate of many smaller a very precise models. Such models may even have been cast mathematically which leads to a false sense of confidence; especially of the math involved is very complicated. Many financial disasters have resulted from such an accuracy failure. LTCM (which included Nobel Prize winners Merton and Scholes on the board) required a bailout by the feds. Lily over at The Honest Dollar described the progenitor of the subprime mess which was another case of systemic failure due to inaccurate (albeit very precise) understanding of the risk. This failure also required a bailout from the feds(*). This article shows that the motivation behind the government stimulus package also constitutes the use of an inaccurate (albeit precise) model. Like any inaccurate model or idea, this will eventually fail because it is not in accordance with reality. In my opinion other inaccurate models include the idea of using corn based ethanol as a fuel substitute and using passive index investments to build retirement savings through a demographic hump. However, since precise and accurate models are usually not available it is not possible to say when and even to some degree how an idea will fail. The only thing to do is to position oneself accordingly and wait for it.

(*) Fun fact: The Federal Reserve is neither federal, nor do they have reserves. Does that mean that the name is inaccurate or imprecise?

What is your story of the world i.e. how do you explain something? How have you verified it? How have you validated it? Is your perception of the world precise or accurate?

Originally posted 2008-02-13 07:22:15.