Standing Inside the Outside Box

In his book Empirical Model-Building and Response Surfaces, George Box made the now semi-famous statement “All models are wrong, some are useful.” In summing up mathematical modeling and statistics, this comes about as close to ground truth on the subject as possible. Aside from those of us suffering from the side effects of cool-aide poisoning (dispensaries have sprung up at universities across the country) many of us who have worked through the detailed mathematics that underlie statistics and formal modeling realize the metaphoric nature of the structure and the perilousness of the assumptions behind it. Despite that, we continue to oversell the “truth” of what we do.

We excuse this behavior with a simple expedient; give the customer what he wants. The general wants an answer, so he is provided with a point solution to a broad spectrum question. He’s happy, we’re happy, everyone’s happy. At least until they start bringing the boys home in boxes. But don’t worry, how will the bullets in those corpses ever be traced back to that product handed to the general. It’s what he wanted, after all. Conscience clear.

Not in my book.

We’re accosted on all sides. In the press we continually see where someone or other has invented some new method for predicting the future (which usually turns out to be something already done wrapped in new clothing, e.g. Gourley, BDM, etc.) and the money starts to flow. Hard not to be envious of that.

When we read reports about some new wizbang method for predicting the future, our first reaction should not be one of optimistic hope and excitement. Instead, our first impulse should be to reach for our crap detector and to start figuring out what it’s doing, what context it works within (and what context it doesn’t), and then to firmly place it within our hierarchy of tools in such a way that it is optimized to perform only on the issues on which it is effective.

Our other challenge comes in the form of poorly defined problems being posed by the technically incompetent (who, unfortunately, usually hold the checkbook). “If I could predict what the terrorists were going to do next, I could stop them and better defend the country.” How many times have you seen some formulation of that sentence/sentiment?

It’s mostly our fault. In the mad dash for social sciences to gain respectability by donning the trappings of the scientific method, we’ve made promises we can’t actually keep. Of the list of things that all good models are supposed to do, predict is one of them. This comes right out of formal methods 101, and it’s based upon a gross misunderstanding of reality. As Stephen Downes-Martin has so effectively pointed out, F=MA is a very effective model at predicting things, except that it is not a social science model.

In quantitative academic literature on civil wars, insurgencies, and non-state actor violence we see a broad range to empirical evidence used to back all sorts of claims. One of my favorites is the notion that suicide bombings are primarily the result of occupations in foreign countries. As a result of this correlation, Robert Pape has recommended rather broad policy prescriptions that have gotten traction in some circles. Except that the correlation is so obviously spurious and driven by a particular political perspective. Beyond that, try and reproduce the results. You can’t. Pape won’t share his data (I’ve asked him more than once for it, and so have other professors. Pape won’t even respond to the request). Another favorite is the work of James Fearon and David Laitin out of Stanford. They claim to have empirical evidence that the presence of mountainous terrain is a primary determinant for insurgency. This author has gotten their data and run their regressions. If the original R Squared value of low twenty percent doesn’t give you pause, then when I tell you that if you take the lagged dependent variable out of the model the whole thing collapses it should at least raise an eyebrow.

But beyond the shortcomings of quantitative work being done in the social sciences (there is good work out there, it’s not a complete calamity), we have a problem with approach. “If I could only predict what the terrorist is going to attack next” is really not the problem we need to be thinking about. What if we instead ask “what can the terrorist do that can really hurt me?” and it’s corollary “what are the responses to bad events that are most appropriate?” These are questions we can actually answer with some degree of confidence, and by doing so, mitigate against their occurrence. Similarly, if I ask “what COA will defeat whatever Red does against it?” I have an intractable problem. But instead what if I ask “How can Red defeat my COA?” then I have a problem that is more tractable by not relying on attempting to predict what he might do, and instead causing me to think about making my own actions more flexible and responsive to potentially changing situations. The wargame response to the latter question is not to have accurate predictive PolMil models informing the game, but rather to harness the creativity of a large number of red cells as they attack a single Blue COA.

By understanding and cataloging our own vulnerabilities, we don’t get a tool to predict, but we do get a tool that anticipates and mitigates. But there’s no panacea here. We live in a sea of risk, and there’s no real way to create certainty out of chaos. In trying to form a future based upon predicted outcomes we create a system highly vulnerable to black swan events that could easily destroy the very future we wish to create. But by anticipating events by understanding what we are vulnerable to and then mitigating against them, we create a ship that can weather most storms.

Advertisements

About Jon Compton

The Harlan Ellison of Wargaming!
This entry was posted in Community Development. Bookmark the permalink.

5 Responses to Standing Inside the Outside Box

  1. Stephen Downes-Martin says:

    Great post Jon.

    I believe that I can predict something very accurately … that the enemy we need to worry about (i.e. the intelligent one) will NOT attack us against our strengths but will wait until he can see how to surprise us and then attack. I therefore predict we will eventually be caught by surprise no matter how hard or smart we think through the problem and that this surpise will be the one that is most dangerous to us. (We can reduce the probability of catastrophic surprise perhaps, but not remove it completely). So a question worth thinking about is “how can we construct our forces and society to survive the shock and huge casualties of the initial surprise with enough resources to hold the enemy at bay and enough agility to adapt to fight back and win, without knowing the details of the surprise?” The issue is not predicting what weapons or technology will exist or will be used, or predicting what the surprise will be, or about not suffering major initial damages and casualties (we will suffer those casualties), the issue is all about generic agility and adaptability to fight back after massive losses.

  2. Brant says:

    “By understanding and cataloging our own vulnerabilities, we don’t get a tool to predict, but we do get a tool that anticipates and mitigates.”

    Having been the intel guy and the red force waaaaay too many times, I would argue that this is more important, even if the commanders don’t seem to always recognize it as such.

  3. Rex Brynen says:

    A challenge arises, however, when you’re working in a shop that is asked and expected to predict—which is typically the case with a very great deal of pol-mil intelligence assessment. Certainly decision-makers know (or ought to know) that analysts do not have a crystal ball, and that their assessments are constrained not only by limited information but also the adaptability and inventiveness of others (including, it should be said, allies—prediction isn’t always about opponents). Indeed, communicating probabilities and confidence levels is so central to the function of most national-level intel assessment that learning about appropriate estimative language is one of the first lessons that the trainee analyst goes through.

    In that case, what is the role for wargaming? Some ICs use it periodically, or use a variety of alternative analytical techniques that serve as wargame proxies. Others never ever use it. I’ve yet to detect any systematic difference in overall analytical quality between those who do, and those who don’t (although I can hardly claim to have undertaken any sort of substantive empirical study). Does this mean that wargaming is poorly used in such contexts? Or does it mean that it doesn’t offer any real comparative advantages compared to all of the other things one could be doing to deepen and broader analytical perspective?

    Just being provocative.

  4. Paul Vebber says:

    “Certainly decision-makers know (or ought to know) that analysts do not have a crystal ball,…”

    My experience is that yes, decision-makers often DO expect analysts to have a crystal ball, and too often the analysts oblige. Particlularly in acquisition circles where a real system not meeting its analytically predicted goal often gets 3 and 4 stars infront of Congress. Once there, armed with still more analyst revised “predictions”, what are sometimes egregious failures are papered over.

    The next most egregious predictors are the intel folks. Ask sometimes to see just how those “weasel words” (that indicate percentage ranges of “accuracy”) are actually computed. Or for a validation analysis of the track record of how often “10%” probable events actually happened and how often “90%” probable events actually happened. Or those in between…

    It would be worthwhile to actually do your hypothetical substantive empirical study. Until such a study is done, I think the answer to “does wargaming make for higher analytic quality in intelligence (or acquisition) products” would be an interesting one, but one that depends on how one measures “analytic quality”.

    If by that one means “more accurate prediction” then I don’t think wargaming vs other activities will do a whole lot becasue I think one would find that “high confidence” outcomes tend to actually occur more often than the “numerical probability” of the weasel words, while those of low proabaility rarely happen at all.

    “In that case, what is the role for wargaming?”

    Wargaming is not about prediction, validation, or other degrees of certaintude.

    wargaming is about exploring a decision space to see what sort of outcomes may be possible. You can’t play a wargame 10 times and be able to say “this result occured 3 times, so its 30% likely”. You play a wargame to say “these are teh 7 different results that occured in the 10 plays and here are 40 or 50 varioatins on those outcomes that could have easily occured if the games had gome a bit differently. There is no other tool for exploration a problem space that can get you to these insights into “what sort of things might happen” as wargaming.

    Unfortunatly this sort of “depth and breadth” of the space of potential outcomes is not valued, either in acquisition or intelligence circels (wih rare exceptions) – analysis stops after one gets to “most likley” and “most dangerous” based on intuition, experience and a process of concensus seeking in a tight time constraint.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s