Saturday, August 17, 2013

Evidence Based HiPPOs

A post over on has me feeling uneasy, which is strange given my job and mandate (Head of Research and Innovation for IIBA, tasked to "transform business analysis into an evidence based profession and practice"). After some consideration, I think the source of my discomfort lies in the implication that opinion is never a good option. The comment below is posted to his blog, but I thought it worth noting here, as well.

(The image is from


As Andrew noted, the evidence we have does show us that more evidence usually improves decision making and performance. But the evidence has to be good evidence. It must be:

- Significant: the signal is clearly distinguished from the noise.
- Accurate: the measures reflect reality with useful fidelity.
- Unbiased: the data cover all relevant aspects of reality (not just the parts we like).

Collecting more evidence addresses the first factor directly: a certain volume is necessary for good evidence, but volume is not sufficient for good evidence. But learning to create and collect accurate evidence takes training and practice - a.k.a. experience. Your writing demonstrates that know that bias is insidious and pervasive, and can be controlled (to a degree) through rigor, discipline, and focus - but these are also characteristics developed through experience.

This leaves us in an uncomfortable situation. Good decisions* depend on good evidence - but good evidence depends on good decisions, and good decisions require experience (or expertise, or both).

So who has the experience needed to make good decisions with the available evidence? Perhaps it is a HiPPO who has lives by a heuristic:

"Evidence trumps reason trumps experience."**

This evidence-based HiPPO goes with experience only in emergencies, when there is no time for reasoning and no evidence available. Even in a crisis, the evidence-based HiPPO uses some rational approach to guide a decision, based on the situation: from argument and debate in some cases, decision tables in others. In all other conditions, the evidence-based HiPPO makes decisions based on good evidence.

The intersection of 'leadership skills' 'evidence-based decision making' and 'HiPPO' may be limited, but it exists, and is worth expanding. The HiPPO might be an endangered species on Vulcan, but they will never be rare on human worlds. This position is part of the way humans - irrational social animals that demand leadership - construct intentional collective endeavors. The power of evidence based practice is enormous, but it's not enough.***

* We're talking about systems that change based on decisions here and not systems that change over time without decisions (evolutionary systems, being an obvious example). Our species demands agency in most events, and finds it very hard to accept the idea that 'things just happen'. This makes changes that avoid agency appear to be wasteful in ways that make them hard to adopt in human endeavors. A/B/N testing is a rare exception in the business world.

**The pithier "evidence trumps experience" is usually how I express this idea, as one of four principles that guide how I live my life.

*** Summed up in another personal principle: "emotion motivates; data doesn't."

No comments: