In the two years before 2018, we created 90% of the world’s data, and this rate is increasing exponentially, according to a statistic is from a Forbes article by Bernard Marr.
Is all this data leading to better decisions? I would argue that, in general, it is not.
We have been conditioned to believe that more data leads to better insights, which in turn result in better decisions. We have equated more data to risk reduction. This belief has resulted in greater confidence in our conclusions, but this confidence is misplaced if we have not fully developed our ability to ensure that our conclusions are valid.
This can cause us to have greater confidence in poor conclusions because they have been derived from a larger data set. Instead, we should be deriving greater confidence from our ability to draw valid conclusions from the data, rather than the volume or immediacy of the data points.
This sentiment was echoed recently in an article by data scientist Kalev Leetaru
BACK TO BASICS – REVISITING INSIGHT
Visual interpretation of Bill Duggan’s Strategic Intuition by Ellen Di Resta
The first step in deriving robust conclusions is to become adept at uncovering deeper insights that make sense of what we are observing in the data. I wrote a short post (link: https://ellendiresta.com/insight/ ) about this years ago. It was applied to repeated questions about the value of deep research on small data sets (via ethnographic and other deep dive methods) to derive insights that can then be quantified. (This was later described in depth in Bill Duggen’s book Strategic Intuition, and illustrated in the diagram above.) I’ll paraphrase the points made in that post here:
The word insight is often described as something to be observed, as in “We observed several insights on that consumer interview (or in that data).”
There are several entries in dictionary.com for the word insight. In different ways, they all describe insight as an understanding of the true motivational forces that drive actions; they “define underlying truths, shed light on, or help to solve a problem.” They are not the solution, and they cannot be observed directly. They need to be inferred or derived by thinking critically about the observations we make.
Observing behaviors or data sets and describing or categorizing them is a fairly straightforward exercise. Deriving insights requires the ability to observe, formulate a theory about why the observation occurred (the insight), and validate the conclusion.
We now have algorithms that help us to make sense of the mind-boggling volumes of data available to us continuously. However, the decisions we make are only as good as our ability to create and use the algorithms appropriately to derive the right conclusions from the data.
When making an important decision, do you feel more confident when you have more data, or when you have more trust in the validity of the conclusion? If it’s the latter, how do you go about ensuring you this trust?
ABOUT
Ellen Di Resta is Founder and Managing Director of Pearl Partners LLC, a boutique consulting form that partners with senior leaders to create new sources of value, by aligning their organizations to new opportunities.