In my last post I described what I call the Customer Experience Dilemma. The tension created when you need to develop a new offering with the ability to disrupt the market, and the competing forces often imposed by those that measure potential success based on existing metrics. I was recently speaking to someone who experienced this frustration when she worked in new product innovation at Pepsi. As she tried to develop products that had the potential to create new categories, she was constantly pressured to ensure that these new offerings passed BASES tests used for successful line extensions. She aptly pointed out that even a product as successful as RedBull would fail the BASES test she was required to use as her metric.
How do I mitigate this tension? It all comes down to metrics. Many people in this situation get frustrated with metrics, and try to eliminate them, but I think that's a mistake. We need to remember that it's not the metrics that are at fault, it's the failure to recognize what they were created to measure, and use them as they were intended. BASES is a great test – for evolutionary line extensions. It is a horrible test to evaluate something completely new.
The key is to understand what customer motivation your new offering is intended to satisfy, and to find (or more likely, create) the appropriate new metric to evaluate it. This happened when I was doing some work for Procter and Gamble. Their current metrics would have killed a new idea my team was working on, so the current business unit president said "If you don't think our metrics are appropriate, then create new ones. But let me be clear – these ideas will be tested, and even though the tests may be different, they will be rigorous!" Point taken. And with that, we developed a series of contextual evaluations that were the basis of many new metrics I use today.
One of the most important things to remember in creating evaluative metrics, is how to establish the right level and type of rigor for your situation. A useful resource in this regard was written by Roger Martin many years ago, and he talks about the difference between reliability and validity. In short, the difference between them is that Validity is ensuring that you are doing the right things; Reliability is ensuring that you are doing things right. You want reliability metrics when you are evolving your current offerings (Are we doing this right? – as in staying close enough to what we know already works.) . You want validity metrics when you are trying to do something new (Are we doing the right thing? – as in knowing we won't fail at something no one has done yet, but will be able to improve and optimize later.).
Reliability metrics such as BASES can be found off the shelf, learned and applied the same way in many situations. Validity metrics may have foundational similarities, but will be different for each situation. This may sound like a lot of reinvention, but as I always like to remind people "You can't lead by finding someone to follow." It's as simple and complex as that. To avoid the Customer Experience Dilemma, you need to figure out what motivates your market's behavior (no shortcuts there either), and then develop metrics that measure the validity of your ideas.