Most companies have methods to evaluate the potential success of a new product or service. Sometimes they come in the form of a focus group, or an online or in-person panel of consumers, or even an internal group of people who are thought to be "in the target market." These types of evaluation typically have one element in common. They usually are asking consumers directly about their preference of a new product over one, or several, alternatives.
This type of testing is useful if you are evaluating an incremental improvement, or correcting a problem with your current product. These are things you can learn about directly from consumers, and during an evaluation they can directly tell you whether or not your solution hits the mark. Companies feel good about these types of evaluation because they eliminate the guesswork in developing new products.
The problem arises when you are trying to evaluate a new product or service that challenges accepted paradigms of behavior within a category. When asked about preference, consumers generally have a strong bias toward what is familiar. Any new product or service that challenges the familiar runs the risk of performing poorly in a preference test. There are countless examples of companies who missed an opportunity to be first to market because a product did not fare well in a preference test.
Evaluating a paradigm changing solution requires different methodology. The test needs to be designed to evaluate how well a new solution solves specific criteria, and consumers need to be immersed in that criteria to be able to give meaningful responses. We can discuss many ways to do this, but what's important is that consumers are seldom asked to state their preference.
The next time you are struggling to understand why your new product is failing the standard preference test, think about whether it's the solution or the test that needs to be changed.