People talk about the high rate of new product failures (about 85%) as if there is no way to change it, and no way to understand the market. Like innovation is some kind of hit and miss affair, or magic. It's not.
Nielsen, a well known quant-focused research group, has been deconstructing and quantifying innovation successes for a few years now with the Breakthrough Innovation studies. I recently heard Lauren Galletta present at a professional meeting on this topic, and it was terrific stuff.
They have studied tens of thousands of product launches, and have strict criteria for being a winner -- there has to be something new happening (distinctiveness) not just a reformulation. The sales of the new thing need to be big enough to be relevant. And the sales have to continue at close to the launch levels for at least a second year.
Their framework for how to create these kinds of wins is this: Demand driven insight, demand driven development, and demand driven activation. Let's look at what they mean by this.