Every time I talk about advertising measurement as a 'black art' I get objections from people telling me how their normative metrics are well tested and very useful, blah, blah, blah. Let me be clear here: I do qualitative testing on advertising, and I'm not too positive on that approach either.
Why so negative? Well, there is a lot of good evidence that our actual preferences and behavior are not all that well related to what we say.
A brilliant article, "Questioning the Nature of Research" reports on an interview with Rory Sutherland, chair of Ogilvy UK, who makes a strong case that our current measurements are not strongly rooted in behavioral economic theory.
Sutherland's point is not that research is a bad thing in general, it's that the current norms for how research is done on advertising don't work.
...as you get closer to the point of decision, there are factors at work which never really appear in conventional market research. They are contextual factors, social factors. No-one in any research group would ever say, “If there are four brands of shampoo, I’ll buy the one that has most bottles on the shelf”
This is true, by the way, not just of survey research, but also of qualitative.
The best qualitative research approaches come at the topic from right angles -- indirectly. Like a magician's deception, we want to focus your attention on something so that you respond as naturally as possible. Or we try to mimic the actual circumstances of your decision. Still, no self-report in the world is as good as watching a thing happen. Or setting up experiments that show you what is actually happening.
What we have learned from behavioral economics research is this: "Your preference is a product of your behaviour and not the origin of it." If you customarily go to one drycleaner, you will be able to provide a rationale for that choice, (e.g. quality, service) which is likely to have little to do with the real reason (e.g.location).
A useful and thought-provoking article to get your synapses back to school and making new connections.
