In the previous post, I was attempting to draw an analogy between perceptual illusions and puzzles and why we need to look carefully if we want to understand what is really going on. There are many roadblocks to the truth, however. Let me give you an example:
I belong to a business club that has speakers at breakfast and dinner, workshops, and that sort of thing.
They are facing declining membership numbers, and are trying to understand what is going on, using member surveys, among other things.
Some long-time volunteers on the executive are of the view that some people join just to be able to list membership in this prestigious organization on their resume. And indeed, there is a group of members -- about 20% -- that renew annually, but never attend events. So the instincts of the leadership are probably right -- some members obtain value from membership independent of actually attending the educational events.
The organization wanted to test this notion in their membership survey. This is the question they used, in a list of reasons for joining:
"I joined XYZ ... (pick two from a list)... to be recognized as someone that belongs to a reputable organization."
Only 3% of the respondents to the survey selected this as a reason for joining. That shouldn't really surprise you, because most of us don't want to admit to shallow motives, even to ourselves, even in an anonymous survey.
Asking questions sideways
If we really want to find the answer to this question, we are going to have to come at it sideways somehow. Most commercial survey research uses multivariate techniques to tease out these kinds of findings. Like connecting the strength of your agreement with attitudinal statements (e.g."it is important to belong to prestigious business organizations") with your stated likelihood of renewing your membership.
In qualitative research, we cook up exercises of various sorts, instead of just asking the literal question directly. [It's also because most of us really like to play with colored markers and Post-it notes, although that's just a corollary benefit.] Sometimes, we may just ask what you think others do, not what you yourself do.
Cool method called randomized response technique
In social policy research, sometimes the question is so sensitive, there is a concern that respondents will lie to avoid sharing a socially unacceptable response, even with the interviewer. Topics like domestic abuse, for example, are unlikely to receive the unvarnished truth from respondents. A clever method called "Randomized Response Techique" was devised to get around this problem.
Here's how it works. I want to ask you to answer a question about your belief in capital punishment. So I have a card with two questions on it, which I hand you. I also hand you a box that has one white and one black marble in it. I invite you to pick a marble from the box, but not to show it to me. Then I ask you to answer the question on the card, but not to tell me which question you answered:
A (white marble). Flip a coin. Do you support capital punishment? Say yes if you agree, and no if you disagree.
B (black marble). Flip a coin. Did you get heads? Say yes if you agree, and no if you disagree.
Even the interviewer does not know which question you answered. They just write down whether your response was yes or no. Using statistics we extract the answers to the innocuous question, and are left with the answers to the real question. Way cool, huh?
There's always bias
Most of the time, you don't need something quite this tricky. But you do need to consider that people will bias their responses toward things that are socially more desirable. They will state a higher income range than they have, they will say they watch less TV than they do, they will say they read more books to their kids, they will say they eat healthier food, drink less wine, exercise more often ... you name it.
So the more you can observe directly, the better off you are. And your questions, when you ask them, need to be clever.
