Customer Crossroads: New study confirms power of qualitative in public policy

Canada's national marketing research association, MRIA, has just released the report of a study on government use of public opinion research.

For the most part, the report confirms what a regular reader of a quality newspaper would already know -- that our federal government now uses research and data of all kinds to confirm, rather than inform, their thinking, when they use data at all. Less data is collected, and what is collected is often of lower quality. Notably, the abandonment of the census, replacing it with a costlier, less valid, and less useful household survey.

But the leaders were pretty smart about what makes for good research and what they would like more of.

How to research executives and opinion leaders

The report used individual depth interviews of 45 minutes, mostly conducted by telephone. This is really the best way to conduct research with senior individuals -- executives, senior politicians, and similar. It's a waste of their input to make them answer a survey, and would be difficult to get them to do it anyway.

The MRIA study likely needed to be pretty tightly controlled, because it was planned to be made public. (And a committee probably wrote the questions.) Which is why you will see such a lengthy interview guide, that reads almost like a survey. If you have an option, I recommend not to conduct executive interviews this way.

Most of the interview guides I have used for executives are much shorter than this, and are designed to feel more like a conversation, where the direction of the interview is more fluid, and makes few assumptions going in. It is the difference between starting with a framework, as this study did, and attempting to discover what factors are important.

More importantly, you are dealing with really smart, really thoughtful, really well-informed people. You want them to give you the benefit of all that brainy goodness, not let them coast along in second gear. As we will see, once these folks were given free rein, they came up with some interesting stuff, and more on that below.

And for purely logistical reasons, you should avoid having the researcher take up a lot of time asking questions.

Leaders really want insight into the why, not just the what

Some of the things the interviewees wanted shows that leaders are often more sophisticated in their approach to research than their research department may give them credit for. They are well aware of the many sources of bias that can damage research quality, from poor sample to poor design to weak analysis, inadequate knowledge of the subject area, and deliberately misleading questions.

What they wanted more of also speaks well of these leaders.

They wanted more longitudinal research, to understand the evolution of topics and issues over time.

They wanted better linkage of knowledge and engagement to opinions. This is a huge flaw in a lot of public opinion polls, where people are asked about some complex issue, like pipeline construction or environmental policy, and not asked if they have ever even considered the topic before, or spent any time engaging with it. There is no attempt to gauge their level of knowledge on the topic. This is a very valid criticism, and I could have hugged them for saying so.

They wanted better linkage of opinions to values and beliefs, to understand not just what people think, but why they think this. (Yes, they want great qualitative! Another hug!) Minimally, they would like more open ended questions on surveys.

Some expressed concerns over the validity of "online focus groups" versus telephone interviews or in-person focus groups. I was very much wishing that the study probed on this -- what do they think an online focus group is, for example?

There was also a desire for more integration in the data used -- to pull in behavioral data, or database information, as well as primary research.

Understanding bias

One gap in this study, from my point of view, was not to share a summary of how the interviewees lined up based on their political leanings. The report says an effort was made to control for this, but a small table summary (not identifying the individuals) would have been helpful. Because as I read it, I did find myself suspecting that a relatively small group of supporters of the current government were included.

Further, there was no analysis reported on this aspect. Perhaps obvious, but one should either say that there was no difference of opinions across the political spectrum, or say what that difference is.

You can't eliminate bias in research. But it is always good to have an informed opinion of what the sources of bias are.

Thanks to the volunteers

A researcher was commissioned to conduct the study, and somewhat unusually, got their name on the cover of the report.

Unfortunately, the volunteer members of the Government Relations Committee who undoubtedly spent a great deal of time on this, including coming up with lists of contact names for the sample, were not acknowledged.

I'd like to point you to the online list of these good folks, but was unable to find it on the association website. So let me just say thank you for some good work here.