Customer Crossroads

New study confirms power of qualitative in public policy

Canada's national marketing research association, MRIA, has just released the report of a study on government use of public opinion research.

For the most part, the report confirms what a regular reader of a quality newspaper would already know -- that our federal government now uses research and data of all kinds to confirm, rather than inform, their thinking, when they use data at all. Less data is collected, and what is collected is often of lower quality. Notably, the abandonment of the census, replacing it with a costlier, less valid, and less useful household survey.

But the leaders were pretty smart about what makes for good research and what they would like more of.

Are we asking too much of survey participants?

I've had a Vimeo Pro account for a few years, and have been really happy with them. I recently completed a survey for them. This was the last page.

I found this screen quite problematic for a few reasons:

[1] They didn't tell the truth about their intentions up front. The purpose of the survey was stated as (broadly) understanding our experience and improving the service.

[2] They want to impose a pretty high standard on the individual offering the testimonial. Contacting them if your opinion changes, for example. Can you imagine actually trying to do that, and how much aggravation you would encounter?

[3] Completely one sided. They get to choose what my testimonial is and how to use it, including my avatar, and I get ... umm, well, nothing.

Seeing this actually changed my impressions of the company. I found them to be less trustworthy than I had thought. They came across as grasping and unpleasant. Involvement in research generally leads people to form a better opinion of an organization. In this case, the reverse is true.

My final issue is this language: "Your testimonial is truthful..."  I am seeing this start to appear in other research documents, at the request of clients. I'm not sure how the lawyers got involved in research, but this feels like a really hideous trend, and one we should all endeavor to stop in its tracks.

 

Anne Crassweller + Frank Graves #NetGain8 - using IVR systems for research

I've been talking to Anne Crassweller of NADbank about research and the challenges faced by companies like hers, so I know this is going to be an interesting session.

NADbank collects the data about readership for the daily newspaper industry, how many people read a newspaper every day and throughout the week, and all the related data.

Frank Graves is a major figure in political polling, and has been a big advocate of IVR polling. This is where you pick up the phone, and get to respond to a very small number of polling questions through an interactive voice response. I kind of enjoy these, they are fast, and usually interesting. And faster than a real person, frankly.

Anne: It's all about the sample. But that doesn't mean we can do, or want to do things, the way we did them in the past. Consumers do not all use land lines, so land line methods are inherently biased now.

She says her stakeholders want the data really fast, overnight if possible. So they have been working with Frank Graves at Ekos to develop alternative approaches.

Frank Graves is now up.

[BTW - first presenters to use Prezi]

Online panels are not reflective of households, and have a gazillion other problems for sampling. Challenges for declining response rates, sampling, commercial pressures for faster and cheaper ... So, they have been playing around with IVR, "robo-calls."

They have been refining the methodology to get better and better samples.

Best lines of the day: "Now that's an exciting table!" "Prediction is really hard, particularly about the future."

Anne is back up, talking about how the HD-IVR delivers results for them.

• Gender results are a closer match to census than any of the other methods compared
• Age results are much closer to census than any of the other comparison methods. This is the only method to really get 18-24 people effectively
• Education is very close to the census, again much better than the other methods compared to.

Part of the challenge is that the CATI surveys are much longer, so they have been trying to figure out what questions to shift to IVR.

Cell-phone only households are the challenge, because so many people are getting rid of their landlines. The debate is, does cell-only define your media behaviour? Their tests have showed that, NO, your choice of being a cell-only household is not a major determiner of your media habits.

My quant buddies are probably following the stats quicker than me, but I think this is quite interesting from the standpoint of just getting good, real, data. There really is no substitute for actual facts, is there?

 

 

Caroline Fletcher on mobile self-ethnography at #NetGain8

I'm a big fan of having people take videos and pictures of events in their lives, whether that is using an ATM machine or walking their dog. And with smart phones, handhelds and tablets, all of this is just so much easier to do. So I'm of course interested to see what kinds of things Caroline Fletcher will be showing us.

Why use mobile video ethnography:

• genuine behavior
• creative expression
• cost effective
• pre-post task - review footage together

There are so many ways to approach qualitative research projects, when do these approaches make sense?

• when people would not want a researcher there. e.g. carving the Thanksgiving turkey, other private or special moments
• capturing pain points, like setting up HDTV, setting up home internet WiFi, IKEA furniture assembly

Saw a nice sample of some edited video.

 

#NetGain8 Tom deRuyck on Consumer Communities

I met Tom deRuyck at the speakers dinner last night. We had a lively discussion about what "community" means in the context of marketing research, and found that we largely agreed with each other!

He says his firm has done more than 500 communities in the last 8 years. Based on that, these are the things that companies need to survive:

• being open - embracing the empowered consumer
• being agile - moving quickly to respond to consumers

The average CEO finds this scary, and some bravery is needed to be open and agile. Most companies do not have two-way dialogues with consumers, even when they claim to. They are concerned about what will be said if they actually open up a dialogue.

deRuyck asserts that marketing (in campaigns) and research (ad hoc projects) are not responding to the continuous nature of experience. Silos in departments, and process-oriented marketing result in lack of agility. Projects are passed from silo to silo for each phase.

His recipe:

• work on project teams instead of departments
• stop working on ad hoc research
• stop working on ad hoc campaigns
• Bring consumers in to the business and put them at the heart of the business

Examples of companies that are doing this...

[1] His favorite cookie, PiM's, a Belgian cookie. He says he and other brand fans are more knowledgeable about the product than the brand manager who has been in the job a short period of time. "Your fans are emotional shareholders because they are emotionally connected to your brand."

[2] Ben and Jerry's wanted to develop a brand ambassador strategy. Found people who score 9 to 10 on a brand fan-ship scale. In a community, these people were the toughest audience. They became angry when a brand team did something wrong with "their brand." They truly want to help you out.

[Aside, @Tom de Ruyck has the best slides ever. Love the typographic design.]

A community - consumer consulting board - a closed online platform on which you bring together a larger group of people for a longer period of time. Closed = invitation only. closed = your secrets are safe. Efficiency of online. Most important charaacteristics - more people for more time. Allows for clashes of ideas, which can be very interesting. More time allows you to think along with consumers while you are developing a marketing campaign.

[3] Heineken - wanted to be known as a design brand. Did a year-long community. Did some mobile ethnography. Got immersed in the world of clubbing. This served as an immersion for a design team of 15 people. A year after the first day of the community, Heineken wanted to actually build the club of the future -- the ideal clubbing experience, as a pop up club in Milan.

After the first immersion, they started sketching. Then they got feedback on the concepts. They worked across silos, brought down the walls between different departments.

He suggests that communities be used to tackle all your qualitative questions. Use for all business objectives. Generating insights. Developing new concepts. Guiding marketers when they are launching a new product.

How to make it happen? Recipe for success:

• first element is getting the right people on board. the high performing community members are more into the brand OR more into the topic than other people are. It is NOT representative. Not a bad thing (this is qualitative). We work with people who are interested and interesting to listen to. You want people to inspire you!
• How many members? Bigger is not better. Their communities are small - they have studied this with Maastricht U. AFter 30 answers, you know what you need to know. In a short-term commuity, 50 people participating intensely will give you this. In a longer-term, more permanent community, you get 150 people. As you approach 150, it becomes more difficult for the moderator to interact and to probe.
• We brief people and create engagement. The incentive is not money -- it is recognition, feedback, and from time to time a small gift. e.g. from Heinz, every quarter they get a basket of products that they have worked on.
• Co-creation of the IKEA catalog around the world. People want to do this! Make it fun to engage people.
• Help keep people engaged. Gamification -- use the principles and psychology behind games to make a boring task more fun. At the question level, the individual level, and the community level. A question is turned into a challenge. e.g. have ten teams compete against one another, based on cities -- show us that your city is the coolest on Earth. (done for MTV globally) At the individual level, give people a status badge for quality and achievement. On a group level, organize battles of arguments -- create tensions, but do it in a game way. At the community level, you want to make the community act as a we, as a group. Set goals for whole community feedback

Chiquita smoothie example is kind of cool. They had regular fruit eaters and "unhealthy ones" switch their diets for a week. At the end of the week, they listed the benefits of healthy eating. Then they had to argue to convince people to change their diet.

Mobile tools are an obvious good fit for communities. But so is going into the social media where people already are. They created a Facebook page that only community members had access to, so they could do second screening during a TV program. So they could help people capture real time feedback (because they were going to be on FB anyway)

Interesting that they sought feedback on ther communities. Campbells' community had a professional moderator, and a participant co-moderator. this person helps to guide asking better questions. Co-moderated communities get 25% more on topic discussion.

Another thing they did was get analytical help from people to understand stories and other input that they don't understand. 20 to 40% additional insights, depending on the topic. Get more insights. The insights that they come up with, that you didn't find, those are the most interesting things.

Have also tried recruiting observers to just watch a community. Then they do first run of analysis, and run it by the observers. The observers provide feedback and correct the problems of the analysis. This freed the people from social pressure and cultural challenges. (This example was research done in China.)

He has a free book on the internet at www.consumerconsultingboard.com

 

 

Annie Pettit at #NetGain8 on Behavioral Economics

Love the title of this presentaiton - "Behavioral Economics: New, or just new to you?" You will be familiar with Annie Pettit from her @lovestats Twitter feed and terrific blog of the same name.

She thinks it's a whole lotta hooey. Interesting start! [Can't say I agree totally, but I am a little tired of people trying to explain all of human motivation through loss aversion tendencies.]

Definition: Behavioral economics is the application of insights and research from psychology to economics.

Basic tenet behind BE: humans are irrational. No surprise there.

Next up: she shows a number of historical figures dating from Adam Smith and covering the next 300 years. People who all documented how human behavior is not fully rational.

Her real argument here, I think, is that there is a lot of documented psychology (e.g. Asch conformity studies in the 50s, MIlgram obedience experiments, Zimbardo situational studies with randomly assigned guards and prisoners) that show how people do things that aren't fully rational, and the power of the pull to conformity. Situational ethics and norms.

She did a little experiment with us about the number of peanuts in a jar. We had to guess. But she planted the first dozen guesses, to show that we would tend to conform to the prior guesses.

Cognitive dissonance. Festinger and Carlsmith, also in the 50s, showed that we evaluate a task as boring or fun based on how much compensation we received. (If less compensation, evaluated as more fun)

Halo effect, Thorndike in the 20s. Our overall impression can be based on one trait. (This has been re-demonstrated in many fields thousands of times. e.g. effect of appearances on perceived intelligence of children, impact of name gender on resume, and many other approaches.)

Impact of all of this - her take:

• people want to give you the truth, but they can't always do that.
• they misunderstand their own behavior

So -- stop asking questions like:

• would you...
• why do you ...
• hypotheticals of any type

Instead, ask what I have done. Ask about past behavior. What did you buy, when, what coupons did you use, etc. THIS is GREAT ADVICE.

 Brian Singh, in his question, suggests that experimental design is something that is growing, because of wider acceptance of human irrationality.

 I think Annie's point is a good one overall -- that psychology wasn't just invented, and we need to pay more attention to the substantial actual science that exists, and has existed for years.

 

 

 

 

#NetGain8 - SAS's Chris Long

We're all fuelled up on chocolate-filled croissants and coffee, and ready to learn more. The gentleman beside me has an Elite luggage tag, which instantly made me wonder many things. Great badge value in those tags, isn't there? It says "I travel a lot," "My company pays for it" and "I always have lounge access." Such is the life of a qually, always observing and overanalyzing.

Next up, Chris Long from SAS, who is talking about marketing analytics and where that is going. Most of his slides are kind of complicated, and have a zillion logos on them, representing the MANY players in social media and marketing analytics.

I do like this graphic, a chart that explains social media.

• Twitter - I'm eating a donut
• Facebook - I like donuts
• YouTube - how to eat a donut
• Instagram - vintage-looking donut
• LinkedIn - my skills around donut eating
• Foursquare - where I eat donuts
• Pinterest - a donut recipe
• Last.fm - now listening to "donuts"
• G+ - I'm a google employee that eats donuts
• I might add, Quora - "how to build a donut business"

Mr. Long says Canada is 1.5 to 2 years behind the global technology curve on social media. From a business / marketing / analytics use perspective. Which suggests there will be more adoption of social media analytics by business, by mainstream media.

How companies will use social media: foresight, brand protection, marketing, sales.

He says we need to understand four dimensions of social media. Conversations, content, context, connections. For context, think Marshall McLuhan, "The medium is the message". Very true - seeing something on Facebook vs Twitter says something to you just because of the context.

My biggest takeaway from this presentation: the slide people will remember and talk about is the one I captured in the bullet points above. It's the one that tells a human story.

Questions on how to use social media data yield surprisingly obvious recommendations -- like monitoring and responding to what people say about your brand. It seems like you don't really need this kind of sophisticated tool to do something like this. My take: we are at the nascent stage in learning how to effectively use all this analytical horsepower.

 

#NetGain8 - Mark Michelson and mobile research

Mark is presenting virtually from Atlanta, which got snowed in yesterday. I know Mark as a fellow quallie, but he is current Exec Director of the Mobile MR Association. MMRA.

Smart phones are the big enabler of mobile MR, both surveys and qualitative. [I recently completed my first true mobile qual project, and it was great. Technology worked well, participants loved it, and we got a video diary of moments through a week in their life, plus Q+A discussion.]

Some other parts of the trend in mobile:

• voice possibilities, live video chats
• passive data collection possibilities
• location-based data
• app-centric communities
• QR and scanner codes to engage and classify
• real time and asynchronous
• Augmented reality - superimposed imagery in the real world (Vision Critical has done some of this apparently)
• gamification - survey questions inserted into the game in order to earn in-game rewards like lives

As wearable computing comes in, many more possibilities for data gathering will be possible, including biometrics.

The challenges now are actually in design, because you get a lot of data from studies of this type. Images, videos, etc.

Other challenges:

• responsive design to fit screen size
• user experience and on-boarding (training)
• keeping participants engaged - balance engagement with bothering people
• participant privacy
• validity of self-reported data, client trust in findings, and participant trust in using the apps
• device ID and interface optimization for specific devices -- there are so many different mobile devices

Relatively few companies are using mobile. General Mills is an industry leader.

MMRA has developed a new set of guidelines for privacy (full standards plus a short form that can be shared with participants in research). These integrate with existing ESOMAR standards, but expand them.

 Best practices:

• be transparent and respect respondents - share intent, expectations, in advance.
• avoid participant burn-out. Not too many questions. Respect time zone - don't send notices out in the middle of the night
• make the study appropriate for the screen size
• channel dictates question type (channel being native app, SMS, WAP, etc.)
• use micro-surveys, shorter length of questionnaires (e.g. 3 to 5 questions) Use data stitching techniques to bring these together.
• use mobile friendly tools - sliders, dials, graphic interface. Avoid typing where possible.

Clearly, the future of mobile research is great, and it will enable many more participants to engage than traditional online methods, especially when we consider global markets.

 

Mark's slides can be downloaded from mmra-global.org

 

 

NetGain8 - leading edge developments in Marketing Research

I'm at #NetGain8 today, and thought I would blog live. These may be fairly brief, but this conference is usually a great one. Plus, I have to moderate the final panel, so I do need to pay attention.

THE FUTURE OF RESEARCH - Jeffrey Henning

Jeffrey Henning from Researchscape International is talking about the future of the MR industry. bitly/NETGAIN8 has the slides, he says. He drew a beautiful metaphor about how roads and transportation have changed, but we still need to go from place to place faster than our feet will carry us. Methods change, fundamental needs do not.

Surveys are a victim of their own success. There is a lot of pressure on surveys, including length, badly written, too complex, declining response rates. And even more, behavioral economics has made it clear that direct questioning methods may not be very useful, because they rely on us being good observers of our own behavior and intentions.

However, some of the things driving growth are: mobile surveys, micro-surveys, DIY systems, automation, and cool features (think feature wars among software and solution providers) like the ability to capture emotions.

Proprietary analytics is one of the ways companies are competing in this environment. Better dashboards that reduce the need for human work to provide specific views, like single location results, for example. Automatic weighting, which is not very good now, is likely to become better as companies keep iterating. Automatic coding of open-end responses is getting better, especially where the answers are very short (e.g. what brand, what tv show). Statwing.com, for example, automatically applies the appropriate statistical method to the data, meaning someone without statistical knowledge can do good analysis. Crowd-shaped surveys change the questions based on early answers. Better weightings, text analytics of long verbatims, and a lot of aggressive innovation will characterize the supplier environment.

Things he thinks we should do to differentiate are interesting. Several of them are already being done by good qualitative people. e.g. infographic reports, posters, storytelling, running workshops to help internalize findings, and increased qualitative analysis. [Personally, I do a lot of these things, and many of my colleagues do as well. So my conclusion here is that the statistical part of the MR community is catching up to the qualitative side. Not sure if I like that trend, but it's still a good idea.]

 

Closing the feedback loop on surveys: tell people what you learned

I get regular impassioned pleas to complete surveys for some product or group. Often multiple requests. I bet you do too.

But how often do you hear back from anyone about what they learned? Rarely, right?

A recent survey from AeroPress.com (sellers of the amazing human-powered espresso maker), had a surprising and engaging follow-up message: they shared results!

There are a few things I like about David's approach here.

He speaks in a human voice: "Such a ton of great feedback, thank you!"

He says thank you repeatedly.

He offers a download of the results.

He invites further feedback!! 

He goes on to tell the reader what they are going to do with the feedback they received -- launch a new product that responds to unmet needs. Now how cool is that?

In an age when every business is striving to engage customers, this approach really stands apart for actually BEING engaging.

Well done Aeropress.com.

What can you do with this idea?

How can you take this idea back into your own business? Are you doing surveys? (Silly me. Of course you are.) Do you ever go back to the surveyed people and provide them feedback?

For research with business people, it's often important to offer some feedback on what was learned in order to actually get participation in the research. They may like hearing what others said, or compare their own situation with the average situation, for example. So providing some feedback doesn't compromise anything secret, it just offers something of real value in exchange for their time and opinions. This is a recommendation I have made to many clients, and the smart ones follow through on it. (Hey, sometimes they even ask me to write the memo! Which is fine with me.)

If you used a panel to conduct the survey, you likely don't have permission to e-mail these folks directly. But you could still post something on your social media sites, liked Facebook, LinkedIn, Twitter or company blog. You could put something on your web site. You could issue a press release.

In your actual survey, you could ask people if they want a copy of the results, and to provide their e-mail address for that purpose. Then you could actually e-mail them a copy of the results, or even a subset of the results.

I'd love to hear from you if you like this idea (or don't like it!) or if you have ever tried it. What happened? Did your customers respond well? Was it hard to convince management to do this?

Live blogging from IIeX - Day 3 - Rebirth of Marketing Research

Larry Friedman of TNS spoke on Big Data, Little Data, New Data and the Future of Market Research.

Marketing research came about because the data needed to make decisions were rare. So the expertise of the industry developed around creating that data. Now, it is more important to find the data that are already out there and learn to use it effectively, by linking it and analyzing it.

Kantar (parent company of TNS) has compared typical survey data measures (such as brand awareness, purchase intent, and purchase in the last three months) with actual retail scanner results. Turns out that the correlation between some of these measures is close to zero.

However, this company now has methods to measure ACTUAL ad exposure, ACTUAL purchase, and similar, by utilizing "tracking" capabilities with their panelists.

New realities:

• Assuming what people tell us is true, is often wrong
• Market Research's role in Marketing needs to be broader than it has ever been.
• Someone has to do something....it's just incredibly pathetic that it has to be us. (Quoting Jerry Garcia)

Friedman has written a post on the Greenbook blog about this topic, which has a ton more information than I captured here.

 

 

Posted with Blogsy

#IIeX live blogging - Qualitative Stream

Dana Slaughter is next up, presenting a number of mini-case studies of qualitative mobile around the world: In the Moment with Mobile. Case studies came from a dozen or so individual researchers who contributed cases.

Mobile is now typically an add-on to an online qualitative platform, although there are a few dedicated mobile-only platforms.

• For grocery shopping, participants were asked to use their smart phone for tasks related to grocery shopping, such as price comparisons, using bar codes.
• Capturing activity in context, such as drinking beer at Oktoberfest.
• Breakfast experience in limited service hotels, to understand the breakfast context for a waffle batter provider
• Seasonal merchandising in the retail environment. What do people notice in terms of assortment, items, signage, etc. What is adding or taking away from the shopping experience?
• Personal care occasions, such as leg care/hair removal for women
• Moments of connection for new parents -- an intimate moment that would be hard to capture through other approaches
• Employee communications over time -- multiple experiences captured in the moment
• Events - Superbowl-watching experience

Mobile can be used at the front end, at the back end, or as the major part of a study. Mobile offers potential for spontaneity, for insights captured in the moment of experience, and for insights captured right in the context of experience. More transparency and fewer boundaries.

An interesting audience question: can you do live mobile, with probing in the moment? According to Dana, not all platforms permit probing. Some of it can be pretty close to in the moment, but right now there is a lot of online followup.

Coffee-break conversation insight: You cannot define qualitative and quantatitve by the method of data collection. It is more useful to define them as methods of analysis.

Next up, a panel discussion with Susan Saurage-Altenloh (Saurage Research), Ben Smithee (Spych Research), and Kathy Doyle (Doyle Research Associates), all qualitative specialists with varying perspectives. One of the biggest changes is that there are so many choices in technology for study design, and a lot more data. Better access to respondents is possible.

In the consumer's quest to find information, they provide a lot of information to us. In that sense, consumers are driving the insights process more than ever before. Instead of asking questions, we are now listening more -- this is a huge shift, which is leading to more effective research.

The data collection piece is becoming easier, so as researchers we have to ask what value we add to the process. Methodology is not as important as insight -- how can you find out what we need to solve?

Lines are blurring, (between qual and quant) or are they? Quali-quant became multi-method begat hybrid. But the possibilities for hybrid methodologies are much greater today because of new methods and technologies. You can analyze social media qualitatively OR quantitatively. It may be that researchers are the ones holding on to these distinctions.

Communities designed for insight purposes are still building brand loyalty (if branded) or category loyalty (if unbranded) -- even if marketing advocacy is not the goal, it happens anyway.

Interesting quote: "If we want people to call us brand partners, we need to stop calling them clients."

Ricardo Lopez of HIspanic Research is going to show us The Tools of a Qualitative Research Techie

Evernote: Gives you access to all your information, everywhere. All information is indexed. Take a picture with your phone, and immediately put it on Evernote, and use this system to instantly capture project photos (e.g. flipcharts.) Evernote recognizes the names of photos, even in handwriting. E-mail forwarding lets you store and save mail that you might want to later reference. You can even assign it to a folder when you do this.

"If you were to go back to the 1970s and show someone one of these Star Trek communicators that we carry around, it would look like magic."

Zendone: A new company that has taken the GTD organizing approach and has applied it to Evernote. He uses it, along with Google calendar and Evernote to process his inbox and manage to-do lists.

Livescribe Echo pen works with desktop software. Livescribe Sky pen works with Evernote only, not via the desktop. It connects to the internet with no cable. This technology is very useful for conducting interviews. It also integrates with Evernote. You can take notes later when you are playing back, and those notes are also linked to the recording. This makes the tool very useful for the analysis.

Video for qualitative has always been extremely useful. With bulletin boards, Ricardo asks all his questions using video to increase participant engagement. He told a great story about recording a video introduction in his office, when his two-year-old walked into the room. Ricardo posted his son singing a Christmas song, and everyone responded by posting videos of their kids during the introduction. It completely changed the whole perspective and tone. So now he does this as a regular practice!

Depending on the platform, you can force respondents to respond in video.

It really pays to have a high-end webcam, for good quality video. Don't use the one that comes with your laptop. A cell phone is even superior, usually.

A challenge with respondent video is the number of different formats. The platform may capture in Flash, a difficult format to work with later. He typically asks them to send the video, and he will figure out what to do with it. Handbrake is software that will convert video formats, but it takes some learning.

He cautions that clients often have old versions of software, and to be cautious about embedding video into PowerPoint. Instead, he uses a Vimeo account, and puts the links into PowerPoint.

For basic video editing, he says your on-board editor is probably fine. To create a video report, he likes Power Director (Mac).

 

Posted with Blogsy

#IIeX - Socially intelligent research, and crowdsourcing to diagnose disease

Philip McNaughton and Andrew Needham (both of Face), are talking about "From Socially Intelligent Businesses to Socially Intelligent Research"

Consumers are doing things that companies previously paid their employees to do. Value creation is moving outside the walls of the corporation. They have a cool hexagon graphic showing how technology platforms enable value creation.

What clients need: make more, better decisions faster, instead of trying to make perfect decisions. (Sounds like Agile to me)

So what is socially intelligent research? It seems to involve "sexy metadata" and "continuous, integrated, holistic view of the consumer." I'm a little unclear, actually, but I still like the concept (System 1 thinking at work!) There is a white paper on Slideshare.

Putting data in context -- look at social data not just in terms of what is being said, but who is saying it, and who they are saying it to. Move away from how many tweets, to what are foodies saying? Instead of how many negatives, ask what is the brand equity?

Question about how many people involved? They start with 50 people through screening and do some early work to get down to about 16. This is a very different approach than the 400, or 4,000 member insight communities we hear about.

My take: If we agree with the view that more value is being created outside the firm (and I do), then how to we actually start to think about adding value to our customers by shifting value creation outside our walls? Is this a strategy we can actually use to drive innovation, instead of a just a trend to observe? This has been done in the past.

Observation: The rooms were switched, but the online agenda is actually correct, so those of us who thought we needed to adjust went to the wrong place. Arggh.

Next up, John Barrett of CrowdMed is going to talk about using prediction markets in healthcare. I met John last night on the deck, and thought this would be fascinating. Stay tuned ...

The company founder wanted to help his sister with her depression problems. (So much innovation comes from this kind of motivation!) He considered The Wisdom of Crowds by James Surowiecki, and the prediction markets that Infosurv had been doing in marketing applications, and in political prediction.

A new person is introduced -- Stacey, who had irregular periods and mild depression. Plus a bunch of other problems. Over 970 days a number of experts could not figure out what was going on. So, a crowd solution was sought. The group of people was given the same information and facts that the family and doctors all had. They were invited to use any resource at their disposal. Most of these people made their contribution within 12 to 15 minutes. They could earn additional rewards for being correct, or being closer to the result.

A number of ailments were offered up. Five were weighted. A second market was run, with the same background and information, and given the chance to invest in only these five opportunities.

The results of the crowdsourcing came up with FXPOI (an ovarian insufficiency), which Stacey took to her doc, who confirmed the crowd diagnosis, and was able to treat her.

This is a start-up situation for founder Jared Heyman, who is still running proof of concept cases, and hopes to take this to a different kind of market. They are not clear where the commercialization comes in this application.

What is important to make crowdsourcing work? A controlled panel produces much better quality information than a river sample.

Posted with Blogsy

More live blogging: co-creation with Bacardi

OK, I got lost in the agenda changes, and found myself in the wrong room, having missed Nils Schillewaert. But now I'm back on track, with Kevin Lonnie of KL Communications talking about Co-Creation.

Kevin likens the setup for co-creation to setting up a party -- lots of detailed planning in advance is needed to create a really good party. The Bacardi community has about 400 people. They were asked to create and post ideas, then others vote up or down, and add their own refinements. They monitor whether the original challenge is being addressed, and refocus if needed. Top ideas are selected and further time is spent on these.

Interesting that Bacardi is running this through the sales arm of the company, so is looking to support the purchase channels, not just consumers. One example of a challenge was to create a better shelf design for the spirits section of a liquor store. Discoveries made as a result included in-store lighting displays, and rearranging shelves by flavour. (Good idea!) Bacardi is now testing this in 10 stores, a real market test.

Why the appeal of LED lighting? On the shelf, there is enough signage and clutter (e.g. price stickers) that you can't see the product as well as you might like. Consumers wanted the new products to be highlighted with lighting, so they notice it.

Another challenge was to create a VAP (Value Added Pack aka gift pack.) Members are asked to design the pack. It turns out that glassware is the key -- people collect them. They also liked having sampler bottles of other flavours.

The two challenges mentioned above took 3 to 4 weeks in total. (That makes sense to me -- you want real co-creation, it might take a bit longer than the 24 hours we were being told about yesterday.)

Key message: Insights come from "truly listening."

A questioner says that part of co-creation is to feedback to the consumer. (Yes! Totally agree!)

Posted with Blogsy

Live blogging from IIeX 2013

OK, I confess that the session on Training the Next Generation of Research Pros by Lisa Courtade of Merck, was actually pretty good. I just don't have a lot of patience for that topic, so didn't take notes. My key takeaway: we need to be polymaths (think Leonardo DaVinci)

On to Imran Anwar, of Microsoft, who has included two whole slides on his bio and disclaimers. Quite witty, actually, but rather a profligate use of time when you only have 15 minutes? (Yes, that is a quallies' take -- we budget time like other people budget money.)

He summarizes five decades of technology: mainframe > PC + Apps > Client/Server > Internet > Cloud. The future: "It will all be wireless."

You will be competing with a lot of other things begging attention. We are beginning a new decade of disruption. It is scary and also exciting ...

He advises us not to be held back by what we think technology can do, or what we are told it can do, because it can probably do whatever we want/need.

Even the people who can't work in safe manufacturing facilities now have the resources to buy second phones because they like the color, for example. That is the change wrought by the price and penetration of technology -- a major democratization of access.

BYOD (Bring Your Own Device) will yield to BYOF (Bring Your Own Future). Love that. Whatever your view of the future is, you can come up with it.

He says micropayments will allow the monetization of almost anything. He sees this helping the #MRX industry. Are we doing surveys, or are we delivering valuable, actionable information that clients can immediately use? He urges us to seek insights into the insights process. To innovate in how we target, measure, manage, deliver, and adapt. To be decisions drivers and change agents -- to be the platform for business innovation.

(Side note: Why do hotels insist on having noisy doors in meeting rooms? Surely this is as important as serving Starbucks coffee? Get out the duct tape!)

Posted with Blogsy

Live blogging from IIex 2013 - Three key areas of behavioural science

John Kearon of brainjuicer says he is going to distill Daniel Kahnemann's work on system 1 and system 2 thinking (Thinking Fast and Slow) and its application to #MRX.

"We think much less than we think we think."

We use our system 2 thinking mostly to rationalize the decisions made with system 1. Proof offered: more car advertising is read after the purchase. (Really?)

"Thinking is to humans what swimming is to cats. We can do it if we have to."

Three key areas to think about from behavioural science:

Environmental: how things are framed. The example used was estimating the age of Mahatma Ghandi's death. If you start from 66 and count up, you estimate lower than if you start from 90 and count down. Rolls Royce sells hardly any cars from their showroom. They sell more at boat shows. Because when you are looking at expensive luxury yachts, their cars seem cheap.

"The whole of quantitative marketing research is a system 2 edifice."

Social: We should be called homo mimicus, the copying one. He cites Crocs as sure evidence. Wisdom of crowds are better estimators of what other people will do than we are at predicting our own behaviour. Predictive markets in elections are more accurate than polls for predicting British elections.

Personal: How we feel dictates what we do. Feel nothing, do nothing. Feel lots, buy lots. Famous advertising that creates the biggest commercial return works ... by creating overwhelming emotion. Message triggers system 2, and it undermines system 1. We don't feel as much. The more we can make people feel, the bigger the commercial return.

According to John, no one has all the answers. But we are at a moment in market research history where we can try new things. His suggestion: run a deprivation study with your own tracking study for a year, and redeploy the money to try new things.

[Observation: We still need a better mic solution in large rooms. The football mic to toss during the Q+A period. Maybe the drop-down robotic mic hooked to the ceiling. Perhaps the drone mic, that will fly to the questioner. Because the current system clearly does not work.]

Posted with Blogsy

#IIeX day 2 - Live blogging starting with Mark Earls

Mark Earls started the day with his topic, I'll have what she's having, and the observation that we are more like Capt Kirk than we are Spock (Behavioural Economics distilled to one slide.)

Social influence and social learning is the most important factor shaping human behaviour. The UK government is working with behavioural economics insights to shape policy (seriously, data and science? innovative idea. Note to self, find the study publication, also called "I'll have what she's having".)

Several categories are more influence by social than independent thinking: alcohol, music and voting, for example. Deodorant and insurance purchases are more independent.

Paul McDonald of Google Consumer Surveys, a very good speaker, is talking about the future of surveys. They'll be mobile. They'll be chunked up into small bits and then reassembled. (Think packet data applied to surveys.) They are talking about having a billion person panel. They want users to install the app on their phone, and then earn free music downloads and similar things as incentive to respond to surveys. (Kind of sounds like a panel that just has better validation via their background analytics. Oddly, the more familiar this becomes, the more familiar it seems.)

According to Paul, we should be asking ourselves, "How can I take advantage of Google's Technology?" So I did ask if they could help me out with Hangouts as a qualitative product. He says that's a different division of the company. Oh well.

They are about to announce a website satisfaction product. Their pitch, clearly, is to the DIY end user, but maybe also to smaller research firms that might want different tools. Certainly, Google is not going to be providing analysis of the data.

They are offering free samples. So sometimes, Google is a conventional marketer, just like the rest of us. Speaking at conferences, offering free samples.

 

 

Posted with Blogsy

Live from IIeX

Best quote of the morning was "In God we trust; all others bring data. If you don't have any data, you're operating on opinion. If it's a contest of opinions, we're using mine."

Observation: one way to ensure everyone has their head down during your talk is to put the conference program on an app.

Next up: Google Consumer Surveys (John Sadow) and Gut Check (Matt Warta) promise to talk about their case study after we listen to their pitch. Odd approach, because most people here would have heard about Google's survey offering, and we'd all be more interested in the pitch after we heard their case study.

I had stopped by the Google booth earlier, because I have this crazy idea that Google Hangouts might be a good real-time webcam research platform. Not sure they really got it, but they did concede that some people are already using Hangouts that way.

The promise of the case study is doing a full quali-quant study in 24 hours with equivalent quality to traditional methods. The category and product: Cover Girl foundation makeup.

Matt says that they don't do a lot of preliminary design -- they dive in and do some iterative stuff. First iteration was 25 in-depth interviews (not yet defined exactly what he means by in-depth interview. Not clear how GutCheck is recruiting people. Looks like it is some kind of panel.)

In the end, they took 72 hours to gather data, and a week to provide a report. I'm guessing this was with dedicated individual people resources. Still pretty fast.

What is appealing here -- the ability to iterate quickly, such as with concept development work. They're calling the approach Agile research, a nice label.

Rajan Sambandam of TRC: Behavioural Conjoint: Can Behavioural Economics and Conjoint Analysis Coexist?

Conjoint analysis is an approach that uses statistical modeling to evaluate tradeoffs. A conjoint study will present you with a number of scenarios where you need to choose one of two options, where feature levels and prices are varied. Example: Folger's in a bag for $9.99 or Maxwell House in a can for $8.99. Choice is modeled as a function of each feature being tested.

One difference is that conjoint uses a within-subjects analysis, whereas in behavioural economics, it is a between-subjects design. There is generally some subterfuge in behavioural economics -- the thing being tested is not supposed to be obvious. In conjoint, it is completely obvious.

Another way to look at this: behavioural economics looks at how environmental stimuli affect decision making.

What they did, was run the same study two ways, to determine whether the presence of a picture in the stimuli affected the results of the study, where every other factor is held constant. What did they find? The presence of a picture (even a fairly bland picture of coffee) beats a plain text stimuli across all other factors (such as brand, features, price).

(I think this means that we like anything presented with graphics, more than we like things without graphics. Probably why most advertising has graphics, not just words. But now we have proof. I'm being a bit unfair, because this guy is clever, and a really strong presenter. He would make a good prof, which I guess explains why he is adjunct at Columbia.)

Last up today, Rudy Nadilo of Daprosey talking about Using Data Visualization to Drive Stakeholder Value. He argues that reporting tools have not changed much in 20 years. "Instead of static slides, what if you could provide dynamically generated information in a much more visually engaging way." They're nice infographics, and they appear to be created within their tool. A pitch, but a nice pitch.

Off to cocktails now. And no, I won't be live blogging from there.

 

Posted with Blogsy

When surveys don't work

A recent international study about blood donors shows why asking direct questions of people is not always a good basis for decision making.

The background

International guidelines prohibit monetary incentives for blood donation, based on concerns that any form of incentive may actually reduce motivation, at the same time as it puts the blood supply at risk from those motivated only by the cash rewards.

Extrinsic rewards (i.e. cash) can sometimes interfere with intrinsic rewards (feeling good about yourself).

The surveys showed one result

These policies have had support from surveys that ask people if they would donate blood based on receiving cash rewards.

"Policies that prohibit monetary incentives have been based primarily on population surveys in which respondents were asked if they would donate blood if offered a cash reward, and the majority nixed that idea. But Lacetera said responses were based on a hypothetical premise, not on studies of actual potential donors who were offered an incentive."

Nico Lacetera quoted in Globe and Mail

The real world experiments showed another

Actual experiments showed very different results. Donations increased for all sorts of things, like a day off work (Italy), a $5 lottery ticket (Switzerland), a $10 gift card (USA.)

What it all means: be careful with hypotheticals

We need to be careful when asking people direct questions about hypothetical situations. Human beings are great at coming up with answers, but our own ideas about our behavior are not always accurate. The futher removed we are from the real world situation the more this is true.

Running small experiments is a much better approach, whenever possible. In digital marketing and in things like loyalty programs, it is pretty easy to do this, and marketers are learning a lot from testing approaches.

But what if you can't run an experiment? 

You should still try to avoid just asking the direct question. As a researcher, what I try to do is set up tasks that are similar, ask indirect questions, observe actual behavior, and otherwise try to overcome this problem.

Topics that are most likely to give you problems are direct questions involving motivation, hypothetical situations, and any topic where there are strong social beliefs or taboos, or where people might desire to paint themselves in a positive light. (A great many things, in fact!)

Resources

Here's a more detailed discussion from researcher Nicola Lacetera written in 2008

Here's the source of the scientific article in Science: Economic Rewards to Motivate Blood Donations published in 2013

Finding insight: what art projects have in common with insights projects

The art project, now documented as a book, Mapping Manhattan: A Love (And Sometimes Hate) Story in Maps is an interesting exercise in insight finding.

The artist, Becky Cooper, was initially asked to document the locations of public art in the city as a paid project, and realized how we create our own internal maps of places. How each space is formed of our memories and our experiences. So she created some maps on a printing press, and asked people to fill them in.

These three samples from the Tumblr site are wonderful examples of how people responded in highly diverse ways. As a researcher, I can tell you that this is EXACTLY the kind of thing I see when I ask people to produce a projective exercise* of this kind, such as a collage or a mind-map, or a video, or a story. 

You can see how much rich information is here to be mined, should one choose to do so. 

One could easily argue that all great art is about innovation and insight, actually. If you are ever in need of inspiration, look at what contemporary artists are doing. 

Resources:

I learned about this work via BrainPickings, a somewhat unfortunate name for my new favorite e-mail newsletter.

*Projective technique: has it's origins in Freud and Rorsach. Great definition from my buddy Monica Zinchiak: an "ambiguous stimulus designed to evoke inner thoughts and emotions".  The purpose is to start a different kind of conversation.

Are people gaming your customer satisfaction surveys?

We love our car dealer, we actually do. They look after us well. You could picnic on the floor of their service department, it's so clean. The organization prides itself on service, and wants to do well on the customer satisfacton index.

In fact, they are so focused on this that they help their customers anticipate and respond to the questionnaire even before it arrives!

"...please take the time to complete the survey with all 10's if we met and exceeded all your expectations"

It's hard to say what impact this approach is actually having on the feedback they receive. Certainly the letter provides a clear indication that they are trying hard.

One thing I do wonder about is this -- does their approach mean that their actual results are almost useless to them in terms of monitoring and improving? It might. You'd need a statistician to help you determine this based on comparing sets of data.

There's an interesting thread on LinkedIn discussing the challenges of actual cheating on this kind of survey, and how it can be detected using statistical methods.

My take:

If you want to know what's really going on with your customers, you probably need to go beyond this kind of automatic approach. Or at least supplement it with some telephone interviews or some good qualitative research.

Another problem with expectations is that they are always moving up, based on your past performance. So exceeding expectations becomes exceedingly difficult.

Related articles

Gathering customer feedback: the good, the bad, the ugly

Write Better Surveys Now: Secrets to Successful Surveys

We've just launched our newest e-book designed to help people write better surveys. It's written for average business people, consultants, freelancers, coaches, professional speakers, workshop leaders ... you get the idea. Anyone who is looking for a fast focused resource to help them right now, and save a ton of time.

This goes way beyond the FAQ in your online survey provider, I promise. Plus it's organized all in one place.

End the survey torture now - write better surveys.

I decided it was time to stop complaining and invest some time in a solution. There are just so many bad surveys out there.

So I decided to write a nice focused e-book  for the people who need it most. Not research professionals (who do sometimes write terrible surveys, of course, but should know better). I wrote it for all those people who are not in the survey business, but find themselves needing to do a survey anyway.

Consultants, mostly. But also facilitators, coaches, speakers, trainers. Volunteers on association committees and boards. Association executives. Analysts who haven't been sent on training yet. Managers outside the research function who find themselves doing a survey. In short, all those folks who have surveys thrust upon them and would like to do a better job at them.

There's lots more about the book here, including the table of contents. You can buy the e-book for immediate download using any credit card you want, from Gumroad, a nice simple service that just does this one thing very well. No excuses, it's seriously good value.

Research Tools for Consultants: The Survey

Think of it as buying us both a really large latte, and me telling you everything I know for about two or three hours. Well, as much as I could put into 42 pages, anyway, and still keep it readable, fast, and focused.

So please. Be part of the solution. Stop the survey torture. Read the book. Follow the simple rules and checklists. (And don't embarass yourself with any more awful surveys.)

Gathering customer feedback: the good, the bad, the ugly

I wanted to boost the usefulness of my iPad with a Bluetooth Keyboard, and ultimately got the sleek and delightful Logitech Ultrathin last weekend.

The Good

As it happens, I used BestBuy Canada's online option that allowed me to reserve the item for in-store pickup. You don't have to pay in advance, but you do have to provide basic contact info, including your e-mail address.

The purchase itself was uneventful. Go to store, pick up item, pay for item, head home. Worked well, I would do it again, except...

The Bad

Yesterday, five days after the purchase, I started receiving requests for feedback via e-mail.

On Thursday, I deleted 15 such requests. This morning, I see there are 11 more in my e-mail. EEK!

In self-defense, I decided I should answer the darn thing, provided by ForSee.

The Ugly Deconstruction of the Survey

The e-mail promises a 5 minute survey. This in itself seems crazy to me. It should be more like a 30 second survey, with about 3 questions. But wait till you see it.

There are 34 questions, and 32 of them are marked * meaning you can't finish the questionnaire unless they are answered. This is an odious practice in itself, because it's so one-sided. I might be willing to answer a few questions, but these questions are silly.

They've decided there's a model involving a lot of factors, and they're going to make me answer every question in the model.

Naturally, I am not going to get a thing for investing my time in this.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I thought perhaps there would be a place for open-ended comments, which there is. However, the survey makes it clear that no one will follow up on my comments.

I love the part that says "If you would like us to contact you about your feedback... visit the contact us part of the web site."

There are several links at the bottom of the page, but none of them go directly to a contact page, or even better, spawn an e-mail or contact form.

Interestingly, as soon as I clicked through to the Forsee site, YET ANOTHER survey opened up!!

Yesterday, I posted a Tweet to @BestBuy about the multiple survey invites arriving. They actually did respond, and copied their response to @BestBuyCanada.

For the record: I am not answering your hideous questionnaire. And if you loved your customers, you wouldn't ask them to do this.It's a hideous process, even if it was working correctly.

Magic, sausages and insightful creation: perhaps best enjoyed from a distance

I've been thinking about cake-baking as a metaphor for the process of report writing. I'm convinced that the process looks fundamentally different from my point of view than it does from the client's.

To read the text that accompanies this picture, you need to bounce over to the new MRIA blog, where you'lI see the full article, called Magic, sausage making, and the baking of insight cakes: or, why I hate providing draft reports

Yes, Apple does research

There's a myth out there that Apple does not do marketing research. This is nonsense, of course, but seems to have tremendous staying power.

I saw something today that pretty much proves they conduct research -- they might call it product development, or usability, or ergonomic, or whatever, but it's research into what people want, what they like, what is comfortable for them. 

When you test 124 different prototypes on 600 people, that's research. Good companies do it. Including Apple.

Insights on a budget: How to do Marketing Research with no money: Part 2

In the previous post, I talked about how to do a few interviews on your own if you have a need for insights (and who does not need insights?) but no money. It's a common problem for small businesses and start-up businesses.

I do recommend avoiding some kind of high-tech thing in favor of just actually talking to people -- well listening to them mostly.

But, if you want to do this online, there are options. Some are designed for the DIYer, others say they can be used by anyone.

Here are a few that I know about. Not a complete list by any means. (Note that I am talking qualitative here.)

Option 1 -- use a free survey tool but ask only open-ended questions

This is actually a pretty good option. Think in terms of maybe five questions, and give people lots of space to answer. This is not about statistics, this is about using technology to get yourself a mini-qualitative study.(Seriously, forget statistics in this instance, just go for actual insights.)

If you think this sounds feeble, let me tell you that Jackie Huba (yes, that Jackie Huba) used exactly this method to conduct some of the advance planning for the Dell Customer Support Think Tank that I've been telling you about. She had about five questions, all open ended. It was a very effective approach.

Top tips: Be short. Rewrite. Rewrite again. Use plain language. Ask a "big" question. Leave a big space for an answer. 

Option 2 -- consider these other tools and resources

There are a few services out there designed to work for the DIY researcher. Full disclosure: I have not used all of these services, and I am not getting paid for this. (Maybe a drink at the next industry conference, she says hopefully?)

You can find a very good directory of online qualitative platforms on the Greenbook site -- the NewQual Directory, as well as lots of good articles about how this stuff works, written by some online experts, Jay Zaltzman and Betsy Leichliter.

Gut Check is a service designed to help you find respondents and complete -- fairly quickly -- small scale projects on fairly narrow topics, like testing an advertising concept. They will help with recruiting participants from their panel, or you can recruit your own. They provide a library of video tutorials and other forms of help.

ITracks is also a platform that provides leading edge professional grade tools to quallies. But they have a package called Reality Check that gives you the ability to do a quick concept test and get responses quickly. It's stripped down, but looks promising.

2020 Research has a suite of online qualitative tools. These are leading edge professional grade tools, but there's nothing stopping you from renting their platform, reading through their extensive resource base, and running your own project. There are a couple of articles on their site about the pros and cons of running your own projects.(I thought they had a tool specific to DIY, but I'm not seeing it there.)

BlogNog is an online research platform that has a lot of features, but they also promote it as a good tool for DIY research.

QualVu is an online video platform, that also promotes a DIY option. I'm not clear on how the DIY option is priced or handled differently than if someone like me was just to rent their platform.

IdeaScale is a tool that is designed to encourage the submission of ideas from a group of users, and let them vote up, vote down, comment and enhance the ideas. Depending on what you have in mind, it might work well for you.

There are MANY more tools, as you will see from the NewQual directory. Just like going to Home Depot, buying the power tools is actually the easiest part of the project. Just remember that carpentry adage: measure twice, cut once. 

Have you had experience with DIY? I'd love to hear from you on this. Was it a happy experience? Would you do it again?

Guerrilla marketing research: how to find information on a budget Part 1

My regular workout buddy is an outsourced marketing consultant. As we sweat it out in the cardio area we yak it up about many things. A frequent topic is the desire of her small business and start-up clients to get some kind of marketing research insights on essentially no budget.

Recently, she was telling me about a start-up who wants to get 30 interviews done across North America with various types of business owners, and has a budget of $5K for the whole thing. As you can imagine, a range of expressions crossed my face when asked for advice on how to do this!

It's a common problem, and I'm not unsympathetic. In fact, I keep thinking about putting out some kind of e-book on this topic, as I get the question so often. (What do you think about that, dear reader? Is this a good idea? What needs to be in it?)

What advice do I give?

For right now, here's what I usually say, or variations on this theme. (I'm thinking B2B here, because it comes up more in that context for some reason.)

1. A few really good in-depth interviews with the right people are better than a whole lot of interviews.

2. 30 interviews is not some kind of "random sample" or "more representative." The ONLY reason to do that many interviews would be to systematically cover a few people in many different categories. e.g. ensuring you have some people from each of six industries, for example, and subsets of "users" and "non-users" in each group. Or smaller and larger. Or more sophisticated / less sophisticated. Or some other dimension. (These are research design considerations.)

3. For business buyers in North America, regional location is only important because it affects industry, regulation, and access to solutions. Silicon Valley business is different than Chicago business because it's Silicon Valley industries, not because of the warm climate, unless you are talking about landscapers.

4. Research done with friends of the owner is dangerous, because they are anything but disinterested and unbiased. This is the worst kind of convenience sample (Yes, that's a real research term, and it means "people we could easily get hold of.")

5. You can get people to participate in research without a monetary incentive, but there really needs to be something in it for them. Examples of "something" include: a copy of the findings, a chance to network with peers and talk about shared concerns, a free meal in a nice setting, a chance to influence development of a new business concept, free stuff from the sponsor.

The reason professional researchers don't do it this way is that a) it's not fast/efficient enough and b) it's not reliable enough. It's not because it's impossible.

6. Assuming you have the right people to talk to, it's all about the questions you ask, and the environment you create, and how skilled you are at listening. You don't generally hire a carpenter just because they have tools -- they also know how to use the tools. If you decide to build your own deck, at least keep it simple. This leads to point 7 --> how to use the time in the interview.

7. Spend most of your time just learning about them, their business, their interests, what kind of solutions engage them. Do not start with a bunch of direct questions. Soak, soak, soak up the information about who they are and how they go about being successful. What are their pains? How do they deal with those pains today? This is always the key to developing good value propositions.

8. Plan to spend some time going over your notes in detail, and figuring out what the differences between people are, as well as what the similarities are. Going for only similarities is a very common newbie mistake that is made even by trained researchers.

9. Don't go all high tech, that won't substitute for a lack of finesse on 6 and 7. Just phone people up using that good old OLD tech, the telephone. If you are going to record the interview, tell them.

Planning help

I have a short e-book -- it's FREE -- that will help you think about what you are trying to figure out. Check it out. I get a lot of good feedback on it from people who take 30 minutes to work through the questions.

Your challenges

If any of this resonated with you, I would love to hear from you, either in the comments, or send me an e-mail with your thoughts -- susanabbottarc[at]gmail.com.

Research industry trends: under the covers in marketing with the GRIT report

Marketing is a function that runs on information much more than most people realize. From the outside, it looks like a beer commercial -- lots of cool people doing creative things. Under the covers, it's much more analytics driven.

Research is a big part of that analytics process -- so research trends tell us a lot about what's going on in the world of business.

Greenbook's Industry Trends Report (GRIT) comes out twice a year and tells the story. There's a wonderful infographic with the current issue, available here. The graphics below are extracted from same - the red circles are things I added.

Marketing wants results -- if innovation gets there, fine. If not, forget it

Innovation is on the list, but not if it compromises feasibility, budget and speed. Nobody in corporate life can afford to take a lot of risks these days -- so they're not trying out innovative methods unless they solve some other problem in a big way.

Social media analytics aren't just buzz -- it's happening

The longstanding buzz about social media measurement is finally warranted. People are using it. Which means that business is taking it very seriously indeed. Everyone is still learning, it's all still changing, but it's here to stay.

Budget isn't everything -- because research questions are getting tougher

Businesses are adopting some very expensive approaches, one of which is MROCs (aka insight communities). MROCs are an innovation that IS solving a problem for clients -- these communities provide more in-the-moment access to people as they go about their actual lives. But they require real skill to set up and manage, and they are anything but cheap.

They may be difficult and costly, but they are working and they are gaining traction. Clients are finding real value there, which is why their use is growing. The growth in MROCs also shows the need for deeper insights -- when you are swimming in data (as most large organizations are these days) what you need is deeper insights. Simple questions can be answered from all kinds of internally available data. What's left for research are relatively difficult questions.

I think Gamification is a trend that suffers from sounding "bigger" than it is -- it is really just making surveys more fun. Focus groups and other qualitative methods have been moving in this direction for a long time -- mostly we call this "projective techniques".

Mobile research is also coming of age, because it is solves a problem for clients (better insights in the moment) and leverages a technology that is ubiquitous. Unlike some of the other methods discussed, however, mobile research promises to be inexpensive (eventually). Mobile will include surveys, in-the-moment reactions on site ("how did we do today?") and mobile qualitative ("take a picture of what's happening right now, and send it to me with your thoughts about...")

This infographic is evidence of another important trend affecting everyone -- we all want information that is easy to consumer and easy to find.

Another great job from Lenny Murphy and the GRIT team.

 

How to save money on qualitative research

There's a great discussion evolving on the Greenbook blog, started by Jason Anderson with his article "The 7 BS Stories of the Market Research Business".

I don't agree wtih everything he says, however his perspective is likely reflective of what a lot of research buyers feel. And his vantage point from inside the video gaming industry (life doesn't get much faster moving than that) is certainly interesting and unique.

Be sure to check out the comments too -- some good value added material is posted by other researchers in response.

Is good research expensive? Sometimes it is. There are ways to ensure you are getting the most from your budget, however. I published an article on exactly this topic (link below).

Link to PDF: Good to the Last Drop: 10 Ways to Save Money on Qualitative Research, by Susan Abbott, published by AMA MarketingPower.com

Must read article: Rory Sutherland of Ogilvy UK on how to measure advertising

Every time I talk about advertising measurement as a 'black art' I get objections from people telling me how their normative metrics are well tested and very useful, blah, blah, blah. Let me be clear here: I do qualitative testing on advertising, and I'm not too positive on that approach either.

Why so negative? Well, there is a lot of good evidence that our actual preferences and behavior are not all that well related to what we say.

A brilliant article, "Questioning the Nature of Research" reports on an interview with Rory Sutherland, chair of Ogilvy UK, who makes a strong case that our current measurements are not strongly rooted in behavioral economic theory.

Sutherland's point is not that research is a bad thing in general, it's that the current norms for how research is done on advertising don't work.

...as you get closer to the point of decision, there are factors at work which never really appear in conventional market research. They are contextual factors, social factors. No-one in any research group would ever say, “If there are four brands of shampoo, I’ll buy the one that has most bottles on the shelf”

This is true, by the way, not just of survey research, but also of qualitative.

The best qualitative research approaches come at the topic from right angles -- indirectly. Like a magician's deception, we want to focus your attention on something so that you respond as naturally as possible. Or we try to mimic the actual circumstances of your decision. Still, no self-report in the world is as good as watching a thing happen. Or setting up experiments that show you what is actually happening.

What we have learned from behavioral economics research is this: "Your preference is a product of your behaviour and not the origin of it." If you customarily go to one drycleaner, you will be able to provide a rationale for that choice, (e.g. quality, service) which is likely to have little to do with the real reason (e.g.location).

A useful and thought-provoking article to get your synapses back to school and making new connections.

 

Are you capturing the insights of the Introverts?

Our society has a strong bias in favor of extraversion -- those who are energized by being around others, who can process information quickly, and blurt out opinions as fast as they can blink. But there is a sizable minority of the population that does not fit this pattern -- people with a preference for introversion.

I wrote recently on the NewQual Blog about the advantages of online asynchronous research methods for capturing the insights of Introverts. And boy, this really resonated with people!

A couple of people sent me private e-mails [as introverts would do, not wanting to be loud and out-there, for the most part].

It's not just qualitative research that suffers from this problem -- virtually all marketing research does. Consider the telephone survey, where you are asked a battery of questions, many with multiple choice answers or agree/disagree scales on relatively complex topics.And you need to answer pretty much immediately.

A lot of online shopping has time limits that test our patience -- we have to choose among options quickly, or the system "times out". But with many options to choose, even the speedy must find it challenging to react quickly enough.

For more on this topic, check out the other article here:

Introverts get their day: When discussion forums and bulletin boards can be more inclusive

What's your take?

Is the world stacked against introverts? Does anyone give you enough time to consider before expecting you to choose, react or opine?

To probe or not to probe: Active listening versus poking with sharp thingy

More and more qualitative research is happening online. One of my favorite tools for this is the discussion forum, also known as a bulletin board. This is an asynchronous environment -- everyone is not necessarily present at the same time. [I so wish there was a better word for asynchronous!]

It's not quite the same as face-to-face. In person, if I hear something intriguing, I might ask, "Can you say more about that", or something similar. This is known, in researcher parlance, as a "probe". Although we try to make it gentle -- drawing out more information and ideas -- sometimes I think it feels more like a sharp poky thing.

In the discussion forum environment (asynchronous), you definitely can probe and ask follow-up questions.  The technology support for making this easy has improved a lot – we can now flag things, send a mobile message, or e-mail to the respondent.

My approach on this has changed a lot over the years -- I try to be more of an active listener, and less of an inquisitor.

Treat participant energy as a scarce resource

I treat follow-up questions as if I have only a very limited number I can use, and I must use them only when really important.  Here’s why:  I really love it when participants start to engage with each other.  But they do this less if I’m constantly butting in.

Not only that, but asking people to go back to something they said before is sort of like trying to rewind a conversation by hours or days — people have moved on. The energy is now somewhere else.  So you spend a lot of participant energy asking them to go back.  When people do go back, their responses are often not worth the cost in energy.

The virtual head nod

Probing does have one clear benefit — it shows people you are listening.  And listening is hugely important.  Participants see what you say, and see whether you are listening and understanding them or not.

If I don’t have a definite need to follow-up for additional detail, I sometimes restate what I am hearing, just as I might do in a F2F discussion.  This approach generally results in feedback from the participants that they really felt “heard”, which is so important.  Who wants to talk to an empty room?

The virtual head nod is even shorter — just an indicator that I’m there, reading and listening.

The challenge

Clients distracted by other work may log in and read something intriguing that happened earlier in the week. Then they want me to post a follow-up.

To keep everyone happy, I tend to bring this kind of thing back in with a fresh topic later on, rather than ask people to go back.

I do make my presence regularly felt, as often as possible given the demands of eating and sleeping, but limit the number of probes.

There is another school of thought that you can run these projects with no probes at all — I think that would produce significantly reduced output. However, there are some new DIY qualitative platforms that essentially take this approach. There is no active listener.

What’s your view?

Does this resonate with anyone out there? Either as a participant in some form of interview or discussion, or as a moderator of same.

Portions of this were previously published on the NewQual Blog.

How to save on qualitative research

No matter how large your research budget, you probably wish you had more. Here's my list of ways to save money on qualitative research -- some more obvious than others. You can get a PDF of the full article, with more details on each method, here.

Here's my top 10 ways, in brief:

1. Keep a tight focus on objectives
2. Reduce the number of participants
3. Cut the travel
4. Manage the number of markets you touch
5. Manage back-room catering demands
6. Decide if you need transcripts, video and other goodies
7. Work with realistic deadlines
8. Remember that the screener is not a survey project
9. Consider your written report needs
10. Be a valued client, easy to work with

These are times when organizations need insight more than ever to compete, but every
expenditure is under scrutiny. A collaborative approach with a skilled researcher will help you
get more for your research dollars without sacrificing quality of insight.

Resources

This piece was published by AMA in their newsletter as "Good to the Last Drop: 10 Ways to Save Money on Qualitative Research".  You don't need a subscription to read the article, and there's a nice PDF here.

New Qualitative 101

Greenbook, in conjunction with my buddies at QRCA, have just come out with a directory of New Qualitative providers.

Quite apart from the fact that I am listed in it... you would find it useful for a few other reasons:

• Great basic articles telling you what you need to know about various methodologies. Well written by experts Betsy Leichliter and Jay Zaltzman
• Details and contact information for the major platform providers
• Listings for researchers experience in these methods

You can get a hardcopy for $29, but check out the digital version for free, here.

There's also a blog (I'm writing there too), which has some fantastic contributors, all people who've done a fair bit of heavy lifting in new methodologies, and will be sharing best practices, new ideas, and whatnot.

It's good news to finally have a resource in one place, and I'm sure it will grow as providers seem to be expanding continually. Many of these providers were simply lost in the giant directory that has everything from survey sample to focus group facilities listed.

So now we have resources in one place, plus information too. And that's just great news for practitioners and clients!

 

Conversation in slow motion: How to observe a bulletin board focus group

I'm currently moderating an extended online B2B project involving business execs in a discussion forum. It's somewhere between a bulletin board project and a community, because the same people have been involved for several weeks.

It's demanding for me, but hey, I designed it, so I only have myself to blame.

But extended online qualitative is really challenging for the clients, who have many demands on their time in a typical workday. When the data really starts to build, after a few days in a project, keeping up can be overwhelming.

It's all well and good for me to have insights and distill them into nuggets of wisdom. But they're as good as useless if the client doesn't learn anything. And the best place to start the learning is in the backroom, just as it is with F2F research.

Here's what I've learned that seems to help people engage:

Tips to help the team stay engaged, and help your moderator manage.

[1] Log in early and often

The first day, it often feels like nothing is happening. Team members who logged on expecting activity can get discouraged, and when they look at the transcript later, it can feel a bit depersonalized, and a lot overwhelming.

If you log in frequently, you will see the same ebb and flow the moderator sees. Some topics attract immediate interest, others are slower to build. It’s easier to see where participants are spring-boarding off others’ ideas. Even though the transcript is time-stamped, seeing the evolution is a richer experience.

[2] Schedule team meetings to review insights

One team scheduled daily debriefs in a conference room where they projected the discussion and talked, just like they would in a backroom setting. Snacks included.

[3] Divide up the work of observing

Assign individual team members to cover off one or two segments in a multi-group discussion forum, then report back to the team on their segment. Another approach is to divide up by days (or weeks, for longer projects).

One project team held a group meeting where they projected the evolving discussion and reviewed interesting things together, just as they would in F2F research in the backroom. What a great idea! 

[4] Add comments

Observers can usually add comments to the transcript that are invisible to participants. This is a great way to capture a budding idea, and can help the moderator see possibilities for probes.

[5] Ask for follow-up probes right away

The faster we can ask a follow-up question, the better. Going back more than one day in a discussion tends to sap the group’s energy, and often does not work anyway.  Have a plan in place for how you will direct these to the moderator, to prevent frustration on both sides.

As a moderator, I may make the decision to bring something back in later to the discussion -- even days later. Or I may know that the topic already came up earlier. So if the moderator doesn't respond to your request, try not to get frustrated -- they are likely not ignoring you, just trying to manage the flow in the best possible way.

Acknowledgements

Portions of this article were previously published on the NewQual Blog as Bulletin Boards from the Backroom: 5 Tips for Observing a Conversation in Slow Motion.

If you haven't seen the NewQual blog or directory, you should check out this new resource: NewQualitative  The directory (finally!) brings together all the providers of online research platforms, and it's a good place to discover other resources and content.

Is DIY research going to put us all out of work? The debate rages on!

There's an interesting debate raging around about Do-It-Yourself Research. I posted this comment earlier today on Power to the People, an article by Joshua Hoffman on the Research Access Blog.

As a quallie, I am well aware that lots of people think what i do looks not just easy, but dead simple. Relatively few clients want to do their own, although some do. The usual reason appears to be cost, from what I can see, as recruiting is no faster or slower for them or me.

I find client's analysis of qualitative is usually of the fast kind -- first impressions. If they did more analysis, many of them could come up with what i come up with.

The real issue for me is more one of raw capability. People who have the capability to do from an analysis perspective what I do certainly exist inside client companies, but they are typically senior managers and executives, and are primarily in a managerial or strategic role.  If client companies were prepared to hire people at a senior level to do this work, they could likely obtain great results.

A couple of other things are often missing when clients DIY on qualitative. The longer the person is in the job, the less breadth of exposure they get, the less ability to bring in contextual learning from a broad base of clientele. 

For some methodologies, it could be difficult to get people to tell you the unvarnished truth -- i say this because even as an independent, I (and my colleagues) often have to go to some lengths to assure people of true anonymity in their feedback. I struggle to see how this could be done by an employee.

I also -- as do many consultants -- spend a truly massive amount of time and money on my professional development every year. I attend conferences, I speak at conferences, I attend training events of various sorts, I participate in several professional organizations, I read widely.

My clients, on the other hand, are as busy as small mammals, scurrying on their wheel, often without the time to really think about anything in depth. Their voicemail says they are "in a series of meetings", and that menu rarely seems to change. Of course they have access to learning opportunities, but most are mandated by the company, not chosen by themselves for what they bring.

Even large research organizations have relatively few qualitative people involved in the world outside the corporation, which I would guess to be for cost containment reasons. They send one person to a conference with the idea that they will share the learning.

There is a lot to be said for people who strive to invent new things every year to compete. If I worked inside the client organization, I might not feel the same level of pressure to keep up and be inventive to get ahead. It would be easy to fall into patterns. As it stands now, I am constantly trying to figure out how to either add more value, or ratchet down the third party costs somehow.

Finally, I think the cost argument is upside down where qualitative is concerned. The senior people in qualitative would be prohibitively expensive to hire on a full-time basis, with the benefits and associated overhead.   Just think of the lost productivity in meetings alone!  

It is great to see the tools getting better to do technology-supported qualitative. I love it. I hope that the price drops, as surely it must if DIY is going to take off. Because feature-creep in online research has led to price creep, when it should be going the other way.

Some of the things raised in this article and others are relevant to the qualitative world as well.
* I wish that I could have more access to the learning from the CRM system when i am in the planning or reporting stages. I think clients themselves are struggling to integrate and effectively use all their many sources of data.
* I wish for better long-term collaboration. The best do it, the rest should.
* I wish for better semantic analysis tools -- not to replace me, but to make the work faster and cheaper.

But the good news in all of this is that clients want and need insights. And they don't want to pay 24 carat prices for fool's gold. And that's a good thing for those who can embrace the new reality.

What do you think about all this? I would love to hear your comments, and I know others would too.

[Hey, if nothing else, you can avoid writing that report. And that's gotta be good, right?]

mobile qualitative - ready for your phone today

This week I am participating in a demonstration of mobile phone texting as a method of gathering insights.

Not a new methodology, incidentally -- people have been using mobile methods for a few years now, but it's really starting to come into its own with the proliferation of smart phones.

In fact, I recall a large-ish student project of a few years ago, (bulletin board methodology), where i gave up using e-mail to communicate with students because they responded so much better to texts!

Back to the demo. Mobile methods for qualitative now come in many flavors, from simple Q+A gathered up into a spreadsheet at the end, to fully integrated texting in a discussion forum platform. 

Here's what I'm noticing, two days into the study:

- Far from being an irritant, I am finding the quick text messages a welcome interruption to my work a couple of times a day.

- I'm curious to see what the researcher is asking next. (Yes, it seems that a good and engaging question is as crucial here as it is anywhere, perhaps even more so!)

- I got a "great idea" response from the researcher that made me smile (the usual pats on the back we give to reinforce participation worked just fine on me too!)

I'll be interested to see the aggregated results of this demo study in a webinar later this week (put on by the Online SIG of QRCA.)

I love the idea of using this approach for a warm-up or follow-up to a more in-depth method. It's in the moment, very unscripted from the standpoint of the respondent, and reaches me where i am right now. The possibilities suggested are really wonderful!

If you have experience with this method -- either as a participant, or as a researcher / client, I'd love to hear your thoughts here in the comments.

Focus groups are a waste of time: or are they?

Researcher Chris Kann, (pictured left) and I spoke a while back about the reasons that people bash focus groups. The result of that discussion was an article that was published in Greenbook, called Don't Blame the Focus Group.

One of the most frequent negative examples trotted out about focus groups is the Herman Miller Aeron Chair story, a story of a very innovative product that was so foreign to consumers when introduced, that initial reactions were not all that positive. This story is often told as a cautionary tale about research -- it shouldn't be.

"That Herman Miller management chose to go ahead with a launch of the chair despite these findings and generated a massively successful product, is taken to mean the focus group did not do its job.

But perhaps this is a story of a focus group doing its job well, and management doing its job well too. Management knew what initial reactions to the new product would be and could plan accordingly. The history of the Aeron chair on the Herman Miller website in fact makes it clear that extensive consumer research, including focus groups, informed the development of the product."

We think it is pretty ironic that companies are invited to listen to consumers more, but only in certain ways.

I'm pretty sympathetic to the challenges of management in client organizations. So is Chris. The truth is, we've both been on the other side. Managers have to make a lot of tough calls, and they hope the research will help. And it often does help. But no research, regardless of method, can save leaders from the pain of making actual decisions, and some decisions will inevitably be wrong.

Bad research won't help your decisions much. But good research will save your hide over and over -- save you from making expensive mistakes, help you see the potholes in the road to success, and point to new opportunities.

Focus groups are not the be-all and end-all. But there's a reason they have stayed around, and it's because they are a good tool in the hands of a skilled researcher.

I'd love to hear your thoughts and experiences.

 

Resources:

Don't blame the focus group for leaders' shortcomings, by Susan Abbott and Chris Shields Kann, Greenbook, 2011.

About Bill Stumpf and Don Chadwick, designers of the Aeron Chair

Company history of the Aeron chair, including video with the designers.

Focus Groups: how can a table and some chairs cause such controversy?

Dan Pallotta created an interesting range of comments in response to his trashing of focus groups on his HBR blog post, Real Leaders Don't Do Focus Groups.

The comments are a much better read than the actual post, which coughs up the same tired mythologies you've already heard a hundred, if not a thousand, times. [e.g. Henry Ford and the "faster horse" soundbite, the "Apple doesn't do research" myth, and more of the same.]

Mr. Pallotta talks about his experience creating a successful charity fundraiser ride as if he created it in a complete vacuum, but it was successful because it was bold.

There are truths to be made on both sides of any controversy.

Truth #1:  Good ideas are sometimes killed with bad research. I've experienced this personally, although I'm bold enough to claim that I haven't contributed the bad research, just some good ideas that died at the hands of others.

Truth #2: Leaders who claim to not need insight research are either idiots, or they have a strong pipeline of connection to customers that they are calling "not research." Sometimes the connection is a simple one -- they are part of the target group personally. More often, in large organizations, a lot of people are actually gathering insights all the time. But they might not call it that.

Truth #3:  The focus group is only as good as the people using it. [aka: It's a poor carpenter who blames his tools.]

A few of the more insightful comments:

From Tony Flanders:

"The new model is --- engage, listen, dialogue, synthesize -- oops; that's actually the old focus group model." 

[Tony has posted an interesting article on focus groups on his blog, here]

From Nick Coster:

"The opportunities for innovation exist when we are able to strip what the market says it wants back to it’s underlying needs.This removes any given solution from the discussion (ie horse or internal combustion engine car) and abstracts it to I need to get from A to B in less that 1 hour."

From Richard Harris: 

"First, congratulations. I think you're the first person to post something on the HBR blog that pissed people off.
Second, I would have been curious to know what market-sensing mechanisms you do use to decide things like how bold to be. Something made you choose 7 days/600 miles/$2000 as your standard. It sounds bold, but you weren't so bold as to go for the round trip. How did you know how far to push the envelope?"

From Matt Towers:

"On the surface, it is sort of a flat Earth mentality that advocates that less information is better than more. It bespeaks a lack of confidence in understanding how to use information to make decisions."

And this gem: "Apple does do research, they just don't talk about it."

From Steve Willson:

"The key point that runs through the discussion is that to be successful an organization needs to listen to it's customers. They are talking, so you need to pay attention.

Whatever methods you use to engage them in a constructive dialog are better than shutting yourself off from the world and emerging with a set of features that are based on what you can build, not on what people want."

Care to add your own experiences? Please do.

Truth, lies and feedback (Part 4 final amazing installment)

I've been posting for a few days on this topic of how to get truthful feedback. Here are the links to Part 1 which poses a question, Part 2 about how to seek face-to-face feedback, and Part 3 which is about a questionnaire virtually designed to elicit falsehood.

Today, I want to look at the most challenging of all truth-finding situations, the police investigator.

This is a situation that makes the work of commercial interviewers of all sorts look like absolute child's play. People generally volunteer to talk to us, most times we actually pay them. The worst downside they are likely to experience is cognitive dissonance or perhaps embarrassment, possibly boredom.

The legal investigator, of course, is faced with someone highly motivated to lie for a number of reasons, social, cultural, psychological and rational. Everything is working against them.

This is why I found a recent presentation I attended by Donna M. Pence, a retired Special Agent who is now an educator, so totally absorbing.

If I was marketing her course, I'd probably call it something like impossible interviews, but she called her topic "When getting the truth is critical: Interviewing techniques used by law enforcement professionals".

It's not like on TV

One of the most interesting things Ms. Pence mentioned was the length of time spent preparing for, setting up, and actually interviewing.

There's the interview-before-the-interview, which is designed to develop a sense of who the subject is, how they respond to non-threatening questions, and even what their general comprehension level is. Then they seek to understand the answer style when under pressure, in a state of fear or anxiety. The protocol she teaches is called PEACE: Preparation, Engagement, Account, Closure, Evaluation.

She was very clear that every question asked has to have a defensible purpose that will withstand questioning in court. No "fishing expeditions" allowed.  This is a key difference from insight  research work -- our questions need to have a purpose, and do, but we are almost always going on a fishing expedition.

When you expect everyone to lie, you plan differently

Like a good insight seeker or facilitator, the good law enforcement interviewer sets the stage in terms of the environment, privacy, clothing, tone and manner, the actual interview space. They consider gender, ethnic and racial matching (matching or deliberately unmatching).

Unlike insight researchers, they are often seeking to disarm or destabilize. We saw her shift roles and accents like a skilled actor, and honestly, I was really glad I wasn't interacting with her professionally, so to speak!

The level of detail required to support criminal charges is, in her words, "excruciating". So one thing that is probably true from TV is the phrase "let's go over this again".

I truly loved some of her phrasing. Let me give you an example:

"I've never called anybody a liar. I say, 'I think you're not being as truthful as you could be.'  Or for a good ole boy, 'That dawg won't hunt'"

My top takeaways

These are approaches that we can all learn from, whether you are interviewing, or just trying to be a better communicator.

[1] Cognitive interviewing -- show the subject that you see them as an individual with a unique set of needs. In turn, present yourself as an identifiable person. Depersonalizing reduces the amount of information you will obtain/receive.

[2] Interact meaningfully -- don't just ask a list of predetermined questions.

[3] Communicate empathy -- see the situation from their point of view.

[4] Give your undivided attention -- convey that you want to listen and can be trusted. Give reassurance and support.

[5] Let them talk until they run out of steam.

Why does anyone talk to them?

The most amazing part, for me, was Ms. Pence's answer to my question of why anyone really talks to them, despite her persuasive methods. Her answer provided telling insight into the human condition.

For the most part, she says her tone is clear, calm, slow and matter-of-fact. She has often interviewed people who have violated foundational cultural taboos, such as child molesters, rapists, murderers.  She says things like, "You can't offend, embarass or surprise me", "I'm not going to be upset at anything you tell me. I've talked to many people in your situation".

While people are initially motivated to talk to get themselves out of trouble, if possible, they also want to give their side of the story, to provide an explanation for what happened. 

My take

We all live the existential problem -- we are alone in our own experience. The ability to share that inner world -- even a little bit -- is a powerful drug. I often wondered how law enforcement officers get people to tell them the truth, and I think this is the essence of it: empathy.

And on a less sobering note, if you meet someone like Ms. Pence in a police station, you are in serious big trouble, because she is one amazing interviewer.

Resources

Donna Pence recommends the book, Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage, by Dr. Paul Eckman

All of the graphics in this post were created by me, based on diagrams presented by Ms. Pence.

I heard Ms. Pence speak at the annual QRCA conference. If you love qualitative methods, you should join us sometime.

Common research mistakes

I logged in to Constant Contact a few minutes ago to start writing a newsletter. There was a button for completing a survey, so being a naturally curious [some might say snoopy!] person, I clicked.
Constant Contact now provides a survey platform. Sadly, they made several common mistakes. So egregious and common are these errors, it seemed essential to post about it and save others from the same mistakes.

This is a screen shot of the first page of the survey. I see three mistakes here:

First mistake -- ask for my name right up front. If you are going to make this non-confidential, you might want to ask for this information in the last question. Because then if I don't answer, you still get the rest of my data.

Second mistake -- non-exclusive categories. If I am in business 6 years, I have two choices. Same problem with 8 of the response categories. Oddly, the sequence changes from 8-9, and 10+.

Third mistake -- the person who wrote this survey has insufficient knowledge for the task. Two easy solutions. Either take a survey writing course -- there are lots of them around, and some are available online. Or, spend a small amount of money and hire a researcher to write the questionnaire for you. Seriously, this is not a complex questionnaire. It would be likely a half-day of work for an experienced person.

Or, you can just get poor data and look like a goof to the world.

You know, the thing is, the fact that I can use a power saw does not make me a carpenter.

Free e-book: Painless Insight Planning helps you set objectives, avoid wasting money on qualitative research

 

I've refined a number of tools to use in my work over the years. A few of them are tools designed to help clients in one way or another. This free e-book, Painless Insight Planning, is one of these tools with some narrative wrapped around it.

[Download Painless Insight Planning now]

One thing I've found that is often a challenge at the start of a project is getting clear objectives. People may have a problem or concern, but it is often not clearly articulated.

And I can tell you one thing for certain: a good way to waste money on research is not having clear objectives at the beginning.  I know this sounds perfectly obvious, but it is an issue more often than you would suspect.

Sometimes the problem is clear, but other elements are not. Such as -- who could we get insight from, otherwise known as "the research target."

I can't sit down with everyone individually and work through this, as I do with clients. So I thought I would put the whole thing into an e-book that you can download and use as much as you want. And then you'll have clear objectives for your research projects. And the world will be in a better place.

Because we really need insights these days, more than ever. And people inside companies can't afford to waste money or appear to waste money, get fired and be unemployed. Suppliers like me also can't afford to waste client's money. We all need to ramp UP on the insight quality and the innovation / ideation, but keep a lid on costs. 

What's it worth to get this answer?

An even harder thing for many clients is figuring out how much to spend. They ask me, what will it cost? 

WRONG QUESTION!  WRONG QUESTION!  WRONG QUESTION! 

The right question is "what is it worth to get good information on this situation?"

I have created a nice little rule to help figure this out. Not a complicated ROI on research thing. Just a back-of-the-envelope, cocktail-napkin kind of formula that I call the 10 Times Value Rule. It's a great place to start figuring out what you should consider spending to get some insight into your problem / challenge / situation.

[Download Painless Insight Planning now]

Easy worksheets

I hate filling in forms. Most people do. So there aren't forms in this book. There are easy worksheets designed to spur creative thinking to get you started. Here's an example.

Bonus content

If you download the book and actually read it, you'll find a link to some bonus content that will save you even more time. 

A small request - no strings attached

I'm simply putting this handy e-book out there for free with no strings attached. You can even add this e-book (with no changes) to your own site or blog for your own customers and prospects to use.  But I do have a small request.

If you think this e-book is of any value, please add this blog to your RSS feed reader. I also send out a newsletter about six times a year. You can add yourself to that list if you want to hear about other e-books I plan to release, or just get the newsletter. You can check out the previous issues here.

Thanks for reading. And good luck with your insights research projects!

[Download Painless Insight Planning now]

Listening right vs. listening wrong: Google Latitude and sampling

You've likely heard about a new service Google has launched called Latitude, that can enable you and your pals to track each other geographically via your mobile phones. There's a bit of controversy about the potential abuse of the service for stalking, but that's not what I want to talk about. At least not directly. I tested this concept a couple of years ago in research. Let me tell you about it.

About two years ago, I was planning a session with some grad students in a research program. You know the drill -- guest instructors who are practitioners come in and show their stuff. So I did an advance study with the group using an online bulletin board method. Then we talked about it in the class.

To try to find a topic of interest to all the students of various ages, I picked cell phones. Among other things, I cooked up a number of possible features and enhanced services that we concept tested. One of them I called "Friend Finder", which was essentially a product like Latitude.

I thought this Friend Finder concept was pretty clever, actually, and that if I tested such a product concept with a broad base of young adults, that it would likely be well received. I was really surprised when this group of research students reacted pretty badly to the idea, and were REALLY worried about privacy. Freaked-out and creeped-out about privacy, actually.

They also didn't like "Taxi Finder", something I personally would love. And they didn't like "Bus Finder". I don't think there was one concept that tested well.

So I pondered this finding ... and here's what I think.

This was not an average slice of humanity in any respect. Most of these people were into quantitative i.e. they enjoy statistics. A number of them had just finished a four-year degree and were running out of money. Several were mid-life career changers who were also pretty cautious about their spending. And they were in a small city on the edge of the Canadian Shield (lots of big rocks) and they told me cell coverage was often spotty. [Who knew? When I go up there to canoe, I'm pretty much ignoring my cell phone.]

And it was not a well targeted slice either.  A number of the students turned out not to have a cell phone at all, or used it very sparingly. In a real study, the client most likely would have been looking to gain insights into only those people who were already pretty keen on their cell phones and were already using some advanced features.

My colleague that day, Tania Heintzman, was personally VERY keen on Taxi Finder. But she uses taxis a lot and students don't. Off target.

I personally was very keen on Bus Finder, but the bus service in small cities is often terrible and so irrelevant to this group who all drove to campus. Off target again.

Basically, researching any number of the wrong people does not help. A bigger sample of the wrong people does not help. Statistical significance would not get you out of this issue either. The wrong sample is the wrong sample.

Sometimes only a few of the exactly right people can help you dive very very deep into powerful insight. So you really need to figure out who those right people are. Especially when you are trying to see into the future with new concepts for products.

If something like this happened in a real project, it would be a mistake to conclude that the concepts had no potential market. Because clearly, Latitude will find users. And all sorts of people are developing open-source products like my "Bus Finder" idea, even here in the frozen wastelands of Toronto. Soon it will be available via smart phone apps. 

On the other hand, some of the learning would have been valid. For example, the concern about privacy. That's an attitudinal and behavioral segment that you would want to understand. Even knowing about this issue would help you design a better service, and avoid the criticisms Google is getting now.

Innovation breakthroughs: now really IS the time or why recycling does not work for insights.

 

People are cutting back. [Oh, you heard that, too?]

I've had a couple of projects cancelled or postponed indefinitely in the last short while. One of them was to help an organization identify new markets so they could escape the commodity price-cycle trap of their current markets.

This is exactly the kind of project you definitely should not cancel -- projects that will help you crack new revenue sources. So what will this company do instead? I'm guessing, they will just push harder at existing markets, and find themselves in the same place a year or two down the road.

While having these thoughts, I bumped into a nice article in Marketing Profs. [Now that the economy is tanking, maybe we can catch up on our reading.] The quote from Jeffrey Immelt leapt off the page.

General Electric CEO Jeffrey R. Immelt has established a class of projects now well-known as "imagination breakthroughs," consisting of ideas that are "really hard or really important" and might generate significant revenues over a three-year period, the time that GE usually takes to implement a new idea. "Imagination breakthroughs are a protected class of ideas — safe from the budget slashers because I've blessed each one."

If it takes you a couple of years or more to really get an idea off the ground, then the fuzzy front end you cut now will leave you in even more serious glue in two to three years time. And if you kill the stuff already in the pipeline, you're not going to have anything new to launch in 2009 either.

I really wish clients would say instead: "Can you get me something substantive with less money?" That's a better challenge than dropping the idea creation process.

Caveat

The article this came from is filled with good ideas from the Zaltmans. But like many of their ilk, they firmly believe that no approach other than their technique (individual metaphorical interviews) can get to the good stuff. Not true, in my experience.

"These deeper ideas are not available from survey responses to statements such as "financial planning is difficult" or in focus groups where each participant has on average about 10 minutes of air time." 

OK, you are definitely not going to get these deep meanings from a survey. I agree. But individual interviews are not the gold standard either.

There may still be people running focus groups with 12 bodies in them. However, most of the kids I hang with have shifted to small groups when they are using a group discussion method. Small being five or six. And researchers frequently add advance activities such as blogging / electronic diaries / collage assignments / videos / you name it. And there's no rule about 90 minutes or two hours either -- you can keep people humming along for three hours quite easily. And yes, you can uncover metaphorical meanings in focus groups, in online research methods, and in many other ways.

In truth, the generation of insight has a lot more to do with the skill, knowledge and art inside the researcher's head than the method employed. There's no method going that guarantees insight.

As anthropologist Grant McCracken has said, the proof is in the pudding. If you got insight, you got insight. If you didn't, you didn't.

I have friends in the marketing world who have used Gerry's Zmet technique to great effect. And I believe all of methodological infighting and territory claiming must defer to this. If the method works, the method is good. The proof of the pudding has nothing to do with the theory of the pudding or the method of the pudding. The proof of the pudding is a client who says: "This was illuminating. I understand my consumer and my market in ways that I did not, could not before."

Back to the main point

This is no time to be cutting your future revenue streams by shutting down the insight machine. In fact, if you plan to be standing when things turn around, you should be cranking up the insight machine. Certainly, your customers and consumers brains are grinding away on many things right now. Not a good time to recycle some old insights.

Resources:

How to Foster "Workable Wondering", by Gerald Zaltman and Linday Zaltman. In marketing profs, reprinted from HBS.

The Zaltman Method, by Grant McCracken in his blog, 2006.

Cool new tools that enable feedback

I went looking for one thing online, and found a bunch of other stuff. Don't you love doing that on a Friday morning? So of course I wanted to share...

There are just such a variety of interesting online applications for assisting collaboration now. If you can add some others to the list, please do. If you have used any of these tools and have comments about them, please add your thoughts as well.

Mapanno Markup

This markup tool lets you invite comments from others on an image or a project. It feels a lot like Google Maps, where you can add an icon and comments, and stick it to a specific place on the document. Here's a screen shot that gives an idea...

Review Basics

I have actually tried out this tool on a demo basis at least a year ago, and it looks even better now. like Mapanno, it is currently free. But it has a broader range of functionality, and was designed more specifically with marketing folk in mind.

Here's a screen shot of what the markup environment looks like. There are bubbles that you can use to enter text and a variety of marking tools as well.

 

Review Basics also supports common video formats, as well as Word, PowerPoint, and PDF files.
I'm itching to try this out with a research project, but I think it could be really useful on a team internal basis as well. Right now, it's free, just like Mapanno.

Vyew
Vyew is one I just learned about, thanks to GeekBlog. I haven't had time to demo it at all, but it positions itself as a competitor to WebEx and GoToMeeting. In addition to those services, however, there is the potential for asynchronous involvement by multiple reviewers who can mark up documents, and view each other's comments.

Here's a comparison chart that may help.

There are some interesting case studies on the site as well, that make it clearer how one might use such a multi-featured application for things other than web conferencing. Like the other tools mentioned, it is currently FREE, and does not require software installation. There's a review by Guy Kawasaki

Concept Share

This service is specifically targeted to graphic designers. I haven't tried it out, but it looks pretty appealing. There is a free-premium model in place which affects the number of projects you can use, and the amount of online storage you get. Huge amounts of storage are available, which is needed for graphic files.

 

Basecamp

I've been using Basecamp for a few months now. Although it's not perfect, it's a very useful and low cost collaboration tool. I haven't figured out a way to use it for research projects yet. But you can store files, there are whiteboards, a really good messaging function that beats running things on e-mail, and some project management tools.

This stuff all just continues the flattening process where little tiny organizations have access to tools and technology as good as the stuff the big organization's have. Sometimes it's even better.

If you want to read about even more of these tools, here's a reference article from GeekBlog. And their handy comparison table.

Global Village Study on Teens and Technology

About this time last year I was helping to plan a global study on teens and technology use, called Global Village. This idea was cooked up by my colleagues Ilka Kuhagen (Germany) and Corette Haf (South Africa) as a way to demonstrate the power of using bulletin board research to gather global insights.

We sponsored this research as a group, and conducted the research with 12 groups. This was an impressive team of researchers, it was my honor to work with them. 

During the planning process, we had many logistical issues to work through. For example, our platform sponsor, Focus Forums, had to add Mandarin character capabilities to the bulletin board.

A decision was reached to focus our efforts on teens aged 14 - 16 years old. A challenging target in some ways, as ethical guidelines require parental permission to conduct research with this age group, which added to the logistics of the project. But we saw that this group is entering young adulthood with access to a level of technology that they have enjoyed since childhood. We wanted to see where things were heading with this group. So we found teens that had a cell-phone and used the internet relatively heavily.

The bulletin board was a fantastic tool for the project, and the teens loved the format. Age and gender barriers that would have been a major burden in a face-to-face setting evaporated, and we had high engagement and considerable openness and honesty.

In addition to this, the research team was also easily able to watch the research unfold, and share our own discussion about questioning strategies over the project schedule. This meant we could change and shift the later stages of the project based on earlier stages.

It was fascinating to see the style differences among the research team. As findings started to emerge, we were all impressed by how similar, in many ways, the attitudes of the teens were in each country. In picture sort exercises, for example, the same concepts -- and even the same images -- came up over and over again. We weren't really expecting this, it was a fascinating finding.

One thing is for certain -- these teens are joined at the hip to their cell-phones, and this is their life-line. If their house was burning down, it's one of the things they'd grab. The research team found text messaging a much more effective way to send out reminders to participate. (With an adult group, we would use e-mail). Almost none of the teens in the whole study had heard of Second Life. A few had been on Habbo Hotel, but these sites no longer held  -- they are way too busy forming real-world relationships to care too much about the virtual world. Yet.

Overall, a fascinating group of teens. The Canadian group were very much "the New Phoenicians" -- ethnically diverse, open to new ideas, enterprising and optimistic. I wanted to give them all a virtual hug, they made me feel proud to be Canadian. I have no doubt the other researchers felt the same about their own youth.

I'll be presenting on this project with my colleague Betsy Leichliter from New York at the ARF conference next week, and also in Winnipeg at the MRIA conference in May. Other team members are presenting at various conferences around the world, including the upcoming AQR/QRCA conference in Barcelona. The slide at the top came from a presentation done by Diva Oliveira in Brazil.
 

You can also read the write-up that Ilka and Corette did for the QRCA publication VIEWS, which is available free online. How Global is Your Village?  (the link spawns a PDF. Article is on pages 18 - 27)

You know advertising is hot when it's displacing IT among IT research houses

I always thought of Forrester as an IT-oriented research house. And then they were on stuff like supply chains and outsourcing. However, they are now getting talked about for their thought pieces on where advertising agencies are going.

This suggests two things to me:

1. Forrester is good at figuring out what's hot
2. Marketing, advertising, and brand building are hotter than they have ever been, but also under more scrutiny than ever.

I have other evidence of this trend.

The Globe and Mail, my daily must-read newspaper, never deemed marketing topics worthy of business coverage. [Well, any kind of coverage, actually.]  Not long ago -- within the last 18 months for sure -- they have added regular material on this topic. It's thin, but it's there.

Friday, I heard Terry O'Reilly speak at a conference on qualitative research. Mr.O' Reilly is a regular feature on CBC radio, and known for his expertise in advertising, as both a creator and a commentator. Here's one of the notable things he said on the topic of insight and advertising:

"Creative ideas born of intuition are the last legal means of gaining a competitive advantage over your competition."

I thought this was a) brilliant, and b) true.   

Forrester seems to think that advertising agencies are not helping marketing people connect with consumers and customers in a meaningful way. This is the summary of their recent research report, The Connected Agency: Partner With An Agency That Listens Instead of Shouts

"Today's agencies fail to help marketers engage with consumers, who, as a result, are becoming less brand-loyal and more trusting of each other. To turn the tide, marketers will move to the Connected Agency — one that shifts: from making messages to nurturing consumer connections; from delivering push to creating pull interactions; and from orchestrating campaigns to facilitating conversations. Over the next five years, traditional agencies will make this shift; they will start by connecting with consumer communities and will eventually become an integral part of them."

I have not read the whole report, but I am having trouble seeing this as news. And it's far too easy to just blame the agencies, who are under tremendous pressure to deliver results these days. But they can only do what they get a mandate and a budget to do from the marketing people. Surely marketers themselves must take responsibility for connecting with consumers.

And so must executives.

Being in the insight and intuition business myself, I can only agree with Mr. O' Reilly: we need better strategies. We need insights. We need innovation in marketing and advertising.

At the same conference where Mr. O' Reilly was a headliner, I presented with a client a case study on some innovative research we did together. When we got to the Q&A, the biggest question that came up -- which we brainstormed as a group -- was HOW to convince executives to do something less conventional than a focus group or a survey.

These researchers and marketers -- they want to deliver innovation, they want to deliver category-changing insight, but feel challenged to sell it into the organization.

We came up with a flip-chart full of ideas of how to do this. Things like: start small, start with intractable problems where there is motivation to try something different, and so on.

But as one of the research buyers said, "We want innovation, we want to do something new, and we want amazing insights, but we only want it once it's been done before. How do you sell this into the organization?"

Of course, we on the practitioner side have another challenge. Having promised these category-changing insights, we need to stand and deliver.

I only wish these insights that can change the landscape could be produced on demand. [I'll be hunting for a good one this week, wish me luck.]

Facebook Polling: a quick tour and a surprising finding

After holding forth about the future business applications of Facebook, I decided I should put my money where my mouth is, and do a quick little research study.
Facebook makes that easy for you with their polling feature.

Here's what I wanted to know -- what kinds of applications for business would interest people who are already on Facebook? So here's the question I asked:

What application for business would you be most likely to use on Facebook?
[a] A way to post RFPs / find suppliers?
[b] A way to bid on RFPs / find projects?
[c] A project team collaboration function / group?
[d] A way to sell content like e-books / e-learning?
[e] I would not use any of these

Facebook polling is so inexpensive, I thought I could afford to get 100 responses, which would cost about $25.00. [I do want to be clear that this is not a scientific poll, it's just a way to take a quick read on something.]

I couldn't select exactly the group I wanted -- freelancers and SMB people. So I went for age as a proxy, and asked for age 35 - 49. I was surprised that the oldest selection was age 49, actually, since surely there are many people using Facebook over the age of 49. I know several well, in fact.

I sat excitedly at my computer waiting for the colorful dashboard to show results. Which it did, looking like the chart above, with several other colorful charts breaking down my poll by age and gender.

Surprising Finding

When I had 49 responses, Facebook shut down the poll.

Before they did that, here's what I learned:

1. About half would be interested in one of the proposed functions
2. About half indicated no interest in any of them
3. The most popular of the proposed functions was [c], the project team collaboration function. This is the one I want too.

I received an e-mail from Facebook explaining that they had shut down my poll.

Hi, Susan,

Facebook Polls are intended for market research purposes. Facebook Polls created to promote products or services or that are otherwise deemed not appropriate are removed accordingly. Along these lines, Facebook has automatically removed your Facebook Polls (order #Facebook Poll).

To see the final results, go to: link 

Responses Desired: 100

Actual Responses Received: 40

Price per Response: $0.25

Your credit card has been charged an additional $10.00 This completes your order.

The Facebook team

I'm surprised that they thought this was promotion. So maybe it was deemed otherwise inappropriate. Perhaps the only people allowed to do research on Facebook itself are officially authorized. Perhaps using the word Facebook in the poll flagged my little project. As far as I know, they can't stop me from blogging about it.

Anyone have some insight into this?

I was thinking it would be cool to hire some savvy Elance developer to just make me the application I want, but now I'm reassessing.

Holiday notice

I'll be skiing in Banff. If you're there, say hi. I'll be the person doing the face-plant right in front of the lift line, wearing a fuzzy hat with ear flaps [that looks really cute, actually]. So I likely won't have the strength to post after a hard day of skiing. See you in the new year! And thanks for all the reading and the comments, which really make it fun to keep at it.

Jason Oke talks about what's wrong with qualitative research

Jason Oke is a planner for Leo Burnett -- a guy who spends a lot of time thinking about finding leverage points for the creation of great advertising. And is, incidentally, an excellent speaker. He thinks there's a lot of bad research being done, which has created a credibility problem.

Chief among his complaints is that there is far too much direct, literal Q&A going on, which we know -- I mean we really know this -- is not very useful. This kind of technique lacks so much subtlety, it is akin to "strip mining their heads," according to Mr.Oke.

By literal Q&A I mean approaches like asking people to tell us "what's important to you in a vodka", which will generate nothing that can help a vodka marketer. If the target group even knows, which is unlikely, they are hardly going to say things like, "I want to feel like part of an exclusive club" or "I want a wee boost to my social status."

This is not just true for products like vodka, by the way. I just completed a business study in which one group of people were clearly motivated by reducing their fear level. Now just how likely do you think they would be to [a] know that and [b] say it out loud? In fact at least one of them did know that, but she didn't say it, she wrote it down as part of an exercise. [Of course this creates an equally large challenge for the creative team, but that's another article.]

Jason Oke also believes that people are bored by the idea of sitting in a dimly lit room for two hours being poked at with inane questions and limp exercises. He thinks this is a major reason why it is getting difficult, and expensive, to recruit participants for focus groups. This was a new idea for me, and I think he's on to something.

Mr. Oke's recipe to turn the situation around:

[1] Take brain science into account.

People can tell us many things, but they don't have direct access to unconscious motivators. When we ask them questions depending on this access, they will tell us something, but it's unlikely to relate to the problem at hand. It may not even be true. But most people will generate an answer to a question like "what's important to you in a vodka" just because people are very good at generating reasons.

[2] Plan research activities that meet the needs of #1, and are also fun and engaging.

Mr. Oke's solution to the Vodka problem, for example, is to ask small groups of people to design a new vodka and how it will be marketed to people. Exactly. Generates great data, and is a lot of fun for the participants. He showed us clips from the tape, it was brilliant.

[3] Stand up to clients when they ask for bad research. Evangelize for good research.

The thing is, once clients have experienced really good research, they are left unsatisfied by the other kind. At least I think that's the case. I hope and pray that is the case. But it feels risky the first time you get on.

[4] Think of research as a brand touch-point.

Absolutely correct, and great advice.

Resources:

Jason Okes' blog is here

A related Malcolm Gladwell blog post is here

Clips from the Prelinger Archive film about marketing research, here

Book he recommends [I have not read, but it sounds great]: Herd by Mark Earls.

Mr. Oke presented to the qualitative division of MRIA at "Qual in the Fall" on Nov. 9, 2007 in Toronto.