Knowledge@Wharton/George Group Survey
The impact of product and service complexity on business
George Group Consulting and Knowledge@Wharton have teamed up to explore the impact of product and service complexity on corporations: As companies add new products and services to their portfolios, what is the impact on processes, profitability, and growth? Please take a moment to participate in our brief survey via the link below; all information will be kept confidential and will be used only for this study. Each individual who takes part in this study by filling out a complete survey will receive a free copy of the results document. Many thanks for your participation.
Even if you don't subscribe, have a look at the survey questions (registration is free, and K@W can be a great source of background material).
Looks pretty good, doesn't it? Erudite. Professional. Nice clean layout. Easy drop-down boxes. There's just a few problems...
- Respondents aren't qualified, other than being people who read the publication
I'd like a copy of this report. I bet you would too. Unfortunately, guilt overtook me half-way into it, and I just couldn't keep manufacturing some kind of answer. Not only that, I realized the report wouldn't be that useful if it was answered mostly by people like me. Unfortunately, since I am quite interested in the topic of complexity and how it relates to customer experience.
- The attempts to qualify respondents by industry and position aren't all that helpful
Actually, the short list of position types makes a lot more sense than what you usually see in surveys of this nature. And indeed, I am a C level executive, albeit of a rather small enterprise. But what is the function of a C-level executive? Strategy and Planning? Marketing? Investor Relations? Surely it is all of these things and more.
More important, they never ask the size of the organization, or a proxy for that question. It appears that I am a legitimate participant, and so are you.
It's not that this isn't a problem for "real" scientific surveys. It is. A challenging one.
- Question meanings aren't necessarily clear. They could mean many different things to different people
Here's an example. "Our cost accounting systems fully capture the full costs of complexity. Strongly agree, Agree, Neither agree nor disagree, Somewhat disagree, Strongly disagree" . Wow. I bet you could write a PhD thesis on that topic. Maybe more than one.
- Questions don't have a "Don't Know" option.
This is very important, and a common problem with weak survey designs. Even if you are a really smart person, such as the CEO of Oracle, you might want to answer Don't Know. But there's not an option to do that. You are instead forced into the "Neither agree nor disagree" option. And many surveys are constructed in such a way that you must answer all questions.
Okay, let's stop here. You see the problem. This is actually not scientific research. I'm not sure it's even quantitative, even though lots of numbers will be generated from this. It is like asking questions of random people spotted on the floor of a large trade show. Or at a Christmas party.
In fairness, the researcher probably knows this. The purpose of this survey appears to be publicity, since the George Group tells us on their web site that they are "the recognized authority in the development and deployment of Strategies for Conquering Complexity". And perhaps they want to scan the horizon a bit, and aren't too worried about scientific accuracy.
Still, I expect we will still see a future report in K@W with statistics and tables that most people associate with a scientific survey. Except this isn't.
Starting with the K@W readership is not a bad idea. But after that, if you want real research results, you pretty much have to pay the price.
If surveying C-level executives about their opinions and attitudes were this easy, you could also buy brain surgery at Home Depot. So keep your BS detector running when you read reports like this.