Post 19: Giving up on surveys

I’m very glad that Media Watch had a go at coverage driven by vendor-created research this week, because I have reached the decision not to use them any more.

For ages I have been uncomfortable with the way surveys are used to generate coverage, largely because the methodologies used are far from transparent and the intent – generating media coverage – is blatantly obvious.

Recently, however, I encountered the worst, most dishonest piece of research I have ever seen. I label it as such because it simply asked the wrong questions and the researcher was ignorant of the right questions. For example, the survey contained assertions that business data stored on computers is not well protected. But it had not asked those surveyed about the use of the most common data storage and protection technologies, so was simply not reporting on the real world. The research also asserted that businesses cannot recover lost or damaged data at acceptable speed, citing 40 hours as disastrous. Yet when I pressed the researcher on the fact that many organisations deliberately set recovery time objectives of more than 40 hours, the researcher was ignorant of the term “recovery time objectives” and admitted it was something it should look at in the next version of the research.

That’s not good enough.

Yet the research concerned has been cobbled together into a glossy brochure and will be pressed into the hands of prospects and suspects for a year, until the next edition of the research is produced.

I sincerely hope that no-one falls for this flawed study which has (to mix cliches) pissed in the well and broken the camel’s back, because it is such an obvious example of research being abused to prove a point that I simply cannot contemplate trying to find out what is wrong with other research I encounter.

Independent research created by dedicated researchers without commission remains something I will consider. But I simply feel I can no longer trust anything initiated by a vendor. And if I cannot trust it, why would I present it to my readers?


9 thoughts on “Post 19: Giving up on surveys

  1. Good post. I used to find those ‘state-the-bleeding-obvious’ and ‘super-dodgy-methodology’ studies really amusing. But now I just feel shocked and disappointed that they get published. Same goes for most ‘white papers’ (i.e. product brochures) – should go in the same bin.

  2. “But now I just feel shocked and disappointed that they get published.”

    That’s an awesome sentiment Susan, because it shows affection for journalism that makes more effort than recycling suspect information. That’s the kind of journalism I want and want to produce.

  3. Its nice to hear of a journalist making a stand against these phoney ‘research’ results. As a reader it always annoys me when surveys get coverage because they are so transparently promotional. As a PR I am sick to death of them being suggested as a “creative” campaign idea.

  4. Would have been nice to name the company involved so we can ignore all mention of them in every other media outlet that has been dumb enough to publish the “research”.

  5. Responded quite aggressively? To what? You telling the truth? I don’t get it. Did they threaten you? Have you followed? Bashed? All this secrecy only protects the guilty. Thought you had more balls than that.

  6. Kathy,
    blogging is a sideline to my other activities. When, as has happened in the past, I find myself in a situation where I have to defend my writing against a pissed-off adversary, it drains away my time. So I anonymise.

  7. OK, I’ll bite, Simon.

    All surveys are intended to generate a response: coverage, debate, introspection, insight, additional research or marketing funds, etc.

    Also note that there is no absolute truth in any research. Hypothesis and ideas are tested, but the results are either valid or not valid based on methodology and the results.

    I’m glad you were able to poke holes in the study that was presented to you. In my mind, it helps weed out the shonky marketers and researchers.

    When we (Howorth) propose any research to our clients we try and follow accepted statistical practice with the absolute proviso that the outcomes deliver some new insight into the state of whatever the client is trying to elucidate.

    No all clients accept that advice, however, and they usually find themselves working with a more compliant PR company or individual.

    That all said, let’s test the concept of validity and take it out of the realms of IT. Particle physics should do it.

    If you have two sets of physicists, with different testing regimes and stats methodologies, one group will demonstrate that light behaves as a particle and yet others show that light is a wave.

    Which is right?

    What if light is a particle sometimes and a wave at other times — in other words under some circumstances it could behave either way? Would not one scientist’s bias then colour how they describe light’s behaviour? Is that more or less valid than another scientist’s opposing view?

    OK, translate that to vendor research: some have more rigourous testing regimes tha others and some are more biased than others. That said, not all research is trash and some is actually useful because it can be construed as ‘more valid’.

    And since it is valid, it must have some usefulness. By discounting it totally, that utility is ignored. Not a good outcome if it maybe useful to an IT manager or CIO that is looking to justify their own bias or hunch.

    Many so-called bogus research reports were initally dimissed because they didn’t fit acccepted wisdom but eventually panned out. Ken Olsen, the CEO of the thrice-removed DEC (now H-P, via Compaq) derided the idea that PC’s would supercede mini-computers and that UNIX (a.k.a. Linux) was ‘snake-oil’.

    What seemed like bogus (invalid) reseach by Apple and IBM at the time actually turned out to be true. So discounting this apparently invalid research cost Olsen his job and ultimately his company.

    So as I said, it’s worthwhile canning patently crap research, but even research that today looks unlikely, but valid, could in the end make us all fools even if there is a bias (like Apple’s and IBM’s research supporting the rise of PCs and IBM’s push around Linux).

    BTW: Research fuels the media’s need for ‘the new’, as in ‘news’. Those trashing Karl Stefanovic and the butchers’ research should think about the ever more blurring line between journalism and entertainment — a sad by-product of broadcast journalism.

  8. Emilio – your points about epistemology are well made.
    But the problem I encounter as a working journalist is that, to use your physics analogy, a vendor of “light-as-wave” products would simply never release research that in any way supports or validates “light-as-particle” theory.
    Instead, a vendor of “light-as-wave” products would only ever release research that demonstrates the need for their products and denigrates their rivals.
    I’ve seen this time and again over the years and the piece of research I mention in this blog is the final straw.
    As it happens, after I wrote this post I spent some time with folks from a research firm (One a PhD, the other an analyst) that I have some dealings with. Without having read this post, they volunteered their professional dismay at the kind of research they constantly see reported in the media. As it happens, I also once failed to deliver satisfactory work for this group when asked to produce a synopsis of research they had conducted for a client. My journalistic eye and style resulted in an interpretation of their work they, as professional researchers, found unacceptable as a representation of their work.
    This is not to say that I dismiss all research. But until a vendor can satisfy me that they are willing to sacrifice rigour for results (and the outcomes you mention) I’ll be giving vendor-initiated research a miss.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s