Most of what I have seen over the many, many years that surveys have passed my desk tells me there is (almost) always an element of self-selection, espeically where it relates to matters of influence. It has to because it requires bias to support hypotheses.

Provided everyone’s clear on that then generally, I see no harm in using the data. Although one always has at least one eye on where the data might be skewed. This is especially important when using it for your own purposes.

For instance, we are considering a campaign and are testing the likelihood that people would wish to be part of the project. We’ve had many hits to the landing page in question. We’ve seen a lot of cross-reading between linked sites - presumably to view additional insights. A few comments have been posted on all pages we know have been read. Most of which are saying ‘yes’ to the idea. The comment rate is 0.75% of those that landed on pages in question after excluding any attempted spam hits.

What can we assume from that? Anything? We have now added in a few topic heavyweights who we know will add stories and comment. They are passionate about the topic, are at the coal face of execution and are known by both sides of our target audience. Does that mean the people we’d like to influence will read and act up on the message? Especially as we want them to interact on what is a tough area.

The honest answer is I don’t know. The one thing I can say is that having a large number of page reads tells me at least that number are interested in something that’s on the page. That’s more than could ever be calculated for print media. I can also say how the reading numbers for that page compare to other newly minted material on the landing site. So now I have some idea of relative significance. Do those numbers help us make a decision? In borad brush terms, yes they do and they continue to do so as the conversation expands.

What I do believe however, and here I’m sure we’re agreed, it is in the narrative the greatest learning occurs - even in self-selected cases. So if our project IS successful, how will that be judged? The learning from the conversation? The vast number of hits and comments? It’s Long-Tail appeal? No. It will be the actions taken by those we wish to influence.

In the end, we can say as much as we like about what people do or do not know. We can make as many assumptions as we like. We can kid ourselves, if that is indeed what we’re doing. But it is in the action and impact that all things are ultimately measured.

I’m not convinced at this point in time, blogging is anywhere near the maturity it needs to reach for it to have serious influence except in a few highly defined areas. I make no assumptions. It’s a gut feel. It’s certainly at a point where if I hear about one more Blogging-101 for business conference I’ll probably scream.

Survey or not, I believe the value of this medium will come from those that find smart ways to use it and that means taking risks that for many will appear to be flying in the face of conventional wisdom. That’s what we’re doing right now in relation to our project. We have an eye on stats but we know it will need the very best content for it to have impact.

That will come from experts at the coal face, industry people steeped in managing the issue and journalists who have built up expertise in understanding the industry concerned and who can cross the communications divide between one side and another. If it fails, we’ll have learned something. If it succeeds, we’ll have learned something different.
I doubt we’ll think twice about the stats.

I don’t see why my “dissection” is “a tad disingenious.” I’m not responsible for the flawed “research” produced by the PR industry, and I expect that journalists will refuse to “regurgitate” that crap. If all we can do is to say “well, that’s how the world works, the PR guys pretend to do research, the journalists pretend to do reporting”, then we can’t expect to get better research or better reports. Of course, this doesn’t affect only mainstream media; most weblogs are happy to report any “study” or “survey” without questioning their validity - that is, if the results are fitting what “we” think about bloggers.

You say that “it is technically possible to generalise from small samples according to Deloitte” — could you please point me to your source (book, study, article)?

Of course that establishing a sampling frame for bloggers is one of the big problems; then we have all the other problems related to online surveys. I’m not saying that there is an easy answer to these problems - but this doesn’t mean that all one can do is to put out a survey, collect the results, and -hey- that’s it, take it or leave it, ’cause nobody knows how to sample bloggers.

Of course Blog Relations - not BlogAlerts :) - are entitled to draw reasonable conclusions, if by reasonable we understand “limited to the PR pros who participated to the survey.”

Again, my problem was not with the way they reported the results on their blog, but with their lack of reaction to (what I thought to be) inaccurate reporting of their findings.