Archive for Surveys

Japanese bloggers survey published by Edelman/Technorati

Edelman and Technorati Japan have released the results of a blogger survey (PDF, 37 pages) (press release), based on the same questionnaire used for the survey published in January 2006.

It’s good to see that the limitations of the survey are clearly stated this time (the previous study had serious problems in this area).

PS: Memo to press releases fans: they’re big in Japan! :) (details in page 29)

Comments

The Dallas/IABC poll on how to handle angry bloggers — the complete answers and numbers

On July 7, the Bulldog Reporter’s Daily’ Dog has published a piece about a recent poll on how corporate PR should manage “angry bloggers“. The article stated that the online poll

asked more than 300 communicators to choose one response to the question, “Who should respond to angry external bloggers—media relations or customer service?”

The fact that “46 percent of respondents believed bloggers should be ignored, or managed by customer relations departments” caused a storm in the e-cup (thanks for this expression, Richard!).

I contacted Roy Miller, president of Dallas/IABC, and asked him if I can get access to the original questions posted on the chapter’s website. He responded promptly, and sent me not only the original questions, but also the number of people who responded to each question. Here they are:

Number of Voters: 50
Period: June 1st - July 2nd, 2006

Question: Who should respond to angry external bloggers – media relations or customer service?

1. Definitely media relations. They are the company spokespeople. - 21 votes (42%)
2. Customer service. Bloggers are not members of the press and should be treated like any other consumer. - 16 votes (32%)
3. No one. Bloggers should just be ignored. - 7 votes (14%)
4. I don’t know. - 3 votes (6%)
5. Neither. This responsibility doesn’t belong to either of these teams. - 3 votes (6%)

A couple of notes:

  • This was a “quick poll”, not a poll; not a survey, not a study. Dallas/IABC has such surveys posted on its website every single month.
  • Since it’ was on the web, anyone was able to vote, not only IABC members. (I voted two days ago on the keynote speaker for the chapter’s annual conference.) So the results are saying nothing about how IABC communicators think about the question asked in the poll.
  • The press release alluded to this limitation (the quick poll “is an interactive Web site feature that provides a timely snapshot of what chapter members and other site visitors think about current issues and trends.“) However, the acknowledgment didn’t made it to the Bulldog’s article.
  • In fact, Bulldog Reporter’s reporting was totally absent: they just (slightly) rephrased the press release. What about verification?
  • The press release itself is misleading, when it says that “the poll asked more than 300 communicators to choose one response.” False: the poll asked every single visitor -not only the 300 members of IABC/Dallas- to respond to the poll.
  • The release -and the article- added the number of votes for responses to the second and third question. Confusing.
  • Both the release and the article are missing the most important number: how many people responded to the poll, anyway? Only 50.
  • So, it’s not 46% of 300, it’s 46% of 50.
  • In fact, only 7 people (from 50) said that bloggers should be ignored.
  • (rant ahead) Last, but not least: let’s say that a large percent of respondents said that the “angry external bloggers” should be tackled by customer service, not PR. So what? How is that different than what Microsoft developers are doing when they are responding -on their blogs or on other bloggers’ comments- to the complaints about MS’s products? Sure, it’s not the classic understanding of “customer service” - but still. And how many bloggers will want a flack to respond to their complaints, anyway? What a PR person can do is to make the connection between the customer experiencing a problem with the employee that can solve that problem, ASAP. (rant over :)

And another thing: here’s the quote attributed to Roy Miller, president of Dallas/IABC, in the press release:

Corporate communicators must accept and acknowledge a blogger’s ability to spread information– the good, bad and the ugly. Ignoring them and not responding quickly just sets up an organization to be-best case-inaccurately represented; worst case, to be demonized. Treat bloggers with the same respect and responsiveness you’d have for any member of the media.

Prescient words. So, why didn’t Roy commented on the blogs writing about this topic, in order to prevent IABC’s inaccurate representation? I really don’t know.

To conclude: a misleading press release issued for an irrelevant “quick poll,” reproduced -without verification or correction- by an online publication and dissected by bloggers who didn’t had access to the original results manages -without any base on facts- to portray the Texan IABCers as communicators who “just don’t get it.” Nice reputation-building exercise.

Note: I promised Roy Miller that if he wants to post a response to this post (or to any other post on this issue) I will offer him the space of this blog.

Comments

Waiting for Cymphony/Porter-Novelli’s survey results

Back in March 2006, Cymfony was announcing the launch of a survey -done in partnership with Porter-Novelli and Russell Research- on corporate blogging best practices. Now it looks like the preliminary results are in, although the survey continues to be online (hm).

With the risk of becoming a bore with my obsession on survey reporting practices, I’ll say it again:

I hope the final report is going to include a detailed explanation of the survey’s limitations (self-selection, nonprobabilistic sample, lack of control over the survey instrument, etc.), and that the conclusions will be qualified.

I hope Cymphony and Porter-Novelli are not going to pull a Guidewire or an Edelman/Technorati with this survey. (If you don’t know what I mean, please follow the links below.)

Update: Jim Nail responds in the comments.

Related postings:

Comments

What’s missing from the Edelman/Technorati blogger study

Edelman and Technorati have published the final report on their blogging survey, titled “Public RelationSHIPS: Communications in the Age of Personal Media” (PDF). There are two introductory pieces written by Richard Edelman and Peter Hirshberg (Technorati’s Executive VP), and there’s an interesting comparison between the “old” and “new” PR school.

The page dedicated to the survey’s methodology has an incomplete list of limitations. Here’s what’s missing:

  • A warning that the survey is not a typical “scientific survey”; it has used self-selection and convenience sampling, not random sampling. The results cannot be generalized to a larger population; they are relevant only for the respondents to the survey.
  • There’s no such thing as “margin of error” for a survey using non probabilistic sampling. Saying that “the calculated margin of error for this survey is 3.4% based on the sample size” is nonsense.
  • The size of the sample: (at least) 31000 participants, and an unknown number of people who found the survey questionnaire via links posted on various blogs (sources: Richard Edelman’s presentation on October 6, 2005, and the survey results)
  • The response rate, which is very low: less than 3% (821 respondents).

Disappointing, especially since the problem of survey’s limitations has been discussed in the blogosphere. A timeline of these discussions is available here:

Edelman/Technorati blogger survey - aide-memoire

Updates:

  • Kamy Huyse thinks that “some of the comments especially are interesting to read“. Indeed. But the study has no analysis of the responses to open-ended questions.
  • Buldog Reporter’s Daily ‘Dog has a neutral article about the survey; no word about survey’s limitations.

Note: The initial title of this posting was “Edelman & Technorati blogger study released — what’s missing from its methodology“. The entry has been edited.

Comments (1)

Guidewire Group CEO responds to criticism

Mike Sigal, co-Founder and CEO of the Guidewire Group, has posted the following comment in response to my criticism and questions about the Group’s recent survey:

Hi Constantin,

Thanks for the critiques. Our goal as a research firm is to provide information that will explain emerging technology markets, so your feedback will help us develop more robust products in the future. I’m going to post a couple of general comments in response to your post and others as an update to the original post, but I wanted to answer your questions directly:

1) The primary goal of the survey was to explore why and how corporations were adopting blogs. That’s why we promoted the survey via press release and to the blogosphere…because we wanted folks who were blogging to answer it. The fact that such a large percentage of the sample were either already blogging or had plans to blog (and frankly the fact that a huge percentage of the adopters started in 2005) is what lead us to conclude that corporate blogging has already turned the corner into hypergrowth.

2) No guarantee. That’s why we were explicit about the sample and methodology. What surprised us was the distribution of titles/roles that responded to the survey.

3) It’s relevant to the corporate world because there are actual examples of ‘real’ companies that are using blogs and other social media tools for significant business benefit. There were terrific examples at BlogOn. Given that most coverage of blogging outside the blogosphere has been extremely light on corporate case studies, benefits, etc., we wanted those folks trying to learn something about social media to know that SOMEONE is looking into it. Hopefully it will cause MORE people to look into it.

4) The reaction at BlogOn to our annoucement (we only showed the adoption curve…the Executive Summary was put on everyone’s seats so they could read it) seemed pretty positive. In my at-the-show conversations with attendees and speakers, they we’re all pretty enthusiastic that we were drawing attention (both via the show and the survey) to corporate adoption and benefits, rather than the same old debates about journalists vs. bloggers, media transformation, etc. We’ll be posting the podcasts and webcasts from the show in the near future, so between that and the folks you mentioned getting back to their offices and deciding they want to post about the survey…I’m sure you’ll be able to come to your own conclusions.

Again, thanks for the feedback. And if you have suggestions about what we should investigate and/or how to improve our methodology…we welcome them!

Mike Sigal
Co-Founder and CEO, Guidewire Group

Comments (7)

Another pseudo-survey on corporate blogging hits the Web

Try to say this with a straight face:

The vast majority of companies (89%) are either blogging now or planning to blog soon.

Are you LOL? ROTFL?

Anyway, the staggering, unbelievable, hard to believe “statistic” is one of the results of a Guidewire Group “survey” called Blogging in the Enterprise, sponsored by iUpload.

To its credit, the Guidewire Group is transparent about the survey’s methodology (links added):

The “Blogging in the Enterprise” survey was fielded for two weeks in September 2005. The online survey was open to public participation, encouraged by direct email to a random sample of 5000 CMO Magazine readers, a press release announcing the survey, and unsolicited postings in various blogs and blog search engines. 140 individuals responded to the survey.

(Source: PR Newswire press release)

(Missing: the fact that those completing the survey were eligible to enter a draw to win an iPod Nano or a complimentary registration to BlogOn 2005.)

A couple of questions about the survey:

  1. Why would anyone use a “random sample of 5000 CMO Magazine readers“, then give up any control of the survey instrument’s distribution by making public the link to the questionnaire, and by asking bloggers to post about it? The result is that the survey has used a nonprobabilistic sample, and that no generalizations can be made about the data outside the small number of survey respondents.
  2. What’s the guarantee that those who took the survey were in the position of knowing the information they provided?
  3. If Guidewire was aware of the survey’s limitations –as the press release shows– why is ignoring them by talking about the results as if they were relevant for the corporate world? E.g.: Staggering Stats on Blog Adoption (Demo Letter), BlogOn 2005 Opening Remarks (BlogOn 2005 Blog).
  4. Finally, I wonder what was the reaction of participants to BlogOn 2005 when these staggering “statsâ€? were announced. Unfortunately, I couldn’t find anything about it in the entries tagged with BlogOn and BlogOn2005 on Technorati; and there’s no reaction so far from the fellow PR bloggers who participated (most of them as speakers) to the event: Elizabeth Albrycht, Shel Israel, Mike Manuel, Laurie Mayers, Jeremy Pepper, and Lisa Poulson. Steve Rubel has a link (Survey: Corporate Blogging Takes Off) –but no comment– to an article from internetnews.com.

Side note: there’s no difference, essentially, between Guidewire’s survey and the Edelman/Technorati recent survey. In fact, Guidewire was more upfront about the study’s limitations then Edelman (we’re still waiting for Edelman’s official report about the methodology, two weeks after the results were released). But the effects of self-selection are more visible in Guidewire’s survey.

Resources

Reactions from the blogosphere

  • Mike Gotta: “The survey has some potential and initial statistics are interesting but the survey unfortunately does not have enough data points to warrant serious attention. [...] My first concern is the sample size. 140 respondents is small. The respondent group was also self-selecting which can skew results (even though the survey was publicized, it is not the most balanced way of gathering statistically relevant data). Only 19% of the people who responded were from companies with greater than 1000 employees (that’s about 26 people). Around 32 people responded from companies with revenue of over $100 million. It’s unfortunate since the media has picked these statistics up as trends across “corporate america” and clearly that inference is a leap of faith unless you want to extrapolate that based on 26-32 people. You can of course. I would prefer to still classify corporate blogging as embryonic and in an exploratory/pilot phase within the vast majority of companies.”
  • Shel Holtz: “Guidewire Group’s BlogOn 2005 Social Media Adoption Survey suggests 91.4% of these corporations are using blogs internally and 96.6% externally. If you believe these results, I have the deed to a nice bridge we should talk about.”
  • Anu Gupta: “I don’t understand why the general reaction [to the survey] hasn’t been more disdainful - I wonder what the reaction would have been if, with the same sample size, the headline was ‘2% of companies are using blogs’?”

(More to come.)

Media coverage

Comments (7)

Edelman/Technorati blogger survey - aide-memoire

(Just a list of conversations I’m following.)

The survey:

Blog entries:

In the media:

Comments (5)

« Previous entries