Thursday, February 12, 2009

Selection Bias in Online Polls: Twitter vs. Cognitive Daily

There was an uproarious SmackDown on Twitter between @brucewagner and @davemunger over the issue of polling the percentage of gay/homosexual tweeters/blog readers.

You see, there's this small issue called selection bias that prevents reputable pollsters from asking a question in this way: How gay is Twitter? ;) Are you gay?
selection bias

Selection bias comes in two flavors: (1) self-selection of individuals to participate in an activity or survey, or as a subject in an experimental study; (2) selection of samples or studies by researchers to support a particular hypothesis.

Was Bruce actually serious that his Are you gay? poll was completely unbiased?

Yes, that's right, only 54% of all Twitterers self-identified as straight in this fabulous poll, with 28% calling themselves gay and the rest in between. Off just a little?

Sexual Behavior and Selected Health Measures: Men and Women 15-44 Years of Age, United States, 2002

Sexual orientation

In response to a question that asked, “Do you think of yourself as heterosexual, homosexual, bisexual, or something else?” 90 percent of men 18-44 years of age responded that they think of themselves as heterosexual, 2.3 percent of men answered homosexual, 1.8 percent bisexual, 3.9 percent “something else,” and 1.8 percent did not answer the question. Percents for women were similar.

[See Alkaline Earth for more.]

Given the wildly inaccurate Twitter numbers, Dave Munger wanted to prove a point by posting his own Are you homosexual? poll to the readers of Cognitive Daily.

He then compared these results to an earlier "Casual Friday" poll, which embedded a query about same- vs. opposite-sex partners in a series of questions about romantic gifts:
Selection bias and homosexuality

A couple hours ago I posted a quick poll, in what might be construed as an unbiased fashion. I simply asked respondents for their sexual orientation, offering a wide array of choices ranging from "straight" to "mostly gay" to "gay" to "other."

In fact, my poll was biased -- not because the question itself was slanted, but because of the way respondents were recruited: I titled the post "Are you homosexual?" Potential respondents who are homosexual or who don't have traditional sexual preferences are more likely to be interested in the question, and therefore more likely to respond. How do I know this biased the sample? Because I collected similar data last week in the Casual Fridays survey about romantic gifts. In that survey, women reported same-gender partners 5.7 percent of the time, and men reported same-gender partners 3.7 percent of the time.

However, some commenters noted that the Romantic Gift poll was not free of bias, either. Heterosexual partnered individuals were perhaps more likely to answer questions on whether jewelry, perfume, chocolate, flowers, lingerie, etc. constituted romantic gifts or not.

The moral of the story?

Do not give sexy nightwear or sex toys to women on the first date...

Subscribe to Post Comments [Atom]


At February 12, 2009 7:44 AM, Anonymous Anonymous said...

Ha! Nicely done. Best analysis I've seen of this to date.

At February 13, 2009 1:35 PM, Blogger Bruce Wagner said...

Using an obviously derogatory term, like "homosexual", in a poll question.. is the BEST example of poll bias I've ever seen.

This is exactly one of the reasons I created this poll...

Did you point out that my poll has been RT-ed (retweeted, i.e. re-broadcast) to ALL of Twitter at large...? And that the other poll was only posted on one guy's blog...?

His audience is obviously an extremely selective subset of the public - whereas Twitter at large represents a much larger, broader, public sampling.

As I have stated many times, I invite you to broadcast the poll anywhere (public & unbiased) you want to... The more responses we get, the more the numbers seem to stay the same.

Use any Title you want to...

I suggest simply:

"Please take this anonymous poll RT Pls"

If your beef is with the title... CHANGE the title...

If your beef is with the sample group... rebroadcast it to a larger (unbiased!) public...

I only ask that you:

(1) always include the word "anonymous", to address the fear of closetted gay/bi people.

(2) distribute in the most public way possible - not to a specialized mailing list of only southern baptists or anything.

If your beef is simply with the results... Deal with it.

Please take my anonymous poll, everyone! RT Pls"

At February 13, 2009 3:05 PM, Blogger The Neurocritic said...

Bruce - if your contention that...

my poll has been RT-ed (retweeted, i.e. re-broadcast) to ALL of Twitter at large...? And that the other poll was only posted on one guy's blog...?

is true, why does your poll from Feb. 8 have only 395 votes, whereas Dave's poll from Feb. 10 has 1,712 votes? Is the number of Cognitive Daily fans greater than the number of ALL Twitter users?

At February 15, 2009 3:16 PM, Blogger The Neurocritic said...

It turns out that Bruce Wagner is just a spammer and a scammer, as his suspension from Twitter clearly illustrates:

RT @mangaaa @brucewagner had an army of fake Twitter accounts which retweeted on command, Clever yet unethical!


Post a Comment

<< Home

eXTReMe Tracker