Monthly Archives: November 2015

Why You Can’t Depend On Twitter Polls

- Prasanto K. Roy, Head – Media Services, Trivone Digital


Well, mostly — accept to prove what the pollster already believes

Towards end-October, Twitter finally rolled out the ‘poll’ feature on its most popular platforms: the desktop Web, iOS and Android. So now you can ask your followers and their followers, important stuff like who’ll win Bihar, or whether they’d rather date Justin Bieber or a troll, or whether New India’s symbol should be a spaceship or a cow.

How much can you depend on the Twitter poll for real insight, though?

The short answer is: not much.

The longer answer is, it depends on what you’re depending on it for, insight, or influence. If it’s to prove something you already believe to be true, yeah, that works. All you have to do is craft your question and response options in a way that will get most people picking the answer you want, and let selection bias do the rest.

What’s the selection bias here? That’s the big filter: you and your followers. They’re aligned to your thinking. If you’re into tech, or Hindutva, or medicine, you already have many followers with similar interests, and many with similar thinking.

Here’s a simple experiment. A senior journalist with center-right leanings and over 80,000 followers on Twitter posted this poll in early November: In your day-to-day affairs, have you experienced a rise in intolerance lately? The options: a. Yes, there’s venom, and b. None, whatsoever.

As you might expect, only 8% of 1,667 who voted said Yes, there’s venom. Why? One, because of the way it was worded. Indeed: in my day to day affairs, I’m less likely to have experienced intolerance, let alone ‘venom’. (Note also the extreme contrast between the two response options.) And finally, there’s the selection bias of the pollster’s followers being likely right-wing-inclined.

Abhijeet Poll Final

A twitter poll result can change drastically depending on who's asking–and on the exact word

To test this out, I posted a similar, but simpler, question: Do you see rising intolerance in India? Yes or No. To dilute selection bias just a bit, I put in request to RT. Over 180 retweets (18,300 impressions) took it to a more diverse audience. And as against the 8% who said Yes in Abhijeet’s poll (which got 84 retweets), 44% of the 1,891 who voted on my poll said Yes, they saw rising intolerance.

PKR Poll FinalWider sharing can help dilute selection bias (here, 180 RTs, 18,300 impressions)

The pattern was interesting. In the first 30 minutes, with 99 votes, 63% said Yes to rising intolerance. Those would be my immediate followers. As the poll was RT’d, diluting selection bias somewhat, the “Yes” responses dropped to 55%. With further RTs, especially from right-wing handles, the Yes responses dropped below 50% and finally settled at 44% when the poll closed 24 hours later.

PKR Poll Hour 1

In the first hour, with only the pollster's own followers responding, selection bias shows

No, I don’t believe even this 44-66 is necessarily a representative response for India.

The point here isn’t about intolerance, but about how unscientific online polls can be with the best of intentions. The answer to the same question can vary widely based on who is asking, how the question and responses are phrased, and who shared it. Among other factors.

Someone qualified in market research can do things to reduce bias and get more dependable answers. Or, conversely, to manipulate those answers to ‘prove’ virtually anything.

You could argue that this is true of all polls, especially online ones, including SurveyMonkey polls. Well, Twitter makes it so much easier to poll. Expect to see an explosion of them, with everyone becoming a pollster and surveyor. Just as Twitter turned everyone into a journalist, killing mainstream media.

Oh wait. It didn’t.