Tuesday, March 27, 2012

Crowdsourcing a Democracy Index – 2012 edition

It’s that time of the year again – time to crowdsource a democracy index.

First, a bit of context. Last year, I had the idea of using the Allourideas pairwise comparison software to crowdsource a ranking of countries by their degree of “democracy” in 2010. I asked students in my Dictatorships and Revolutions class to set the ball rolling, and then posted the link to the widget here, allowing anybody to vote. Surprisingly, in just a couple of months of voting the results were interestingly close to standard indexes of freedom or democracy: the crowdsourced ranking had a correlation of 0.84 with Freedom House’s widely-used ratings of political and civil liberties for 2010, and the basic crowdsourced ranking was generally plausible (see this post for further analysis of these results) . In fact, by now the correlation has increased to 0.86 (8556 votes total), which is about as good the correlation between Freedom House and Polity IV (0.87).

I am interested in seeing if this kind of crowdsourced measure can be used as a sort of quick and dirty index of democracy. To be sure, crowdsourcing the construction of an index of democracy in this way is usually not a good way of generating reliable social science data. For one thing, the exercise does not impose any restrictions on how the concept of “democracy” should be understood, which means that it implicitly aggregates all kinds of different ideas about democracy, weighting them by the degree to which larger numbers of people consider them important. (It takes a “democratic” approach to concept formation, you might say). But it does have the virtue of being cheap (total cost: about $0, compared to over US$500,000 annually for the Freedom House “Freedom in the World” report, and $120,000 annually for the Polity IV project), aggregating the dispersed information of large numbers of people from all over the world, and making it possible to generate various measures of “uncertainty” around the crowdsourced estimates. So I want to repeat the exercise, and produce a democracy ranking for 2011:

(Click here if you can’t see the widget above. Vote as many times as you'd like, and don't worry if you have to use the "I can't decide" button).

In theory, the more votes, and the more diverse the voting population – the more people from Africa, Latin America, Asia, and the rest of the world – the more informative the results should be. So please vote early and often, and share!

I am also interested in which dimensions of the complex concept of democracy people tend to weigh more when making these sorts of comparisons. Do people put more weight on the presence or absence of elections, for example, than on economic equality? You can help me to figure this out by using the widget below to rank various dimensions or components of democracy in terms of their importance to you (or adding your own):

(Click here if you can’t see the widget above. Vote as many times as you'd like, and don't worry if you have to use the "I can't decide" button).

The “seed” dimensions of democracy for these comparisons are taken from a recent piece by Michael Coppedge, John Gerring, et al. (“Conceptualizing and Measuring Democracy”, Perspectives on Politics 9(2): pp. 42-62; ungated here) that is very much worth reading if you are interested in the issue of how to measure democracy. But I make no claim that these are the only dimensions of democracy that matter; if people have other ideas, you can add them in (I will need to approve any suggestions, though).  You are also welcome to discuss in comments the kinds of considerations that you used to make distinctions between countries, or any other considerations that might improve the usefulness of this sort of exercise.

Enjoy, and please share!


  1. Glad to see you're doing this again, Xavier. It's a great idea. Unfortunately, the widgets are working for my first vote, but I get nothing but error messages after that. I'm using Chrome, in case that's relevant.

    1. Jay, it could be a problem on the server side. the widgets were working earlier for me on Chrome, but aren't loading properly now. Try clicking on the links to go to the Allourideas website directly (http://www.allourideas.org/democracyindex2011 and http://www.allourideas.org/democracyconcept2011). These still seem to be working - I'll check with the allourideas people later.

    2. They seem to load and work now, a few minutes later. (Chromium running in Ubuntu). Hopefully it was just a temporary issue.

  2. I hit "can't decide" on the importance of two aspects of democracy, and was asked to give my reasons. The options were "I like both", "I don't like either", "I don't know enough". This seems like a non-sequitur. You didn't ask which of the two I like better. I like pizza, but it has nothing to do with democracy.

    This seems to reflect the common use of the term "democracy" as a substitute for "good government". This conflation of those two ideas should be avoided, I think, since it often stands in the way of useful discussion.

    1. Hi Yoram - the options are standard, they are not mine. (The come with the software). To change them, I would have to change the code for the allourideas software, though I agree these options are not ideal - democracy is not necessarily "good government" as you note.