Is The Economist democracy ranking a good index?
This article shows that the methodological quality of the DI index is poor
Image credit: PexelsThere is no doubt that the Democracy Index (DI) ranking of The Economist magazine is one of the most influential indices in the world. Every year, when the new scores are published, governments, political parties, the media, and domestic and international public opinion carefully observe which countries move up and down the ranking. There are big reasons for this. A country can gain prestige, a good reputation, and even economic benefits if it ranks well, since a good score can indicate, for example, that the country respects human rights or that it is a good place to live and do business.
The great influence of the indices hides, many times, more substantive questions about their scores. Who decides that a country should raise or lower its note? Under what criteria? Is the modification of the grade in accordance with your methodology? Of course, the quality of democracy will be better measured if the indices that measure it have a good quality as well. And by quality we mean that the index is valid, reliable, and transparent. This article reviews how and under what criteria The Economist has rated Spain in recent years. It is a relevant issue, since the quality of democracy has been called into question by various domestic and international actors and because in the latest report by The Economist, Spain has come to be considered a “weak democracy”. This article concludes that the methodological quality of the DI index is poor.
The methodology of the Democracy Index
The DI ranking is prepared by The Economist Intelligence Unit and scores the democratic quality of countries on a scale of 0-10, being 10 the highest level of democracy and 0 the lowest level. According to its latest report published in early 2022, Norway is the highest ranked country with 9.75 points and Afghanistan the lowest ranked country with 0.32 points. The report also establishes various categories of democracy based on the numerical score each country receives. Thus, countries between 8 and 10 points are classified as “full democracy”, between 6 and 8 “weak democracy”, between 4 and 6 “hybrid government” and less than 4 “authoritarian government”.
The final numerical grade of the index is established on the basis of 60 indicators that are grouped into five dimensions. For example, the dimension of “civil liberties” is made up of 17 indicators that measure, among others, whether the country in question respects freedom of the press, freedom of expression or whether there is diversity of opinion in the media. communication. Each indicator can have three values: 1, 0.5 or 0. For example, a country would receive a 1 on the press freedom indicator if it is fully respected, 0.5 if there are some violations, and 0 if it is not respected. Once the scores are established for each country, these indicators are aggregated at the dimension level and then re- aggregated so that each dimension has the same weight in the final index.
To carry out an analysis of the DI ranking, the first thing an observer encounters is a problem of transparency. The Economist does not offer data at the indicator level, so it is not possible to know if the Intelligence Unit coders have scored a country with 0, 0.5 or 1 in each indicator. The only information available is the score for each country at the dimension level that appears in the tables of the annual report (see summary in the following table), and the wording of the report itself, where on some occasions the text justifies why a country has suffered a variation from one year to another.
Of the numerical variations that Spain has suffered in its levels of democracy, only some of them are explicitly commented on in the latest reports. We are going to analyze two of the variations that, as detailed below, point to a serious problem of validity. These variations are marked with color in the table above. The first variation is commented on in the 2019 report, where it is noted that the improvement in Spain’s score between 2018 and 2019 was due to an “improvement in relation to one question concerning local elections, which were held in May 2019 and were deemed to have been free and fair.” Indeed, if we look at the dimension of ‘Electoral process and pluralism’, we observe that there is an indicator that asks: “Are municipal elections both free and fair?”.
It could be deduced that this indicator fell in 2017 and rose again in 2019, which resulted in a decrease from 9.58 to 9.17 at the dimension level. In 2019 it recovered its initial levels. However, it is difficult to establish the reason for this variation. In the first place, because municipal elections took place in Spain in 2015 and 2019. But there seems to be no reason why they can be considered more or less free1. In other words, municipal elections have been equally free since the beginning of the democratic period, and it does not seem that there has been a sufficiently significant variation in their performance to suggest that the score should be changed.
The most plausible explanation is that this variation is due to the fact that the coders of The Economist considered, for one reason or another, penalizing Spain in 2017 for the events related to the independence referendum in Catalonia and, after these events, decided to return the indicator to its original level in 2019. This variation responds, in any case, to a validity problem, because the modification of the indicator was motivated by a fact that had nothing to do with the operationalization of the indicator.
The second variation analyzed in this article is included in the 2021 report, where it is explained that Spain has suffered a “downgrade in its score for judicial independence, related to political divisions over the appointment of new magistrates to the GCJ”. If we look at the tables, the only dimension that underwent a change between 2020 and 2021 was that of ‘Civil Liberties’, which dropped from 8.53 to 8.24. The only indicator of this dimension related to the judiciary measures the following:
Indeed, the European Commission urged Spain to renew the magistrates of the General Council of the Judiciary. But the indicator measures the degree to which the judiciary is independent of government influence. Precisely, there is currently a significant ideological distance between the courts, which are more conservative, and the government, which is more progressive. For this reason, it is difficult to justify with what is described in the operationalization of this indicator that there are reasons to think that the government has influence in the judiciary. Quite the contrary. Again, this variation responds to a validity problem because the modification of the indicator is motivated by a fact that has little to do with what the indicator intends to measure.
Conclusion
With the examination of the changes that Spain has undergone in the DI index, it is concluded that the methodological quality of the index of The Economist is poor. The index is not transparent, and its validity is strongly compromised, since the variations of the indicators are arbitrary and do not respond to the operationalization established in the methodology. This does not mean that democracy in Spain does not have important deficits. But for the indices to generate the correct incentives, their methodology must be clear, transparent and, above all, predictable. If their variations are subject to arbitrariness, the states will be more concerned with pressuring the arbitrators than with complying with the democratic standards set by a rigorous methodology.
One might argue that the elections in Barcelona were not free and fair, since the Spanish intelligence services and some Spanish media invented a false bank account belonging to the then mayor Xavier Trias. Trias lost the local elections by a short margin of votes to Ada Colau.↩︎
Jordi Mas Elias
Professor of International Politics
My research interests include political economy, international politics, regionalism, and methodology in social sciences.