Governance reform results are as reliable as a coin toss

What do governance reforms and coin tosses have in common? They both yield positive results 50 percent of the time...

If we assume that governance matters and even that governance indicators make sense, what is the record of externally influenced reforms on governance? 

• One would hope that reforms from organizations like the World Bank have helped countries to improve their economic growth records--given the regularly implied argument that good governance promotes growth and development.
• One would expect World Bank reforms to impact World Governance Indicator (WGI) scores directly, given that such interventions and WGI’s are both products of the same external development community. They are the ‘test’ around which reforms may have been shaped.
• One would also be led to believe that reforms sponsored by organizations like the World Bank facilitate improvements in alternative measures like the International Country Risk Guide (ICRG) Quality of Governance (QOG) indicators. These provide potentially more independent yardsticks than WGI scores but ostensibly reflect things we care about when talking about governance.      

Given such measures, the record of economic growth and government improvement varies across a random sample of 40 countries I am looking at. It includes Afghanistan, Algeria, Angola, Argentina, Azerbaijan, Bolivia, Bulgaria, Cape Verde, Central African Republic, Chile, China, Georgia, Ghana, Guinea, Haiti, Honduras, Kyrgyz, Laos, Madagascar, Malawi, Mauritius, Moldova, Mongolia, Mozambique, Nicaragua, Niger, Poland, Rwanda, Samoa, Senegal, Serbia, Sri Lanka, Tanzania, Thailand, Togo, Uganda, Ukraine and Uruguay. 

• While GDP per capita increased in 38 of the countries between 2000 and 2008, growth was more sluggish than the average for comparable countries in 25 cases. Personal incomes increased by about 20 percent in Bolivia in this period, for instance, which was below the mean for lower middle income countries.
• Similarly, 21 of the 40 countries went backwards on WGI Government Effectiveness scores between 2000 and 2008. For example, Cape Verde dropped from 0.35 to 0.05 on a measure that ranges from -2.5 to +2.5. Togo regressed from -0.67 to -1.43.
• Related patterns are seen in the Quality of Governance (QOG) data, where values decreased in 19 of the 28 nations included in the ICRG database in this period. Argentina fell from 0.69 to 0.52 where the indicator runs from 0 to 1. Moldova dropped from 0.55 to 0.44.

There is a fair amount of correlation between these indicators. This overlap ensures that 18 countries languish in the ‘declining’ group of at least two of the indicators. A similar set of 18 countries are ‘improvers’ on at least two indicators.

Interestingly, the governance reform experience looks similar across the two groups:

• There is no statistically significant difference between the groups when one looks at numbers of World Bank public sector reform projects, dollar commitments for such or reform start dates. 
• The same can be said when looking at growth in per capita incomes, WGI Government Effectiveness scores in 2008 and QOG records in 2008. In all cases countries with declining records had similar reform records to those that progressed.

While not statistically different, the average country with declining WGI scores actually had more projects (59) and higher costs ($1.3 billion) than the average country with improved WGI scores (48 projects at $1.1 billion). Reform starting dates were also similar. Of 19 countries that started reforms before 1985, 9 saw declining government effectiveness in the 2000s and 10 saw improvements. Of the 21 that started after 1985, 10 declined in the 2000s and 11 improved.

The message from this data is plain: Countries with similar records of externally influenced governance reform have seen varying results.

Taking the data at face value, it appears that there is about a 50 percent chance that such reforms make governments more effective. Half of the time countries pursue this reform but government gets less effective. Success is as reliable as a coin toss producing heads.

Is this good enough? Is it something we should expect? Is it because the reforms don't work or maybe because 'governance' is a problematic means or end in the development debate?

Comments

[twitter.com/freebalance]

These observations are useful. It's the kind of insight that begs that we peel the onion more.

The problem with the WGIs is that many of the sources are perception-based, overlap of 3rd party indicators across the WGI roll-ups, range of the number of indicators per WGI roll-up, and lack of weighting on what is important. A good anchor none the less.

The next step could be to drill into those countries where the investment was significantly correlated with success and those where is was significantly correlated with reduction of economic development.

There is also the question of the type of investment made. That might provide additional insight.

Governance has many moving parts. The evidence might show that there is a difference between reform as ceremony and government owned reform. For example, PEFA assessment results seem to show that many operational reforms do not take hold. On the other hand, certain types of public financial management automation tends to improve some aspects of governance. Government-led reforms tend to work better than donor-led, from anecdotal evidence - but that gets into a chicken-and-egg discussion. (i.e. if government led the reform, there may have been formal or informal governance mechanisms.)

There's also " modalities " of reform. As much as I dislike the jargon, this seems to be another avenue to consider.

Another problem in analysis: lack of transparency on information (hence #makeaidtransparent - we need to all sign up on this petition. Result? - getting the information to answer the questions that you've raised and make aid effectiveness.)

Not to mention asymmetrical improvements in governance that could be lost in the numbers. Some of the countries that you have mentioned have had excellent results in some aspects of governance but poor in others. The aggregate result hides these successes.

The coin toss might depend on the context.