MattAndrews

The delusion (or deception) of MDGs and measurement

Last week I was involved in some discussions about post 2015 MDGs. In some meetings I was told that all indicators chosen must be measurable and must actually already be reliably measured. A sensible criteria, one might think.
But also one that is delusional or just plain deceptive. For people who have never visited the statistical offices in many developing countries it might sound plausible that reliable measures exist for things like GDP and infant mortality and access to schooling. And this belief gives a sense of legitimacy to the MDGs we have now.
But people who have visited statistical bureaus in many developing countries will tell you that the measures of these and many other concepts are fraught with problems. And should not be trusted in many cases.
Read Morten Jerven's poor numbers if you want a taste of this reality.
I wonder why we insist on pretending the measures and goals we use in MDGs pass the smell test, when in most cases they don't? Is it delusion or deception at work in the international community?

The cost of post 2015 indicator consultations: Money well spent...or not?

I just googled post 2015 governance indicators to make sure I'm keeping up with all the latest on this topic. Wow. Lots and lots to catch up on. Meetings all over the world involving many many organizations and people, from the UN-this and the UN-that, development org-this and development-org that. Not to mention all the bilateral organizations (they can't miss out) and the independent consultants (who are usually not that independent, given the financing of their flights) and now and again some government officials from developing countries. Oh, I should not forget the Eminent Persons Groups, High Level Officials, and other 'most important people who we should listen to very carefully'.

What does all the meeting and negotiation and engagement and travel and dining and chatting and calling and skyping and videoconferencing and otherwise chinwagging cost?

And why is this cost acceptable?

Reading Morten Jerven's book on statistics in Africa (Poor Numbers) makes me think that BEFORE we spend loads of cash on these kinds of development-community-reinforcing-activities we should actually be financing better statistical processes in the developing world (so that if we do have global or regional or country level or local indicators they actually mean something).

But maybe I'm just in a Monday mood.

A great way to evaluate governance reforms

Taryn Vian and William Bicknell from Boston University recently published and article called Good Governance and Budget Reform in Lesotho Public Hospitals: Performance, Root Causes, and Reality (Pre publication version is available at http://www.u4.no/recommended-reading/good-governance-and-budget-reform-in-lesotho-public-hospitals-performance-root-causes-and-reality/downloadasset/3141).

In the article they assess the detailed form and basic functionality of the reform, using the following table. This is an application of an evaluation method I proposed in 2006 and is the kind of mechanism that all reformers should use when introducing complex interventions. Notice the form- function gap that is evident in results:

* plans and budgets kind of exist in form

* data collection and reporting mechanisms needed to make the plans and budgets matter don't really exist

* no one uses the new reforms.

Great work and what I think should be a pathbreaking study on evaluating reforms in developing countries.

 

Table 3: Reform Implementation Progress, Case Study Hospitals

Dimension/ Measure	Hospital				Average
	A	B	C	D
Dimension 1: Evidence that Performance Plans Exist
Availability of plan	●●	●●	●●	●●	Modest
Organization of plan	●	●●	●●	●●	Modest
Accountability structure	●●●	●●	●●	●●	Modest
Defined activities/outputs	●	●●	●●	●	Modest to little
Participation	●●	●	●●	●	Modest to little
Baseline data	○	○	○	○	None
Institutionalization	○	○	○	○	None
Acceptance	●	○	●	○	Little to none
Dimension 2: Evidence that Performance-based Budgets Exist
Availability of budget	●●	●	●●	●●	Modest
Organization of budget	●	●●	●●	●●	Modest
Output/activity costing	○	●	●	○	Little to none
Institutionalization	○	○	○	○	None
Acceptance	○	○	○	○	None
Dimension 3: Evidence of Performance Monitoring & Reporting
Data systems oriented to PBB	●	●	○	●	Little
Performance reporting	●●	○	○	●	Little
Financial reporting	○	○	○	○	None
Dimension 4: Evidence that Performance Information is Used
...To make or change work plans	●	○	○	●	Little to none
...To develop budgets	●	○	○	○	Little to none
...To monitor efficiency and effectiveness	○	○	○	○	None
...To communicate within hierarchy (internal accountability)	○	○	○	○	None
...To communicate to the public (external accountability)	○	○	○	○	None

Key: ○ = no progress; ● = little progress; ●● = modest progress; ●●● = strong progress. Refer to Table 5 for criteria used to evaluate.

Posted with Blogsy

What is the story with MDG performance?

There have been some really interesting recent reports on achievements in respect of Millennium Development Goals (MDGs).  I read them--and the way different authors frame results--and wonder what we could really say about the impact of global goals.

First, the Kaiser Family Foundation (http://kff.org/news-summary/new-one-report-examines-correlation-between-spending-progress-on-mdgs-media-analysis-examines-goals-usefulness/) provides quotes from a new ONE report (in the Guardian) and the Christian Science Monitor.

From the Christian Science Monitor we get the message: These goals made a difference ... but don't ask us to quantify what difference they made. Just do the exercise again!

“Aid officials, academics, politicians, and many ordinary people are trying to figure out what will come next,” the Christian Science Monitor writes, continuing, “There will definitely be a continuing global development agenda. … A consensus has emerged because of the MDGs that these sorts of global exercises are worth doing, even if we’ll never be able to measure their impact precisely” (McClanahan, 5/26).

From the ONE report we get a more nuanced perspctive: These goals are useful but many countries are still not coming to the table (which makes one wonder what the goals are useful for?)

“African countries that are allocating a greater share of government spending to health, education and agriculture are making faster progress on achieving the U.N. Millennium Development Goals (MDGs) but many nations are still failing to meet their commitments and are lagging behind,” according to the ONE Campaign’s 2013 Data Report released today, The Guardian reports.

Then there is an Oxfam blog (http://blogs.oxfam.org/en/blogs/13-05-16-are-governments-meeting-their-mdg-spending-targets): It tries to keep upbeat but notes that most countries have not spent what was needed to meet the MDGs (so, again, what did the MDGs achieve?)

"The vast majority of developing countries are spending much less than they have committed to, or much less than international organizations have estimated are needed to change lives. Only one third of countries are meeting any education or health goals, and less than 30 per cent are hitting targets for agriculture and Water and Sanitation (WASH) initiatives. To top it off, falling aid commitments, low execution rates, and low recurrent spending have combined to put a real threat to any existing progress."  

Here is the picture they use to show how many countries are meeting the MDG spending targets:

Then there are data from the ONE report (Ben Leo's super useful index of MDG achievement... http://www.guardian.co.uk/global-development/poverty-matters/2013/may/29/millennium-development-goal-progress-data). Again the report tries to be upbeat and notes that a number of African countries have seen progress in the last few years, but by my calculation,  when looking across all countries and all MDGs, over 35% of the targets are TOTALY OFF TRACK and about 55% are TOTALLY OFF TRACK OR only PARTIALLY ON TRACK.

It seems to me that the MDGs that are on trck are those that have been influenced by economic growth.... So the message I see is that most of the goals will not be met and we have no real reason to say that the MDG initiative had anything to do with the goals that will be met. 

so, what is the value of the MDGs? and what evidence do we have to say that we need to have post 2015 MDGs?

Just asking. Because it's a pretty important question.

 

Good or Good-looking governance? That is the Question

I am in London and Manchester today, presenting on my new book. Tonight I'm at ODI in London and tomorrow I'm at the University of Manchester. Details here. (http://www.odi.org.uk/events/3162-matt-andrews-pfm-governance ; http://www.policy.manchester.ac.uk/events/2013/professor-matt-andrews-esid-seminar-and-book-launch.htm)

We all like to look good, especially in the eyes of others. This is why so much was made earlier this year when Afghanistan scored 59% on the Open Budget Index (OBI) measure of budget transparency. Its score made it look about as good as the more developed Italy, which registered 60%. This performance generated applause from a variety of observers, who reflected on the reforms that made such score possible. These included producing a new pre-budget report, including output information in budget documents, and creating a citizens budget that explains funding commitments in layman’s terms.  The Ministry of Finance, a variety of donors, and some contractors got major pats on the back for these endeavors.

And they are all good things, and constitute a good achievement.  And the OBI statistics are important for all of us thinking about transparency and accountability and governance. But when one looks at the OBI assessment in full (and not just the average score) there are some significant caveats to note (that are not picked up in reports of the average scores). In particular, while Afghanistan was transparent in telling citizens what it intended to do with money, it did really poorly when asked about transparency in its budget execution process. It did not do well in bringing out mid-year reviews, or producing year-end reports or audits showing what money was actually spent on, when, how and with what effect. When I calculate the scores it received for transparency in this area of the budget process—where money is actually spent—Afghanistan only registers 39% compliance. This contrasts with 67% when looking at transparency in budget preparation.  The result is a gap of 28 points between transparency in budget formulation and execution.

This gap is evident in the OBI data and discussion documents. It means that while Afghanistan produces great looking budgets, we have very little idea about what it actually does with its money after these budgets are published. Incidentally, the gap is only 1.8 for Italy according to the OBI data. While Italy may not look as good as one might expect for an OECD country, it is just about as good as it looks. Afghanistan, on the other hand looks as good as Italy but what you see is not what you get (wysinwyg).   It has good looking governance but I’m not sure if you can say that this kind of governance is also ‘good’. And this is after a decade of concentrated and expensive reform.

 

The above table shows the OBI data and the calculation of gaps between transparency in formulation and execution for Afghanistan, Italy and a few other countries.  All of the countries have similar OBI scores but very different gaps, such that Italy, Bulgaria and Chile look about as good as they are but Uganda and Afghanistan look much better than they are.

When looking at various governance indicators I find that a large proportion of developing countries seem to be producing ‘good looking governance’ like this. I see this with Global Integrity’s measures of processes and systems to combat corruption, for instance. Uganda surprises everyone on this index to yield a 72% score, similar to that of Italy, Chile and Bulgaria—and higher than neighboring Kenya. However, Global Integrity also publishes an ‘implementation gap’ (which is the difference between the quality of a government’s laws and the implementation of the laws) and Uganda scores more than 45 on this measure. Italy scores 12, Chile scores 20, Bulgaria 23, and Kenya 33 (as shown below). Again, we see developing country governments getting strong average scores that suggest all is good and well but actually only reflect ‘good looking governance’.

 

One could look at this kind of evidence and respond in a number of ways. One could say that the gaps I observe are probably only apparent in some countries, so it isn’t a real problem. One might also argue that you could expect gaps when reforms are new and that these close over time, so it isn’t a real problem. My recent research suggests both of these responses are not valid. By my estimates the vast majority of developing countries have gaps and ‘good looking’ governance problems and the gaps are growing in many countries after reforms—not closing.

I argue in my recent book that this is because many reforms are adopted as signals—to garner better governance scores and ensured continued support from outsiders—and not real efforts to improve governance. This approach to reform results in new laws, systems and processes being introduced that do not fit the local context, demand more capacity, political will and other content than is available, and are led by groups far too narrow to ensure diffusion and deepening of change. The result is a reform limited to changes in form, but without functionality. Good looking governance rather than good governance.

I believe that there are ways we in the development industry could do reform differently and better. The approach I suggest is (i) driven by real problems in the contexts undergoing reform, (ii) involves multiple agents in iterative processes of experimentation, learning and adjustment, and (iii) ultimately results in the production of hybrid structures that may or may not yield better scores on indicators but provide a more functional set of rules for governing. The process is called PDIA (problem driven iterative adaptation) and is something I and colleagues Lant Pritchett and Michael Woolcock have identified by looking at principles of successful change in developing and developed countries. It offers an approach to fostering development that could be the future.

If, of course, we value governments that actually perform; where transparency matters and laws are implemented and budgets are executed. Right now, I think we care about looks a lot more than this kind of real, performance. And this is why we continue getting good looking governance more often than anything we could honestly call good (or good enough) governance.

Governance Looks Different in Different Countries: A Chinese Translation

I'm really happy that my 2010 Governance article looking comparatively at governance regimes is now available in Chinese, at Chinese Translation .

I'm taking this from the Governance Blog, April 2012 (see below).

The GOVERNANCE blog

Governance: An international journal of policy, administration and institutions

Download translation of Andrews article on good governance

哈佛大学Matt Andrews教授所著《"因国制宜"话善治（Good Government Means Different Things in Different Countries）》的中译版现在可以免费下载，其英文原文发表在2010年第1期《治理（Governance）》，该翻译由北京大学政府管理学院博 士生臧雷振完成。下载译文.

Like this:

Like

Be the first to like this.

Written by governancejournal

April 30, 2012 at 12:46 pm

Posted in Translations

« Free download of "Explorations in Governance"

Elitism, Meritocracy, and Governance in South African Companies

I want to get back to the messiness of governance.

The issue of meritocracy and elitism in good governance

'Meritocracy' is a key element of most 'good governance' models. The idea behind this is that governments, companies and the like should hire those who are most competent to do the job, not those who are most connected. The assumption is that we can measure—objectively—what it means to be 'most competent'. We say these are the ones who have the 'merit' for the job. Hence meritocracy.

Elitism is often presented as the opposite of meritocracy and is something that many 'good governance' advocates reject. These good governance advocates don't like the idea that some people get key jobs because they are elites—connected, wealthy, enjoying power.

Good governance, as many suggest, requires having principles of meritocracy that replace practices of elitism in key appointments, contracts etc. in governments and the private sector. The idea is to have authority in the hands of those who have the merit, which means that authority is not concentrated in the hands of the elite and is in fact located in a distributed group of agents who enjoy the skills and 'merit' to get the job done.

Meritocracy and elitism in South Africa's listed firms

A few years ago (in 2008) I did some research into the background of directors of South Africa's listed firms. These are people who have the kind of positions many would say matter in society (they have power over how resources are allocated, for instance, and authority). You would want to make sure these folks were appointed on merit, so that they did what was required to keep their companies growing and being successful. South Africa scores pretty well on most good governance indicators (given its income levels) and so one would expect that directors would be appointed based on merit.

Some might expect that this means they have qualifications like advanced degrees, in accounting, engineering and management perhaps. And this is what one finds. As the table below shows, 772 of the 873 directors (89 percent) I looked at in a random sample of 97 listed firms had advanced degrees from universities. 562 of these (64% of the sample) had what I would call targeted advanced degrees (in accounting, law, business, and engineering) that one would expect of directors with 'merit'. There is an issue with gender and racial composition in South African boards, so I broke the data down to show how 'meritocratic' directors in different groups were. One can see that more than half of the directors in all 4 sub-groups I looked at (white women through to non-white men) had these targeted degrees. Over 85 percent of the directors in all sub-groups had tertiary education. The lowest percentage was 86 percent (for white men).

	Tertiary qualification	Chartered Accountant	Law	MBA	Engineer	Targeted degrees (CA, Law, MBA, Engineer, Medicine)
Number	774	285	98	121	78	562
Percent of total	89	33	11	14	9	64
White women (ex 31)	27	6	2	4	1	14
White men (ex 560)	484	219	66	66	58	409
Non-white women (ex 80)	77	16	7	16	4	44
Non-white men (ex 212)	188	44	23	35	15	119

So far, so good. Directors of South African firms seem to reflect the kind of 'meritocratic' profile good governance advocates would support. But this is not the full story.

Directors are not just hired because of their qualifications. Other factors matter, which companies communicate in annual reports to shareholders (which boast about the backgrounds of directors). When one looks at these other factors things get messy and one starts to see the social and political complexities of South African society come out. One also starts to see variables that speak to the presence of 'elites'.

• The primary historic South Africa 'elite' variable must relate to race; one sees this squarely in that 68% of directors were white (592 in total)…14 years after apartheid ended.

• Gender variables also reflect 'elitist' tendencies; 87% of directors were men (762 in total).
  • Interestingly, white men had enjoyed more and more appointment over time. 26 % of the directors in 2008 were white men appointed after 2003. (This is greater than the share of non-white male directors appointed in all periods (23%)). 41 % of directors in 2008 were white men appointed after 1998. (This was greater than the share of non-white directors appointed in all periods (32%)).
• Another 'elitist' variable could relate to age; over 90% of directors were over the age of 40.
  • Interestingly, the average appointment age of a director increased from 41 and 46 between 1994 and 2008. Also, the appointment age of white male directors is significantly higher (statistically) than the age at appointment of other groups in most recent periods (post 2003).
• Another 'elitist' variable could relate to access to capital and ownership of the firm. 36% (317) directors came to their positions in this way.
  • Interestingly, 69% of all non-white directors (183) were members of a Black Economic Consortium that held partial ownership in the firm.

There are some variables I captured to reflect on the many times an annual report would speak of a director's past experience when introducing their profile. I was not sure if many years of experience is a mark of merit or of elitism, but the influence of this variable on director selection mattered a great deal.

• 560 of the 873 directors (64%) had big business background (ten years or more working in a listed company, most often the company in which they became a director or a similar company in a similar industry).
• 167 (19%) had experience in the public sector, which involved a mix of political office and public office (being part of the public administration) (100 had direct, publicly advertised political affiliations). Most of these were non-white directors, over 55% of whom had a past public sector career.

So much data. I know. I'm sorry. Sort of. But let me get to the point.

When one looks at directors of firms in South Africa, one sees a group of people who have both the attributes of meritocracy and elitism. And both sets of attributes seem to matter in getting them where they get to. There are very few white male directors who got to be a director without being able to say that they had merit (lots of experience and advanced degrees). Non-white directors may have found an entry way through the capital of an elite-owned Black Economic Empowerment Company, but most of these directors also had tertiary education and advanced degrees and high level experience working in government.

In short, elites are often also the ones who claim points on meritocracy scales. They go to the best schools, get the most targeted degrees, enjoy opportunities to get lots of work experience, etc. And they often define what it means to have merit as well…. Elitism and meritocracy are thus not mutually exclusive but actually form two sides of the same coin.

Pushing meritocracy as part of a good governance agenda is thus not a simple antidote to concentrated power in societies.

I'd love thoughts on this….

Post 2015 MDGs should be shaped around country-specific problems, not global agendas

Yesterday I wrote a short post on the post-2015 MDG goal setting exercise. I got loads of responses online and offline. Just to be clear, I think the exercise of setting global goals raises many questions: about the role of  governments versus international organizations, the idea of 'eminent persons' making high level policy decisions versus normal people doing the implementation, and the gap between high level planning and actual implementation.

One of the reasons I focused on governments identifying their problems (note problems, not goals) and then beginning a process of finding solutions is because this is the best way to mobilize the kind of energy required for implementation. And no matter how much people want to spin the story, the MDGs lacked implementation power in many places (local money, energy, political authority etc. were not provided to push the global goals forward). In my other blog (that supports my book) I argue that problems are the entry point to real change, which also requires experimentation and broad-based ownership (see a link to the blog here http://matthewandrews.typepad.com/the_limits_of_institution/2013/01/looking-like-a-state-the-result-of-limited-institutional-reforms-in-development.html).

One of the responses to my blog yesterday reflected on this issue really well. It is from Natalia Adler at UNICEF and I include the full comment below (emphasis is mine).  Her references to Mozambique reflect our common experience there, and are relevant in many other countries. Thanks, Natalia:

"These types of global goal-setting initiatives   [post 2015 MDG type initiatives] are rarely able to inspire or mobilize concerted action for change, particularly in everyday actions in public management at the country level (never heard government officials saying they have to change course because of the MDGs...).

But the problem is not necessarily related to whether these indicators are globally or nationally agreed upon. I think the same logic applies to national goals as well... The problem is that it's just difficult to translate these outcome-level indicators into a series of operational interventions that will eventually contribute to their achievement. 

As you know, Mozambique is/was developing a national planning system aimed at bringing synergies among all these plans/goals in hopes of making them more strategic, coherent and linked to the budgeting process. But making even well-designed plans and their goals operational is hard work. Plus, politics and power relations influence the extent to which these plans yield concrete results. In addition, the goal-setting agenda carries the assumption of a ‘clean slate’ often neglecting existing arrangements and commitments (to other goals), which cannot be easily brushed aside to give place to new plans/goals, whether they are agreed nationally or globally.  

I personally don't have a problem with the post-2015 discussion identifying a set of 'global' indicators (e.g. child mortality being one of them). What should change are the targets, which, I agree, should be country specific/defined and focused on existing implementation issues and a realist understanding of the pace of development efforts.  I agree with you. The focus on solving problems can trigger change better than goal-setting exercises..

A Post 2015 Menu of Development Indicators, With Fitted Goals?

I'm going to veer off the topic of governance for a short post this morning.

Last week saw a high level meeting in Monrovia to discuss the post 2015 Millennium Development Goals (if that is what they are called). Yesterday the ever-sharp Charles Kenny and some others were active on twitter discussing some of the ideas emerging from the meeting (and in other spaces where post 2015 indicators are being discussed). Charles was questioning the number of proposed goals that not only identify the specific indicator that will hold for all countries but will also note a wildly perfect set of goals--infant mortality to be reduce to zero, for instance.

My immediate response to this is to scratch my head and wonder what all the discussion of country ownership in development really means. Surely countries should be choosing the specific goals they care about themselves. This meas choosing the area and field they want to see particular impact in and choosing the goal in that field.

I understand the international community wants to have international and global goals, but the goals will be met by actions in individual countries and it is vital that these countries are the ones:

• identifying the problems
• accepting baseline measures of the problems
• defining goals
• counting the cost (direct and opportunity) of setting these goals
• looking for solutions
• owning the process of change
• ensuring feedback and learning throughout the process of change

I would propose that the high level meetings focused on post 2015 indicators develop menus of potential goals, with baseline data for as many countries as possible in all of these menu areas. Then let countries choose the areas they want to focus on and what they see as do-able targets, and let the countries drive the discussion about how they will get there.

If ownership matters in development and change, let's allow for ownership through choice.

Economic Governance: Using unemployment rates as an indicator

This morning I suggested that we could use comparative GDP growth stats to shed light on economic governance (how governments are exercising the authority citizens give them to support the goal of economic growth). I noted that GDP growth should be one of a number of indicators of economic governance as citizens are not only concerned about growth.

Unemployment is another potential indicator of economic governance. It reflects on whether government is using its authority to ensure jobs are provided for those needing them. Note I am not suggesting how government uses its authority to do this, just that it has authority and unemployment rates show if this is being used (could be in providing jobs through fiscal expansion or changing regulations to promote small business, etc. etc.).

I used a similar method as the one discussed in the last post to compare unemployment rates in countries. First, I looked at whether unemployment rates were higher between 09 and 11 than they had been in the prior decade (as a measure of whether unemployment had crept above levels considered acceptable). My own country South Africa shows lower unemployment rates (23%) in the 09 to 11 period than the higher rate (27%) between 00 and 09. Second, I looked at whether the unemployment rate was higher or lower than that in comparable income countries. When compared with forty countries around South Africa's income level, the nation turns out to have higher than average rates (about 11% higher).

As with the analysis this morning, one end up with four categories:

• Countries in green (with scores of 3) have unemployment rates in the 2009-2011 period that were below the ten year average before. They also have unemployment rates lower than those in comparable income countries. The analysis does not suggest that this is because of 'good' governance but governments in these countries are bound to face less pressure than others. A critical outcome citizens often look for (jobs) is in good shape, compared with the past and with other countries, indicating that government is using its authority appropriately. Denmark and Norway are here right now.
• Countries in orange (with scores of 2) have unemployment rates that are lower than they were in the prior decade but higher than they are in comparable countries. Governments in these countries face the challenge of exercising authority in a way that enhances the country's economic competitiveness. They probably have the political room to do this given that unemployment rates are down from the past--more people are in jobs than before in relative terms.
• Countries in yellow (with scores of 1) have unemployment rates that are higher than they were in the prior decade but lower than they are in comparable countries. Governments in these countries have the challenge of dealing with a higher percentage of people out of work who will likely perceive that government is not using its authority to create jobs.
• Countries in red (with scores of 0) have higher unemployment rates than they had in the past and that comparable countries are experiencing. Governments in these countries are likely under a great deal of pressure to show that they can use the authority citizens have given them to deal with this problem. It does not suggest 'bad' governance as the causes of unemployment are not always directly attributed to the actions of government. This situation does create a severe economic governance challenge, however, that is common in many countries right now. The USA, Portugal, Ireland and Georgia are in this territory right now.

We should always question unemployment data like this, and there are some countries I am unsure of when I look at the data, but this kind of comparative analysis is useful--I think--in revealing what kind of governance challenges different countries are likely facing. The differences in colors should show how much variation there is in challenges. These varied challenges should be the basis of varied reforms.