DO COUNTRIES BOX AT, ABOVE OR BELOW THEIR WEIGHT? Contextually-fitted outcome measures to drive thinking about governance

OUTCOMES ARE USEFUL AS GUIDES FOR GOOD GOVERNANCE

BUT ONLY IF WE CONTROL FOR CONTEXT, BEING REALISTIC ABOUT EXPECTATIONS AND DRAWING LESSONS FROM RELEVANT COMPARATORS

Outcomes as a doorway to better governance indicators

Yesterday I did a post on creating outcome measures that could be useful in facilitating focused governance indicators. I focused on child health and suggested that we look at the survival rates of children to see where countries are experiencing good or bad governance. I took some steps:

First, I argued that we need a theoretical frame for the indicator that ties principals to agents in the production of welfare. Hence the statement,

Citizens grant public organizations the authority to ensure optimal provision of child health care, in whichever way appropriate, reflected in the cost efficient production of the highest possible survival rate of all children under five. Governance systems are ‘good’ when they ensure relatively high survival rates at relatively low average cost.

Second, I suggested that we construct a measure that reflects effectiveness (what level of service is produced, for example) as well as equity (how many citizens have access to the service) and cost (how much do citizens have to pay). I argued that the last consideration is particularly important because efficient production of one good allows citizens resources to consume other goods (further improving welfare).  Inefficiencies in one organizational field will impact effectiveness in others.

Following these steps, I created a two-dimensional measure that allows one to see the zones in which countries are performing (see figure 1 below). There is a (potentially) good governance zone, in the bottom-right hand corner of the figure (see below). Here countries have child survival rates above the global average with costs below the global average. There is also a (potentially) poor governance zone, in the top left corner, where countries are producing survival rates below the global average but spending more than the global average. There are two additional zones that suggest mixed performance (high survival rates at high costs or low survival rates at low costs).

Figure 1 from yesterday: Different countries, different outcomes

 

This is not a governance indicator. It is a two-dimensional measure of outcomes that starts to raise questions about governance. We might wonder if countries in the bottom right zone exhibit specific management or institutional or organizational mechanisms that others do not have, for instance. Perhaps country performance can help us say something about governance.

I asked what more information we needed before saying this, however. Today I will address one set of such issues. Put simply, we need to consider other factors that might explain outcomes. Contextual factors may dominate governance in explaining how well countries perform. We must be careful not to move to a discussion of 'governance' making a difference when we don't actually know if it does.

Many governance indicators have not controlled for contextual factors, and this undermines their construct validity. I discussed this a few weeks ago and related to studies by Melissa Thomas and Kurtz and Schranck. These authors argue that many governance indicators are really just different measures of economic development. World Governance Indicators, in particular, have extremely high correlations with country GDP per capita. We do not know if the WGI's are just capturing a country's level of development or other factors driving social, political and economic development. Consider the graph below, which shows that WGI scores are systematically higher in higher income country groups.

 

Controlling outcome measures for context: What results should we expect?

Some contextual factors impact outcomes in a reliable fashion and muddy the study of governance. We know, for example, that income levels have a significant impact on mortality rate statistics (Pritchett and Summers 1996; Filmer and Pritchett 1999), and that even income scores from 25 years ago can explain inter-country differences in under-five mortality rates.

The fact that Pakistan has a lower survival rate for under five year olds than Singapore may thus be more about its level of development than its governance quality. Indeed, only 3 of the 64 countries in the bottom right ‘good governance’ quadrant of yesterday's figure are low income countries (Vietnam, Uzbekistan and the Kyrgyz Republic). Fifteen out of 24 countries in the top left (inferior governance) quadrant are low income, with the highest income country in this quadrant being from the upper middle income country group (South Africa).

Higher income countries are also disadvantaged by development status, given that health care costs per capita are much higher in these countries. Higher income countries account for 32 out of the 57 countries in the above figure's top right quadrant (better than average survival rates at higher than average costs). This could imply higher costs (and hence lower welfare) in these countries, or a health care product that ensures significantly more than survival for under-fives (perhaps because more developed citizenries demand more than just survival and also because health care cost indicators used in relation to child health include a cross subsidization of health costs for the aged).

These kinds of dependable contextual variations are important to consider in both assessing governance through outcomes and in identifying where different countries should look for lessons on how to improve. Like must be compared with like as an approach to control for context in governance indicators.

Roger, Jerrett and I took a simple approach to doing this, starting first by observing which variables do have a dependable influence on the outcome in question. As discussed, in the case of child health, this variable is the nation’s income level (Filmer and Pritchett 1999). Figure 2 below shows just how much variation there has been in mean scores for survival rates over the years, across four ‘income leagues’—lower, lower-middle, upper-middle and higher (based on United Nations  classifications).

Fig. 2. Average survival rates across ‘income leagues’, 1960-2005

 

The figure shows that average under five survival rates in 2005  among lower income countries were lower than the survival rate average in higher-income countries in 1960. Figure 3 shows that the deviation across income leagues is also different, being higher in the lower income group. Higher and upper-middle income groups have seen greater convergence around high under five survival rates.

Fig. 3. Standard deviation in survival rates across ‘income leagues’, 1960-2005

 

Given such data, we should not expect that a lower income country will score a survival rate as high as a higher income country, even if the lower income country has the best form-basedgovernance in the world. Wealthier countries will also likely have higher per capita health costs (even as a proportion of GDP) regardless of whether they govern their systems well or not. 

Given these observations, we create a two-dimensional governance outcome measure that controls for income leagues present across the world, using the United Nations Classification as a guide.

The underlying argument is simply that governance in the field of child health is best assessed through outcomes when comparing lower income countries with other lower income countries, and so forth. It is still comparative, but the comparison is now within income groups and not across all countries. So we ask how far each country’s survival rate and cost scores are from the averages in their league, measured in standard deviations to allow comparison across all countries (applying an approach similar to Anderson and Morrissey 2006 and Lawrence 2006).

Z-scores are calculated for all of the countries, using the score for each country (of the survival rate and cost), and the mean and standard deviation (for both measures) for the relevant country set (low income, lower middle income, upper middle income and higher income (OECD and non-OECD)).  

Figure 4 shows the results, with the USA survival rate ending up 0.13 standard deviations above its high income country group survival rate mean, and 2.64 standard deviations above the high income country group cost average. South Africa stands -2.65 standard deviations below its comparator group (high middle income) in regards to survival rates, and is also 1.12 standard deviations above average mean costs in this group.further research into why outcomes vary between countries, and will help to focus research on real governance and government impacts.  

Fig 4. How outcomes look when controlling for expectations

 

This graph shows where governance systems ensure the provision of child health quality at, above or below their income level; much as one might asks whether a boxer boxes at, above or below his weight level.

As in Figure 1, the lower right-hand quadrant shows those countries that ‘box above’ their income levels by producing higher survival rates than the average of their income group (at zero, the vertical line) at lower costs than the average of their income group (the horizontal line). One should note that some countries change their position given the contextual control:

• Pakistan moved from the lower left quadrant in Figure 1 to the lower right quadrant in Figure 4, for example (because its survival rate was lower than the overall world average but higher than its comparator lower income group average). Pakistan’s performance is below average in comparison to the entire world, but above average when compared with countries in its income group. Much like a middle-weight boxer may routinely lose against a good heavyweight but be the best of the middleweights.
• The USA “boxes at its income level” in terms of the survival rate produced (0.13 standard deviations above the average of higher-income countries) but it “boxes below its income level” in terms of cost of production, with costs over 2 standard deviations higher than the high-income country average. The USA is an economic heavyweight boxing at the middleweight level because of high health care costs. Using the language of governance, we conclude that authority in the US child health field is not being exercised in a manner that maximizes welfare (given the high costs of producing services).
• The situation is more dire in South Africa, where weak governance in this sector is reflected in weak service delivery (lower than average upper middle income group survival rates) and high costs (higher than average upper middle income group cost per capita). South Africa may be an economic middleweight, but its child health care field struggles to compete as a welterweight.

The approach to measuring governance in Figure 4 helps one to identify the top contenders or better governed countries (in this children’s health field) in all income groups:

• 16 low income countries fall into the bottom right quadrant: Pakistan, Comoros, Mauritania, Bangladesh, Eritrea, Madagascar, Papua New Guinea, Lao PDR, Uzbekistan, Kenya, Yemen, Tajikistan, Nepal, Tanzania, Vietnam, and The Gambia.
• 24 lower middle income countries are in this quadrant: Indonesia, Philippines, Thailand, Algeria, Vanuatu, Syrian Arab Republic, Sri Lanka, Azerbaijan, Peru, Cape Verde, Mongolia, Samoa, China, Tonga, Ecuador, Morocco, Armenia, Tunisia, Colombia, Guatemala, Albania, Dominican Republic, Egypt, and Honduras.
• 15 upper middle income countries are also in the good governance area: Libya, Malaysia, Fiji, Mauritius, Venezuela, Romania, Russian Federation, St. Kitts and Nevis, Chile, St. Lucia, Lithuania, Latvia, St. Vincent and the Grenadines, Poland, and the Seychelles.
• And 10 high income countries box above their weight level: Brunei Darussalam, Kuwait, United Arab Emirates, Singapore, Bahrain, Estonia, Cyprus, Korea, Slovak Republic, and the Czech Republic..

It is interesting to note that while we have not controlled for the income bias completely, the above lists show significant variation in income levels amongst the strong performers in each group.

• Six of the fifteen low-income countries listed have per capita incomes below the average of low-income countries. (Using GNP per capita calculated according to the Atlas Method for 2005, available in the World Development Indicator database; Eritrea, Madagascar, Tajikistan, Nepal, Tanzania, and The Gambia  have income levels below the low-income group average.)
• Five of the top ten lower-middle income countries have per capita incomes below the average for their group. (This includes three of the top six performers; Indonesia, the Philippines and Sri Lanka.)
• Four of the top seven in the upper-middle income group also have per capita incomes below their group average. (Fiji, Venezuela, Romania and Russia).
• All ten high income countries boxing above their weight level had per capita incomes below their group averages. (Bahrain, Brunei Darussalam, Cyprus, Czech Republic, Estonia, Korea, Kuwait, Singapore, Slovak Republic and the United Arab Emirates.)

A number of factors emerge as intuitive answers to questions about why different countries fit into these different ‘governance quadrants’. Many of these are not about the processes of governance but rather about the governance context:

• It seems, for example, that country size might matter in influencing governance outcomes (Singapore has better survival rates at lower cost than the United States, a high-income comparator). Is it easier to govern a small health sector cost effectively than a large one?
• Newly-rich countries also seem to do better on this indicator than their other high-income comparators, producing high survival rates at much lower costs. Do newly rich countries have a leapfrogging benefit, whereby they build capacities and relational mechanisms on the basis of the latest technology and lessons from older wealthier countries, who are locked into their more outdated systems?
• Countries with higher Gini coefficients, relative to their income league, seem to perform differently, perhaps reflecting the complexities of governing unequal countries. The USA has the highest Gini for high income countries, and a much higher cost of producing health care; countries like South Africa might argue that their inequality puts them in a higher income league than appropriate (upper-middle) which results in an unfairly harsh benchmarking.
• Higher-income countries seem to do better when there is a greater direct public sector expenditure outlay, which may speak to the complexities of governing market-driven systems. The high cost of a private sector-led US system stands in contrast to other lower cost government led systems in the OECD, for example, suggesting different costs drivers and abilities to contain cost in these systems.

These and other second-stage observations provide important avenues for further research into why outcomes vary between countries, and will help to focus research on real governance and government impacts.  I will explore these more in coming weeks but wonder what you think of the approach in general?

 

Comments

[David Jacobstein]

I like this approach overall, it's very helpful to focus on the standard deviations rather than the set distance. To reduce the importance of the specific benchmarks, I wonder if you would consider dividing up the total countries into more groupings (say, 3/4/5 categories of income, or 3/5/10) and creating three maps, and then averaging those out for final positions = average standard deviation from cohort as variously defined. This would further clarify the relative positions and reduce the weighting of the income categories.

I look forward to further posts getting into how one might effectively use this more accurate comparative data to make statements about specific patterns of sector governance.

[Natalia Adler]

Hi Matt,

Somehow I always find myself one or two (or more!) steps behind your entries… I like the suggestion about the linkages between governance and field-specific outcomes (and the proposed approach) – but wonder if there’s still room to raise questions about the purpose of this (or any) kind of measuring exercise.

What will be the implications of developing indicators for governance that actually satisfy the construct validity criteria mentioned earlier? More aid for countries faring better?... In your earlier blogs, you criticized some of these indexes that ranked countries according to different indicators... Were you questioning them because ‘the indicators didn’t correlate with the thing being measured?’ Or could it be an indication that perhaps this kind of ranking doesn’t make sense to begin with… To clarify: I’m not against measuring things. But worry that perhaps we might be too boxed in this results-based framework (even if it’s measuring the rights things with right indicators)…

The pressure for results among donors is growing – and that’s very real. In my earlier comment, I was hoping to find ways to curb this appetite for results by strengthening the linkages to outcomes – and not short-term results that often relate to form rather than function. However, now I wonder whether we shouldn’t be questioning this entire framework in the first place…

Indicators influence the nature of reforms/services/activities in development work. Choosing the right indicators (especially outcome-oriented ones), can help direct services towards concrete results. But is that really a good "influence?" Doesn't this framework end up limiting the use of innovations and experimentation for problem-solving in public service? Don't we run the risk of departing from the focus identifying/addressing bottlenecks in the service delivery chain?

[matt andrews]

I did a comment that somehow was lost. Essentially, it noted that my personal reason for doing this has two dimensions. First, I am trying to provide a positive critique of the governance indicators we already have, which lack validity and are responsible--in my opinion--for driving an error-laden governance reform agenda. Second, I believe governance matters and would like to see if there is an approach to thinking about and measuring governance that is more valid, reliable and relevant. I am less interested in how my work drives what donors do or do not do. While I recognize this matters I have also found myself in too many situations where donor agencies are pressing for flashy measures, indicators, change theories etc. to do use in advocating some or other set of activities. I am not sure that the incentives of donors, their short time horizons and the like are conducive to serious work in this area. In response to David, I agree with the ideas and will be moving in this direction in coming weeks.