MattAndrews

One thing to note about contexts: There are no clean canvases...and what's there matters

Following my last blog entry, let me suggest one thing we should always remember about the importance of context in development and governance, especially when we are contemplating institutional reform. The idea is implicit in my forthcoming book on the topic.

There is no such thing as a blank canvas, and what's there matters and may not be well identified, appreciated or understood by outsiders.

 Let me explain what I mean, and why it is important.

Most serious new institutional theorists (including Douglass North) argue that all social groups have rules of the game; including small communuities, sports teams, scout groups, ministries of education, tax collection offices, and classrooms of students.  These rules differ in size and shape across contexts and through time, but they are always there, as incumbents that have emerged to direct behavior in the context. They influence how agents think and make decisions, impact patterns of resource allocation, and shape the kinds of capacities that exist and do not exist.

In a way, they are like the paint marks an artist has already put on her canvas. Sometimes they make sense to external observers, and appear ordered and purposeful (maybe like the flowers below), and other times they make a little less sense to observers, looking disordered and messy (the Pollock painting under the pretty flowers may look like this to some, certainly I have a hard time working it out).

 

It turns out the functionality of different incumbent institutional forms may be as subjective as art, and as difficult to make sense of and appreciate for outsiders who have not learned about such mechanisms. Rules of the game in a poor country with no transparency may resemble messy and 'in need of reform' painting marks to you and me, for instance, but they could merge to provide some kind of chaotic order that works for important groups of people in the country (often even for the poor).

More importantly, the people in the country may not see any need to change incumbent rules of the game, or may lack capacities to change given the entrenched dominance of incumbents. So any new reforms you introduce become like added paint on the Pollock canvas; which has to fit in with what's already there or risk adding to the chaos; or being overwhelmed by the incumbent patterns (as seems to happen to so many reforms).

This basic observation is an important starting point for thinking about context and governance and institutional reform: Institutions always exist, in all contexts, in many different forms that have some functionality for some people in the context, and that need to be considered when thinking about reforms, but probably are never properly considered by outsiders.

It is an obvious observation, you may say, but is it always respected when development agencies craft their reform programs for developing countries, or when we as outside specialists conceive our indicators of what constitutes a good, or better, or best practice?

It seems to me that many reforms are crafted with only a passing glance at the context; which is followed by a much more serious presentation of externally devised, 'better' and 'right' rules. It seems, further, that the act of creating generic indicators of the 'right rules' turns a blind eye to the fact that different places already have rules in place and that relevant governance solutions are more than just a little likely to look different across different places (if only because the opening slates are so different).

In both instances it seems that we in the development comunity think it appropriate to craft reform solutions and think about 'good governance' ideas as if there are a bunch of clean or clean-able slates on which to work our masterpieces. The reality is that you cannot 'buy' a clean slate on which to paint new governance solutions (like those shown below) and that we should stop blaming countries when reforms fail because the reform slates were filled with other incumbents that we either did not see or did not take seriously.

We need to remember this. And stop pretending that the world is our oyster. And start developing the tools, capacities, and sensibilities to better understand how to appreciate 'what's already there'.

What do you think these tools, capacities, and sensibilities should or could look like?

 

Context, history, governance and development

I have been away from the blog awhile, finishing a couple of book manuscripts. It is time to pick up, however, and I'd like to do so by asking some simple questions about context, governance and development.

Last year I wrote a bit about institutional reforms in Malawi . I looked at the creation of Malawi's Anti Corruption Bureau (ACB) in 1994. The reform turned out to be a problematic failure; not once, but on multiple attempts. The experiences raise some important questions about context, history, governance and development.

As a refresher on Malawi's story:

• The ACB was created in the mid-1990s at a time when Malawi had hoped of being a new, vibrant democracy and economy. It was modeled after the Hong Kong anti corruption commission which had also influenced the Botswana version.
• The ACB initially opened thousands of investigations into corruption. Very few investigations went anywhwere, however, and corruption got progressively worse in the country.
• Measures and analytical work done by organizations likeTransparency International and Global Integrity panned the work done by the ACB and noted that the action behind anti corruption laws was limited.
• Donors depressed with the results attempted to launch a new version of the commission in the late 2000s, with new legislation to boot.
• A 2010 NORAD evaluation noted that the anticorruption reforms (past and present) were “Affected by a political context where politics and institutions are highly personalized and a political culture which tends to discourage impartiality.” The report argues that such contextual factors undermined the impact of impartial, formal anticorruption laws and law-enforcers.
• This resonates with comments made over a decade before, which suggested that 1990s reforms should address contextual “reasons why the formal rules governing behaviour in the public sector have broken down, and why informality—of which corruption is one manifestation—has taken over.”
• It appears that these contextual realities were not addressed, however, either in 1994 or in 2009. In both instances reforms focused instead on introducing new and improved formal rules and rule-makers. The result is history repeating itself, with contextual limits emerging again and again to thwart change.

Going beyond Malawi, to generic questions:

The institutional reform experience in Malawi is not peculiar. It is an example of the many reform  failures that seem to result from the impact of stubborn contextual constraints that are not
effectively considered in most reform designs.

•  Isabelle Werenfels points to such in arguing that Algerian privatization was undermined by, “a country-specific complex interplay of … forces [that] decisively affect the path of any institutional transformation.”
• Andrew Wilder writes similarly about governance reform in Pakistan, where donors consistently push for change “without investing sufficiently in understanding the social, cultural and political contexts within which the
  civil service functions.”

Werenfels’ and Wilder’s comments—and the story from Malawi—raise two important questions:

• What should external actors like donors know about the context in which institutional reform takes place?
• Why do these external actors under-invest in understanding such things?

What do you think?

PS. I'm happy to give my sources to anyone who asks but choose not to include them here because they just make the blog post too bulky.

 

What do externally influenced governance reforms achieve?

I have not written for a while. So, this post is me getting back to business. Those who follow the blog will know that a while back I discussed some research showing that only half the countries pursuing externally supported governance reforms see better government effectiveness scores over ten years. I thought I would re-visit this commentary today, as I am working on why this is the case and what could be done to address it. So, let me backtrack a bit.

The 50:50 problem

The argument rests on some work I did last year, where I started looking at whether countries with World Bank sponsored reforms in the Public Administration, Law and Justice Sector (PAL&J reforms, which typically target institutional improvements) had better World Governance Indicator scores on Government Effectiveness.

The WGI scores have obvious problems (and I myself am a critic of using them too much), but they are commonly used to reflect on the quality of governments in developing countries and are the ‘internal’ measures development organizations like the World Bank used to think about the issue in the 2000s. One would therefore expect World Bank reforms to impact these WGI scores directly, given that such interventions and WGIs are both products of the same external development community. In this way, they are the ‘test’around which reforms may have been shaped and, using a term from the education literature, one would expect external organizations to ‘teach to the test’.

Interestingly, WGI data suggests that reform results are mixed.Some of the 145 countries I looked at--all of which adopted institutional reforms with the World Bank--have seen improved WGI government effectiveness scores in the past decade. Many have seen scores decline, even though they adopted reforms through costly projects. Afghanistan and Rwanda recorded the most improvement between 1998 and 2008 at +0.96 (with Afghanistan going from-2.27 to -1.31) while the Maldives dropped the most, at -1.39 (from 0.95 to-0.35). While not matching the Maldives in the extent of its drop, 72 countries had declining scores in the period. An almost equivalent number, 73, saw improved scores.

I find the story startling: Half of 145 countries that have had donor sponsored government reforms in place saw declines in donor devised indicators of government effectiveness over a recent ten year period.Given such evidence, organizations like the World Bank cannot be too confident in their reform agendas. Imagine a new president asking her World Bank country representative, “What are the odds that proposed reforms will help us get better government effectiveness scores in a decade?” Based on this kind of data, the answer should surely be, “No more than 50-50 Madame President …as good as getting heads on a coin toss.”

These are also the odds of reforms yielding improvements in a smaller 40 country sample I worked with at a higher level of detail. 21 of the 40 countries went backwards on WGI government effectiveness scores between 1998 and 2008. A smaller set of 19 countries saw improved indicators. Cape Verde dropped from 0.35 to 0.05 even though the country had pursued reforms in 28 World Bank projects that started in 1992 and cost $122 million. Senegal’s scores dropped by a quarter of a point even though it had undertaken 75 projects with PAL&J content, which cost over $1 billion in total. Senegal’s reform activities were consistently pursued as well, with 16 projects in the 1980s, 26 in the 1990s and 21 between 2000 and 2008. $526 million was committed to projects between 1998 and 2008 alone, the period in which government effectiveness scores fell. Nicaragua engaged in World Bank sponsored institutional reforms costing $355 million between 1998 and 2008, building on about $290 million worth of prior engagements, but saw government effectiveness scores dip by more than half a point.

It’s worse than 50:50 on the Quality of Governance measures

As noted, I expect some to question this source of data, noting that the WGI measures lack validity or reliability. While standing up as a critic, it is important to note that many many people use these indicators, so one should not dismiss them as a legitimate source of data. Even academics use the data to capture hard-to-measure concepts like government effectiveness, leading to thousands of Google Scholar citations to the ‘Governance Matters’ paper series.

Even given the multiple WGI citations, it is reasonable to ask whether other measures paint a different picture of government quality or effectiveness between 1998 and 2008. One alternative is the Quality of Governance (QoG) measure produced as part of the International Country Risk Guide (ICRG). It is generated by the Political Risk Service Group (PRS), a rating agency that has provided data for developed and developing countries over the last few decades. Its indicators are used by international firms, donors and international financial institutions. The QoG measure is a subjective index created by summing three sets of data on corruption in government, bureaucratic quality and the rule of law. These three dimensions are actively targeted by external institutional reforms. When summed, they are presented as an indicator ranging from 0 to 1. This indicator is widely used in academic research, with over 500 references on Google Scholar between 2009 and 2011 alone.

I looked at QoG data for 107 countries to illustrate the story about governance quality and ‘reform’ between 1998 and 2008. All 107 countries had World Bank sponsored institutional reforms in place during this period. Most also had reforms predating 1998 and most are expected to have also worked with other agencies on reforms to improve the quality or effectiveness of government.

Even with such reforms, however, over 70 percent of these countries saw QoG scores decline between 1998 and 2008. This is a larger number of decliners than was evident when considering the WGI government effectiveness data. This evidence suggests that the odds of a country doing externally influenced reform and seeing improved quality of governance was less than 50-50 for the period. Given this, most countries doing reform should expect that government quality will still decline over time, regardless of reform.

This finding is reinforced when looking at the smaller sample of 40 countries. QoG data over the entire period are only available for 29 of these countries. The QoG scores declined for 19 of these, stayed the same for one, and improved for the other 9. The biggest gain was in Serbia, which increased its score from 0.34 to 0.47. At the other extreme, Argentina fell from 0.69 to 0.52. Serbia had adopted 22 World Bank projects with institutional reform content in the 1998-2008 period, costing $217 million. Argentina engaged in 46 such projects in the period, costing over $4 billion.

Looking at per capita income as well

Mixed results like this are evident when looking at other measures of government effectiveness as well. Economic growth could be considered such a measure, not because governments drive all growth but because this is a bottom-line indicator that many believe citizens care about. Ronald Reagan’s 1980 election campaign famously noted that an administration’s success was reflected in whether people felt better off because of the government’s presence. Bill Clinton’s 1992 election slogan reinforced the idea, proclaiming ‘It’s the economy, stupid’ when describing what citizens expect of their governments.

Given such, it is important to note that most countries’ economies grew over the 1998-2008 period, and most of the world’s citizens could access more value in 2008 than they could in 1998. However, many countries grew at rates slower than their comparators, suggesting variation in factors like the quality of institutions. GDP per capita, measured in constant terms, increased by about 20 percent in Bolivia between 1998 and 2008, for instance, less than the mean growth rate for lower middle income countries. Latin American lower middle income neighbors like Brazil grew by significantly larger amounts. One could also look at Malawi, where citizens saw personal incomes grow at about 10 percent over the entire period, way below the average of low income countries. Incomes increased by about 50 percent in comparable income countries like Laos and Mozambique, the latter sharing Malawi’s southern border. Countries like Bolivia and Malawi were performing less effectively than comparators. Using a sports metaphor, they were boxing below their weight—looking like a middleweight but punching like a flyweight.

It seems that 60% of 132 developing and transitional economies underperformed in this way. They grew at rates lower than their comparators showed was possible in the period. One cannot attribute this directly or completely to low quality governance or ineffective government, but the poor performance does reflect partly on such—at least subjectively. As with many other countries, Malawi’s government did not ensure the same growth as neighboring Mozambique between 1998 and 2008, a lackluster performance that many would surely call ‘ineffective’. This weak record was achieved even though the country had embarked on 23 World Bank projects with PAL&J content amounting to about $550 million in the period, building on over 40 operations prior to 1998.

Twenty five of the 40 randomly selected countries identified in chapter one had economic growth rates less than those evident in comparable countries. This is a similar proportion of underperformers as one sees in the full set of 132 countries. It suggests that government is comparatively ineffective in about 60 percent of the countries where externally influenced reform was meant to make government more effective.

Governance, outcomes and context: What we learn from boxing

My last post continued thinking about outcomes and governance. I am trying to see if we can tie governance indicators to outcomes, such that we have better construct validity of these indicators.

Basic ideas to date, and questions

I argued that this needs to be done in fields where we can focus on outcomes. Child health is one such field. I argued that we need a measure that captures outcomes citizens care about. Under 5 survival rates is my example. I argued that a second dimension of this outcome, costs, is also important. It is not a great outcome if all children are kept alive but it takes all your resources.

Putting all this together, the idea is that good governance in the child health arena will be reflected in high survival rates at low costs. Such outcome indicates that governments have taken the authority given to them by citizens to set up mechanisms and structures that produce welfare value to those citizens. These mechanisms will differ between places and we are not even getting into the detail of what they are...just whether they have delivered.

I made one additional adjustment. I said that we need to control for big contextual drivers that affect the expectations citizens might have of what outcomes should be--and the realities of delivering such. One could call these relevant indicators. In the health area, we know that levels of per capita income are such a variable. So, we need to compare wealthy countries with wealthy countries, poor countries with poor countries, and so forth.

This is where I got to in the last blog. I showed that when we do this we can see how well countries like the USA are performing relative to other wealthy countries (not well) and how well countries like Pakistan are performing relative to other poor countries (well). So, we have a contextually relevant measure of outcomes that tell us something about governance.

But does it? This is the question Alan Hudson asked and which is pregnant in my mind and I am sure many others. Alan raises two dimensions of the question. First, is the measure fully contextualized? There are contextual factors that matter--other than income--that we have not controlled for. Second, is the indicator capturing variations in 'governance'? If we are not fully controlling for context, and country outcomes vary because of contextual factors, aren't we jumping the gun in saying this is about governance?

Great questions. I think they raise empirical questions about measurement and theoretical and applied questions about what governance is. I don't think I will answer them completely here, but let me start. And let me start by going back to sports...

Boxing and governance: can we learn something

I have likened the indicator of income adjusted, costed survival rates as measures of the ability countries have to box at their income weight. They ask a lightweight boxer what his record is in the lightweight division. This is a good indicator of how well that boxer performs against appropriate opposition.

Such indicator is really different to conventional governance indicators. These indicators identify a set of characteristics in the overall best performers and then say these are the basis of good governance. Think of Doing Business indicators that emphasize deregulated industry, or World Governance Indicators that give countries credit for having things like meritoractic civil service systems and decentralized governments. To me, these indicators are akin to having a measure of 'good boxing' that looks at the weight, height and arm reach of a fighter.

This kind of indicator might suggest that good boxers are all over 200 pounds, 6 foot 2 and with an arm reach of 200 cm.  Any boxer that is less than 150 pounds, 5 foot 8 or with a reach less than 180 cm must be bad. Certainly the latter boxer would not succeed against the former, in what the industry calls a 'mis-match'. Like the one shown below.

It is not useful to say that taller, heavier men with longer arm reach will win bouts against significantly shorter, lighter men with less-long arms. I've had some of those scraps on the rugby field and I know they don't work out in my favor! The indicator does not capture how good the fighter is, just how different their basic attributes are--given some intervention in their past (like their birth).

Professional boxing knows this. So it controls for the most defining attribute--weight level--to try and give a better perspective on who is good and who is bad at boxing. Every boxer falls into a weight division and must compete with others in that division. Light middleweights are those weighing between 147 pounds and 154 pounds, for example. They tend to be between 5 foot 6 and 5 foot 11, and their arm reach is between 160 and 190 cm on average.

We know that a fighter is 'good' or 'bad' because of the record they have in their weight division. This is simply the number of wins compared with the number of losses. Floyd Mayweather is 5 foot 8, weighed in the region of 154 pounds in 2007 and 2008, and has a reach of only 183 cm. But he has a record of 42-0. Forty two wins, and no losses (according to wikipedia...I am no boxing genius). Tony Pep, a 6 foot 1.5 inch boxer from Canada, has a record of 43-10-1 (43 wins, 10 losses and 1 draw). One of his losses came against Mayweather.

Mayweather is a good boxer. Pep is less good--but not by any means bad.

Now, if we get to the heavyweights we do the same analysis. Wladimir Klitschko is 6 foot 6, weighs about 210 pounds, and reaches 206 cm with his arm. Wow. He has a fight record of 56 wins and 3 losses and is currently considered world champion. A good boxer.  Axel Schulz was also a heavyweight. Standing at 6 foot 3, he also weighed over 200 pounds and had an arm length of 193 cm. His record was 26 wins, 5 losses and a draw. Not a bad boxer, but not as good as Klitschko.

One wonders if Schulz would have beaten Mayweather. I am not sure if boxing aficionados will jump on me in saying I expect he would have. he just had more size. And in boxing size matters. But that does not mean he was a better boxer than Mayweather. Actually many believe that Mayweather is the best of all time, across all weight divisions.

All this to say that we need to compare like with like, controlling for the broadest factors that affect performance.  This is something we learn from boxing. And it works in governance indicators as well. The approach I have taken shows this.

• The United States is an income heavyweight that does less-well than the countries we can compare it to in the area of child health--its survival rates are just above average but its costs are way above average.
• Singapore is an income heavyweight that does really well when compared against its comparators (its survival rates are higher than average and its costs are lower than average).

See both in the two dimensional graph below, with the horizontal axis showing how many standard deviations all these rich countries are from the average rich country survival rate; the vertical axis shows how far the countries are from the average cost of producing this outcome.

Now, one may say that this does not account for all of the contextual influences on under 5 mortality rates and thus we cannot say "the USA child health sector is governed less-well than that in Singapore."  One would certainly be right in some respects.

Some contextual variables emerge as important.

• First, you will see that bigger countries tend to be located in the upper right quadrant with the USA (relatively high survival rates but relatively high costs as well). The lower right quadrant (with relatively high survival rates at relatively low costs) is in contrast dominated by smaller countries.  When I look at a single measure of outcome performance (as per the last blog), the top 25 child health countries average out with populations of 12 million. The bottom 25 have 31 million. The difference in averages is statistically significant (at 5%) suggesting that bigger countries have a harder time in this area.
• Second, the better performing countries tend to have higher degrees of population density. Statistical analysis suggests that ten times more people are located in equivalent land areas in the better performing countries than the poorer performing countries (again, a statistically significant difference in means).  So it seems that child health outcomes are easier to provide at lower costs in rich countries with smaller, more densely populated populations. Island states are the place to be!
• Third, you can see that the top right quadrant is dominated by the old guard of the OECD. This shows up in the stats. The average number of years' membership in OECD is 41 for the 25 worse performing countries. It is 9 for the 25 better performing countries. It is a statistically significant difference. I will look to interpret it in future posts but some possibilities are: Rich countries that are more newly rich have better child health performance than others; rich countries with newer child health service delivery mechanisms perform better than others; rich countries with younger populations perform better than others.
• Fourth, if we look at the data we find that countries that perform better than others have lower numbers of doctors. The top 25 performing rich countries have about 2.5 doctors per 1,000 while the bottom 25 performers have 3.2 doctors per 1,000. So there is something here about the size of the medical profession (that could be related to the third point on OECD membership and also to the size and spread of population in the first and second points above).
• Fifth, I looked to see if there is any variation between conditions of children in these countries, and it turns out that the better performing countries have lower levels of undernourished children (5 per 1,000 as compared with 5.3 per thousand).

Now one may say we should control for all these factors--and a bunch of others like gini coefficients, number of years kids are in school, teenage pregnancies, and a range of other factors we think or know impact on a country's performance. One may say that we only get to governance issues once these have been controlled. 

So, without taking a firm position, let me say why this may not be a great idea. I will use the boxing metaphor to help me.

• First, boxers in various divisions have something in common--they fall into a particular weight group. Beyond this, they have many things that are not in common. Some heavyweights are taller than others, for instance.
• Second, these variations matter. If we went to do a statistical analysis explaining why some boxers win the heavyweight championship and others do not, we are bound to find that height matters. Apart from Mike Tyson, there has not been a heavyweight champion in the past few decades who is shorter than 6 foot 2 (he stands at 5 foot 10, I believe).
• Third, heavyewights who are shorter than 6 foot 2 still have to compete with other heavyweights. They get no special 'control' to help explain that their losses were against taller opponents. This may have helped David Ruiz improve his record from 44 wins and 9 losses...given 2 of the losses came in decisions against Nikolay Valuev who stands at over 7 ft tall and weighs over 320 pounds (see picture). His record could have been 44 wins, 7 losses, 2 losses against the 7 ft tall guy. No. It was 44 and 9. His context was controlled for in terms of weight, and beyond that he was simply expected to compete. (In fact he did compete, with the losses being controversial...)

• Fourth, one of the reasons why boxing allows someone like Ruiz to be compared and competitive with someone like Valuev is because, when one gets into the ring, factors other then height matter. Ruiz mastered techniques that fighters like Valuev did not. Mike Tyson had aggression, speed and power that many taller fighters lacked. Some fighters have more organized management teams, cut-men in their corners, or preparation approaches. 
• Fifth, many of these other factors are variables that the fighters themselves can control. The techniques they adopt, the number of hours they put in at the gym, their selection of manager. They actually need to control and manage these factors in relation to the factors they cannot control. If a tall fighter wants to use his height as an advantage, he should choose a specific technique, for example. If a fighter has natural speed he needs to engineer a fighting technique that works towards this advantage.
• Sixth, good fighters are the ones who put all the pieces together. They build on their different natural strengths by adopting relevant techniques etc.

I would like to suggest that these sixobservations are important for our thinking about governance:

• Put together, they tell me that there are many factors that an econometric analysis will show explains which rich countries are more likely to provide good child health outcomes.
• Some of these are natural or entrenched contextual advantages  or disadvantages that countries cannot do that much about.  The United States cannot be a small island state like Singapore.
• But there are defining factors that are dynamic and reflect strategic choices countries make to do better. Countries can do something about the medical technologies they adopt, for instance, or the degree to which their children are under nourished.
• The choice or dynamic action is contingent on the natural factors. Countries with large and deconcentrated populations need to adopt different mechanisms to improve their child health performance than Singapore does.
• Governance is about whether governments use their authority to make dynamic adjustments that fit the natural contexts in which they are found. These adjustments should counter the contextual constraints and build on the contextual strengths in place. The USA does not get a pass on poor performance because it is large and dispersed. Good governance is achieved in the USA when it makes the policy, organizational, etc. decisions needed to improve its child health performance given its context. 

 In this analysis, the quality of governance is indicated--not necessarily measured--by the outcomes in evidence. They tell us if the country is performing at, above or below expectation (given its income level), which is an indication of the degree to which authority is exercised in such a way to build on natural strengths and compensate for natural constraints.

What do you think?

The USA lags Pakistan and the Comoros in its governance of child health

CONTEXTUALLY CONTROLLED OUTCOMES GIVE ONE A DIFFERENT PICTURE OF GOVERNANCE

My last post explored the idea of a two-dimensional approach to measuring governance in the child health area. A couple of comments agreed that this is appealing from a policy perspective. It shows visually where welfare is being produced and lost between the quantity and cost elements of the outcome. This is something that you won't currently find in any approach to governance (If I am wrong, please tell me...I'm always looking for new measures).

I also believe that a one-dimensional governance indicator might be calculated through the proposed approach, however, and be something for consideration. It would be a single measure that one could point to in referring to the overall governance outcomes of a field or sector. Again, it does not reflect on governance processes--just the degree to which governments exercise their authority in a way that ensures children stay alive to 5 years and that does not crowd out other outcomes because of excess expenses.

This would essentially involve combining the two elements introduced in my last blog posting to see how far an individual country departs from its income group on both survival rate and cost measures. The combined deviation-from-mean scores are shown in the figure below. The z-score for cost in each country has been subtracted from the z-score for survival rate; for the USA this results in 0.13-2.64=-2.51. The USA fits into the area to the left of the vertical line, which reflects increasingly weaker outcomes, welfare for citizens and hence governance.

   

The vertical axis in the above figure is the same as that used in the figure below (which illustrates World Governance Indicator scores). This shows that the new indicator is not biased by a country’s income group (as most governance indicators are). Actually, the focus on income-group controlled survival rates and costs leads one to identify three lower income countries as having the (relatively) best governance in the 181 country sample (Comoros, Pakistan and Bangladesh). The worst ten performers come from an assortment of income groups, including higher income (Equatorial Guinea and the USA), upper middle (South Africa and Gabon), lower middle (Kiribati, Timor Leste, and Djibouti) and lower income (Sierra Leone, Rwanda and Burkina Faso).

WGI government effectiveness scores for 181 countries, 2007

 

Readers might ask the obvious questions: is governance in the child health sector really better in Pakistan and the Comoros than it is in the United States?

The answer is yes, given the different contexts of these countries.

While Pakistan and the Comoros have lower under five survival rates than the USA, these survival rates are high relative to the income group of which they are a part, and the cost of generating these survival rates is relatively low, given their comparators. The USA has average survival rates, relative to its high income reference group, produced at very high costs relative to its peers. These high costs cause the United States to score substantially below the high income league averages on the combined u5mr/cost measure—as shown in the top figure. 

Put differently, Pakistan and the Comoros  may be lightweights but they box excellently at that weight level. The USA is a heavyweight that is struggling in its weight division...

Observers might argue that the method adopted as a basis for this indicator is biased against wealthier countries, given that these countries are all abutting the upper limit of child survival (100%) while costs have no such limit and the health care costs in wealthier countries are likely to be higher than in others. The backward tilt in the top figure may reflect this problem, with fewer wealthier countries able to score a combined high positive standard deviation and a large group of (mostly OECD) wealthier countries scoring in the negative standard deviation area.

There is variation in the high-income group of countries, however, and some countries (like Singapore) score better than their comparators in both the survival rate and cost data. Better performing countries in this high income group all have low health care costs per capita, as well, and on the basis of our work we argue that costs may actually have a practical, unofficial upper boundary. Eighty-six percent of the high income countries have costs below 10% of GDP, which we would suggest seems to emerge as a reference point to think about as some kind of comparative limit. (This is calculated on the basis of per capita health spending in 45 OECD and non-OECD countries. All of the wealthier non-OECD countries have per capita figures below 10% of GDP and 83 percent of the OECD countries are below this figure too. A total of 93 percent of the 178 countries in our data set have per capita health spending below 10% of GDP.)

The United States is the only high income country to score more than two standard deviations below its peers on the overall indicator. We suggest that it could learn from a host of other high income countries how to govern its child health arena better (especially countries like Japan and Korea, both of whom are more than one standard deviation above the peer mean). These countries appear to have better governed child health fields, where comparatively strong under-five survival rates are achieved at relatively low costs. This is not to say that the governance structures here have cultural, political or other transferability, but just that they appear to offer an example of good practice within the United State’s income league and our indicator points to lessons the USA could learn from them.

In future posts I will relate the performance of higher income countries with other higher income countries,  and so forth, and discuss how this comparison might be used to test for governance impacts on relative outcomes.

 

 

DO COUNTRIES BOX AT, ABOVE OR BELOW THEIR WEIGHT? Contextually-fitted outcome measures to drive thinking about governance

OUTCOMES ARE USEFUL AS GUIDES FOR GOOD GOVERNANCE

BUT ONLY IF WE CONTROL FOR CONTEXT, BEING REALISTIC ABOUT EXPECTATIONS AND DRAWING LESSONS FROM RELEVANT COMPARATORS

Outcomes as a doorway to better governance indicators

Yesterday I did a post on creating outcome measures that could be useful in facilitating focused governance indicators. I focused on child health and suggested that we look at the survival rates of children to see where countries are experiencing good or bad governance. I took some steps:

First, I argued that we need a theoretical frame for the indicator that ties principals to agents in the production of welfare. Hence the statement,

Citizens grant public organizations the authority to ensure optimal provision of child health care, in whichever way appropriate, reflected in the cost efficient production of the highest possible survival rate of all children under five. Governance systems are ‘good’ when they ensure relatively high survival rates at relatively low average cost.

Second, I suggested that we construct a measure that reflects effectiveness (what level of service is produced, for example) as well as equity (how many citizens have access to the service) and cost (how much do citizens have to pay). I argued that the last consideration is particularly important because efficient production of one good allows citizens resources to consume other goods (further improving welfare).  Inefficiencies in one organizational field will impact effectiveness in others.

Following these steps, I created a two-dimensional measure that allows one to see the zones in which countries are performing (see figure 1 below). There is a (potentially) good governance zone, in the bottom-right hand corner of the figure (see below). Here countries have child survival rates above the global average with costs below the global average. There is also a (potentially) poor governance zone, in the top left corner, where countries are producing survival rates below the global average but spending more than the global average. There are two additional zones that suggest mixed performance (high survival rates at high costs or low survival rates at low costs).

Figure 1 from yesterday: Different countries, different outcomes

 

This is not a governance indicator. It is a two-dimensional measure of outcomes that starts to raise questions about governance. We might wonder if countries in the bottom right zone exhibit specific management or institutional or organizational mechanisms that others do not have, for instance. Perhaps country performance can help us say something about governance.

I asked what more information we needed before saying this, however. Today I will address one set of such issues. Put simply, we need to consider other factors that might explain outcomes. Contextual factors may dominate governance in explaining how well countries perform. We must be careful not to move to a discussion of 'governance' making a difference when we don't actually know if it does.

Many governance indicators have not controlled for contextual factors, and this undermines their construct validity. I discussed this a few weeks ago and related to studies by Melissa Thomas and Kurtz and Schranck. These authors argue that many governance indicators are really just different measures of economic development. World Governance Indicators, in particular, have extremely high correlations with country GDP per capita. We do not know if the WGI's are just capturing a country's level of development or other factors driving social, political and economic development. Consider the graph below, which shows that WGI scores are systematically higher in higher income country groups.

 

Controlling outcome measures for context: What results should we expect?

Some contextual factors impact outcomes in a reliable fashion and muddy the study of governance. We know, for example, that income levels have a significant impact on mortality rate statistics (Pritchett and Summers 1996; Filmer and Pritchett 1999), and that even income scores from 25 years ago can explain inter-country differences in under-five mortality rates.

The fact that Pakistan has a lower survival rate for under five year olds than Singapore may thus be more about its level of development than its governance quality. Indeed, only 3 of the 64 countries in the bottom right ‘good governance’ quadrant of yesterday's figure are low income countries (Vietnam, Uzbekistan and the Kyrgyz Republic). Fifteen out of 24 countries in the top left (inferior governance) quadrant are low income, with the highest income country in this quadrant being from the upper middle income country group (South Africa).

Higher income countries are also disadvantaged by development status, given that health care costs per capita are much higher in these countries. Higher income countries account for 32 out of the 57 countries in the above figure's top right quadrant (better than average survival rates at higher than average costs). This could imply higher costs (and hence lower welfare) in these countries, or a health care product that ensures significantly more than survival for under-fives (perhaps because more developed citizenries demand more than just survival and also because health care cost indicators used in relation to child health include a cross subsidization of health costs for the aged).

These kinds of dependable contextual variations are important to consider in both assessing governance through outcomes and in identifying where different countries should look for lessons on how to improve. Like must be compared with like as an approach to control for context in governance indicators.

Roger, Jerrett and I took a simple approach to doing this, starting first by observing which variables do have a dependable influence on the outcome in question. As discussed, in the case of child health, this variable is the nation’s income level (Filmer and Pritchett 1999). Figure 2 below shows just how much variation there has been in mean scores for survival rates over the years, across four ‘income leagues’—lower, lower-middle, upper-middle and higher (based on United Nations  classifications).

Fig. 2. Average survival rates across ‘income leagues’, 1960-2005

 

The figure shows that average under five survival rates in 2005  among lower income countries were lower than the survival rate average in higher-income countries in 1960. Figure 3 shows that the deviation across income leagues is also different, being higher in the lower income group. Higher and upper-middle income groups have seen greater convergence around high under five survival rates.

Fig. 3. Standard deviation in survival rates across ‘income leagues’, 1960-2005

 

Given such data, we should not expect that a lower income country will score a survival rate as high as a higher income country, even if the lower income country has the best form-basedgovernance in the world. Wealthier countries will also likely have higher per capita health costs (even as a proportion of GDP) regardless of whether they govern their systems well or not. 

Given these observations, we create a two-dimensional governance outcome measure that controls for income leagues present across the world, using the United Nations Classification as a guide.

The underlying argument is simply that governance in the field of child health is best assessed through outcomes when comparing lower income countries with other lower income countries, and so forth. It is still comparative, but the comparison is now within income groups and not across all countries. So we ask how far each country’s survival rate and cost scores are from the averages in their league, measured in standard deviations to allow comparison across all countries (applying an approach similar to Anderson and Morrissey 2006 and Lawrence 2006).

Z-scores are calculated for all of the countries, using the score for each country (of the survival rate and cost), and the mean and standard deviation (for both measures) for the relevant country set (low income, lower middle income, upper middle income and higher income (OECD and non-OECD)).  

Figure 4 shows the results, with the USA survival rate ending up 0.13 standard deviations above its high income country group survival rate mean, and 2.64 standard deviations above the high income country group cost average. South Africa stands -2.65 standard deviations below its comparator group (high middle income) in regards to survival rates, and is also 1.12 standard deviations above average mean costs in this group.further research into why outcomes vary between countries, and will help to focus research on real governance and government impacts.  

Fig 4. How outcomes look when controlling for expectations

 

This graph shows where governance systems ensure the provision of child health quality at, above or below their income level; much as one might asks whether a boxer boxes at, above or below his weight level.

As in Figure 1, the lower right-hand quadrant shows those countries that ‘box above’ their income levels by producing higher survival rates than the average of their income group (at zero, the vertical line) at lower costs than the average of their income group (the horizontal line). One should note that some countries change their position given the contextual control:

• Pakistan moved from the lower left quadrant in Figure 1 to the lower right quadrant in Figure 4, for example (because its survival rate was lower than the overall world average but higher than its comparator lower income group average). Pakistan’s performance is below average in comparison to the entire world, but above average when compared with countries in its income group. Much like a middle-weight boxer may routinely lose against a good heavyweight but be the best of the middleweights.
• The USA “boxes at its income level” in terms of the survival rate produced (0.13 standard deviations above the average of higher-income countries) but it “boxes below its income level” in terms of cost of production, with costs over 2 standard deviations higher than the high-income country average. The USA is an economic heavyweight boxing at the middleweight level because of high health care costs. Using the language of governance, we conclude that authority in the US child health field is not being exercised in a manner that maximizes welfare (given the high costs of producing services).
• The situation is more dire in South Africa, where weak governance in this sector is reflected in weak service delivery (lower than average upper middle income group survival rates) and high costs (higher than average upper middle income group cost per capita). South Africa may be an economic middleweight, but its child health care field struggles to compete as a welterweight.

The approach to measuring governance in Figure 4 helps one to identify the top contenders or better governed countries (in this children’s health field) in all income groups:

• 16 low income countries fall into the bottom right quadrant: Pakistan, Comoros, Mauritania, Bangladesh, Eritrea, Madagascar, Papua New Guinea, Lao PDR, Uzbekistan, Kenya, Yemen, Tajikistan, Nepal, Tanzania, Vietnam, and The Gambia.
• 24 lower middle income countries are in this quadrant: Indonesia, Philippines, Thailand, Algeria, Vanuatu, Syrian Arab Republic, Sri Lanka, Azerbaijan, Peru, Cape Verde, Mongolia, Samoa, China, Tonga, Ecuador, Morocco, Armenia, Tunisia, Colombia, Guatemala, Albania, Dominican Republic, Egypt, and Honduras.
• 15 upper middle income countries are also in the good governance area: Libya, Malaysia, Fiji, Mauritius, Venezuela, Romania, Russian Federation, St. Kitts and Nevis, Chile, St. Lucia, Lithuania, Latvia, St. Vincent and the Grenadines, Poland, and the Seychelles.
• And 10 high income countries box above their weight level: Brunei Darussalam, Kuwait, United Arab Emirates, Singapore, Bahrain, Estonia, Cyprus, Korea, Slovak Republic, and the Czech Republic..

It is interesting to note that while we have not controlled for the income bias completely, the above lists show significant variation in income levels amongst the strong performers in each group.

• Six of the fifteen low-income countries listed have per capita incomes below the average of low-income countries. (Using GNP per capita calculated according to the Atlas Method for 2005, available in the World Development Indicator database; Eritrea, Madagascar, Tajikistan, Nepal, Tanzania, and The Gambia  have income levels below the low-income group average.)
• Five of the top ten lower-middle income countries have per capita incomes below the average for their group. (This includes three of the top six performers; Indonesia, the Philippines and Sri Lanka.)
• Four of the top seven in the upper-middle income group also have per capita incomes below their group average. (Fiji, Venezuela, Romania and Russia).
• All ten high income countries boxing above their weight level had per capita incomes below their group averages. (Bahrain, Brunei Darussalam, Cyprus, Czech Republic, Estonia, Korea, Kuwait, Singapore, Slovak Republic and the United Arab Emirates.)

A number of factors emerge as intuitive answers to questions about why different countries fit into these different ‘governance quadrants’. Many of these are not about the processes of governance but rather about the governance context:

• It seems, for example, that country size might matter in influencing governance outcomes (Singapore has better survival rates at lower cost than the United States, a high-income comparator). Is it easier to govern a small health sector cost effectively than a large one?
• Newly-rich countries also seem to do better on this indicator than their other high-income comparators, producing high survival rates at much lower costs. Do newly rich countries have a leapfrogging benefit, whereby they build capacities and relational mechanisms on the basis of the latest technology and lessons from older wealthier countries, who are locked into their more outdated systems?
• Countries with higher Gini coefficients, relative to their income league, seem to perform differently, perhaps reflecting the complexities of governing unequal countries. The USA has the highest Gini for high income countries, and a much higher cost of producing health care; countries like South Africa might argue that their inequality puts them in a higher income league than appropriate (upper-middle) which results in an unfairly harsh benchmarking.
• Higher-income countries seem to do better when there is a greater direct public sector expenditure outlay, which may speak to the complexities of governing market-driven systems. The high cost of a private sector-led US system stands in contrast to other lower cost government led systems in the OECD, for example, suggesting different costs drivers and abilities to contain cost in these systems.

These and other second-stage observations provide important avenues for further research into why outcomes vary between countries, and will help to focus research on real governance and government impacts.  I will explore these more in coming weeks but wonder what you think of the approach in general?

 

Choosing Outcomes To Shape Governance Indicators Around

Today's Question to Guide the Design of Governance Indicators: Which Outcomes Should Government Be Engaged in?

I'm back in Boston and into thinking about governance, governance indicators and outcomes. I have replied to a couple of great conments on past posts--thanks to those who are engaging so thoughtfully.

One of the comments, by the ever-sharp Clay Wescott, suggested that we need a step by step approach to connecting governance indicators to outcomes. I agree. I think my starting step is even earlier than what Clay proposes, however. I think we need to start by asking what outcomes we have in mind when we think about governance. Given we are thinking about governments in particular, the question I ask is: What outcomes do citizens commonly authorize governments to help facilitate?

This question emerges from the assumption that principals delegate authority to agents in order to achieve outcomes. Mirroring basic theories of public finance, I argue that citizens delegate authority to governments when faced with restrictive costs or complicated problems of cooperation and coordination. I could extend my perspective to sociological theory on the state, agreeing that sometimes citizens give governments authority because they beleive it is appropriate to do so (regardless of the presence of cost or coordination-type problems).

I am arguing that citizens are still looking for outcomes in return for the delegated authority, however, no matter why they give such. Further, the quality of public sector governance is reflected in whether public organizations exercise authority in a way that facilitates outcome and value production.

This does not mean that the governments produce services directly. They could regulate a sector. They could subsidize a sector. Etc etc.

Given such thinking, I suggest that public sector governance indicators should focus on domains where most would agree that citizens accord governments authority to influence outcomes. This could include “public goods” interpreted narrowly (such as defense) and more broadly (maybe the Millennium Development Goals (MDGs)).  I wonder what areas quickly come to mind: Perhaps macroeconomic management, policing and defense, education, health care, and environmental protection.

I would love to see a top 5 from readers. What outcomes do we think governments around the globe are authorized to  facilitate?

I think the question is important and takes us from dominant approaches to governance that have assumed that 'small' or 'limited' governance makes sense--regardless of country or outcome being considered.  It calls us  into the debate on the functional role of government--not necessarily its form. 

As an example, in the coming weeks I will be discussing an area in which governments are always involved--child health--and an indicator that cannot be achieved without the enaggement of governments--child survival to five years old.  What other examples would you like to explore?

Stop saying 'Country x has a governance score of y': It makes no sense

If we focus on Governance Outcomes, we need to focus indicators on specific Organizational Fields: Whole of government indicators—or government only indicators—make little sense

Last week I did a blog post on my ideas about linking governance indicators to outcomes. A range of people have responded positively to the idea, though most via email. Try writing comments in and let’s get a conversation going on the topic.

Anyway, I thought I would provide some more discussion on what I see as the implications of focusing governance indicators on outcomes and share specific examples of how this may be done. I will do this in the next few days. I aim to be a little provocative and get some responses…so fire away, please!

What is the appropriate unit of analysis in governance?

To kick things off, I thought it might be useful to think about the way in which an outcomes focus might challenge the unit of analysis in governance discussion and measurement. Given the theoretical approach I outlined last week, it is my belief that governance can, by definition, be assessed only in the context of specific fields in which outcomes are produced. I mean something specific about ‘field’ here, building on the definition in organizational theory: “The particular set of organizations that, in the aggregate, constitute a recognized area of institutional life” (see DiMaggio & Powell, 1983). The field includes “suppliers, resource and product consumers, regulatory agencies, and other organizations” involved in getting things done in a particular functional area.

Such a system is an appropriate unit of analysis for governance research because it is the domain in which outcomes are produced (yes, a bit repetitive but important). Note a few things. First, The Government is in a sense too large a unit of analysis for a governance indicator. Governments are engaged in multiple fields producing multiple outcomes. Governance indicators that pretend to say something about an entire government are problematic from this perspective. Do they refer to governance as it relates to child health outcomes or to educational outcomes or to safety and security outcomes. Or do we think that one set of questions gives valid information about governance issues relating to all three? Second, The Government is in a sense too small a unit of analysis for a governance indicator. Fields involve multiple players, particularly principals and agents, who are constantly engaged in allocating, receiving and managing authority to do specific things, with particular functional results expected. The quality of governance is reflected in how well the entire field engages to produce these results. National governments (the focus of many governance indicators) are usually only one player in any given field. We need to go beyond measuring governance within and through national governments to link outcomes and governance indicators.

The implication, therefore, is that it really makes no sense to have a ‘governance score’ for governments in Mauritius or Botswana or China or Bolivia. We should have indicators for the fields in which outcomes are produced in these countries, thus ‘the child health sector in Mauritius’ and ‘the safety and security domain in Botswana’ etc.

If we focus on fields, we need to expect variation and complexity

One should note some additional important features of organizational fields (as discussed by Richard Scott in his classic text on organizations). They come in different sizes and with different relational structures (often driven by resource access). This can yield fields that are placid and do not change very much with time, disturbed or turbulent and subject to many shocks, randomized, reactive or clustered (See Emery & Trist, 1965). In each variation the relational and normative governance challenges may be different, and one should expect very different governance processes to have become embedded. One should also expect different structural and process solutions in each.

Organizational fields also reflect different authority typologies—or ways in which authority is cognitively and normatively influenced. In some places one should expect social choice to determine who has authority and how it is exercised. In others authority will be more coalitional, and in others it will be federative. Authority structures could also be unitary. Once again, the different authority typologies will probably yield different solutions to governance problems. These differences also emerge because of physical differences in fields—some being geographically centered and some being dispersed. A central organizing mechanism may make the first work. A decentralized mechanism may be appropriate to facilitate best outcomes in the second.

Building on this idea, governments can be expected to play different roles in different fields, largely because the sets of players and institutional logics differ in different fields, rates of change in fields vary, and demands on controlling and shaping behavior are not always the same. The education and health care fields look very different in the USA, for example. The roles of federal, state and local governments differ between these two fields, as does the relative size and scope of engagement by private and non-profit players.

These differences matter because public sector governance is about the exercise of authority by government agents, on behalf of citizen principals, to maximize the welfare of those principals; and all of these elements are field-specific. In some fields citizens delegate authority to government to regulate, in other fields to provide services directly.

So, governance indicators should focus on fields…where outcomes emerge…but with flexibility to reflect context.

As a result of the expected variation in structures between fields within and across countries, and the implications of such for governmental roles, we believe that governance indicators should be defined by the field in which outcomes are produced, players can be identified, and authority structures delineated. This allows governance indicators to look like specific rather than general health measures. Prominent governance indicators pay indirect homage to this, including the WGI, which separates “government effectiveness” from “rule of law” in its components; but these are categories of interest and not defined areas in which authority is exercised—the key driver of the governance definition. As a result, the measures become difficult to interpret. Practitioners, policy-makers and researchers would be better served with indicators focused on specific outcomes in specific fields that make immediate sense and can be acted upon.

I see the child health arena as one field, with identifiable outcomes and navigable organizational fields—where one can identify key agents, organizations, organizing logics and systems connecting and structuring such (even though these do vary across countries and over time, shown in studies on changes to health care fields). Child health is the complex product of multiple players bringing multiple inputs to bear in these fields, under specific authority structures, with different effects in different settings. (Other field-based studies include Beaumont, 2003, on anti-poverty systems and Parto, 2005, on waste subsystems in the Netherlands and the UK and Knill & Lehmkuhl, 2002, on regulation of the internet.)

Making good good governance indicators--the construct validity question

IF YOU HOPE PEOPLE WILL DO WHAT YOU MEASURE

MAKE SURE YOU MEASURE WHAT YOU WANT THEM TO DO

I was asked to give some thoughts on construct validity in governance-type indicators. Here are some thoughts and references to sources that help me in developing such thoughts.

First, I would start with the critique Melissa Thomas provides of the world governance indicators. She asks simply, 'what do they measure' and in so doing raises a major construct validity question, with two dimensions: (1) is there underlying theory about how the things measured in the indicator affect the outcomes designers care about? (2) is there evidence that the things being measured are related to these outcomes?

See her full paper at, http://www.sais-jhu.edu/sebin/q/r/What%20Do%20the%20Worldwide%20Governance%20Indicators%20Measure.pdf

See also the response by Kaufmann, Kraay and Mastruzzi. http://info.worldbank.org/governance/wgi/pdf/kkmresponseejdr2final.pdf They argue a blend of 'construct validity isn't relevant' and 'you don't prove we have a construct validity problem anyway.'

Whatever you think of Thomas's paper and the response, the issue is simply whether the thing being measured correlates with the thing it is meant to be an indicator of. As with simple ideas, however, there is more complexity to consider. Indeed, there are multiple dimensions of this construct validity test.

1. Construct validity requires content validity: does the measure reflect on all facets of the construct? Lawshe proposed a way of testing this, suggesting that panels of experts be given a list of all the questions one intends to ask about a construct, and asked whether each question relates to an essential, useful bot not essential, or Unnecessary aspect of the construct. They could also propose questions that have been left out.

With enough feedback, one can then get an idea about the core content of a construct. If a question is roundly endorsed as 'essential' for example it would be part of the non-negotiable part of the indicator. If it were anything less, one would have to decide if the question was a negotiable, cutting edge or unnecessary element in the indicator.

I have some cautionary notes. First, make sure the experts explain their results- providing theoretical reasoning to back their argument. Make sure they are not just making a normatively biased call. Second, make sure your panel of experts has a good understanding of the construct you are trying to measure and why you are measuring it (see yesterday's discussion on indicators and outcomes). Third, make sure your panel is not biased, reflecting only one normative opinion of the construct. This is one of the problems I think the world governance indicators and other governance indicators has encountered. Kurtz and Schranck and others simply say (with justification) that these indicators reflect neoliberal bias and the perspectives of private sector operators. Fourth, I think you need to go beyond lawshe to actually test the correlation with questions and outcomes, continually, so that over time you build evidence in support of your content decisions.

I have included a couple of links to studies that do this ( in areas like education and health care).

• This is a citation to a study on Ccnstruct validity of 3 indicators of psychological distress in relation to perceived health by Koopmans and Lamers. http://www.mendeley.com/research/assessing-construct-validity-three-indicators- psychological-distress-relation-perceived-health-physical-illness/ The study asks about how each indicator relates to the outcome in question. It uses regression analysis to test the relationship. Combining theoretical and empirical work like this helps foster content validity.
• See also Willis, Stoelwinder and Harris. Their work focuses on interpreting indicators in health care. The authors test the assumption that quality indicators reflect on the quality of care in hospitals. http://www.ncbi.nlm.nih.gov/pubmed/18603538
• See also Mathews, Hackett and Pennell on construct validity of accountability measures and performance indicators in schools.http://www.learningpt.org/sipsig/mathews.html

2. Convergent validity is another aspect of construct validity. This asserts that a measure parading as an indicator of x should converge with other indicators of x. The Tests here are quite simple. Look at indicators that purport to reflect on the same construct and see if they are correlated. An example is Townsend and Kaiser who looked at indicators of fruit intake and psychosocial health. http://www.ncbi.nlm.nih.gov/pubmed/16029687

The world governance indicator folks have done this, and argue emphatically that their indicators correlate highly with other indicators of constructs implied in the broad idea of good governance. There are some problems with this, though. First, the 'good governance' concept is so broadly presented in this work that one is bund to found overlap. Second, the world governance indicators are calculated on the basis of many of these 'related measures'. So, one should expect correlation. But this is mathematical convergence, not theoretical convergence.

Another approach to testing construct validity of a new indicator involves taking the new indicator and testing it side by side with a prior indicator that is acknowledged as having some construct validity already. I like Burke, Burke and Crowder in this regard. http://aei.sagepub.com/content/31/4/1.abstract

The article looks at how student skills of a new indicator of early literacy scores correlated with an older indicator of a similar construct that was already known to predict Later reading efficiency.

3. Discriminant validity is the third aspect I will draw on now. It notes that a measure meant to indicate x should not indicate y, z and other constructs. It is kind of like the opposite of convergent validity in that you should be able to see that it does not correlate with (and rather diverges from) indicators of other things.

This is a dimension where world governance indicators have struggled. The argument of people like me is that these indicators correlate with measures of economic well being, such that it is hard to say they are not just another way of measuring this. Neumann and Graeff suggest this is one of the questionable parts of the good governance indicators they look at. http://www.springerlink.com/content/t417158753238u2q/

An empirical consequence of this problem is that governance indicators typically correlate with other economic, social and political indicators. This means that there could be measurement problems with regressions with governance indicator scores as explanatory variables and democracy, GDP level, etc also as explanatory variables.

Governance indicators and governance outcomes: How do they relate?

 Alan Hudson asked yesterday how I relate my comments about constructing governance indicators to prior work I have done that argues governance is all about producing outcomes. I replied in comments, but let me draw a bit from a 2010 paper I wrote on this subject (in Oxford Development Studies). Think about this as the way I am conceptualizing the process of 'theorizing' governance indicators.

We need to have some idea what governance is, before starting. In a general sense, most see this as the exercise of authority. Which requires us asking: who the authority is exercised by, on behalf of, and for what reason?  Answers to such questions will help guide us to appropriate measures of the phenomenon.

I find the literature on corporate governance in publicly traded companies useful in addressing these questions. Tirole defines corporate governance in this context as “the design of institutions that induce or force management to internalize the welfare of stakeholders”. Consider the basic elements of governance implied in this definition: (1) it is focused on how mechanisms (institutions) regulate (2) the way that authority is exercised by one set of agents (managers) (3) who act on behalf of a broad group of principals (stakeholders) (4) with the goal of maximizing the welfare of these principals (stakeholders).

Combining these elements, it can be seen that good governance emerges when specific agents exercise delegated authority in such a way that their specific principals enjoy improved welfare.

This definition has parallels in the political science and public management literature. Kooiman’s characterization of governing, for example, points to “the totality of interactions, in which public and private actors participate, aimed at solving societal problems or creating societal opportunities”. Hill & Lynn describe public sector governance from a management perspective, as “Regimes of laws, rules, judicial decisions, and administrative practices that constrain, prescribe, and enable the provision of publicly supported goods and services through associations with agents in public and private sectors”.

The idea of delegated authority emerges across these definitions, as does the focus on outcomes as the purpose of delegated authority (maximizing stakeholder welfare, “solving societal problems or creating societal opportunities” and ensuring the “provision of publicly supported goods and services.”)

In the public sector context we are obviously dealing with governments as the agents to whom authority is delegated, by citizens (as principals), with the explicit goal of maximizing various kinds of social welfare (as the outcome). Governments can use this delegated authority in many ways: to garner and allocate resources, build capacities (human and physical), regulate via laws or force, and so forth. Governance is superior when authority is exercised in such a manner that social welfare is maximized and governance is inferior when authority is exercised in a manner that does not maximize welfare.

In this way of thinking, governance indicators should assess the degree to which organizations structure their authority relationships to produce outcomes--not just the degree to which they comply with a generic set of best practice processes.

Some roles, interactions and mechanisms might be more effective than others in facilitating better governance outcomes. Some may have generic import to all organizations producing the outcome, some may be relevant contingent on specific contexts, others may be cutting edge ideas we think will have value but are still untested.

These kinds of practices can only be identified after we look at specific organizational fields (e.g. education or health care), measure the outcomes of interactions in these fields, and consider contextual differences between countries. Function must lead form in this respect.

We need to theorize our indicators to make this happen. This starts with us having clear ideas about the outcomes we are looking at (and our normative approach to thinking about such) and the fields in which we are focused. It moves to having some theoretical idea about why we think some practices may facilitate better outcomes. It then goes further to the evidence we have about such relationships. The evidence helps us see which practices fall into which categories, such that we can say:

• “The following governance dimensions (what we measure as the generic base of our indicators) are non-negotiable institutions that we have evidence induce or force agents to produce welfare-maximizing outcomes for principals—in all cases”.
• “The following governance dimensions (what we measure as the contingent segment of our indicators) are negotiable institutions that we have evidence induce or force agents to produce welfare-maximizing outcomes for principals—in a contingent way (being relevant in specific contexts)”.
• “The following governance dimensions (what we measure as the cutting edge segment of our indicators) are untested institutions that we believe may induce or force agents to produce welfare-maximizing outcomes for principals (and we include to facilitate some testing)”.

In all three dimensions, we include the form because we theorize that it leads to better functionality and improved outcomes. I would like to say we are constructing indicators of more effective government...