« Making good good governance indicators--the construct validity question | Main | Can governance indicators focused on outcomes still be about governance? »

09/26/2011

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Robert Klitgaard

Consider the generalization: when do aggregate country measures have (some) utility? "For what?" is the obvious follow-up.
More generally still, consider aggregate measures of all kinds:
http://www.cgu.edu/PDFFiles/Presidents%20Office/Contested_Summary_Measures_10-07.pdf

MJ

Hi Matt,

I would agree with you to a large extent, but I think it should be possible in some way then to abstract to the government score. For instance, in different fields, e.g. education and health (both of which aggregate many sub-sectors), the government has a major role to play. Further, its role will have some key commonalities, e.g. in allocation of budget, and promulgation and enforcement or regulations. These could be - and in existing indicators I assume are - scored in some kind of way. (Your previous criticisms about frames of reference, of course, remain 100% valid.) The scores for these areas of commonality may well be of interest to potential outsiders, e.g. businesses considering investments. (When you consider such consumers of this interest, situating governance indicators within a single frame of social reference may actually be a good thing; the businessmen may better understand such indicators.) The end result is necessarily crude, obscures a huge amount of detail, and is difficult to relate to on-the-ground specifics, but is not completely worthless.

I think the point you raise about degree of decentralisation is, in some ways, more important. In a highly centralised consumers of the indicators (whether citizens or outsiders) can hold the central leadership to account for poor scores. In more decentralised societies such direct accountability is diluted, although the simple moral response is that one should be accountable for one's own role in matters, and for working to persuade others to take similarly enlightened approaches.

Keep up the thought-provoking posting!

Clay Wescott

Hi Matt,

I think that there is a role both for summary indicators, as Bob points out, and for analytics focusing on fields where outcomes matter. Here's a stab at the latter in Bangladesh, looking at rural infrastructure.

http://www.worldbank.org.bd/WBSITE/EXTERNAL/COUNTRIES/SOUTHASIAEXT/BANGLADESHEXTN/0,,print:Y~isCURL:Y~contentMDK:22351464~pagePK:141137~piPK:141127~theSitePK:295760,00.html

matt andrews

Thanks for the comments, all of which are useful and send us to great conversations and examples. Bob is insightful as always, and I have to agree with your ideas--I am trying to provoke here and glad to get this set of your thoughts as the product of my provocation. Let me respond with an open stream of thought (hopefully it makes sense).

I do want to locate my comments in the earlier blog entries as a strating point: I am arguing that, if we agree good governance is about structures that facilitate better outcome achievement (as per prior posts), the 'country' or 'government' is an inappropriate unit of analyzing governance.

I am not arguing that aggregate country level indicators of other things lack value. I am also not arguing that aggregated indicators--of governance and other things--lack value altogether.

To emphasize this, I am one of many who use aggregate indicators all the time. I am not suggesting we do not aggregate measures into indicators. I am just asking whether the 'country' or 'government' is an appropriate unit or level of aggregation for governance indicators. As MJ notes, focusing on outcomes in the education or health sectors also requires aggregation--but the idea here is that you are putting pieces of data together that relate to specific outcomes.

When we combine information across outcome areas (like health and education), I think we get too 'summary' for our own good (to use Clay's language). And it becomes hard to theorize effectively about what the indicator is and why it matters--in respect of outcome achievement.

To be honest I use aggregate governance indicators all the time--WGIs and TI anticorruption stats, and Global Integrity data, and Quality of Government measures. They do have value and they do have some meaning. In the case of the WGIs, people like Kurtz and Schranck argue that the indicators are an alternative measure of development--pretty highly correlated with levels of GDP but not with levels of growth. This is an interesting and valuable indicator. I ask if this is a governance indicator, though, that captures "the way authority is exercised by a set of agents to advance the welfare of a set of principals"? In particular, is it a governance indicator that is relevant for developing countries--given that it iss correlated with a static level of GDP and not a dynamic improvement in GDP (or related outcomes)?

I would need to agree that high levels of GDP are an important outcome that certainly advances the interests of citizens. If we think that this is the goal of good governance and that we can attribute it to government structures, then Ok. This is how I use these indicators, given that it seems to have some validity. As MJ says, it may be general but it may capture what business-people looking for opportunities think generally matters. But it does not capture growth in GDP very well...so most developing countries will have weaker governance scores automatically...even if they are producing growth--arguably the outcome people carea about in such contexts. This could be why many of the higher-growth developing countrie shave the lowest country-level governance stats.

I also think the general indicators are valuable because they have been legitimated and hence influence behavior. Like standardized tests at schools. They measure a host of things, aggregating all into single scores, but have become really important because they drive behavior. Like 'Teaching to the Test'. I use WGIs in particular because I see major overlaps between the content of best practice reform agendas and ideas in the WGIs. So, I am interested to see if the international community's reforms have an impact on 'governance' as defined by the international community. How well do they teach to their test? (not so well, it turns out....see earlier blogs)

At the end of the day, though, I am not sure that teaching to the test means that much when we are talking about governance--as defined in my prior blogs. And I am not sure that the things countries are doing to get better TI results, WGI scores, QoG numbers and such lead to any specific and sustainable improvements in outcomes that matter to developing countries. I think they are signals--maybe important ones, but maybe not--but do we know if they signal better outcomes in a dynamic sense?

All of this leads me to a more nuanced position on the issue: Country level indicators may make sense, but are the current country level indicators (that drive reform behavior) too general to make theoretical sense or provide valid measures of what matters to key developing country outcomes and what does not?


David

Very thought-provoking post, as intended. I would echo the idea of focusing on meaningful "fields" or "sectors" to review governance in terms of its outcomes. It may be worth remembering, however, that part of the value of having a governance score is that it can draw attention to aspects (authority and its exercises, norms and rules of the game) that can be subsumed or made implicit if the focus moves to welfare and performance in a field. Part of the genesis of governance as its own focal area within development seems to me to have been the relative underattention to governance issues within fields, as compared with resource constraints, poor planning, lack of technical skills and knowledge, and immediate humanitarian emergency responses.

So my provocative question in return would be, if governance indicators are situated within different fields, how to ensure that the governance aspects of the measured performance do not get lost among field specialists (e.g. doctors working on health) focusing on other, often equally valid, limitations on performance? Does governance only matter as one of several inputs to outcomes in a field, or are there particular externalities related to getting the governance of a field right that are worthy of additional or special attention? (e.g., if the education outcomes are poor due to a combination of brain drain, poor infrastructure for delivery of information, outdated curricula, and lack of local input into resource allocation decisions, is there something intrinsically valuable about analyzing the governance failures that clearly relate to the last cause, perhaps contribute to the outdated curricula, and are remotely at best causes of poor infrastructure and brain drain?)

matt andrews

Thanks David for your posting. I like the primary question you ask a lot and am crafting a full post to respond. I will separate your follow up question and commentary a bit here, however.

As a 'governance person' I am thrilled that Dani Kaufmann and others put the topic on the map. My generation finds active audiences of people interested in this area of work, because people like Dani, Bob Klitgaard, Clay Wescott and so many others did to ensure it got attention (I menton Bob and Clay specifically because they contributed comments as well, with typical quality).

I do wonder if it has become too much of a stand-alone area, however, where indicators are developed without people asking 'how do they connect to things we care about?' and reforms dominating agendas with little to no evidence that they will help governments govern better. It must be true that there are externalities associated with doing governance reform. Where effective, changs to authorizing mechanisms will surely have interactive effects across social systems--given changes to incentives, cognitive frameworks etc. It is also true, however, that governance mechanisms are just one part of the story of effective states and governments. We know enough to say that contextual variables explain a lot of the variation between places that most of us would say are working and other places.

Perhaps more importantly, we also know that the contextual variables are really important in knowing what kinds of governance solutions will work in a given place. So, governance is not only part of the story, but is also contingent on other parts of the story.

Given this, I think it is a mistake to keep discussons about governance separated from other discussions. It is messy, I know, but when we talk about governance we have to be asking 'of what? by whome? where? when how? etc.' I think the relevance of such questions makes governance less stand alone as it has been portrayed to be.

The comments to this entry are closed.

Most Recent Photos

  • Slide1
  • Slide1
  • Slide1
  • Image
  • Culture, signaling and the transparency fad
  • Limits getting good reviews: not quite the RedSox, but...
  • Hirschman told us that implementation involves a journey
  • MDGs-spending-620-color
  • Gint
  • Obi
  • Dtogdp2010
  • Dtogdp2006