Stop saying 'Country x has a governance score of y': It makes no sense

If we focus on Governance Outcomes, we need to focus indicators on specific Organizational Fields: Whole of government indicators—or government only indicators—make little sense

Last week I did a blog post on my ideas about linking governance indicators to outcomes. A range of people have responded positively to the idea, though most via email. Try writing comments in and let’s get a conversation going on the topic.

Anyway, I thought I would provide some more discussion on what I see as the implications of focusing governance indicators on outcomes and share specific examples of how this may be done. I will do this in the next few days. I aim to be a little provocative and get some responses…so fire away, please!

What is the appropriate unit of analysis in governance?

To kick things off, I thought it might be useful to think about the way in which an outcomes focus might challenge the unit of analysis in governance discussion and measurement. Given the theoretical approach I outlined last week, it is my belief that governance can, by definition, be assessed only in the context of specific fields in which outcomes are produced. I mean something specific about ‘field’ here, building on the definition in organizational theory: “The particular set of organizations that, in the aggregate, constitute a recognized area of institutional life” (see DiMaggio & Powell, 1983). The field includes “suppliers, resource and product consumers, regulatory agencies, and other organizations” involved in getting things done in a particular functional area.

Such a system is an appropriate unit of analysis for governance research because it is the domain in which outcomes are produced (yes, a bit repetitive but important). Note a few things. First, The Government is in a sense too large a unit of analysis for a governance indicator. Governments are engaged in multiple fields producing multiple outcomes. Governance indicators that pretend to say something about an entire government are problematic from this perspective. Do they refer to governance as it relates to child health outcomes or to educational outcomes or to safety and security outcomes. Or do we think that one set of questions gives valid information about governance issues relating to all three? Second, The Government is in a sense too small a unit of analysis for a governance indicator. Fields involve multiple players, particularly principals and agents, who are constantly engaged in allocating, receiving and managing authority to do specific things, with particular functional results expected. The quality of governance is reflected in how well the entire field engages to produce these results. National governments (the focus of many governance indicators) are usually only one player in any given field. We need to go beyond measuring governance within and through national governments to link outcomes and governance indicators.

The implication, therefore, is that it really makes no sense to have a ‘governance score’ for governments in Mauritius or Botswana or China or Bolivia. We should have indicators for the fields in which outcomes are produced in these countries, thus ‘the child health sector in Mauritius’ and ‘the safety and security domain in Botswana’ etc.

If we focus on fields, we need to expect variation and complexity

One should note some additional important features of organizational fields (as discussed by Richard Scott in his classic text on organizations). They come in different sizes and with different relational structures (often driven by resource access). This can yield fields that are placid and do not change very much with time, disturbed or turbulent and subject to many shocks, randomized, reactive or clustered (See Emery & Trist, 1965). In each variation the relational and normative governance challenges may be different, and one should expect very different governance processes to have become embedded. One should also expect different structural and process solutions in each.

Organizational fields also reflect different authority typologies—or ways in which authority is cognitively and normatively influenced. In some places one should expect social choice to determine who has authority and how it is exercised. In others authority will be more coalitional, and in others it will be federative. Authority structures could also be unitary. Once again, the different authority typologies will probably yield different solutions to governance problems. These differences also emerge because of physical differences in fields—some being geographically centered and some being dispersed. A central organizing mechanism may make the first work. A decentralized mechanism may be appropriate to facilitate best outcomes in the second.

Building on this idea, governments can be expected to play different roles in different fields, largely because the sets of players and institutional logics differ in different fields, rates of change in fields vary, and demands on controlling and shaping behavior are not always the same. The education and health care fields look very different in the USA, for example. The roles of federal, state and local governments differ between these two fields, as does the relative size and scope of engagement by private and non-profit players.

These differences matter because public sector governance is about the exercise of authority by government agents, on behalf of citizen principals, to maximize the welfare of those principals; and all of these elements are field-specific. In some fields citizens delegate authority to government to regulate, in other fields to provide services directly.

So, governance indicators should focus on fields…where outcomes emerge…but with flexibility to reflect context.

As a result of the expected variation in structures between fields within and across countries, and the implications of such for governmental roles, we believe that governance indicators should be defined by the field in which outcomes are produced, players can be identified, and authority structures delineated. This allows governance indicators to look like specific rather than general health measures. Prominent governance indicators pay indirect homage to this, including the WGI, which separates “government effectiveness” from “rule of law” in its components; but these are categories of interest and not defined areas in which authority is exercised—the key driver of the governance definition. As a result, the measures become difficult to interpret. Practitioners, policy-makers and researchers would be better served with indicators focused on specific outcomes in specific fields that make immediate sense and can be acted upon.

I see the child health arena as one field, with identifiable outcomes and navigable organizational fields—where one can identify key agents, organizations, organizing logics and systems connecting and structuring such (even though these do vary across countries and over time, shown in studies on changes to health care fields). Child health is the complex product of multiple players bringing multiple inputs to bear in these fields, under specific authority structures, with different effects in different settings. (Other field-based studies include Beaumont, 2003, on anti-poverty systems and Parto, 2005, on waste subsystems in the Netherlands and the UK and Knill & Lehmkuhl, 2002, on regulation of the internet.)

Comments

[Robert Klitgaard]

Consider the generalization: when do aggregate country measures have (some) utility? "For what?" is the obvious follow-up.
More generally still, consider aggregate measures of all kinds:
http://www.cgu.edu/PDFFiles/Presidents%20Office/Contested_Summary_Measures_10-07.pdf

[MJ]

Hi Matt,

I would agree with you to a large extent, but I think it should be possible in some way then to abstract to the government score. For instance, in different fields, e.g. education and health (both of which aggregate many sub-sectors), the government has a major role to play. Further, its role will have some key commonalities, e.g. in allocation of budget, and promulgation and enforcement or regulations. These could be - and in existing indicators I assume are - scored in some kind of way. (Your previous criticisms about frames of reference, of course, remain 100% valid.) The scores for these areas of commonality may well be of interest to potential outsiders, e.g. businesses considering investments. (When you consider such consumers of this interest, situating governance indicators within a single frame of social reference may actually be a good thing; the businessmen may better understand such indicators.) The end result is necessarily crude, obscures a huge amount of detail, and is difficult to relate to on-the-ground specifics, but is not completely worthless.

I think the point you raise about degree of decentralisation is, in some ways, more important. In a highly centralised consumers of the indicators (whether citizens or outsiders) can hold the central leadership to account for poor scores. In more decentralised societies such direct accountability is diluted, although the simple moral response is that one should be accountable for one's own role in matters, and for working to persuade others to take similarly enlightened approaches.

Keep up the thought-provoking posting!

[Clay Wescott]

Hi Matt,

I think that there is a role both for summary indicators, as Bob points out, and for analytics focusing on fields where outcomes matter. Here's a stab at the latter in Bangladesh, looking at rural infrastructure.

http://www.worldbank.org.bd/WBSITE/EXTERNAL/COUNTRIES/SOUTHASIAEXT/BANGLADESHEXTN/0,,print:Y~isCURL:Y~contentMDK:22351464~pagePK:141137~piPK:141127~theSitePK:295760,00.html

[matt andrews]

Thanks for the comments, all of which are useful and send us to great conversations and examples. Bob is insightful as always, and I have to agree with your ideas--I am trying to provoke here and glad to get this set of your thoughts as the product of my provocation. Let me respond with an open stream of thought (hopefully it makes sense).

I do want to locate my comments in the earlier blog entries as a strating point: I am arguing that, if we agree good governance is about structures that facilitate better outcome achievement (as per prior posts), the 'country' or 'government' is an inappropriate unit of analyzing governance.

I am not arguing that aggregate country level indicators of other things lack value. I am also not arguing that aggregated indicators--of governance and other things--lack value altogether.

To emphasize this, I am one of many who use aggregate indicators all the time. I am not suggesting we do not aggregate measures into indicators. I am just asking whether the 'country' or 'government' is an appropriate unit or level of aggregation for governance indicators. As MJ notes, focusing on outcomes in the education or health sectors also requires aggregation--but the idea here is that you are putting pieces of data together that relate to specific outcomes.

When we combine information across outcome areas (like health and education), I think we get too 'summary' for our own good (to use Clay's language). And it becomes hard to theorize effectively about what the indicator is and why it matters--in respect of outcome achievement.

To be honest I use aggregate governance indicators all the time--WGIs and TI anticorruption stats, and Global Integrity data, and Quality of Government measures. They do have value and they do have some meaning. In the case of the WGIs, people like Kurtz and Schranck argue that the indicators are an alternative measure of development--pretty highly correlated with levels of GDP but not with levels of growth. This is an interesting and valuable indicator. I ask if this is a governance indicator, though, that captures "the way authority is exercised by a set of agents to advance the welfare of a set of principals"? In particular, is it a governance indicator that is relevant for developing countries--given that it iss correlated with a static level of GDP and not a dynamic improvement in GDP (or related outcomes)?

I would need to agree that high levels of GDP are an important outcome that certainly advances the interests of citizens. If we think that this is the goal of good governance and that we can attribute it to government structures, then Ok. This is how I use these indicators, given that it seems to have some validity. As MJ says, it may be general but it may capture what business-people looking for opportunities think generally matters. But it does not capture growth in GDP very well...so most developing countries will have weaker governance scores automatically...even if they are producing growth--arguably the outcome people carea about in such contexts. This could be why many of the higher-growth developing countrie shave the lowest country-level governance stats.

I also think the general indicators are valuable because they have been legitimated and hence influence behavior. Like standardized tests at schools. They measure a host of things, aggregating all into single scores, but have become really important because they drive behavior. Like 'Teaching to the Test'. I use WGIs in particular because I see major overlaps between the content of best practice reform agendas and ideas in the WGIs. So, I am interested to see if the international community's reforms have an impact on 'governance' as defined by the international community. How well do they teach to their test? (not so well, it turns out....see earlier blogs)

At the end of the day, though, I am not sure that teaching to the test means that much when we are talking about governance--as defined in my prior blogs. And I am not sure that the things countries are doing to get better TI results, WGI scores, QoG numbers and such lead to any specific and sustainable improvements in outcomes that matter to developing countries. I think they are signals--maybe important ones, but maybe not--but do we know if they signal better outcomes in a dynamic sense?

All of this leads me to a more nuanced position on the issue: Country level indicators may make sense, but are the current country level indicators (that drive reform behavior) too general to make theoretical sense or provide valid measures of what matters to key developing country outcomes and what does not?

[David]

Very thought-provoking post, as intended. I would echo the idea of focusing on meaningful "fields" or "sectors" to review governance in terms of its outcomes. It may be worth remembering, however, that part of the value of having a governance score is that it can draw attention to aspects (authority and its exercises, norms and rules of the game) that can be subsumed or made implicit if the focus moves to welfare and performance in a field. Part of the genesis of governance as its own focal area within development seems to me to have been the relative underattention to governance issues within fields, as compared with resource constraints, poor planning, lack of technical skills and knowledge, and immediate humanitarian emergency responses.

So my provocative question in return would be, if governance indicators are situated within different fields, how to ensure that the governance aspects of the measured performance do not get lost among field specialists (e.g. doctors working on health) focusing on other, often equally valid, limitations on performance? Does governance only matter as one of several inputs to outcomes in a field, or are there particular externalities related to getting the governance of a field right that are worthy of additional or special attention? (e.g., if the education outcomes are poor due to a combination of brain drain, poor infrastructure for delivery of information, outdated curricula, and lack of local input into resource allocation decisions, is there something intrinsically valuable about analyzing the governance failures that clearly relate to the last cause, perhaps contribute to the outdated curricula, and are remotely at best causes of poor infrastructure and brain drain?)

[matt andrews]

Thanks David for your posting. I like the primary question you ask a lot and am crafting a full post to respond. I will separate your follow up question and commentary a bit here, however.

As a 'governance person' I am thrilled that Dani Kaufmann and others put the topic on the map. My generation finds active audiences of people interested in this area of work, because people like Dani, Bob Klitgaard, Clay Wescott and so many others did to ensure it got attention (I menton Bob and Clay specifically because they contributed comments as well, with typical quality).

I do wonder if it has become too much of a stand-alone area, however, where indicators are developed without people asking 'how do they connect to things we care about?' and reforms dominating agendas with little to no evidence that they will help governments govern better. It must be true that there are externalities associated with doing governance reform. Where effective, changs to authorizing mechanisms will surely have interactive effects across social systems--given changes to incentives, cognitive frameworks etc. It is also true, however, that governance mechanisms are just one part of the story of effective states and governments. We know enough to say that contextual variables explain a lot of the variation between places that most of us would say are working and other places.

Perhaps more importantly, we also know that the contextual variables are really important in knowing what kinds of governance solutions will work in a given place. So, governance is not only part of the story, but is also contingent on other parts of the story.

Given this, I think it is a mistake to keep discussons about governance separated from other discussions. It is messy, I know, but when we talk about governance we have to be asking 'of what? by whome? where? when how? etc.' I think the relevance of such questions makes governance less stand alone as it has been portrayed to be.