Can governance indicators focused on outcomes still be about governance?

MEASURES SHOULD CAPTURE THE SYSTEMS

BUT ONLY THOSE THAT HELP PRODUCE THE OUTCOMES.

A reader, David, asked an important question in response to my blogs about governance and outcomes. Essentially, he noted that the world of development has come a long way in recognizing that governance matters. Nowwe have a focus on administrative structures, rules of the game and the like, under the title 'Governance'. If we start saying that governance must be about outcomes, could we lose this focus...

A great question (which I hope I have represented fairly). Here are my thoughts on it.

While my past work may seem to suggest I see outcomes as measures of governance themselves, I do not. Outcomes come into the picture because governance must be about outcomes. So, governance is about the institutions, structures, systems etc. that shape authority mechanisms principals use to ensure that agents produce outcomes that lead to welfare gains for the principals.

Consider the figure below. The indicator should measure institutions, structures, etc. that we believe help achieve outcomes (or otherwise enhance the welfare of principals). In the case of governance in the child health arena, the indicator would measure whether a country has systems etc. that we believe facilitate the exercise of authority in such a way as to maximize the survival rate of under five children. It would aggregate many components for sure, to be a complex measure of the governance structures deemed necessary to keep kids alive. But it would not be a vague summary indicator of something that someone somewhere can postulate about.

Note that the outcome is central here, and the test is whether the indicator, and its components, reflect on the outcome. See the discussion on construct validity from a few days ago. You cannot assess construct validity unless you know what the governance focus is. And you cannot vouch for the construct validity of your indicator unless you have some way of testing whether its components are actually related to the outcome you care about.

 
 If one thinks of governance indicators--and the relationship between governance, indicators and outcomes--in this way, it should alleviate concerns that the focus on institutions, systems, strcutures etc. will be lost in the future. They are still front and center. But they cannot be introduced in a stand-alone fashion. In this way of thinking they are chosen and included in the indicator and in our lexicon of reform because we believe that they help improve the way society produces outcomes.

I have used the word 'believe' above to be a little open and fluffy. I would love to see it replaced with 'know' or 'believe, on the basis of theory and evidence'. All the systems and structures and institutions we include when constructing an indicator should be things we believe, on the basis of theory and evidence, facilitate the exercise of authority by governments (and other entities) in such a way as to produce outcomes that advance the welfare of citizens. Sorry such a mouthful.

If we had this kind  of theoretical, evidence based story, we would not weaken the governance agenda--or dilute it. We would surely strengthen it. Being able to say that having a multi-year budget in the health sector is empirically associated with having better  child health outcomes is surely the best way of ensuring that these reforms gain traction. But we cannot say this. So, we depend upon the development community accepting that multi-year budgets have some intrinsic, stand-alone value...things to be done even if we don't know their value. I cannot really think of too many governance practices to put in this category.  

I know I have not addressed the fullness of David's question, which asks about the impact of other factors on outcomes, potential interactions of these factors with governance reforms, etc. I hope  the commentary following this--and future posts--will go some way to do this. In the mean-time, as usual, what do you think about the perspective?

 

 

 

Comments

[David]

I think that this is a clear explanation, and I certainly echo your perspective. I have long felt, working on governance issues, that (to some extent, by no means predominantly, but still) the "institutional forms" measurement of core governance indicators had led to a bit of a focus on form over function. So I've been very much supportive of the broader trend of a) people from other "fields" paying attention to governance and b) integrating governance as a means to get at other outcomes. However, as a natural cynic, now that there are these successes, I begin to worry about the pendulum swinging too far towards governance as only instrumental, with limited/no intrinsic value.

I liked your commentary on construct validity, and particularly that this forces us to further articulate the underlying logic linking the indicator to the purpose. So part of why I asked the question that prompted your response, is that I think that the more deliberately and carefully we can link the governance agenda to field-specific outcomes, the more likely that we can start to isolate some of the positive externalities that come from working on governance. If we will claim that there is something there, more than just one of several means to an ends, good indicators (and the work to make them valid) should point us not only towards governance reforms' places as steps in a causal chain, but also help us to differentiate: when and why governance approaches matter more to field-specific outcomes; or when and why they have positive spillovers in other fields; in ways that can be better tested. I'm very excited by what such approaches can help us to learn.

[Clay Wescott]

I support the need for a theoretical, evidence based story better linking the governance agenda to field-specific outcomes. But we will need to work step by step. The first step is to better develop theories linking specific institutional processes to impacts on the ground. This is where we are with enterprises like PEFA, where indicators have been constructed, inter alia, based on theory. There are outcome indicators measuring two key outcomes of public financial management: fiscal discipline and strategic resource allocation, so the theory can be tested in these respects. The third key outcome, efficient use of resources for service delivery, is more difficult to test, and that is where we should focus our efforts going forward.

[Philipp]

To my mind this is both a question of what’s “real” about governance and a question of how much we can know about it.

I would think it’s pretty much consensus in the field to say that development depends a lot on how a place is run (i.e. its governance). But we mostly identify this ex-post, i.e. by observing countries that had a successful run, excluding some obvious other candidates (say, investment, aid, etc), and then focus on governance as a convincing case-by-case explanation. We then look at cross-country comparable manifestations of good governance to be better able to extrapolate from our couple of cases to the population of countries. At the same time we know that in those well run countries, good governance expresses itself through all sorts of informal practices, cultural norms, patterns of behavior which we often can’t fully grasp, which often differ between countries, and even if we could fully measure them their complexity in one place, we’d be overwhelmed by the task of making them comparable.

I fully agree with the argument made elsewhere by Matt and others that we have to keep in mind the difference between governance (which we know is out there but we can’t measure it directly the same way we can do with child mortality) and governance indicators (which are proxies, some better than others, but they are not the same as governance itself). I read the call for governance indicators to relate to outcomes in this context. If development assistance focuses on changing governance indicators narrowly defined, assuming that governance will automatically follow, it’s putting the cart before the horse.

I don’t, however, think that coming up with governance indicators that relate closely to development outcomes will necessarily make a big difference, unless we know a lot more about how one relates to the other over longer time periods. For one thing, this is clearly a normative undertaking that reflects why we are interested in specific governance arrangements in the first place (as a means to an end). However, actors in country x might pursue institutional change with completely different objectives.

If European history is any guidance (and it might not, I just happen to know it), then our Western good governance structures owe a lot to mean, tiny elite groups bent on war, exploitation and glory. More than one effective tax state brought its citizens to the brink of starvation (i.e. produce development outcomes decidedly more rotten) before those same governance structures enabled the funding of social safety nets a couple of generations down the road.

In an emergency situation, it’s often effective to import institutions wholesale to quickly help people, even though that might not be the best way to grow endogenous capacity. Over time, ‘governance for results’ has a lot to do with a state that is able to assess, decide and implement autonomously in a complex environment. Do we already know what tangible governance institutions today would get us such a state tomorrow?

[Clay Wescott]

Taking the European history story a bit further, consider the case of Belgium's King Leopold in the late 19th century, as told by Bruce Bueno de Mesquita. In Belgium he helped workers, reduced poverty, and supported massive public works. On the other hand, in Belgian Congo, he supported enslavement, and a policy of murder and terror to help him stuff his personal bank account with the proceeds of the colony's riches. Why the split personality? Because the incentives were different. In Belgium, he was a constitutional monarch, and had to work within a democratic system to stay in power. In Congo, there were no constraints, so he went for broke.

[matt andrews]

Thanks to Philipp and Clay for their comments. A really nice discussion indeed. I would say that the governance discussion takes three lines, all related: About governance, governance indicators, and governance reforms. One would hope that all three related seemelessly--such that reforms are reflected in indicators and these make sense given the way we think about governance (with supporting evidence). I think that discussions on governance reform and governance indicators have been quite connected in the past 15 years or so, but these have been disonnected from serious empirical work on governance. This kind of work is quantitative and qualitative, and in both domains involves the kind of historical analysis Philipp and Clay mention. Emerging literature is very very important in this respect. I think it tells a number of stories. For example, institutions emerge in contexts and are contingent on contextual factors. Also, institutions are seldom 'good' in an absolute sense, having a 'fitted' relevance instead. I also find the literature constantly reminding me that governance is complex and good or bad governance cannot be easily explained. Building on the example of Belgium and the DRC, for example, I would agree that the incentives of a monarch are important but would also say that there is mcuh more going on than this. Not all colonial masters disregarded their colonies in the way Belgium did in DRC. This was partly because different countries had different perspectives on what they wanted from the colonial engagement, on how power structures looked in the colonies, on how political power structures looked in the home countries, and a lot more. Philipp is right in saying we often get 'normative' in interpreting governance solutions. I think this is something we do partly because of our own cognitive limits, but we need to be more aware of it than we are. And serious cross-disciplinary research on what kinds of governance arrangements lead to what kinds of outcomes can really help us in doing this. Such work may not yield indicators. It will yield more humility, though, at least in my opinion. And this makes it worth while.

[matt andrews]

I have a specific comment in response to Clay's comments on the PEFA indicators. One of the things I am trying to explore is how we can theorize indicators better. For those who are not close to the PEFA indiactors, could you explain how you see the theorization of PEFA? In particular, how were the outcomes you list identified as key and generally valid? How were they defined and how are they measured? How were the indicators and dimensions chosen and what evidence was provided that these dimensions actually tie to outcomes? How is the theoretical and evidence basis of each indicator or dimension explained to potential users? It would be great if others could answer similar questions in respect of other indicators as well. A repository of such knowledge could help inform the entire 'universe' of indicator designers and users. Thanks.