Principles for developing governance indicators

I had an interesting meeting this morning to discuss a new initiative to devise indicators of good practice in observing human rights. It got me thinking about principles I see as important in devising indicators like this. Think of it as good practice for good governance indicators, to make sure there is some sense to the box ticking now dominating development.

 Here goes:

1. Be clear what you are devising the indicators for: Is it to hold organizations accountable? Is it to help organizations know how to do things better? These and other objectives are not necessarily mutually exclusive but not all indicator sets work to do both. If indicators are meant as guides on doing things better, don't use them to facilitate coercion. It could undermine their value.

2. Be clear about the indicator's normative bias: Indicators always have some kind of bias. Human rights groups develop indicators that are about observing human rights. The World Bank World Governance Indicators have a distinct neoliberal flavor (touting the norms of limited government). This should not be hidden, or considered a good or bad thing. It is, however, important for users to understand what the indicators are about.

3. Theorize the indicators: What I mean is simply that the designers should explain where each indicator measure, dimension or element comes from, and why measures are being incorporated (why do we think x is important as a contributor to good governance, for example). What is the theoretical argument underlying it? What is the empirical evidence supporting it? What examples can you provide as to why it is included? Too many indicators assess things that are not well theorized, leaving organizations in the dark as to whether they should comply and how.

4. Distinguish between dimensions in the indicator set: In most indicator sets there are issues being raised that one could theorize as non-negotiable good governance conditions. Others are interesting conditions or practices that are actually negotiable--working well in some conditions and not in others. A third set of practices could be considered innovative and new, but untested. Where indicator sets don't distinguish between the three, organizations face the same pressure to conform with non-negotiable, negotiable and cutting edge elements. This is problematic.

5. Assess de jure and de facto aspects of key practices: If an indicator set has an impact, organizations will face pressure to conform with required practices. This conformity can involve 'ticking boxes' by making changes in form--when in fact the change intended is in function. When this happens, indicators can foster a gap between form and function that undermines legitimacy of the prescribed practices. To counter this, measure aspects of the practice that involve de jure changes in form as well as aspects of the practice that involve de facto implementation. Then measure the gap between the two. 

6. Allow indicator analysis that has quantitative and qualitative dimensions: I don't like indicators that are just about numbers. A 1.5 this year and a 1.5 next year may not be the same. How do I know if there are side stories that need to be told? How do we learn about the fit of some indicator questions to context? I like indicators that provide both numbers and qualitative descriptions of the analysis underlying such. Contextual texture is important to understanding the indicator results.

Just my thoughts. I like Global Integrity data, Bertelsmann Transformation Index reports, and Economist Intelligence Unit reports because they satisfy quite a few of these criteria. 

Comments

[Alan Hudson @alanhudson1]

Thanks Matt. Very helpful, and I share your liking of Global Integrity and the Bertelsmann Transformation indices. But, I'd really appreciate a short explanation as to how you see the continued drive for governance indicators as fitting with:

a) the recognition that "good governance" means different things in different places

and

b) the related realisation - made most clearly by you - that what matters is not the form of governance by the sectoral outcomes that it produces.

I am stumbling towards a decent answer. Can you help? Or can other readers?

[matt andrews]

Hi Alan: Good questions.

First, I think that there are dimensions of good governance--a few high level dimensions--that may have generic import. My point in '4' is that these are the non-negotiables. The things we can say with certainty are important to making an organization or government work. I will leave it to future posts to identify these, but we might think of an example being 'having a budget' or 'reporting on funds received from citizens'. If you don't do these things at all, there is little scope for governing effectively. Then, I think there are the negotiable dimensions and the cutting edge dimensions. These will vary with context and are at the heart of 'good governance means different things in different countries'. This could get into certain types of budgets, or reporting mechanisms, etc.

Second, what matters is surely what the government or organization does. This is why I believe we need to do '3' above--theorize the indicators. This involves explaining why we believe a dimension matters in facilitating better performance. Dimensions should be included because we see them making a difference in performance, not just because they look good. Include a process requirement because we really think it helps ensure health services are better provided, for instance. These will be identified in the stage 2 assessments I recommend in the 2010 article on u5mr as governance indicators.

The focus on results is also why I think we need to do '5' above--measure de jure and de facto aspects of indicator dimensions, and emphasize the gap. We must make sure that we have an argument (with evidence to back it, hopefully) about how compliance leads to better performance, and we need to make sure compliance is complete (function and not just form). It seems to me that this gap is the key part of the difference between developed and developing country governance regimes. We must capture it, and keep building the evidence about how specific practices, effectively implemented, can improve results.

Hope useful!