I had an interesting meeting this morning to discuss a new initiative to devise indicators of good practice in observing human rights. It got me thinking about principles I see as important in devising indicators like this. Think of it as good practice for good governance indicators, to make sure there is some sense to the box ticking now dominating development.
1. Be clear what you are devising the indicators for: Is it to hold organizations accountable? Is it to help organizations know how to do things better? These and other objectives are not necessarily mutually exclusive but not all indicator sets work to do both. If indicators are meant as guides on doing things better, don't use them to facilitate coercion. It could undermine their value.
2. Be clear about the indicator's normative bias: Indicators always have some kind of bias. Human rights groups develop indicators that are about observing human rights. The World Bank World Governance Indicators have a distinct neoliberal flavor (touting the norms of limited government). This should not be hidden, or considered a good or bad thing. It is, however, important for users to understand what the indicators are about.
3. Theorize the indicators: What I mean is simply that the designers should explain where each indicator measure, dimension or element comes from, and why measures are being incorporated (why do we think x is important as a contributor to good governance, for example). What is the theoretical argument underlying it? What is the empirical evidence supporting it? What examples can you provide as to why it is included? Too many indicators assess things that are not well theorized, leaving organizations in the dark as to whether they should comply and how.
4. Distinguish between dimensions in the indicator set: In most indicator sets there are issues being raised that one could theorize as non-negotiable good governance conditions. Others are interesting conditions or practices that are actually negotiable--working well in some conditions and not in others. A third set of practices could be considered innovative and new, but untested. Where indicator sets don't distinguish between the three, organizations face the same pressure to conform with non-negotiable, negotiable and cutting edge elements. This is problematic.
5. Assess de jure and de facto aspects of key practices: If an indicator set has an impact, organizations will face pressure to conform with required practices. This conformity can involve 'ticking boxes' by making changes in form--when in fact the change intended is in function. When this happens, indicators can foster a gap between form and function that undermines legitimacy of the prescribed practices. To counter this, measure aspects of the practice that involve de jure changes in form as well as aspects of the practice that involve de facto implementation. Then measure the gap between the two.
6. Allow indicator analysis that has quantitative and qualitative dimensions: I don't like indicators that are just about numbers. A 1.5 this year and a 1.5 next year may not be the same. How do I know if there are side stories that need to be told? How do we learn about the fit of some indicator questions to context? I like indicators that provide both numbers and qualitative descriptions of the analysis underlying such. Contextual texture is important to understanding the indicator results.
Just my thoughts. I like Global Integrity data, Bertelsmann Transformation Index reports, and Economist Intelligence Unit reports because they satisfy quite a few of these criteria.