How Transparent Are Open Budgets?

A couple of weeks ago Paolo de Renzio posted a great response to my post of a few weeks ago. The post in question used Open Budget Index (OBI) data to argue that governments are generally more transparent about the promises they make in formulating budgets than they are in actually executing budgets. As is usual, Paolo's response was thought provoking and indicated the seriousness of OBI generally. The tone of engagement was also positive and welcomed the discussion about how to make indicators like OBI useful. The one thing I would like to stress is that I already think they are incredibly useful; my post was not a critique of OBI but rather an illustration of how OBI's data can be used to support a pretty fundamental hypothesis about public financial management reform (that upstream change happens easier then downstream change…or formulation reforms are typically easier than execution reforms).

In his commentary, Paolo noted that some of my numbers did not gel with what they could calculate and he also raised questions about the general argument I was making. Paolo is right that I used the data a little differently to the way he anticipated when trying to replicate my method. When looking at transparency in formulation I only looked at the executive budget process (not the citizens budget or even the pre-budget statement, both of which I believe to be delinked from the formal budget preparation process in most of these countries and hence less relevant when talking about budget formulation and the theory about reforms in the formulation and execution processes). When looking at execution I examined in-year, mid-year, end-of-year, and audit report transparency. But I did not just look at the production of documents. I also included some data points reflecting the degree to which citizens were empowered to engage with the documents in the formulation and execution processes. (These are found in the question section between 113 and 117 if I remember correctly). Most countries scored poorly on these questions and did worse on the questions asking about execution processes. These are the main differences in our approach.

Beyond these differences, I do see some bugs in the data set I ended up analyzing, which manifest in the examples Paolo presents of Benin, Chad and Niger, but I think the general story I emerging from my analysis holds when using alternative methods. For example: Let me use a variant of the method Paolo himself adopts—looking simply at the production of documents in the budget formulation and execution processes. I use the OBI data explicitly here. Comparing how 21 countries did between 2008 and 2012 to get some flavor of patterns across countries and across time (all of these countries were assessed in 2008, 2010 and 2012). The following shows scores for different areas in 2008 and the change in scores between 2008 and 2012. OBI is the average overall score; EBP is the executive budget process; IYR is the in year reporting transparency; MYR is mid year reporting transparency; YER is year end reporting transparency; AR is audit report transparency; EXECUTION is overall execution transparency (the average scores across IYR, MYR, YER, and AR).

Country	OBI   2008	OBI   change	EBP   2008	EBP   change	IYR   2008	IYR   change	MYR   2008	MYR   change	YER   2008	YER   change	AR 2008	AR   change	EXECUTION   2008	EXECUTION   change
Angola	3.98	23.72	0.00	29.84	0.00	79.25	0.00	0.00	13.30	3.30	0.00	0.00	4.43	1.10
Botswana	62.00	-12.48	69.73	-8.78	75.13	-75.13	0.00	0.00	46.80	-20.10	57.14	-4.71	34.65	-8.27
Burkina   Faso	14.48	8.31	0.00	28.69	50.00	-50.00	41.50	-41.50	36.60	-13.20	33.29	-33.29	37.13	-29.33
Cameroon	5.45	4.73	0.00	1.72	0.00	75.13	50.00	-50.00	0.00	0.00	0.00	9.43	16.67	-13.52
Chad	7.98	-5.17	0.00	3.45	0.00	0.00	0.00	0.00	50.10	-50.10	19.00	-19.00	23.03	-23.03
DRC	0.74	17.36	0.00	26.33	0.00	25.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Ghana	49.64	0.16	67.24	-9.81	79.38	4.00	0.00	50.00	0.00	0.00	9.57	61.86	3.19	37.29
Kenya	57.66	-8.54	61.89	-6.72	83.50	0.00	0.00	0.00	40.00	-40.00	33.29	14.29	24.43	-8.57
Liberia	2.89	39.93	0.00	46.59	0.00	79.38	0.00	33.00	10.00	29.90	0.00	0.00	3.33	20.97
Malawi	27.53	24.79	43.64	21.33	0.00	83.50	0.00	50.25	0.00	0.00	4.71	28.71	1.57	26.32
Namibia	46.38	8.38	68.51	6.11	16.75	0.00	0.00	0.00	6.60	43.50	38.00	0.00	14.87	14.50
Niger	26.10	-22.58	32.11	-27.51	0.00	0.00	0.00	0.00	33.40	-33.40	0.00	0.00	11.13	-11.13
Nigeria	19.21	-3.45	30.91	-22.31	0.00	62.38	0.00	0.00	0.00	26.50	0.00	0.00	0.00	8.83
Rwanda	1.45	6.29	0.00	4.03	0.00	0.00	0.00	0.00	0.00	30.10	4.71	9.57	1.57	13.22
STP	0.73	28.04	0.00	37.33	0.00	71.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Senegal	2.89	7.28	0.00	0.00	16.63	79.25	0.00	0.00	0.00	0.00	14.29	-14.29	4.76	-4.76
South   Africa	87.46	2.63	88.09	4.33	91.75	8.25	100.00	0.00	80.10	3.30	76.14	0.00	85.41	1.10
Sudan	0.36	-0.36	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	4.71	-4.71	1.57	-1.57
Tanzania	35.52	11.18	40.58	11.75	79.38	4.13	0.00	0.00	0.00	0.00	38.14	4.71	12.71	1.57
Uganda	51.09	13.87	58.18	13.73	25.00	4.13	0.00	41.50	43.40	13.30	42.86	23.86	28.75	26.22
Zambia	48.17	-43.96	44.84	-44.27	79.13	-79.13	0.00	0.00	49.90	-49.90	38.14	4.71	29.35	-15.06
Averages	26.27	4.77	28.84	5.52	28.41	17.67	9.12	3.96	19.53	-2.70	19.71	3.86	16.12	1.71
Median	19.21	6.29	30.91	4.03	0.00	4.13	0.00	0.00	6.60	0.00	9.57	0.00	11.13	0.00
Number   countries scoring 0	0	0	10	2	11	6	18	15	10	8	7	8	3	2
Countries   with improved scores (out of 21)		14		13		12		4		7		8		10

What do I see in this table?

• First, 14 countries improved their OBI scores overall in this period. That is just under 70%. The average increase was from 26 to about 31 points. The median score improved from about 19 to about 25.5. Pretty impressive.
• Second, thirteen countries improved their EBP scores (transparency in formulation), with the average increase from about 29 to about 34.5. The median scores rose from 31 to about 35. Even better.
• Third, there has been remarkable improvement in the in-year reporting transparency, with 12 countries recording gains, and the mean rising from about 28 to about 46. Incredibly impressive.

  But now comes the worrying observations.

• Fourth, the median scores from in-year reporting seem incredibly different from the mean scores. The median score in 2008 was zero and this only improved to about 4 in 2012. That seems strange. But the reason is simply that there were many countries scoring zero in 2008 and many countries were not improving scores over time (a few pulled the mean up).
• Fifth, the 2008 scores and 'improvements' since 2008 look really worrying when one considers mid-year and end-of-year reporting and audit reporting; MYR mean scores increased from about 9 to 13 (but only 4 countries actually improved their scores…and 14 of the 18 countries scoring zero in 2008 still scored zero in 2012); EYR mean scores actually declined from about 19.5 to below 17 (with only seven improvers and 6 declining score countries); Only 8 countries had improved transparency in audit reporting. Ouch.
• Sixth, the performance in budget execution overall reflects these latter patterns; overall only 10 countries improved their scores with the mean going from 16 to about 17.8. Ouch again. (Especially considering the mean EBP went from 31 to 35…even worse when one considers medians as averages).
• Finally, most countries with higher OBI scores in 2012 had higher improvements in EBP scores than they had in budget execution scores (the countries with OBI change scores in orange or red fall into this category…which accounts for 9 of the 14 improving score countries)

So: A pattern seems evident (though not absolute). Transparency in execution seems to be lower than transparency in formulation in both absolute static terms and when thinking about dynamic reform and change.

But let me look at this a little differently, building the two-by-two matrix I introduced a few weeks ago.

I used OBI's averages for the Executive budget preparation process described above as the 'Transparency in Formulation (TIF)' number (Ghana scores 68, 68, and 57 in 2008, 2010 and 2012). Note I don't use pre budget statements for the reason stated simply above (I suspect this will be a point of comment and difference moving ahead, but I simply don't think that pre budget statements and citizens budgets are as central to the process as the other processes and believe that including them in this analysis biases the study). As in the table above, I calculated the average of the average scores for in-year, mid-year, and end-of-year reports, and audits as the 'Transparency in Execution (TIE)' number (In 2012 Ghana scores 51 on this measure, given relative scores of 83.375, 50, 0, and 71.4286). Note that averaging the averages means that all four areas get equal weight, which is not achieved if one sums up all the individual scores (as I did previously) because the different areas (in year reporting and auditing, for instance) have different numbers of questions.

Now I hope sincerely there are no glitches in my calculations. Given this hope, the following graphs capture country scores for 2008, 2010, and 2012 (looking at 23 countries with more than two years' data in the period). I did not include 2006 here because the sample was quite a bit smaller. (The summary table is below).

• In 2008, 7 out of 19 countries had higher scores for transparency in execution (and two scored the same). They are shown in bold. Note that they are all countries that scored low on both TIF and TIE. They are in the quadrant where they are simply not looking transparent at all. Ten countries had higher TIF scores than their TIE scores (and hence the kind of downstream gap I am focused on). Four countries scored above 50 on TIF but below 50 for TIE (Ghana, Uganda, Kenya and Botswana).
• In 2010, 6 countries scored higher on TIE than TIF (and 4 scored the same). They are shown in bold too, and all but one (Liberia) is in the low TIF and low TIE quadrant. Contrast this with 12 countries that exhibited a downstream gap. The countries with TIF scores above 50 but TIE scores below 50 now numbered 7 (Botswana, Ghana, Uganda, Malawi, Tanzania, Namibia, and Kenya).
• Now look at 2012, where only 4 countries had higher TIE scores than TIF scores (one had the same scores on each) and 17 countries had higher TIF scores than TIE scores. 8 countries had TIF scores greater than 50 but TIE scores lower than 50 (in the top left quadrant). Note that the distribution of countries is pretty much exactly the same as I showed in my prior post.

This story across both my analyses gels well with Paolo's analysis of the number of countries producing different kinds of documents. The story is not simple, but there is generally something here about countries (generally) scoring better on transparency in formulation than transparency in execution. And it appears that the gap between TIF and TIE either stays high or increases over time in many countries. (Looking at the numbers below, 3 countries had stubbornly high gaps and ten countries saw growing gaps between the first year of analysis and the final year of analysis).

Country	TIE 2008	TIF 2008	Gap	TIE 2010	TIF 2010	Gap	TIE 2012	TIF 2012	Gap
Angola	3.3	0	-3.32	6.675	35	28.33	23.96	29	5.04
Botswana	45	69	24.23	23	64	41	20	60	40
Burkina Faso	40	0	-40.34	13	0	-13	6	29	23
Cameroon	12.5	0	-12.5	0	0	0	21	1.7	-19.3
Chad	17	0	-17	0	0	0	0	3.4	3.4
DRC				5.2	0	-5.2	6.25	26	19.75
Ghana	22	67	45	47	68	21	51	57	6
Kenya	39	62	23	30	55	25	32	55	23
Liberia	2.5	0	-2.5	60	40	-20	38	47	9
Malawi	1.18	44	42.82	32	58	26	42	65	23
Mali				19	49	30	14	55	41
Mozambique				2	41	39	42	50	8
Namibia	53	69	16	46	72	26	48	75	27
Niger	24	32	8	0	0	0	5	5	0
Nigeria	0	31	31	16	13	-3	22	9	-13
Rwanda	1	0	-1	15	0	-15	11	4	-7
Senegal	8	0	-8	8	0	-8	24	0	-24
South Africa	87	88	1	90	93	3	90	92	2
STP	0	0	0	0	0	0	18	37	19
Tanzania	29	41	12	32	56	24	32	52	20
Uganda	28	58	30	36	62	26	48	72	24
Zambia	35	35	0	42	44	2	24	39	15
TIF > TIE			10			12			17
TIF<TIE			7			6			4

This isn't to say that there is a linear relationship here or even that I'm trying to say the OBI creates incentives for countries to focus on the TIF dimensions. But it does suggest that a long-standing argument in the public finance literature is being reflected in this data: governments use their budget formulation processes for signaling and do less so in their execution processes. This argument would have us expect that governments would generally score higher marks for transparency in formulation, and this is what we see. And we should expect that transparency in execution processes will be lower. And this is what we see (generally, in most countries). Hopefully the analysis is clearer here than before and it is apparent that the data does show a pretty interesting picture, and is supportive of Paolo's own analysis.

…Matt's claim that countries are more transparent in budget preparation than in budget execution does have some foundation. A more useful way to look at this is to simply check whether countries publish the various documents or not. The table below shows data for the 30 African countries included in the Open Budget Survey 2012, indicating how many of the three budget formulation documents, as well as of the four budget execution documents, are publicly available (PA), produced for internal use only (IU), or not produced at all (NP).

According to the table, it is indeed true that countries are on average more open about their planned budgets than they are about their actual ones. Sixty-seven percent of budget formulation documents are publicly available versus only 44 percent of budget execution documents. Two further things are also worth noting. First, two thirds of countries do publish In-Year Reports – as many as those that publish the Executive's Budget Proposal. Therefore, across Africa, some information on budget execution is publicly available as often as information for budget formulation. Second, 24 percent of budget execution documents are not even produced. This could be the result of conscious political choices but is also likely to reflect gaps in the institutional framework (i.e. the non-existence or non-functioning of a Supreme Audit Institution) and capacity weaknesses (i.e. lack of systems that allow for the production of timely and comprehensive fiscal reports).

Now unless my methods are terribly wrong, it seems like we are saying similar things. The question I have to follow is: "Does it matter if a country is more transparent in formulation than in execution?" I think so, but suspect that we may not all agree.

I agree the reasons for these observations are complex and do relate to measurement, institutional, capacity etc. issues. But let me push back on two of these reasons:

• If it is possible that OBI data is skewed towards having higher formulation scores (as Paolo suggests but I had not considered) because the survey asks many more questions about the executive budget process, I ask why this is so? Do we have fewer questions to ask of transparency in the execution process? Or is it easier to assess transparency in the formulation process, or to propose 'solutions' in this stage?
• If transparency is weaker in the execution process because countries have weaker institutional capability in execution, surely we need to be more vocal about such a situation. Execution processes are crucial and if they are lagging in functionality then one has to ask why and spend more time focusing on ensuring such functionality improves.

So, I anticipate that this will not be the final word on such subject, and that we might go around a few times looking at data etc., but beyond the methodology etc. I just have two lingering questions:

• Do we see stubborn gaps between TIE and TIF in many places?
• Does this matter?