Does bottom-up accountability work? Evidence from Uganda

Under-five mortality rates in low-income countries are about 14 times higher than in high-income countries.¹ In sub-Saharan Africa, more than half of these deaths are caused by diseases that could be treated or prevented if children had access to relatively inexpensive services.² Improving local access to these services is therefore a priority for the region. But how best to do this?

A bottom-up approach

In recent years, the development sector has embraced a potentially promising approach: the bottom-up monitoring of service providers by community members. The idea is to provide citizens with information about what is not working at their local clinic. This could be: 

• how frequently staff are not showing up, 
• how often the clinic is out of key medicines, 
• which services that are supposed to be offered are not being provided, or
• how satisfied or dissatisfied patients are with their care. 

Along with this is information allowing them to compare local outcomes with national standards and with outcomes in other communities. It is thought that receiving such information will put citizens in a better position to monitor and apply pressure on underperforming service providers. This, in turn, will generate improvements in service delivery and better health outcomes. 

Power to the People

A landmark 2009 study called Power to the People (P2P), which randomised the provision of precisely this kind of information across 50 rural health centres in Uganda, generated results that seemed to strongly validate this bottom-up accountability approach (Björkman and Svensson 2009). In health clinics that were exposed to the intervention: 

• infant weights increased, 
• under-five mortality declined by 33%, 
• immunisation rates rose, 
• waiting times fell,
• staff absenteeism dropped, 
• utilisation increased, and
• communities became more engaged and monitored health providers more extensively.  

A follow-up study showed that these positive effects persisted four years after the initial intervention (Björkman Nyqvist et al. 2017).

The P2P findings generated enormous interest.  In an attempt to confirm the findings, and also to better understand the mechanisms that underlay them, we teamed up with Innovations for Poverty Action and GOAL to assess the impact of an intervention modelled on the P2P programme in a similar setting – public health centres in rural Uganda – but on a larger scale. The programme, Accountability Can Transform Health (ACT Health), expanded from the prior study’s 50 government-run health centres to 376 centres, distributed across 16 districts in all four regions of the country (Raffler et al. 2019).

Breaking down the ACT Health programme 

We expected to find similar results, and also wanted to understand the mechanisms through which they worked. We collected data on a host of intermediate outcomes, such as whether: 

• citizens became more efficacious, 
• they took greater responsibility for monitoring healthcare providers’ performance, and
• the relationship between citizens and clinic staff improved. 

We also broke the intervention into its parts to try to better understand which aspects of the complex P2P programme were responsible for the dramatic improvement in health outcomes.

We launched ACT Health exactly ten years after the original P2P programme, a period during which Uganda had seen a significant decline in child, infant, and neonatal mortality rates. Under-five mortality rates were cut in half between 2006 and 2016. Nonetheless, the country still falls in the bottom quartile of the world distribution on most basic health measures and health service delivery is generally poor. Although public health services in Uganda have been free of charge since 2001, health staff are underpaid and facilities are poorly funded. Resource management tends to be weak, resulting in missing medicines and high absenteeism rates among health workers. In short, notwithstanding the significant improvements in health outcomes and service delivery, there was still ample room for improvement at the onset of our study.

We conducted a randomised evaluation to measure the impact of the full ACT Health programme (as well as individual components of the intervention) on utilisation rates, treatment quality, patient satisfaction, and health outcomes, including child mortality. The ACT Health intervention consisted of three components:

1. Information. Citizen report cards (CRCs) were provided to health centres containing information about citizen attitudes about their care and treatment, utilisation rates, and services offered at the health centre, alongside district averages for comparison. 
2. Mobilisation. The CRCs were then discussed during community meetings with local leaders and village health team members, leading to the development of community-specific action plans detailing steps to improve service delivery. Parallel meetings with health centre staff were also organised to identify steps that the staff could take to improve care and treatment. 
3. Interface. The health centre staff were then brought together with representatives of the community to discuss how they might work together to improve the quality of healthcare in the community. The output of the meetings was a social contract between the citizens and healthcare workers laying out specific steps that each could take to contribute to improving health outcomes. 

ACT Health implementation teams returned to the treated health centres every six months (for a total of three additional visits before endline data collection). They met with community members and healthcare staff and checked on the progress made toward the commitments in the social contract.

We randomly assigned the 376 health centres and their catchment areas to one of four groups:

1. the full ACT Health programme comprising of the information, mobilisation, and interface interventions,
2. only information and mobilisation,
3.  only the interface meeting, and
4. a control group, with no intervention.  

The research team conducted three rounds of data collection using a household survey, a health clinic survey, and administrative data from the health centres (verified with physical checks of drug stocks and storage conditions). Surveys were conducted before the programme was rolled out, and one and two years later. 

Marginally better care

Overall, communities participating in the full ACT Health programme reported receiving marginally better care and were slightly more satisfied with their care than communities that did not receive any programme. However, in striking contrast to what was found in P2P, no version of the programme improved health outcomes (including child mortality) or increased the number of people who sought care. Equally important from the standpoint of the mechanism thought to be driving the effects in P2P, we find no evidence that exposure to the programme increased citizen monitoring or bottom-up pressure exerted by community members.

Explaining the contrasting results of the ACT Health and P2P programmes

We hypothesise that the most likely reason is the big difference in baseline conditions across the two projects. Health conditions and treatment quality improved markedly in the ten years that separated the two studies. It may be that the positive treatment effects found in P2P were simply more difficult to achieve once baseline levels had risen beyond a certain point.  Indeed, when we restrict the ACT Health sample to health centres whose baseline child mortality rates were within one standard deviation of those reported at baseline in P2P, we find positive treatment effects on child mortality (although not on utilisation, treatment quality, or other health outcomes). 

Also consistent with this line of argument is the fact that Christensen et al. (2018) find treatment effects on utilisation and child mortality (but not treatment quality or other health outcomes) in a P2P-style intervention in Sierra Leone – a place where child mortality rates are even higher than in Uganda at the time of P2P. One possibility is that these interventions work by getting more people to utilise the formal health sector. The implication seems to be that information-based, bottom-up accountability interventions become less effective as baseline health conditions, utilisation, and the quality of health service provision improves. 

Policy implications: Bottom-up pressure has important limitations

Changing bottom-line health outcomes and mobilising citizens to apply bottom-up pressure to affect such changes, is extremely difficult. Citizens often have limited leverage over healthcare providers, low effort by health workers may not be the binding constraint, and collective action problems are hard to overcome. As much as it is tempting to believe that one can harness transparency and citizen engagement to improve service delivery, this may not be the most powerful strategy for improving the quality of healthcare and other services in developing country settings. It may move the needle when health conditions are extremely poor, as they were at the time of P2P.  But wringing improvements out of health systems that have the (higher but nonetheless inadequate) levels found in places like Uganda today – and in the majority of contemporary developing countries – may require a different approach. 

References

Björkman, M and J Svensson (2009), “Power to the people: Evidence from a randomised field experiment on community-based monitoring in Uganda”, The Quarterly Journal of Economics, 124(2): 735-769.

Björkman Nyqvist, M, D de Walquen and J Svensson (2017), “Experimental evidence on the long-run impact of community based monitoring”, American Economic Journal: Applied Economics, 9(1): 33-69.

Christensen, D, O Dube, J Haushofer, B Siddiqi and M Voors (2018), “Healthcare delivery during crises: Experimental evidence from Sierra Leone’s ebola outbreak”, Working Paper.

Raffler, P, D Posner and D Parkerson (2019), “The weakness of bottom-up accountability: Experimental evidence from the Ugandan health sector”, Working paper.

Endnotes

[1] Source: WHO (https://www.who.int/gho/child_health/mortality/mortality_under_five_text/en/)

[2] Source: WHO (https://www.afro.who.int/health-topics/child-health)