The importance of monitoring, evaluating and corrective feedback

Growing labour market segmentation in the West, and the perceived political and social benefits of youth employment in other parts of the world, have increased the demand for public programmes to improve worker skills. Recent comprehensive reviews of skills development programmes, however, show poor returns and emphasise the importance of better programme design. Impact evaluation techniques are argued to be a useful learning tool to improve policy rollout. Pakistan Punjab’s ongoing Punjab Skills Development Fund (PSDF) has utilised these techniques to increase the impact of skills training. The results are encouraging.

The government of Pakistan’s skill development fund

In response to growing fears of militancy spilling over into Punjab from the restive tribal areas bordering Afghanistan, the government of Punjab and the UK Department for International Development (DFID) set up a £50 million skills development fund to improve employability of vulnerable young adults in the four poorest districts of South Punjab. The five-year programme was given two objectives:

• to train 145,000 vulnerable young men and women from relatively poor income groups to acquire skills that increase their income; and
• given poor performance of the established public training programs, to develop private sector capability to provide training.

The fund was launched in 2011, as a not-for-profit government company. It has an independent, private sector-dominated board that provides policy oversight and assesses company performance. Operations are carried out by a market-hired CEO and his team of competitively selected employees.

Programme evaluation and course-correcting learning

The PSDF board added a third objective to introduce a culture of course-correcting learning to meet programme objectives. This entailed carrying out rigorous, ongoing evaluations of all critical stages of the training process to ensure that the two PSDF objectives are met in a cost-effective manner. This was made possible due to a unique collaboration between PSDF and a local research centre – the Centre for Economic Research Pakistan (CERP) – that drew on leading academics from Harvard, Princeton, the Institute of Development and Economic Alternatives (IDEAS), and the Lahore University of Management Sciences. The evaluations span demand for training modules, content of the courses offered, private training firms’ training bids, the quality of course delivery, trainee attendance, and the income impact of training. Endorsing the third objective, a well-funded technical assistance component (£1.5 million) was added to the programme. This, and the fact that leading academics have turned their attention to rigorous evaluations for improving development policy outcomes, has resulted in a robust culture of learning that informs the rollout of PSDF training modules.

The largest impact from evaluations is known to be at the first and the last stages of the training rollout – that is, a correct assessment of the demand for training and the impact of training on trainee income. In the poor districts, the baseline survey revealed strong household demand – 90% of the households responded that they will enrol at least one young adult, male or female. This was followed up by the offer of vouchers, a stipend of 1,000 rupees (about $10) per month per trainee for the four-month long courses, and information on the location of training centres. Since the vouchers could only be cashed at the end of the training month, the young adults would need to commit to the training sessions in order to receive payment. But, in the last stage evaluations, it was found that only 47% of the households took up the vouchers. Of those, only 5% enrolled in the program. The strong demand expressed earlier for training did develop into strong enrolment rates.

Subsequent interventions and their impact

Based on information gathered from households in a subsequent survey on the reasons for the low uptake, the research team carried out a pilot small-sample randomised controlled trial to compare three interventions:

• standard information to households on the objectives of the training program and its design;
• community-based social mobilisation to encourage enrolment; and
• in-village training.

Female potential trainees were targeted, because they had the lowest uptake and enrolment rates. The results are encouraging (see Figure 1). Compared to the standard procedure (information only), social mobilisation managed to raise enrolment (completion) by a factor of six (14), while village-based training had an even more dramatic effect – increasing enrolment (completion) by more than a factor of 13 (31)! These results have led to subsequent design refinements of the programme, by introducing additional interventions such as group transport to address access constraints faced by women (Cheema et al. 2017).

Figure 1 Training course enrolment and completion by treatment type

Ongoing assessments of the impact of training on income also suggest that interventions that link the trainee (who acquires skills to produce for the market) to the buyer are needed to realise the full welfare gains of training. While the impact evaluation is ongoing, preliminary trainee tracer surveys revealed that the provision of training alone can increase income. However, initial results suggest that this increases further with interventions that bring the trainee closer to the market by facilitating subcontracting arrangements with buyers. The income impact of training is therefore likely to be substantially higher with further deepening of these links, once established.

Programme report card and the impact of evaluations

PSDF completed its five years in June 2016. A total of 147,599 trainings have been delivered (the target was 145,000). Female trainings were 38% of the total (the target was 40%). Total private training companies engaged by the program to deliver training increased from 29 companies in 2011-12 to 214 companies in 2015-16. The operational cost of the program, capped at 7%, was under 5%. This would not have been possible without the course-correcting evaluations that improved the design of the programme.

Editor's Note: A version of this piece was originally published on the Brookings Institution’s Future Development blog.

References

Cheema, A, A Khwaja, F Naseer and J Shapiro (2017), “Crossing boundaries: Experimental evidence on access constraints for rural women”, CERP Discussion paper.