Engaging with teachers unlocks the potential of technology in schools: Evidence from Pakistan

Education Technology (EdTech) is a very attractive solution to the complicated problem of ensuring effective learning for all students. Despite improving access to schooling due to the removal of barriers to education including tuition and fees, children and adolescents (even those who are attending school) are not learning effectively, and are often behind grade level (Lucas and Mbiti 2012, World Bank 2018). At least some of schools’ underperformance and students’ lack of learning could be attributed to variations in teacher preparation, capacity, and ability (Andrabi et al. 2008, Muralidharan 2013, Jackson et al. 2014, Bold et al. 2019).

Existing methods attempt to address potential deficiencies in teaching by using one of two approaches. Some programmes bypass existing teachers by hiring assistants or tutors, in some cases creating separate, parallel schooling systems. Other programmes attempt to engage existing teachers through extensive, and often expensive, in-service training and monitoring, which has often had limited success unless it is implemented alongside intensive involvement from a non-governmental organisation. In Beg et al. (2022), we test these two ways in which technology could improve student outcomes, by either bypassing or engaging with existing teachers.

Engaging and bypassing teachers with education technology

In 2016, we partnered with the government of Punjab province, Pakistan, to test two alternative ways to increase student achievement in middle school by using education technology. The overarching government programme, eLearn, included brief videos in which teaching experts explained math and science concepts from the official middle school curriculum. The videos both explained concepts and demonstrated compelling ways to explain complex ideas, potentially teaching both teachers and students the material, and providing a teaching model that teachers could emulate. These videos, along with electronic copies of the textbooks, multiple choice review questions, and animated simulations, were loaded onto tablet PCs. Teachers received these tablets, along with training on how to use the tablets and integrate their use and content into their teaching practices. The way in which students accessed this content varied.

Using two randomised controlled trials, we considered two implementation models of eLearn – eLearn Classrooms and eLearn Tablets. Both versions had similar math and science content tied to the existing curriculum, but targeted students and teachers of different grade levels. Grade 8 teachers implemented eLearn Classrooms, while grade 6 teachers implemented eLearn Tablets.

In eLearn Classrooms, teachers were engaged as the content providers. In addition to the tablets and training, classrooms in the eLearn Classroom schools received 40 inch LED television screens, which were installed adjacent to or above existing chalk or white boards. Teachers projected the eLearn content from their tablets to the screens. The videos, which averaged 9 minutes, were not long enough to replace an entire lesson, and were instead designed to accompany in-person teaching on the same topic.

In eLearn Tablets, in addition to teachers receiving tablets and training, each student received a tablet for their personal use. Classrooms did not receive screens and teachers could not display the content to the students. Teachers could instruct students to view the short videos on their own either in class or at home. The tablets were also open for students to use for other purposes, such as downloading games from the internet.

We evaluated these interventions through two randomised controlled trials (RCTs), which allowed us to separate the effects of the two programmes from other factors that may have influenced outcomes within schools. Intuitively, simply observing schools, recording whether and how they used technology, and collecting student test scores, would provide the correlation between technology use and test scores. But schools that have access to technology might be different than schools that lack access – the school might also have better resources on other margins such as more effective teachers, more teaching staff, or an after-school tutoring programme. In a randomised controlled trial, receiving technology is determined randomly, almost like a coin flip, and therefore having technology is no longer related to school characteristics. Now that technology is unrelated to these other school characteristics, we can determine the effect of technology on student outcomes.

Our study included schools from three districts in Punjab. In one sample of schools, we randomly allocated half of the schools to receive the eLearn Classrooms intervention and the other half to be control schools i.e. remain with business as usual. In the other sample, we also randomly allocated selected schools to receive the eLearn Tablets treatment, while the remaining schools continued with business as usual.

Effects of EdTech on student learning

After approximately 4 months of exposure, our results show that eLearn Classrooms increased student learning, while eLearn Tablets decreased student learning. eLearn Classrooms increased students’ test scores on an exam that included both math and science content by 0.3 standard deviations (SD) relative to business as usual. Students in the eLearn Classroom schools learned 60% more than those in the control schools over the same period. In contrast, eLearn Tablets decreased test scores by 0.4 SD relative to the control schools. In the control schools, test scores increased by about 0.45 SD over this period, meaning that eLearn Tablet students learned about 95% less than control group students. Figure 1 shows these magnitudes.

Figure 1: Test score changes from baseline to follow-up

Notes: The tops of the orange bars illustrate how much the control group learned over this period – about 0.5 SD in each case. The tops of the blue bars are the total learning by the treatment groups – the eLearn Classroom students learned about 0.8 SD overall while the eLearn Tablet students only learned 0.05SD. Source: Based on Beg et al. 2022 Table 2.

Classroom centred EdTech impacts teachers’ effort and teaching practices

The effects on student learning were the sum of any direct effects, such as watching the provided content, and any other technology induced changes in behaviour. For example, teachers might change their own effort or teaching practices after seeing others explain complex material in a compelling way, or teachers might shirk, using technology as a replacement for their own engagement. Figure 2 shows the differences and similarities between the two interventions and highlights the potential direct and indirect effects.

Figure 2: Direct vs. indirect effects of the interventions

Source: Beg et al. 2022, Figure 2

The overall test score gains in eLearn Classrooms, and the relative test score reductions from eLearn Tablets, are both influenced by the direct effect of students and teachers engaging with the material. This factor was likely the same across the two interventions, as the effect of seeing a video should not be inherently different, whether it is on a personal screen or a larger classroom screen. Therefore, how the interventions were used and what other changes the programmes induced, i.e. the indirect effects, likely resulted in the large disparities we observe between these interventions.

These indirect effects include both teacher and student effort. Despite not directly observing teacher and student classroom behaviour, multiple other pieces of evidence point to teachers investing more effort and changing their own teaching practices due to the eLearn Classroom intervention. First, it is unlikely that such substantial test score gains were the result of only brief video content (and the negative effects of eLearn Tablets provides supporting evidence). Second, both teachers and students in eLearn Classroom schools were more likely to be present, and teachers increased their self-reported effort. Third, the largest test score gains were in the schools with the lowest baseline test scores, and for the students with the lowest baseline test scores. These were the schools and students for whom the video material (which was grade-level) was likely the least well suited without additional engagement by the teacher. Finally, the test score increases were larger in schools in which teachers had less experience and did not have a grade-level peer. Less experienced teachers could benefit more from seeing an expert teacher, and those without grade-level peers had less opportunity to learn from others in the absence of the videos. Therefore, the complementarity between existing teachers and the eLearn Classrooms intervention was likely crucial for its success in increasing test scores.

EdTech’s impact depends on its design and integration with existing teaching practices

The positive effects of eLearn Classrooms, relative to the negative effects of eLearn Tablets, shows the importance of integrating technology into existing teaching practices if policymakers want to unlock its full potential benefits. In the eLearn Classroom version, teachers could play the videos for the entire class, pausing as necessary, replaying if students were confused, and noting how students responded to different teaching methods. It combined positive aspects from previous studies on teacher professional development where the content created its own impetus for use, teachers had a skilled virtual colleague teaching the same grade level and material, and the program provided instant feedback on students’ understanding through review questions (Jackson at al. 2014, Jackson and Bruegmann 2009, Jackson and Makarin 2018, Papay et al. 2020, Berry et al. 2020).

eLearn Tablets decreased test scores even though it had similar content, potentially because students were not comfortable with the technology, used the tablets at home for non-scholastic purposes, or did not have appropriate spaces at home to use the tablets effectively (Fairlie and Robinson 2013, Escueta et al. 2017, Government of Pakistan 2014). We also heard from teachers that time was wasted during the class if students’ tablets were malfunctioning, not charged, or left at home. Importantly, eLearn Classrooms was a far more affordable intervention, as only the teachers received the tablets, costing $9 per student at scale, while eLearn Tablets would cost $131 per student at scale.

Conclusions

Technology that engaged teachers (eLearn Classrooms) improved student learning, while providing similar content on technology but bypassing teachers (eLearn Tablets) did not. The eLearn Classrooms intervention was effectively a stealthy form of professional development – it taught both the teachers and students curriculum level material, and likely developed teachers’ pedagogical skills.

Many government systems are struggling to make existing teachers more effective. The eLearn Classroom videos, and accompanying screens, effectively sent a high-quality teacher to each classroom, while also engaging with the existing teacher, highlighting the importance of the complementarity between technology and teachers for governments aiming to improve learning.

References

Andrabi, T, J Das, A I Khwaja, T Vishwanath, and T Zajonc, (2008), Learning and Educational Achievements in Punjab Schools (LEAPS): Insights to Inform the Education Policy Debate. Washington, DC: World Bank. 

Beg, S, W Halim, A M Lucas, and U Saif (2022), “Engaging Teachers with Technology Increased Achievement, Bypassing Teachers Did Not”, American Economic Journal: Economic Policy, 14(2): 61-90.

Berry, J, H Kannan, S Mukherji, and M Shotland (2020), “Failure of Frequent Assessment: An Evaluation of India’s Continuous and Comprehensive Evaluation Program”, Journal of Development Economics, 143: 102406.

Bold, T, D Filmer, E Molina, and J Svensson (2019), “The Lost Human Capital—Teacher Knowledge and Student Achievement in Africa”, World Bank Policy Research Working Paper, 8849.

Escueta, M, V Quan, A J Nickow, and P Oreopoulos (2017), “Education Technology: An Evidence-Based Review”, NBER Working Paper 23744.

Fairlie, R W, and J Robinson (2013), “Experimental Evidence on the Effects of Home Computers on Academic Achievement among Schoolchildren”, American Economic Journal: Applied Economics, 5(3): 211–40.

Government of Pakistan (2014), National Assessment Report. Islamabad: UNICEF.

Jackson, C K, and E Bruegmann (2009), “Teaching Students and Teaching Each Other: The Importance of Peer Learning for Teachers”, American Economic Journal: Applied Economics, 1(4): 85–108.

Jackson, C K, J E Rockoff, and D O Staiger (2014), “Teacher Effects and Teacher-Related Policies”, Annual Review of Economics, 6: 801–25.

Jackson, C K, and A Makarin (2018), “Can Online Off-the-Shelf Lessons Improve Student Outcomes? Evidence from a Field Experiment”, American Economic Journal: Economic Policy, 10(3): 226–54.

Lucas, A M, and I M Mbiti (2014), “Effects of School Quality on Student Achievement: Discontinuity Evidence from Kenya”, American Economic Journal: Applied Economics, 6(3): 234–63.

Muralidharan, K (2013), “Priorities for Primary Education Policy in India’s 12th Five-Year Plan”, In India Policy Forum, 9(1): 1-61, National Council of Applied Economic Research.

Papay, J P, E S. Taylor, J H Tyler, and M E Laski (2020), “Learning Job Skills from Colleagues at Work: Evidence from a Field Experiment Using Teacher Performance Data”, American Economic Journal: Economic Policy, 12(1): 359–88.

World Bank Group (2018), World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank.