SPDI in Education: Online Tutoring

SPDI in Education: Online Tutoring

How can education retention be maintained, and even improved, despite a global pandemic? Researchers from the Harvard Kennedy School and the Laboratory for Effective Anti-Poverty Policies (LEAP) at Bocconi University in Milan sought to address this challenge by designing a free online tutoring program - Tutoring Online Program (TOP) - delivered by volunteer university students. Researchers found that the program had a positive impact on students' learning outcomes, socio-emotional skills, and psychological well-being. TOP now provides one-on-one and small-group tutoring services to thousands of students in Italy, the Dominican Republic, and Brazil.

 

Step One: Identify

The first step of the SPDI framework is to identify the policy problem at hand. Researchers saw that the Covid-19 pandemic was leading to learning loss in education across the globe, particularly within vulnerable populations, as schools were shutting down and turning to remote learning.

Step Two: Diagnose

The issue of learning loss was associated with slow responses from schools, a lack of support and supervision at home, overall disengagement and the psychological impact of lockdowns, and distanced learning being less effective than in-person learning, especially for younger students. Students from lower-income families were disproportionately affected, as they often had limited access to technology, received less support from parents and faced lower quality of remote learning from schools.

Step Three: Design

Michela Carlana and Eliana La Ferrara from the Harvard Kennedy School, with the support of the Laboratory for Effective Anti-Poverty Policies (LEAP) at Bocconi University in Milan, created the Tutoring Online Program (TOP) which went into effect in March of 2020. TOP offered free, completely virtual tutoring for sixth through eighth grade students from disadvantaged backgrounds who were identified by school principals and teachers among those lagging behind during distance learning. Tutors for the program were volunteer local university students.

Step Four: Implement

Throughout the first two weeks of March 2020, Italy experienced its first cases of Covid-19 and Covid-related school closures. By March 20, researchers began contacting three large universities in the city of Milan to request permission to advertise recruitment of voluntary tutors for the TOP program. Throughout the end of March and the beginning of April, the recruitment process continued for both tutors and tutees, with outreach outlining the program as well as the various qualifications necessary to enroll. Tutoring activities began in April 2020 and lasted for five weeks, until the end of the school year in June.

Step Five: Test

1,059 students from 76 middle schools from all over the country applied to the program and researchers randomly assigned 530 of them to be tutored, with the remaining 529 students serving as a comparison group. Students were stratified by school and grade, and those who indicated they needed help in more than one subject were randomly assigned six (instead of three) hours of tutoring per week. Online questionnaires were made available to parents, students, tutors, and teachers, asking about the students’ perceived academic outcomes and beliefs, aspirations, socio-emotional skills, and overall well-being.  For students in TOP, standardized test scores improved by 0.26 standard deviations relative to the comparison group. Students who had participated in the tutoring sessions answered more questions correctly, a 9 percent increase relative to an average of 53 percent correct in the comparison group. Researchers found that gains in school performance were greater for students with learning disorders, for students whose parents did not have a college degree, had blue collar jobs, or did not work from home. Researchers found that students enrolled in TOP exerted more effort in their homework and behaved better during online classes compared to their peers, according to reports by students, parents and teachers. Additionally, students in TOP were 4.9 percentage points more likely to enjoy the subjects of math, literature or English relative to the comparison group.

Step Six: Refine

Between June-September 2021, a new TOP pilot program was implemented in the Dominican Republic (TOP-DR). This program consisted of 103 tutors and 307 seventh and eighth grade tutees from 29 different schools. In this implementation, researchers tested group sessions. As in the first program in Milan, training and support was offered to tutors throughout, and additional professional workshops were offered to promote tutor retention. TOP-DR used a survey at the beginning and end of the program, as well as administering Math & Spanish tests, using the 8th Grade National Test (2016), and IQ Diagnostic Tests. From the surveys, researchers found that 68% of students had only one telephone for tutoring, 48% only had access to the internet via telephone, and 75% of tutors reported problems with the internet in at least one session. The lack of access to devices and the internet limited the scope of the intervention, and underscored the need for complementary efforts to improve access to devices and the internet. Between March-August 2022, the second edition of TOP-DR was implemented, this time with over 800 tutors and over 1,600 seventh through ninth grade student tutees. From the previous TOP-DR program, researchers saw it was challenging to coordinate with bigger groups, so this time they kept tutoring groups to one to two tutees per tutor. In this cohort, both students and tutors still faced internet connectivity challenges. As in previous iterations of the program, surveys were provided, and Math and Spanish tests, the 8th Grade National Test (2016), and IQ Diagnostic Tests were administered. Researchers are continuing to expand TOP pilots to new geographies and seeking new ways to address issues of connectivity, tutor recruitment, and tutor retention.