If policymakers learn to use data and research evidence in their decisions, it may improve policy, leading to better outcomes for citizens. Based on this theory, Evidence for Policy Design and other groups offer training to policymakers around the world. Yet such capacity-building efforts often encounter two major questions: how to deliver the content at scale, and how to determine whether it has an effect. EPoD is taking steps on both of these fronts.

First, delivering at scale.

EPoD developed a blended-learning model under a program titled  Building Capacity to Use Research Evidence (BCURE), funded by the UK’s Department for International Development (DFID) from 2013 to 2017. The training model employs a blended learning approach in which learners first complete online units, followed by in-class sessions. Importantly, instructors can adapt the in-class content based on learners' responses recorded in the digital units, meaning that material is changed based on a cohort's existing understanding of the material. However, it is neither cost-effective or feasible for Harvard faculty to deliver the material in a way that reaches a broad base of policymakers in developing countries.

EPoD achieved scale in India and Pakistan through its Training of Trainers program, empowering local faculty to teach BCURE content. The group brings local instructors either to HKS or regional workshops where Harvard faculty train them in the BCURE content and methodology. These instructors then serve as “ambassadors of evidence” by teaching in their ministry or civil service academy. Through this method, EPoD has trained both directly and through local trainers over 4500 civil servants in India (supported by the Gates Foundation) and in Pakistan (with support from the United Nations Development Program).

The most recent Training of Trainers event was held at the Kennedy School on July 29–31, 2019. The 20 participants from government, academia, and nonprofits in Pakistan and Bangladesh are now equipped to deliver courses in the use of data and evidence.

Second, the question of determining the effects of capacity-building efforts.

The BCURE units appear effective in imparting new knowledge to trainees. According to a learning assessment conducted at course completion, participants show average learning gains of a substantial .4 standard deviations. But as EPoD Senior Training Manager Charlotte Tuminelli points out, “We don’t have any sense of how these gains persist over time, or, importantly, how trainings are applied in the real world.” EPoD has gathered compelling stories from former trainees on how they used their new knowledge on the job, but a rigorous evaluation of the effects of training is not yet in reach.

A preliminary step may be to understand how to give policymakers information in a way that leads to better decision-making. To answer this requires looking closely at how people retain and employ information.

To place this question in context, private and public entities often delegate decisions to groups, but classic economic theory predicts that rational actors with complete information will make the same decision whether in groups or alone. The empirical literature shows that this is not the case: sometimes groups can make better decisions if they are able to effectively process information and eliminate bias; but sometimes they make worse decisions, for example if they fall prey to “group-think”, conforming to one biased opinion. Alongside, higher education research shows group work improves learning.

So, experimentation may lead to a better understanding of which forces win out, and whether learning in a group or alone leads to better decisions.

In June 2019, EPoD researchers started to shed light on this question when they piloted a classroom experiment with a group at the civil 60 senior civil servants at India’s premier civil service academy. A BCURE in-class session instructed by EPoD faculty member Teddy Svoronos provided the laboratory. Participants went through a two-stage exercise where they assessed whether a program would be likely to achieve its public-health objectives, first on their own and then, with the same question, in a group. EPoD researchers observed how subjects were able to retrieve knowledge and assess the efficiency of group collaboration by comparing results on how groups did against both the average individual and the highest-achieving individuals.

Preliminary results based on this modest sample show that groups did better than the average individual in retrieving knowledge to inform decisions, but they didn’t retrieve all the knowledge and didn’t retrieve as much as the highest-scoring individuals did.

Tuminelli and EPoD Postdoctoral Fellow Laura Metzger described their pilot and its  initial results at a number of talks around the Kennedy School in September. “It was surprisingly effective, given how senior the policymakers were in this session. We got some pushback at first, people saying, ‘This group will never answer the same questions twice.’ But it was received really well.”

The results from the pilot are compelling and could eventually improve EPoD’s training programs and possibly inform guidelines for evidence use in government – however, the experiment it did not allow the researchers to compare learning in groups against a control group. That is where the recently trained Ambassadors of Evidence come in.

In a series of sessions that those instructors will deliver later this autumn at Pakistan’s National School of Public Policy, the researchers will run the experiment again. This time they will collaborate with the Center for Economic Research in Pakistan (CERP) and Professor Adnan Khan from the London School of Economics. Given that the sample of civil servants will be larger, the researchers will implement a full study assigning certain subjects to a control group. This may offer insights on how people learn in groups and allow the team make training more effective.

Metzger explained during a talk sponsored by the Kennedy Schools Research Administration Office: “We aim to have a  portfolio of projects, and this might start to tell us how group interactions function in a public policy context . Then we can start to systematically manipulate  factors that might affect that: so, under which conditions do group decisions lead to better outcomes.”

The team hopes that this is the first in a series of studies that will shed light on how evidence can improve policy. Tuminelli says, “These pilots are generating buy-in and excitement among our collaborators. We’re hoping to use that to eventually do a rigorous impact evaluation of training.”