Developed countries base policy on hard evidence as an effective method for improving governance. In the United States alone, policymakers have used rigorous trials to reform welfare, set teacher pay, determine minimum wage, and optimize early education for maximum impact. Data analysis has become essential for the Centers for Disease Control to track disease, the Department of Transportation to control traffic at ports, and a number of governmental bodies to detect fraud. 

Given the successes in using data and research in the US government, imagine the possible benefits of evidence-driven policymaking in highly populated countries rising into middle-income status, like India and Pakistan. 

A crucial first step in creating a culture of evidence in government is building policy actors’ skills in its use. Policymakers often possess the drive to use data, but lack the technical abilities and appropriate frameworks to do so effectively. Dan Levy, faculty affiliate at Evidence for Policy Design (EPoD) at Harvard Kennedy School (HKS), describes the benefits of evidence use among policymakers: “Evidence allows policymakers to make better decisions, use public resources more effectively, and garner support for programs and policies that make a difference in people’s lives.” This is the core idea behind the Building Capacity to Use Research Evidence (BCURE) program being implemented by EPoD, with funding from UK aid from the UK government: Life will improve for citizens if policymakers are trained in skills for using evidence. 

EPoD is a team at HKS that seeks to improve the lives of citizens by using the tools of data and economics to design and enable better policy. In 2014, EPoD received funding to launch its BCURE program to build capacity of policymakers in using data and research evidence in policy decision-making. To maximize impact, EPoD needed to be able to offer trainings on a very wide scale. EPoD aimed to sustainably expand their training offerings, and massive open online courses (MOOCs) seemed like a promising way to do just that. 

2014 was a pivotal year for MOOCs. Coursera co-founder Daphne Koller declared 2014 the year that MOOCs would “come of age,” and the number of MOOCs offered internationally increased by 91% in the first half of the year alone. Prominent institutions like Harvard Business School were launching their own MOOC projects, and the marketplace for MOOCs was growing competitive, with edX, Coursera, and Udacity all well-established players on the scene. 

Levy, who is also Faculty Chair of the Strengthening Learning and Teaching Excellence (SLATE) initiative at HKS, had been tasked with making EPoD’s BCURE trainings in South Asia more sustainable. At the time, EPoD’s faculty affiliates regularly traveled to deliver the content in person—a model in which trainees were clearly learning, but at high costs in time and energy. The group wanted to continue delivering some in-person trainings, but they realized they needed another, more scalable, approach as more institutions in more countries requested trainings. Levy saw this as an opportunity for EPoD “to leverage online learning to make our efforts more sustainable.” 

For several reasons, a standard MOOC didn’t seem like the perfect fit. “MOOCs, generally, assume that a learner has 10–15 hours per week to sit in front of a computer, and we knew that for our group of learners, that would just not be possible.” Levy points to evidence that this assumption may be wrong not just for policymakers, but also for the average MOOC user: “You can see, in the MOOC world, the large numbers of people who sign up for a course and never complete it.” When the team started to design EPoD’s online courses, they saw opportunities to present a differentiated offering. 

EPoD and SLATE turned to a blended-learning format, which promised to use online classrooms—and in-person classrooms—in a smarter way. This included acknowledging the needs of the audience: “These were mid-career professionals, and we thought it would be hard to engage them only online.” Pedagogical principles were at play, too: in any classroom situation, Levy says, “You want the learner to be able to push back, and that’s very hard to do online.” EPoD’s decision to use blended learning thus stemmed from the dual recognition that online components would make training efforts more sustainable, but online-only courses would prove a challenging format for EPoD’s training audience. As Levy puts it, “We could achieve the most with a combination of the two mediums.” 

Early assessments conducted by EPoD and SLATE helped to ascertain trainees' needs - a key phase to the project, according to Levy. The team collected survey data, and met with Indian and Pakistani counterparts to think through the learning experience. The survey and meetings drove the selection of topics for the courses, the technology used to deliver content, the ratio of online to in-class learning, and the manner in which the digital units address learners. 

To increase units’ marketability for trainees across contexts, the design team (including faculty Chris Roberts and Teddy Svoronos, learning designers Maria Flanagan and Mae Klinger, and content developers Josh Yardley Catherine Murphy, and Sarah Oberst among others) targeted content and tone to senior folks. “It’s easier to target content to a high-level decision-maker and then have a more junior person consume that content, than working the other way around,” notes Levy. “Junior officers aspire to be dealing with the kinds of problems that more senior people are dealing with.” The resulting BCURE units are relatively short—requiring roughly two hours of learners’ time apiece—and acknowledge participants’ prior knowledge and experience. 

The flipped-classroom model, as it has been traditionally conceptualized, involves two steps: putting in-class content online, and doing something different with “live” classroom time. For Levy, “there is a key step between these two, often missed in implementation: incorporating learner data.” When online courses have an interactive component, instructors can use data to motivate and guide in-class sessions, resulting in a richer learning experience. Quantitative and qualitative data collected in the BCURE units allows instructors to gauge students’ understanding of particular concepts, as well as their interests and attitudes. As Levy puts it, data from the online units provides instructors with a “window into students’ minds before you even step into the classroom.” An added bonus is that participants are often excited to see their own data, further piquing interest in the possible benefits of data-driven approaches. 

Capitalizing on this advantage of blended learning, EPoD paired each digital unit with an in-person classroom session to reinforce the lessons learned. The online component helps trainees learn some rough vocabulary on the topics being discussed, but the in-class component allows learners to have a conversation using that vocabulary. Feedback and test data collected over the past two years indicate that trainees are, indeed, learning the units’ key concepts—and becoming excited to incorporate these into their work. 

Critical to scalability and sustainability, EPoD has also trained local faculty in countries of focus to deliver BCURE in-class sessions, further reducing the need for Harvard faculty and staff to travel for every training. This “Training of Trainers” program empowers local instructors to become experts in BCURE materials, and will play a large role in the future of the BCURE courses. Trainings are currently deployed in Pakistan at an impressive rate—to date, local instructors have trained over 1,300 individuals in-country—and course completion rates are high, especially when compared with average MOOC completion rates. Incentives play a key role in this success: with EPoD’s BCURE content integrated into top training institutions in India and Pakistan, participants are motivated to apply effort to the units as they do to other courses. 

With these incentives in mind, EPoD has sought additional opportunities to embed, improve, and evaluate delivery of BCURE content. Levy notes, “Initially…the goals of the project had to do with [pure] knowledge: do participants know more? We then extended the goal—we wanted learners to then do something differently as a result of learning.” Measuring such changes in practice is EPoD’s next big focus, as the group explores how policymakers are incorporating learnings on evidence use in their decision-making. As trainees’ organizations increasingly recognize the value of evidence-based decision-making, EPoD is collecting more and more stories of increased evidence use and data integration from BCURE trainees. 

Considering the evolving needs of its target audience, EPoD is also exploring tailoring its online content for mobile use. EPoD will continue to adapt its trainings to reflect technological advances, but the group is confident that blended-learning remains the right strategy for meeting learners where they are—and getting them to where they want to be.