Identifying Survey Respondents: Testing Alternatives to a Census
High-quality data are essential in conducting evaluations for development, but broad surveys can be prohibitively expensive. Jessica Creighton and Wayan Suriastini worked with EPoD faculty affiliate Dan Levy to test low-cost innovations, and describe them in this guest post.
While health services have expanded across Africa, Southeast Asia, and other parts of the developing world in recent decades, reductions in maternal and newborn-related deaths – and improvements in maternal and newborn health more generally – have been slow. World Bank estimates put the maternal mortality ratio at 237 deaths per 100,000 live births in low and middle-income countries, compared to just 10 in high-income countries.
The Transparency for Development (T4D) Initiative —a joint effort of the Ash Center at the Harvard Kennedy School and Results for Development— is evaluating the effectiveness of development programs that seek to improve maternal, newborn, and child health. As part of this effort, T4D is collecting household survey data in rural villages in Tanzania and Indonesia.
High-quality data is essential in conducting evaluations, but data collection can be prohibitively expensive. This is particularly true when it comes to identifying survey respondents in countries with poor administrative data, which is often the case in the communities that most need essential aid and policy improvements. Researchers typically identify respondents by taking a comprehensive census of everyone in a sample area through direct home visits, but challenges such as poor road networks, unclear village boundaries, and people not at home make this inefficient. Is there an easier way to identify potential survey respondents? T4D explored two alternative strategies and compared the results.
In late 2013, the T4D research team and local partners CHAI (in Tanzania) and PATTIRO (in Indonesia) selected maternal and newborn health as the focus of the T4D program and evaluation in the two countries. This focus meant unique implications for the research design: 1) endline data collection would be pushed out by at least nine months to account for the gestation period, and 2) we needed a listing strategy to identify mothers who gave birth in the past year. Since we planned to collect both baseline and endline data, we first considered pre-selecting women whom we thought might have a baby in the year after our intervention. One option – to sample from women who were of childbearing age – didn’t make practical sense, as our sample sizes would exceed budget and practicality. A repeated cross-sectional design better met our needs, meaning we would:
- Identify women who had given birth within the past year for our baseline, and;
- Select a new cohort of respondents (using the same criteria) for our endline sample, rather than trying to interview baseline respondents again at endline.
The next challenge was to identify respondents who matched our sampling criteria – women who had given birth in the past year. We obtained quotes from survey firms to conduct a complete village census, but they were beyond our budget. After extensive brainstorming, we determined two options: 1) a modified version of a census – a ‘door-to-door listing’ or 2) a key informant listing, in which we asked knowledgeable members of the community to help identify potential respondents.
The key informant approach was the more cost effective option, but had higher potential for error – the method could miss eligible respondents, or under-sample women who had given birth at home either because of social desirability bias, or because certain informants (such as midwives) may only be aware of women who had contact with the formal healthcare system. Under-sampling would be problematic, as birth at a facility was a primary outcome for the T4D evaluation.
Baseline household listing approach
The T4D team used the door-to-door approach to identify eligible household respondents for baseline data collection. To cut costs, we randomly selected sub-villages from each sample village to carry out the listing. Additionally, rather than conducting a census of each individual at each house, data collectors gathered information only for household members who had given birth in the past year. If no one had, they would simply move on to the next house. A detailed description of the listing approach (along with information on the T4D intervention, evaluation design, and characteristics of the study population) can be found in the T4D Baseline Report.
Mini experiment in Indonesia
To find out whether the door-to-door approach was worth the additional cost, we worked with Wayan’s team at SurveyMETER to embed an experiment into the Indonesia listing.
We randomly selected 30 villages to revisit, and the survey team (made up of individuals who had not previously collected data in these three villages) conducted the listing using a key informant approach. For each village, the survey team identified a list of potential respondents and then compared it to the first listing (which the team had with them, in sealed envelopes). When the lists didn’t match, the team identified the reason for each mismatch, and ineligible respondents were excluded.
The surprising results!
Combining both lists, we identified 925 unique eligible respondents. While the door-to-door approach only identified 773 (84%) of them, the cheaper key informant approach found 849 (92%). Furthermore, the finding on the distribution of home versus facility births was similar for both – home births were 35.2% in the door-to-door sample, and 36.9% in the key informant sample.
We anticipated that the door-to-door listing would be more comprehensive than the key informant strategy, so these results were unexpected. Why did we miss so many respondents in the door-to-door listing? Due to resource constraints, if nobody was home, the survey team asked a neighbor whether a household member met the criteria. It is possible that villagers did not know – or did not want to report – births in their neighbors’ homes. It’s also possible that some homes were skipped due to sub-villages’ often ambiguous borders. However, even if this process could be improved, the key informant listing results are strikingly close to the combined results.
What worked well with the key informant listing, and why?
Firstly, we included informants who were outside of the formal healthcare system, such as sub-village heads and traditional birth attendants (TBAs, a term that refers to uncertified midwives who sometimes help with home delivery). This helped us mitigate against under-counting respondents who gave birth at home. Further, we were careful not to approach TBAs through formal health workers. We worded our questions to TBAs in a way that emphasized pregnancies and babies, and not birth (to guard against TBAs ‘hiding’ home births, which are discouraged by the government). We also checked with the village office for official birth records. Record keeping and availability of informants varied by village, so ensuring we talked to multiple informants was important.
Is an approach other than a village census pragmatic? This experiment illustrated that in some cases, it can be a reliable, efficient decision. It is of course critical to take into consideration a communities’ unique context. For example, it’s possible that the circumstances that enabled a reliable key informant listing are unique to Indonesia, which has a particularly strong nationwide community-based health program known as posyandu — of which one major focus is monthly antenatal care outreach services delivered by a village midwife and community health volunteers. These midwives and health volunteers served as our key informants. Since posyandu does not exist in Tanzania, and may not exist in other countries, we will need to consider this when deciding which approach to take for the endline, and whether to take differing approaches in Tanzania and Indonesia. But a clear lesson was that the more expensive, and ostensibly more comprehensive approach did not yield the advantages we were expecting, indicating that a well-designed key informant process can be equally reliable and less expensive than a door-to-door approach.
About the authors
Jessica Creighton is the Assistant Director of the Transparency for Development Initiative at the Harvard Kennedy School’s Ash Center for Democratic Governance and Innovation.
Wayan Suriastini is Executive Director of SurveyMETER, a research institute and data collection firm based in Yogyakarta, Indonesia.
This article first appeared on Global Partnership For Sustainable Development Data.