In a recent post, Tim Ogden and I discussed the importance of having solid, reliable data on which to base program evaluations and policy decisions. The Journal of Development Economics explored this theme in last year’s Symposium on Measurement and Survey Design which featured more than a dozen papers on improving data quality in development research (Hat tip to Berk Ozler of the World Bank’s Development Impact blog for pointing us to it).
An important discussion at the symposium was the extent to which self-reported data can be considered accurate and reliable. Because study participants are usually asked to report information after significant time has elapsed, self-reported data are often subject to recall bias and can be inaccurate or misleading. This post is the first in a three-part series that will explore the reliability of self-reported data through a discussion of papers featured at the symposium.
Jishnu Das, Jeffrey Hammer and Carolina Sánchez-Paramo investigate the reliability of self-reported health data in “The Impact of Recall Periods on Reported Morbidity and Health Seeking Behavior.” The researchers measured the impact of different recall periods on health data gathered across income groups. Between 2000 and 2002, they administered weekly and monthly-recall questionnaires to 1621 individuals in Delhi, India with questions about morbidity, doctor visits, time spent sick, use of self-medication and days of work or school missed due to illness. In 2008, the team augmented this information with a 2-month experimental evaluation on a sub-sample of households.
The study showed that varying the recall periods produced different self-reported results, and that these results differed across income groups. In the monthly surveys, both rich and poor households reported much lower rates of illness, lower use of health care services and less labor impacts than they reported in the weekly surveys. While this was true for all households, the effects were particularly pronounced among poor households: in the monthly surveys, poor households’ morbidity burden was halved and over a third of doctor visits and almost half of all self-medication episodes were forgotten. The authors hypothesize that this is because the poor, more so than the rich, normalize illness so that episodes of sickness are no longer perceived as “extraordinary events,” but rather as a part of everyday life. Overall, the study showed that self-reported health data are highly subject to recall bias and that these effects are more pronounced among poor households than rich households.
Kathleen Beegle, Calogero Carletto and Kristen Himelein also explore the significance of recall bias in a related study, “Reliability of Recall in Agricultural Data.” For logistical reasons, agricultural data are usually collected during a single visit to a household rather than through multiple visits throughout the farming season. In large multi-purpose surveys that are administered over a 12-month period, households can be asked to recall harvest information from 11 months earlier. Information reported about earlier events may be more susceptible to recall bias and less reliable than data reported about more recent events.
Drawing upon information collected from three 12-month household surveys in East and Southern Africa, Beegle et al. measure the extent to which recall bias affects agricultural data. They regress data about harvest sales and input use (fertilizer and hired labor) on the time elapsed between the harvest and the date of the interview to detect systematic under/over reporting based on the length of the recall period. In general, they find little evidence of any significant recall bias: information about harvest, crop sales, and input use is not significantly different when collected more than 8 months later as opposed to just after the harvest. The authors suggest that while their findings do not speak to the overall quality of self-reported data, they do allay concerns about the length of recall periods in the reporting of agricultural data.
While the first study found that self-reported health data are undermined by recall decay, the second study found that self-reported agricultural data are not particularly affected by recall bias. These differences may be explained by the salience hypothesis. The salience hypothesis states that events that are of greater salience to the respondent are less likely to be affected by recall decay. As Das et al. suggest, households, especially poor households, may normalize illness so that it no longer appears as an aberrant event, but rather as a part of everyday life. Illness therefore becomes less salient and more susceptible to recall bias. We do not see the same degree of recall decay in the reporting of agricultural data because farming represents many families’ main source of livelihood. Agricultural events, even those that occurred 11 months ago, remain quite salient so households have less trouble recalling them after significant time has passed. For these households, agricultural events may be more salient than illness-related events and therefore less subject to recall bias when recalling them several months later.
While recall bias – and the extent to which it undermines self-reported data – remains a concern, these studies shed valuable light on the ways in which recall bias may have more pronounced or muted effects on different types of self-reported data. These findings have important implications for health and agricultural survey methodology and the ways in which recall bias can be mitigated in the collection of self-reported data.
Stay tuned for subsequent posts about additional threats to the reliability of self-reported data and ways in which these threats can be remedied through survey design.