Implausible incarceration data need to be addressed
Authors: Hans Gutbrod, Aaron Erlich
Journal: The Lancet Global Health (2013)

The recent study by Rachel Jewkes and colleagues in The Lancet Global Health—"Prevalence of and factors associated with non-partner rape perpetration: findings from the UN Multi-country Cross-sectional Study on Men and Violence in Asia and the Pacific"1Jewkes R Fulu E Roselli T et al.Prevalence of and factors associated with non-partner rape perpetration: findings from the UN Multi-country Cross-sectional Study on Men and Violence in Asia and the Pacific.Lancet Glob Health. 2013; 1: e208-e218Summary Full Text Full Text PDF Scopus (129) Google Scholar—does an impressive job of disentangling complex and important issues. In addition to providing data, the study advances research with a range of techniques to capture and analyse information about the perpetration of rape. However, on closer examination we were struck by results that we found implausible in ways that need comprehensive clarification. Table 2 of the study (page e211) shows an unusually high number of perpetrators stating that they had been incarcerated in prison. The investigators acknowledge that the "data suggest surprisingly high conviction rates compared with those in other countries", but could have done more to explain this puzzle. Although the CIs for the estimates of the number who went to prison are large, we believe it to be highly improbable that they contain the true population estimate. For example, of the Chinese respondents in the study, 22·7% (95% CI 20·2–25·1) state they have committed partner or non-partner rape. Of those, 15·6% (12·0–20·0) say they have been sent to prison. If these numbers are truly representative, roughly 35 out of 1000 Chinese men should have been incarcerated for rape at some point in their lives—a number that does not match other available data. According to the most recent population data (from 2010),2Chinese Population and Employment Yearbook for 2010 (via the All China Data Center at the University of Michigan).http://chinadataonline.org/Google Scholar a minimum of 7·1 million Chinese men (2·4% of 293 million men between 18 and 49 years of age) should have spent time in prison for rape, which is a conservative estimate that uses the lowest numbers of the CIs for both perpetration and incarceration. These 7·1 million incarcerations do not square with the reported 40 000 rape cases every year, according to recent research.3Ye Q Introduction to the issue of rape in China as a developing country.in: Westmarland N Gangoli G International approaches to rape. Policy Press, Bristol2011: 57-77Crossref Google Scholar At this annual rate, China would need more than 177 years to convict the number of incarcerated perpetrators suggested by the survey. Note that these estimates are extremely conservative in each case, since they are taken from the lowest end of the CI. If we use the actual percentages reported in Jewkes and colleagues' article and the number of rape cases solved in China according to a recent article in the New York Times,4Tatlow DK 2 Countries, 2 Struggles for Justice. New York Times.http://www.nytimes.com/2013/02/27/world/asia/27iht-letter27.html?_r=1&Date: February 26, 2013Google Scholar we arrive at 10·5 million men incarcerated for rape, and about 350 years of convictions needed for the survey data to be plausible. Similar discrepancies exist for Cambodia (about 500 cases of rape, attempted rape, and sexual harassment reported in 2009),5Macan-Markar M Cambodia: rape victims need better protection from new penal code. IPS News Agency.http://www.ipsnews.net/2010/03/cambodia-rape-victims-need-better-protection-from-new-penal-code/Date: 9 March 2010Google Scholar where, for the survey data to be plausible, more than 180 000 men should have gone to prison, again requiring more than 350 years to convict these perpetrators at 2009 rates. In Bangladesh, the data seem more plausible but still high. UN figures6United Nations Office on Drugs and Crime, 2011 data.http://www.unodc.org/documents/data-and-analysis/statistics/crime/CTS12_Sexual_violence.xlsGoogle Scholar suggest that about 12 000 police reports of rape occurred in 2005 (we did not find newer data), when the survey data suggest that 200 000 men should have gone to prison for rape, if the punishment reported in the survey really happened. Although within a more plausible range, the data could still be a stretch, since not all police reports of rape are likely to have led to a conviction, and at least some perpetrators should be presently incarcerated and thus are not part of the sample. The reason why this discrepancy matters is that corroborating data seem to be off by an order of magnitude. From our survey experience, there are three likely causes for large discrepancies of this type—in sampling, fieldwork administration, and questionnaire design. First, sampling could have captured a disproportionate number of previously incarcerated perpetrators. As the authors note, the surveys might not be nationally representative because they focused on particular districts. However, this fact would not explain the order of magnitude in the distortion, especially for Cambodia, China, Indonesia, and Papua New Guinea. Moreover, for the survey in Bougainville in Papua New Guinea—a region with a violent history—the suggested rates of incarceration (316 men out of 1000—ie, nearly a third of all men in the age group) are also extraordinarily high. In a context of ongoing conflict, weak institutions, and displacement of up to a third of the population,7Braithwaite J Charlesworth H Reddy P Dunn L Reconciliation and architectures of commitment: sequencing peace in Bougainville. ANU E Press, Canberra2010http://epress.anu.edu.au/apps/bookworm/view/Reconciliation+and+Architectures+of+Commitment%3A+Sequencing+peace+in+Bougainville/5301/ch07.xhtmlGoogle Scholar such a high percentage of men claiming to be incarcerated for rape cannot be due to a sampling bias. Thus, although sampling might be a concern, alternative explanations are necessary. A second possible bias could have resulted from fieldwork administration by comparatively inexperienced interviewers administering a sensitive questionnaire to respondents who do not participate regularly in surveys. In our experience, formal procurement rules can sometimes direct the award to the least expensive bidder who just about looks technically competent, rather than the most qualified. This situation could have had a significant effect. One easy test would be to do a comparison for interviewer behaviour on the basis of their unique identifying number, to see whether their results show a plausible distribution, to what extent there were logical errors or errors with skip patterns, and whether concerns about particular interviewers were monitored, addressed, and documented. The dataset, especially in its precleaned form, should be instructive in this regard. The third plausible explanation focuses on questionnaire design and translation, and how well particular questions were understood by respondents. Overall, this theory seems to be the most powerful explanation for the large divergence. Respondents might have understood the question about incarceration in several different ways. They might well have understood this question as asking whether prison would be a possible or even deserved punishment for rape. Translation issues could also explain that findings in Sri Lanka seem to be broadly plausible; the translation might have captured intent better in this country. Separate back-translations should be used as another test for this concern. All this said, the potentially troubling consequence of these explanations is that the broader research community must wonder whether this discrepancy is isolated. If, for example, the implausibly high incarceration claims are the result of an ambiguous translation that renders a question as a scenario rather than a report of personal experience, other findings, including the inferential ones, might also be sensitive to the measurement error caused by translation issues. This situation would cast doubt on other results of the survey. Thus, it would seem to be worthwhile to undertake a thorough assessment to establish the source of these unlikely results, ideally with the help of external reviewers, to establish that there is no spillover onto the plausibility of other results in this study. To make the questionnaires and dataset available would be a step in that direction. This clarification is all the more important since the findings have significant policy relevance, have received much public uptake, and will become the benchmark for research into sexual violence in the region and beyond. One general recommendation for The Lancet Global Health would be to publish ballpark budget breakdowns for survey efforts, since these data would make apparent whether sufficient funding has been made available to a study, and how exactly that funding has been allocated. Sufficient funding gives no certainty that surveys will succeed, but insufficient resources are a guarantee that they will fall short. To reiterate, we are writing from the point of sympathising with the survey effort and its concerns, with an appreciation for the challenges of administering a cross-national survey, and with the hope of contributing to the further development of the approach. Ambiguous translation could, for example, have understated other concerns, as much as it might have inflated claims of incarceration. We declare that we have no conflicts of interest. Prevalence of and factors associated with non-partner rape perpetration: findings from the UN Multi-country Cross-sectional Study on Men and Violence in Asia and the PacificRape perpetration committed by men is quite frequent in the general population in the countries studied, as it is in other countries where similar research has been undertaken, such as South Africa. Prevention of rape is essential, and interventions must focus on childhood and adolescence, and address culturally rooted male gender socialisation and power relations, abuse in childhood, and poverty. Full-Text PDF Open AccessImplausible incarceration data need to be addressed – Authors' replyHans Gutbrod and Aaron Erlich raise important questions about the UN Multi-country Cross-sectional Study on Men and Violence in Asia and the Pacific. They seek assurance that fieldwork was undertaken properly and the questionnaire translated accurately. These issues are important for any survey and we can assure them that we addressed these points very rigorously when designing the study. We trained our fieldworkers for at least 2 weeks and deployed them only after pilot data had been gathered and checked. Full-Text PDF Open Access