Do we know the effectiveness of behavioural interventions?
Authors: Sevgi O Aral, Thomas A Peterman
Journal: The Lancet (1998)

During the past decade the increased emphases on "evidence-based medicine", "quality of evidence", and "prevention effectiveness studies" raised the question of how well we know the effectiveness of our prevention efforts. Ultimately, our knowledge of the effectiveness of behavioural interventions is determined by the comprehensiveness of the behavioural-intervention research that has been done, the representativeness and completeness of published research findings, and the scientific rigour and appropriateness of the research. Behavioural-intervention research into the prevention of sexually transmitted diseases (STDs) and HIV started only in the late 1980s and has been greatly under funded. In fiscal year 1996, only 8·1% of the funds allocated to AIDS research by the US National Institutes of Health were for behavioural intervention research, and in fiscal year 1997 this proportion decreased to 7·7% (J Auerbach, Office of AIDS Research, NIH, Bethesda, MD, USA, personal communication). It is reasonable to expect that other research institutes in the USA and elsewhere allocated even fewer resources to evaluating the effectiveness of behavioural interventions that prevent the spread of STDs and HIV. The research that has been conducted on behavioural interventions to date is spread across a variety of populations (eg, heterosexual, homosexual, and drug-using populations), outcome measures (eg, knowledge, attitudes, behaviours, and STD or HIV incidence or prevalence), types of intervention (eg, delivered one-on-one, in dyads, in small or large groups), content of intervention (eg, delivery of information, delivery of materials, skills training, delivery of services, referrals to professional services), intervention duration (eg, less than an hour, 2–4 hours, or 5 hours or more), and number of intervention sessions (eg, one, two, three, or more sessions). Thus, in general, findings are not sufficiently replicated. The quality and completeness of reporting of behavioural intervention research in the peer-reviewed scientific literature is also an issue; published reports may not cover fully all relevant aspects of the studies, and published studies may be a select sample of all studies reflecting publication bias. First, when published, behavioural and biomedical intervention studies need to be completely and reliably reported.1O'Leary A DiClemente Aral SO Reflections on the design and reporting of STD/HIV behavioral intervention research.AIDS Educ Prev. 1997; 9: 1-14Google Scholar One methodological review found that only 72% of studies provided information to replicate the intervention, 85% stated the numbers recruited, 44% provided preintervention data, 76% discussed attrition, 76% discussed all outcomes, and 65% provided postintervention data for all study groups.2Oakley A Fullerton D Holland J Behavioural interventions for HIV/AIDS prevention.AIDS. 1995; 9: 479-486Crossref PubMed Scopus (176) Google Scholar In the absence of adequate reporting, assessing quality becomes impossible.3Schulz KF Chalmers I Hayes RJ Altman DG Empirical evidence of bias.JAMA. 1995; 273: 408-412Crossref PubMed Scopus (3995) Google Scholar Journals and editors could help by allowing space for explicit descriptions of the study population, recruitment and sampling methods, participation rates, power calculations, randomisation procedures, treatment of control groups, blinding procedures, attrition, content of the intervention, exposure to the intervention, cost of the intervention, intention-to-treat analyses, intervention exposure analyses, and outcome measurement.1O'Leary A DiClemente Aral SO Reflections on the design and reporting of STD/HIV behavioral intervention research.AIDS Educ Prev. 1997; 9: 1-14Google Scholar Second, all well-conducted intervention studies, both those with null findings and those that obtain statistically significant intervention effects need to be published. Any tendency on the parts of investigators or editors to fail to publish study results on the basis of the direction or strength of the study findings leads to publication bias.4Dickersin K Publication bias: the problem that won't go away.Ann N Y Acad Sci. 1993; 703: 135-145Crossref PubMed Scopus (323) Google Scholar The earliest study of publication bias reviewed four prominent psychology journals and reported that more than 95% of articles that focused on hypothesis testing reported statistically significant, positive findings.5Sterling TD Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa.J Am Stat Assoc. 1959; 54: 30-34Google Scholar Results were very similar when study was replicated in the same four journals in 1986 and 1987, and 85% of articles in medical journals were found to report statistically significant results.4Dickersin K Publication bias: the problem that won't go away.Ann N Y Acad Sci. 1993; 703: 135-145Crossref PubMed Scopus (323) Google Scholar Since that time evidence has accumulated to show that studies reaching publication are a select sample of all studies conducted.6Begg CB Berlin JA Publication bias: a problem in interpreting medical data.Stat Soc. 1998; (annex 151): 419-463Google Scholar, 7Dickersin K The existence of publication bias and risk factors for its occurrence.JAMA. 1990; 263: 1401-1405Crossref PubMed Scopus (1135) Google Scholar There is now considerable evidence indicative of reporting bias in all stages of study findings including acceptance of abstracts for presentation at meetings, publication of studies initially presented as abstracts, publication of initiated studies, and reporting of published results by lay press. Although no studies have been conducted of the STD/HIV prevention literature, our knowledge of the effectiveness of behavioural interventions may be greatly affected by bias in the reporting of findings from intervention research. The effectiveness of an intervention is the impact an intervention achieves in the real world, under resource constraints, in entire populations, or in specified subgroups of a population. It is the improvement in a health outcome achieved in a typical community setting. Efficacy is the improvement in health outcome achieved in a research setting, in expert hands, under ideal circumstances. Thus, effectiveness of an intervention is the product of its efficacy, the penetration or reach of the intervention into the population, and the compliance of the population with the intervention.8Haddix AC Teutsch SM Shaffer PA Dunet DO Prevention effectiveness. Oxford University Press, Oxford1996Google Scholar In general, effectiveness is lower than efficacy, though theoretically it could be higher, particularly for infectious diseases, due to the effect of transmission dynamics. When any intervention is evaluated at the community level, the impact of the intervention can be measured directly in terms of the reduction in the incidence of a specific adverse health outcome; for example, in the community randomised trial conducted in the Mwanza Region in Tanzania, improved STD treatment reduced HIV incidence by about 40%.9Grosskurth H Mosha F Todd J et al.Impact of improved treatment of sexually transmitted diseases on HIV infection in rural Tanzania: randomised controlled trial.Lancet. 1995; 346: 530-536PubMed Scopus (1194) Google Scholar Most studies do not explicitly measure impact or effectiveness. In these cases, potential impact needs to be estimated to provide an indicator of effectiveness. Preventable fraction can be estimated based on attributable risk or the incidence of the health outcome that can be attributed to a specific risk factor. Adjustments need to be made for compliance of the population with the intervention, penetration of the intervention into the population, and risk incurred before implementation of the intervention. The estimation of potential impact in the community is not straightforward in the case of HIV and STDs in general (panel). Prevention-effectiveness-estimation methods have mostly been worked out for chronic diseases and tend not to take into account the dependence of one individual's health outcome on the health outcome of other individuals.10Koopman J Emerging objectives and methods in epidemiology.Am J Public Health. 1996; 86: 630Crossref PubMed Scopus (50) Google Scholar If an individual's sexual partners are uninfected, the individuals's risk of infection is zero. Each prevented infection also prevents a variable number of secondary infections. To obtain a more accurate estimate of the impact of STD-prevention strategies on morbidity, an additional correction that reflects the transmissible nature of STDs is necessary; without this correction the impact of behavioural interventions tend to be underestimated.PanelChallenges •Validity of measured behaviour change as indicator of STD/HIV risk reduction.•Implementation not always influenced by evidence of effectiveness.•Community trials are hard to do, therefore, contribute too little too late.•Inadequate feedback of community needs to intervention development. •Validity of measured behaviour change as indicator of STD/HIV risk reduction.•Implementation not always influenced by evidence of effectiveness.•Community trials are hard to do, therefore, contribute too little too late.•Inadequate feedback of community needs to intervention development. In the absence of community randomised trials, it is difficult to determine what infection rates would be with and without the intervention; therefore, attribution of changes in morbidity to the effects of an intervention is problematic. Prevalence and incidence of STDs or HIV may increase or decline as a reflection of natural progression of epidemics,11Wasserheit JN Aral SO The dynamic topology of sexually transmitted disease epidemics: implications for STD prevention strategies.J Infect Dis. 1996; 174: S201-S213Crossref PubMed Google Scholar or as a result of changing behaviours where behaviour changes may be unrelated to the intervention. The optimal way to control for the effects of the natural progression of the epidemic and the behaviour changes unrelated to the intervention is to employ randomised controls. What we need to know is the impact of behavioural interventions on STD/HIV morbidity in the population. This impact is ideally measured directly through community randomised trials. In the absence of such trials, potential impact may be estimated based on considerations of attributable risk, penetration, compliance, previous risk, transmission dynamics of communicable disease, and intervention efficacy, where efficacy is measured through individual or group randomised trials with STD incidence as an outcome measure. Moreover, intervention impact probably varies by the profile of the epidemic, behavioural patterns of the population, and existing policies, and needs to be assessed specifically in these three contexts. Available evidence on behavioural interventions indicates that in many study populations, interventions of variable content may lead to desired changes in knowledge, attitudes, perceptions, self-efficacy, skills, and self-reported behaviours including male condom use, number of sex partners, and practice of unprotected sex.12Boyer CB Barrett DC Peterman TA Bolan G Sexually transmitted disease (STD) and HIV risk in heterosexual adults attending a public STD clinic: evaluation of a randomized controlled behavioral risk-reduction intervention trial.AIDS. 1997; 11: 359-367Crossref PubMed Scopus (88) Google Scholar, 13O'Donnell L San Doval A Duran R O'Donnell CR The effectiveness of video-based interventions in promoting condom acquisition among STD clinic patients.Sex Transm Dis. 1995; 22: 97-103Crossref PubMed Scopus (62) Google Scholar, 14Eldridge GD St Lawrence JS Little CE et al.Evaluation of the HIV risk reduction intervention for women entering inpatient substance abuse treatment.AIDS Educ Prev. 1997; 9: 62-76PubMed Google Scholar, 15Peterson JL Coates TJ Catania J et al.Evaluation of an HIV risk reduction intervention among African-American homosexual and bisexual men.AIDS. 1996; 10: 319-325Crossref PubMed Scopus (101) Google Scholar, 16St Lawrence JS Brasfield TL Jefferson KW Alleyne E O'Bannon III, RE Shirley A Cognitive-behavioral intervention to reduce African-American adolescents' risk for HIV infection.J Consult Clin Psychol. 1995; 63: 221-237Crossref PubMed Scopus (292) Google Scholar, 17St Lawrence JS Jefferson KW Alleyne E Brasfield TL Comparison of education versus behavioral skills training interventions in lowering sexual HIV-risk behavior of substance-dependent adolescents.J Consult Clin Psychol. 1995; 63: 154-157Crossref PubMed Scopus (103) Google Scholar, 18Kalichman SC Sikkema KJ Kelly JA Bulto M Use of a brief behavioral skills intervention to prevent HIV infection among chronic mentally ill adults.Psychiatr Serv. 1995; 46: 275-280Crossref PubMed Scopus (119) Google Scholar, 19Kelly JA Murphy DA Washington CD et al.The effects of HIV/AIDS intervention groups for high-risk women in urban clinics.Am J Public Health. 1994; 84: 1918-1922Crossref PubMed Google Scholar, 20Jemmott JB Jemmott LS Fong GT Reductions in HIV risk-associated sexual behaviours among black male adolescents: effects of an AIDS prevention intervention.Am J Public Health. 1992; 82: 372-377Crossref PubMed Scopus (480) Google Scholar, 21Rotheram Borus MJ Koopman C Haignere C Davies M Reducing HIV sexual risk behaviors among runaway adolescents.JAMA. 1991; 266: 1237-1241Crossref PubMed Scopus (288) Google Scholar, 22Leviton LC Valdiserri RO Lyter DW et al.Preventing HIV infection in gay and bisexual men: experimental evaluation of attitude change from two risk reduction interventions.AIDS Educ Prev. 1990; 2: 95-108PubMed Google Scholar, 23Kelly JA Murphy DA Sikkema KJ et al.Randomised, controlled, community-level HIV-prevention intervention for sexual-risk behaviour among homosexual men in US cities.Lancet. 1997; 350: 1500-1505Summary Full Text Full Text PDF PubMed Scopus (416) Google Scholar Most behavioural interventions are delivered to individuals and change in self-reported behaviour is measured at the individual level. Only a few behavioural-intervention studies have measured effect on the incidence of HIV or STD. Project Respect and Project Light are two important studies that have measured biomedical outcomes.24Kamb ML Bolan G Zenilman J Douglas JM Does HIV/STD prevention counseling work?.in: Results from a multi-center randomized trial (Project RESPECT). 12th Meeting of the International Society for STD Research. 1997 October 19–22Google Scholar, 25AIDS. 1997; 11: S1-S63PubMed Google Scholar. Thus, in most cases we have evidence of efficacy of behavioural interventions in changing self-reported behaviours, which may not be valid surrogates for incidence of infection. We need further evidence on: efficacy of behavioural interventions to change the incidence or prevalence of STDs including HIV, indirect assessment of potential impact of behavioural interventions on community infection rates, and evidence of impact of behavioural interventions on STD/HIV morbidity in populations. Efficacy of behavioural interventions can be optimally measured through randomised controlled trials.26Peterman TA Aral SO Evaluating behavioral interventions: need for randomized controlled trials.JAMA. 1993; 269: 2845Crossref PubMed Scopus (10) Google Scholar Bias in the adoption of behaviour change and in intervention assignment is inherently a part of the practice of medicine and public health and cannot be dealt with by simply adjusting for the effects of known confounding factors.27Green S Editorial: the eating patterns study—the importance of practical randomized trials in communities.Am J Public Health. 1997; 87: 541Crossref PubMed Scopus (8) Google Scholar During the past decade, self-reported measurement of sensitive behaviours has improved greatly.28Aral SO Peterman TA Measuring outcomes of behaviourial interventions for STD/HIV prevention.Int J STD AIDS. 1996; 7: 30Crossref PubMed Scopus (67) Google Scholar Nevertheless, use of self-reports of behaviour is problematic in that interventions may change reports of behaviour without changing behaviour itself.28Aral SO Peterman TA Measuring outcomes of behaviourial interventions for STD/HIV prevention.Int J STD AIDS. 1996; 7: 30Crossref PubMed Scopus (67) Google Scholar Perhaps more importantly, the nature of the relation between self-reported sexual or preventive behaviours and STD/HIV outcomes is not straightforward. At the individual level, the magnitude of risk associated with particular behaviours depends on the infectiousness of the partner. At the population level, the relation between behaviour and STD incidence varies across the phases of the epidemic.11Wasserheit JN Aral SO The dynamic topology of sexually transmitted disease epidemics: implications for STD prevention strategies.J Infect Dis. 1996; 174: S201-S213Crossref PubMed Google Scholar Moreover, individuals who change one risk behaviour often change other risk behaviours in countervailing directions, necessitating the development and validation of summary measures that reflect the net effects of these changes.29Peterman TA Lin L Bolan G et al.Does measured behavior change reflect change in STD/HIV risk?.in: 12th World AIDS Conference, Geneva, Switzerland 1998Google Scholar In this context, it is important to remember that the purpose of behavioural interventions is prevention of disease, not behaviour change. Most studies of behavioural interventions focus on individuals. Individual-level effects, even when they measure biomedical outcomes, cannot be translated directly into population-level impact. Individuals who participate in studies may differ from those who do not, and, depending on their placement in sexual and social networks, individuals who change their behaviours and their infection risk may have a greater or a lesser effect on the infection risk of others. Focusing behavioural interventions on individuals has other practical implications. Individuals, after successfully changing their behaviours to reduce risk in the study context, may return to their typical life situation and relapse into earlier risky behaviours necessitating repeat applications of sometimes very costly behavioural interventions. Individual members of new cohorts who continually enter societal situations that foster risk also require repeated application of behavioural interventions. The field of behavioural-intervention research is remarkable in the extent to which it has produced self-critical reviews and responded to them by great improvements in methodology and focus (figure). Among abstracts from international AIDS conferences between 1989 and 1992, there were ten reporting on randomised controlled trials of behavioural interventions.26Peterman TA Aral SO Evaluating behavioral interventions: need for randomized controlled trials.JAMA. 1993; 269: 2845Crossref PubMed Scopus (10) Google Scholar An assessment of articles included in the Cochrane systematic database (CSD) indicated that between 1994 and 1997 there were 23 articles focusing on HIV and STD and five articles focusing on STD excluding HIV that employed randomised controlled designs. Over the past decade, use of self-reported behaviours, rather than only perceptions, attitudes, intentions, and knowledge as outcome measures has also increased: of studies included in the CSD, only two conducted between 1990 and 1991, and none conducted since then, failed to include behaviours as outcome measures. Unfortunately the use of biomedical outcome measures has not shown a similar trend: since 1990 only two studies in the CSD have measured incident STDs as their outcome. Failure to use STD and HIV incidence as the outcome measure constitutes a major weakness in the behavioural-intervention area. Another important weakness is our failure to evaluate basic social structural interventions. We cannot know the effectiveness of interventions that have not been addressed. There is overwhelming evidence that oppression contributes to STDs and many other maladies. Yet we have failed to see prejudice against single-sex marriage, racism, and human-rights violations as issues that require public-health interventions. It was recently suggested that research in general should be evaluated based on how often it is cited in the literature and how much it cost ("citations attracted per million pounds spent on research").30May RM The scientific wealth of nations.Science. 1997; 275: 793Crossref Scopus (464) Google Scholar This idea is attractive for its simplicity, but the goal of prevention research is disease prevention, not citation. Prevention research has a big impact when interventions that prevent disease are developed and then adopted by prevention programmes. Thus, a more meaningful evaluation of the impact of research would be "amount of disease prevented in the population per pound spent on research". HIV and STDs are tremendously important health problems and behavioural-intervention research could score well on this scale. Some interventions have been shown to prevent disease in research settings. They need to be implemented to have an effect on the population. Many more interventions need to be developed, tested, implemented, and refined. Greater resources need to be invested toward this goal. We know more about the effectiveness of behavioural interventions than we did a few years ago, but we still have a lot to learn.