To determine what experts and the available research indicate about the types of reentry programs and substance abuse programs that are effective or cost beneficial for juvenile offenders, we reviewed relevant literature, studies, and federal resources for juvenile justice programs, and interviewed federal officials and 26 juvenile justice experts. Specifically, to identify the types of programs to review, we conducted a literature search for studies and articles, including evaluations of juvenile reentry and juvenile substance abuse programs in the United States that were published from May 30, 1999, through May 30, 2009. We chose this time frame, the past 10 years, because it provided us with an overview of the available research, including unpublished and ongoing studies, which assesses the effectiveness of reentry and substance abuse programs. We also consulted with OJJDP officials who coordinate research on juvenile justice programs and Department of Health and Human Services officials who oversee substance abuse and adolescent programs to obtain their recommendations for repositories—online databases that contain information on effective programs—and research studies and relevant Web sites for identifying types of reentry and substance abuse programs. Using these recommendations, information from relevant literature, and categories of program types used by OJJDP’s Model Programs Guide, we identified five types of juvenile justice programs that are used to address reentry issues and five types of programs that are used to address substance abuse issues for juvenile offenders. Specifically related to substance abuse, we focused on substance abuse programs that involved relapse prevention treatment for juvenile offenders with substance abuse histories. After consulting with experts and reviewing the literature, we excluded juvenile alcohol abuse programs and substance abuse programs for the general juvenile population as well as at-risk juveniles who are prone to, but have not yet developed, substance abuse problems. For instance, we excluded after school or recreation programs, conflict resolution programs, and school or classroom programs. While all of these programs may have a substance abuse component, this component is not designed to address juvenile offenders’ actual substance abuse problems. After identifying the types of programs to be reviewed, we looked at online databases, academic research, and professional organizations to select subject matter experts—researchers and practitioners—to obtain their views on the types of programs that have been shown to be effective or cost beneficial and the basis they used for making such determinations. We specifically identified researchers who focus on juvenile reentry issues or substance abuse issues and practitioners who operate programs that address these issues. We chose 26 experts to interview as a result of this process. Specifically, we selected 13 individuals with expertise related to juvenile reentry programs, 7 individuals with expertise related to juvenile substance abuse programs, and 6 individuals with both juvenile reentry and substance abuse program expertise. We selected these experts based on several criteria, including their employment histories related to juvenile reentry and substance abuse programs and the number of years they spent studying, evaluating, or managing programs addressing juvenile reentry or substance abuse issues. We evaluated their experience by reviewing the studies the researchers had completed and determining the experience the practitioners had managing the types of juvenile reentry and substance abuse programs selected for our review. See appendix I for the list of experts we interviewed. We asked these experts to provide their views about the effectiveness of program types (e.g., drug courts), rather than about the effectiveness of individual intervention programs (e.g., a specific drug court program that was implemented in one county). Because the Model Programs Guide, like other online repositories, contains information about the effectiveness of individual intervention programs, it does not provide information about the effectiveness of program types. As a result, we were interested in obtaining the experts’ consolidated views of the effectiveness of program types. We also asked the experts to identify other program types—in addition to those that we explicitly asked about—that they considered to be effective or cost beneficial, but no additional program types were mentioned. In addition, we asked the experts to identify factors that in their view could help programs to achieve intended outcomes, such as reducing participants’ recidivism, which are summarized in appendix II. While the results of these interviews cannot be generalized to reflect the views of all experts knowledgeable about juvenile reentry or substance abuse programs, we believe the interviews provided us with a good overview of the available research and valuable information about what program types are considered to be effective by subject matter experts. In addition, while we did not assess the methodological rigor of studies and evaluations in our review, we corroborated expert testimony by reviewing and summarizing the studies or evaluations that experts cited as the basis for their opinions. We also provided the experts with a summary of their opinions to review in order to ensure that we correctly captured their views. To identify the extent to which OJJDP has efforts under way to disseminate information about effective juvenile justice programs and assess the extent to which OJJDP ensures the utility of the information provided, we reviewed documentation, such as OJJDP’s annual reports outlining information dissemination efforts, OJJDP publications, and a contract related to disseminating training information on effective programs. We interviewed knowledgeable OJJDP officials, such as the Training Coordinator and communications policy personnel, about OJJDP’s efforts to disseminate information about effective programs. We selected two of OJJDP’s efforts through which it disseminates information about effective programs—the Model Programs Guide and the National Training and Technical Assistance Center (NTTAC), which provides training and support to the juvenile justice field in identifying and implementing effective programs—because they provide information about effective programs across the range of issue areas in which OJJDP is involved, including reentry and substance abuse programs. We then compared these efforts to guidance articulated by the Office of Justice Programs (OJP), which oversees OJJDP, and in prior GAO reports that stresses the importance of assessing whether the information disseminated is meeting the needs of its users. We also interviewed representatives from the two organizations that manage these two information dissemination efforts. Additionally, we asked the 26 juvenile reentry and substance abuse experts we interviewed about their views regarding OJJDP’s information dissemination efforts and their opinions about the effectiveness of these efforts. Although their views cannot be generalized to the entire juvenile justice field, we believe that the experts provided us with a good overview of the utility of the information disseminated by OJJDP. We did not contact recipients of the information OJJDP disseminates for their views on the usefulness of the information provided because of the large volume of recipients and the resulting cost that would be incurred to obtain this input. To assess the extent to which OJJDP has plans in place for its research and evaluation efforts, we reviewed relevant laws related to the office’s role in supporting research and evaluations of juvenile justice programs. We also reviewed relevant DOJ and OJJDP documentation, such as annual reports and strategic plans that contain information on OJJDP’s research and evaluation goals and plans. We interviewed cognizant OJJDP officials about the office’s planning efforts related to research and evaluation. We also reviewed criteria found in standard practices for program management and our prior products that highlight the importance of developing plans to meet goals and help ensure that resources are used effectively, and then compared these criteria to OJJDP’s stated plans. Additionally, we analyzed OJJDP funding and staff data for fiscal years 2005 through 2009 to better understand the resources the office has had available to support its evaluation activities. We chose these years because they provide the most recent overview of OJJDP’s research and evaluation funding. We conducted this performance audit from April 2009 through December 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Within states’ juvenile justice systems, reentry aims to promote the effective reintegration of juvenile offenders back into communities upon release from residential facilities. Reentry is a process that incorporates a variety of programs to assist juvenile offenders in the transition from residential facilities to communities. In addition, reentry is intended to assist juvenile offenders in acquiring the life skills needed to succeed in the community and become law-abiding citizens and can incorporate the use of education, mental health, drug rehabilitation, or vocational training programs. While reentry begins after a juvenile is released back into the community, to help ensure a seamless transition, a reentry process begins after sentencing, then continues through incarceration, and into the period of release back into the community. According to OJJDP, juvenile justice practitioners and researchers believe that providing supervision and services to juvenile offenders returning to the community will reduce the high rate of recidivism among these juveniles. Several types of programs address juvenile reentry issues, as described in table 1. Substance abuse includes, but is not limited to, the use or abuse of illegal drugs (e.g., heroin), prescription drugs, and nonprescription drugs (e.g., over-the-counter medications available without a prescription, such as cough suppressant). Treatment of substance abuse may occur in a variety of different settings, such as in clinics on an outpatient basis or at a hospital. Treatment can also occur in short- and long-term residential facilities that range from secure environments where juveniles’ activities are physically restricted, to group homes, which are nonsecure settings where juveniles live and receive services in a homelike environment. According to the Center for Substance Abuse Treatment, juveniles with addictions to substances can be helped through programs that specifically target the factors associated with substance abuse—such as a family history of such abuse. For example, substance abuse intervention programs, such as cognitive behavioral therapy and family therapy, aim to change a juvenile’s behavior by focusing on improving a juvenile’s response to situations that contributed to prior substance abuse. Substance abuse intervention programs can be provided to juvenile offenders throughout the juvenile justice system: after sentencing, during incarceration, and after release back into the community. Whether treatment occurs while a juvenile is incarcerated or after the juvenile is released into the community, according to OJJDP, effective intervention programs can help addicted juveniles to overcome their substance abuse, lead crime-free lives, and become productive citizens. Table 2 describes types of programs—in addition to cognitive behavioral therapy and wraparound/case management, which are discussed in table 1—that address juvenile substance abuse issues. The Juvenile Justice and Delinquency Prevention Act (JJDPA) established OJJDP in 1974. As the only federal office charged exclusively with preventing and responding to juvenile delinquency and with helping states improve their juvenile justice systems, OJJDP supports its mission through a variety of activities. For example, OJJDP administers a wide variety of grants to states, territories, localities, and public and private organizations through formula, block, and discretionary grant programs; provides training and technical assistance; produces and distributes publications and other products containing information about juvenile justice topics; and funds research and evaluation efforts. In fiscal year 2009, the total appropriation for juvenile justice programs was about $374 million. See appendix III for more detailed information on OJJDP’s enacted appropriations for fiscal years 2007 through 2009. OJJDP, through its various grant programs, has provided funding to states and organizations to support juvenile reentry and substance abuse programs, although the JJDPA does not specifically require OJJDP to fund them. States generally have the authority to determine how formula and block grants are allocated and may use these funds to support a range of program areas, including programs specifically for reentry or substance abuse. For example, from fiscal years 2007 through 2008, OJJDP reported that states used approximately $7.1 million in applicable formula and block grant funds for programs that target reentry and $19 million in formula and block grant funds for programs that target substance abuse, representing approximately 1.8 percent and 4.5 percent, respectively, of such funding for those years. Additionally, from fiscal years 2007 through 2009, OJJDP awarded a total of approximately $33 million in discretionary grants through four juvenile reentry grant programs and three substance abuse grant programs. Specifically, in the area of reentry, OJJDP awarded a total of $25.4 million to 38 grantees under 4 programs, and in the area of substance abuse, OJJDP awarded a total of $7.6 million to 15 grantees under 3 programs. See appendix IV for more detail on funding for these reentry and substance abuse programs. Of the five reentry program types we reviewed, reentry experts reported that there is evidence from available research that cognitive behavioral therapy reduces recidivism. While experts cited a lack of evidence demonstrating that wraparound/case management, aftercare, and vocational/job training were effective in achieving results, such as a reduction in recidivism, they generally provided positive views on the potential results of these three types of programs, based on their own experience or knowledge of them. Similarly, of the five substance abuse program types we reviewed, juvenile substance abuse experts reported that there is evidence from available research that cognitive behavioral therapy along with family therapy are effective at reducing recidivism and show successful results at reducing substance abuse. However, expert opinions regarding other substance abuse program types, such as drug courts, mentoring, and wraparound/case management, were mixed, with experts stating that these program types could be effective, they were ineffective, or there was not enough evidence to determine effectiveness. Furthermore, both reentry and substance abuse experts cited studies indicating that cognitive behavioral therapy and family therapy programs are cost beneficial; however, the experts cited limited evidence for determining the costs and benefits of the other programs we reviewed. Eleven of the 12 experts we interviewed who provided comments based on their knowledge and experience with cognitive behavioral therapy stated that evidence from available research shows that these programs can be effective at reducing recidivism. Cognitive behavioral therapy intervention programs are designed to identify and provide juveniles with the skills to change thoughts and behaviors that contribute to their problems. The underlying principle of these programs is that thoughts affect emotions, which then influence behaviors. These intervention programs combine two kinds of psychotherapy—cognitive therapy and behavioral therapy. The strategies of cognitive behavioral therapy have been used to, among other things, prevent the start of a problem behavior—such as violence and criminal activity—or stop the problem behavior from continuing. A juvenile offender can receive this type of intervention program after sentencing, throughout incarceration, or after returning to the community. For example, a cognitive behavioral therapy intervention program may provide individual and family services to treat a juvenile offender who has mental health and substance abuse issues. The treatment can occur during the juvenile’s transition from incarceration back into the community and help the juvenile lower the risk of recidivism, connect the family with appropriate community support, assist the juvenile in abstaining from drugs, and improve the mental health of the juvenile. Based on their assessment of the available research, these 11 experts stated that cognitive behavioral therapy programs have been shown to be effective. Experts identified two meta-analyses of cognitive behavioral therapy programs that demonstrated effectiveness. One such study concluded that effective cognitive behavioral therapy programs are characterized by the low proportion of juveniles who dropped out of the program, as well as the close monitoring of the quality of the treatment and adequate training for the providers. In addition, this same study also found that 12 months after treatment, the likelihood of a juvenile who received cognitive behavioral therapy not recidivating was about one and a half times greater than for a juvenile who did not receive the therapy. This study also reported that the effects of cognitive behavioral therapy were greater for offenders who had a higher risk of recidivism than those with a lower risk. Specifically, the best results, in terms of recidivism reductions, occurred when high-risk offenders received more intensive treatment that targeted criminal thinking patterns. A second study also reported that among therapeutic interventions, such as skill building, cognitive behavioral therapy was most effective at reducing recidivism. The 12th expert stated that the particular cognitive behavioral therapy intervention program he was using—aggression replacement training®— had not been evaluated at his particular program site, so he could not draw conclusions as to its effectiveness. Despite having generally positive views on the results of wraparound/case management, aftercare, and vocational/job training programs based on their experience or knowledge of these programs, reentry experts reported a lack of evaluations that show conclusive evidence about the effectiveness of these programs. Specifically, of the nine experts who provided comments on wraparound/case management programs, eight offered positive opinions about these programs. For example, two of these experts commented that wraparound/case management can be successful at reducing recidivism, depending on the quality and availability of services provided to juveniles. However, two of these eight experts also stated that there was a lack of evaluations demonstrating the effectiveness of wraparound/case management programs. One of these experts pointed us to a study on a specific wraparound/case management intervention program, Wraparound Milwaukee, that showed potentially promising results related to a reduction in recidivism rates for juvenile offenders. However, another expert cautioned that initial evaluations of wraparound/case management programs did not conclusively demonstrate the effectiveness of wraparound/case management programs. Finally, the ninth expert stated that in her experience, wraparound/case management interventions are not effective because, for example, juveniles are placed into these interventions based on the availability of program staff and resources rather than program services being tailored to the individual needs of each juvenile. In addition, 7 of the 15 experts who commented about aftercare programs opined that aftercare interventions are important reentry programs, in part, because they link a juvenile with the community and provide regular contact with a caseworker. However, 6 other experts said there was inconclusive evidence to determine whether these programs can be effective in achieving results. Three of these experts based their opinions on an evaluation of the Intensive Aftercare Program that showed inconclusive results about program effectiveness. Specifically, the study found no evidence that the program had its intended impact of reducing recidivism among juveniles who were released back into the community under supervision in the three states that piloted the program. However, the evaluation did find that the three states that implemented the Intensive Aftercare Program model successfully incorporated most of its core features, which prepared juveniles to transition back into the community. For instance, these states created new Intensive Aftercare Programs— specific treatment programs that among other things, prepared juveniles for increased responsibility in the community, facilitated interaction with the community, and worked with the juveniles’ schools and families. The state programs had a large percentage of juveniles involved in various treatment services. Despite the inconclusive results of the study, 1 expert credited the aftercare program model with addressing the issue of juveniles interacting with multiple probation officers throughout the entire reentry process because aftercare programs, in general, assign one probation officer to a juvenile as a consistent point of contact. The remaining 2 of 15 experts opined that aftercare intervention programs had not been shown to be effective at achieving desired results because, for example, the treatment a juvenile receives depends on the services available in the community. With respect to vocational/job training programs, 10 of the 11 reentry experts who commented on these programs expressed positive opinions about the programs’ potential outcomes but noted that there had been limited research conducted to demonstrate their effectiveness. Specifically, experts noted that vocational/job training programs could be beneficial if they were applied to older juveniles and if they led to those juveniles getting jobs. The remaining expert said there is little evidence to demonstrate the effectiveness of these intervention programs. For a more detailed description of reentry experts’ opinions about these program types, see appendix V. All of the 13 substance abuse experts we interviewed stated that based on available research, cognitive behavioral therapy effectively reduces recidivism and has demonstrated success at reducing substance abuse. Experts cited six studies to support their opinions, two of which were the same studies cited by reentry experts that demonstrate that cognitive behavioral therapy is effective at reducing recidivism. Two of the substance abuse experts noted that few studies have been conducted to determine whether an intervention program is effective at specifically reducing substance abuse. However, 3 experts also noted that within the last decade, newly emerging research has shown promising results with respect to cognitive behavioral therapy program types reducing substance abuse. For example, these experts pointed us to three studies that report that juveniles who participated in these programs showed reductions in marijuana use. Twelve of the 13 substance abuse experts we interviewed who provided comments based on their knowledge and experience stated that family therapy programs are effective at reducing recidivism or decreasing substance use. Family therapy uses trained therapists to treat juvenile offenders with substance abuse problems by including families of juveniles in the treatment and focusing on improving communication and interactions among family members and improving overall relationships between juveniles and their families. This type of therapy focuses on the family as it is the primary and sometimes only source for emotional support, moral guidance, and self-esteem for juveniles. Family habits, such as failing to set clear expectations for children’s behavior, poor monitoring and supervision, and severe and inconsistent discipline can often lead to juveniles engaging in delinquency and substance abuse, according to OJJDP. For example, family drug use often results in adolescent drug use. Based on their assessment of the available research, these 12 experts provided positive opinions about the effectiveness of family therapy, and 7 of these experts cited 9 studies that support their opinions. These studies demonstrated, for example, that multisystemic therapy—a family therapy intervention program that helps parents identify strengths and develop natural support systems (e.g., extended family, neighbors, friends, and church members)—is an effective intervention program for reducing recidivism and substance use because, for example, juveniles who participated in multisystemic therapy programs engaged in significantly less criminal activity than did nonparticipants. Specifically, multisystemic therapy participants had fewer average convictions per year for violent crimes than those juveniles who did not participate in the program. Additionally, analyses of drug tests demonstrated significantly higher rates of drug abstinence for program participants than for nonparticipants. One study also showed that participants in functional family therapy, another family therapy intervention program, had 50 percent reductions in substance use as compared to juveniles who did not participate in the program. According to the 10 experts who commented on drug courts, 5 stated that there is a lack of evidence to determine program effectiveness, while another expert stated that drug courts are ineffective types of programs because they expose first-time offenders to more serious drug users. The remaining 4 experts stated that drug courts can be effective if, for example, they are combined with other effective intervention programs, such as multisystemic therapy. Similarly, according to the 8 experts who commented on mentoring, 1 stated that there are too few evaluations to determine effectiveness, while 4 stated that mentoring programs alone are ineffective or unsuccessful at achieving desired results and that mentoring intervention programs are more effective at preventing at-risk juveniles from engaging in delinquent behavior. However, 3 experts thought mentoring intervention programs could be effective if the programs adhere to certain factors that have been evaluated and shown to be effective, such as the mentor being properly trained. Finally, experts also had mixed views on the effect of wraparound/case management types of programs. Of the 11 experts who commented on these programs, 7 experts stated that wraparound/case management is effective or can be effective if, for example, wraparound/case management is combined with another intervention program that has been evaluated and has shown to be effective, such as cognitive behavior therapy. Conversely, 4 experts either stated that these programs are ineffective because, for example, the intervention programs lack follow-through as there are no consequences if a juvenile does not show up for treatment, or there is not sufficient evidence to determine effectiveness. For a more detailed description of substance abuse experts’ opinions about these program types, see appendix VI. While program evaluations establish if a program is effective in producing its intended results or effects—such as a reduction in recidivism—cost- benefit analyses use program evaluation to determine if the dollar value of a program’s benefits exceeds the costs to deliver the program. For example, if a program evaluation shows that an intervention program reduces the number of offenses committed by juveniles from three to one, a cost-benefit analysis would first determine a dollar value for each of the offenses. Then, the cost-benefit analysis would estimate whether the savings of going from three offenses to one offense is more or less costly than the amount of money required to deliver the intervention program, as compared to an alternative program the juvenile would have received. The intervention may not always be more expensive than the alternative. For example, if the alternative is incarceration, the intervention program may be less expensive—meaning that the intervention program can be cost beneficial even if it does not result in a reduction of offenses. By applying the same cost-benefit analysis techniques to evaluations of different program types, decision makers can make comparisons among alternatives and determine which program types offer the greatest benefits for the least cost. The results of a cost-benefit analysis are often represented as a net benefit, meaning total benefits minus total cost. Of the 26 reentry and substance abuse experts we interviewed, 19 provided information related to the cost benefits of the reentry and substance abuse program types in our review. These 19 experts identified five cost-benefit analyses of juvenile justice programs consisting of four meta-analyses and one systematic review. The studies demonstrate that various cognitive behavioral therapy and family therapy intervention programs are cost beneficial because they are effective at reducing crime and are expected to produce more benefits than costs compared to the alternative. For example, in one study, the authors reviewed several program interventions that fall into the family therapy program type, such as multisystemic therapy and multidimensional treatment foster care. The authors analyzed three program evaluations of multidimensional treatment foster care and found that this intervention can be expected to reduce crime outcomes by 22 percent. Based on this reduction in crime, the authors of the study predict that the intervention provides about $80,000 worth of benefits per participant. This dollar value reflects the savings per participant that result from a decrease in criminal activity, including savings to crime victims, police and sheriff’s office costs, and juvenile detention costs, among others. The four studies cited by the experts show mixed or inconclusive results for drug courts, vocational/job training, and mentoring program types. For example, one study found that juvenile drug courts are cost beneficial because they are expected to have a net benefit of $4,622 per program participant. The other studies could not determine drug courts’ cost- effectiveness because they either did not include program evaluations of drug court programs or they found mixed results in the program evaluations analyzed and therefore could not determine the net benefits. In addition, two studies found that there are too few evaluations of vocational/job training or mentoring in juvenile justice programs to calculate if the benefits of these program types outweigh the costs. The remaining program types in our review—wraparound/case management, aftercare, and reentry courts—were not analyzed in these studies. Table 3 presents a summary of these studies. In addition, seven experts also commented on reentry and substance abuse programs that were not included in the cited studies. For example, three experts opined that wraparound/case management programs may eventually be proven to be cost beneficial, based on preliminary research and evaluations. For example, one expert cited an unpublished study of a wraparound program pilot project that showed that recidivism of program participants was low, and that program costs were approximately 60 percent of the costs of incarcerating juveniles. Additionally, although experts did not cite cost-benefit analyses of aftercare program types, four reentry experts stated that such programs could be cost beneficial if the intervention program being delivered is effective because the cost of incarceration is so high. Three experts we interviewed stressed that even though some intervention programs that have been shown to be effective are expensive, if they reduce recidivism, they might be cost beneficial because of the high cost of incarcerating juveniles. Consistent with the JJDPA, OJJDP has several efforts under way to disseminate information about effective juvenile justice programs. Two of these efforts—NTTAC and the Model Programs Guide—provide information about effective programs for a range of juvenile justice issues, including reentry and substance abuse issues. Consistent with federal guidelines for ensuring the utility of information, OJJDP has established mechanisms to ensure that the information provided through its training and technical assistance efforts meets the needs of the juvenile justice field. However, OJJDP could better ensure the usefulness of the information it disseminates through the Model Programs Guide by having a mechanism in place to solicit regular feedback specifically related to the guide from the juvenile justice field. According to the JJDPA, OJJDP is authorized, but is not required, to provide information about juvenile justice issues and programs and to provide training and technical assistance to help the juvenile justice field implement and replicate such programs. In accordance with this authority and its mission to support states and communities in their efforts to develop and implement effective juvenile justice programs, OJJDP disseminates information related to these programs through a range of efforts, from those designed to meet the needs of the juvenile justice field as a whole to those that focus on effective programs in a specific issue area, such as gang prevention or girls’ delinquency. OJJDP distributes the broadest range of information on juvenile justice topics through the Juvenile Justice Clearinghouse (Clearinghouse). Through its services, the Clearinghouse offers, among other things, the latest research findings and statistics, publications on juvenile justice issues and programs, announcements of funding opportunities, and other resources prepared by a variety of researchers in juvenile justice. As part of its efforts, the Clearinghouse responds to requests for information about effective programs by directing users to OJJDP efforts that develop and disseminate information about effective programs, such as NTTAC and the Model Programs Guide. Thus, we focused on NTTAC and the Model Programs Guide because they provide information about effective programs across the range of issue areas in which OJJDP is involved, including reentry and substance abuse programs. OJJDP also disseminates information about effective programs in specific issue areas through various centers, such as the National Youth Gang Center and the Underage Drinking Enforcement Center. For a more detailed discussion of these centers and other information dissemination efforts that focus on specific issues, see appendix VII. NTTAC was established in 1995, in part to provide information about effective juvenile justice programs—such as programs that address issues related to reentry and substance abuse—through its training and technical assistance efforts. According to OJJDP, NTTAC works to promote the use of effective programs in the field through training and technical assistance programs. Additionally, NTTAC develops training materials and resources, and customizes the information included in its curricula in an effort to best meet the needs of its training and technical assistance recipients. In terms of its efforts specifically related to program effectiveness, NTTAC provides training and technical assistance for members of the juvenile justice field on how to develop and sustain effective programs, and to help the field understand programs that are effective for various juvenile populations, such as juveniles with mental health issues or female offenders. The Model Programs Guide is an online database that contains summary information about approximately 200 juvenile justice programs, from prevention programs to reentry programs. It is designed to help practitioners and communities identify and implement prevention and intervention programs that have been evaluated and have been shown to be effective. Programs in the Model Programs Guide may focus on a range of issues, including delinquency, violence, youth gang involvement, substance abuse, or academic issues, and can include, but are not limited to, delinquency prevention, community service, drug courts, or family therapy. To be included in the Model Programs Guide, programs are reviewed and rated along several dimensions, including such factors as whether an evaluation of the program established a causal association between the treatment and the outcome. Users can search the Model Programs Guide to find programs that meet their specific needs. For example, users can look for a program that has been shown to be effective for juveniles with substance abuse problems who are first-time offenders, or they can search for a program that has been shown to be effective for juveniles involved in gang activities who are reentering the community. In accordance with federal guidelines from OJP and prior GAO work, OJJDP has mechanisms in place to regularly conduct evaluations and is currently conducting a needs assessment to ensure the usefulness of the information provided by its training and technical assistance efforts. However, OJJDP could better ensure the utility of the information provided by the Model Programs Guide by establishing a mechanism to solicit regular feedback from the juvenile justice field. We have previously reported on the importance of regularly soliciting feedback to assess user needs and satisfaction. Specifically, we have reported that without feedback, an agency lacks valuable information from its users and is hindered in its ability to make improvements to information products that are relevant to users. Additionally, OJP has published Information Quality Guidelines for its bureaus, including OJJDP, that highlight the importance of ensuring the utility of information to be disseminated to the public by continuously monitoring information needs, among other things. OJJDP has mechanisms in place to regularly assess the usefulness of the information disseminated by NTTAC to ensure that it meets the needs of the juvenile justice field. Specifically, OJJDP has established an evaluation process for NTTAC that is designed to collect the data necessary to regularly assess the outcome and impact of the training and technical assistance NTTAC provides to improve the quality of the information it disseminates. Officials at NTTAC explained that after every training or technical assistance event, all participants are given an evaluation form to complete. This form is intended to capture feedback from participants about the quality of the event, as well as feedback about the referrals and resources NTTAC provides. Other evaluation forms are also available on NTTAC’s Web site so that users can provide feedback about NTTAC’s services, as well as feedback about the utility of the Web site. NTTAC then follows up with a sample of these respondents for more in-depth feedback. According to NTTAC officials, NTTAC analyzes the data collected from these forms and then provides them to OJJDP. These officials stated that OJJDP receives this information on at least a quarterly basis, and uses the information to make changes to existing curricula and guide future curriculum development, among other things. In accordance with OJP guidelines and prior GAO work that highlights the importance of assessing user needs, these evaluation efforts allow OJJDP to regularly monitor the usefulness of the information it disseminates in order to develop or modify its information products. In addition, OJJDP is conducting a needs assessment to solicit additional information about the utility of the information it disseminates through NTTAC’s training and technical assistance efforts. NTTAC is administering the needs assessment and, according to NTTAC officials, it is designed to determine the training and technical assistance that would be most helpful to the field. Specifically, the needs assessment is soliciting feedback from members of the juvenile justice field about OJJDP’s existing efforts. It is also requesting information regarding issues of interest to the field, any current training or technical assistance needs, and the specific challenges that the juvenile justice field is facing in its work. OJJDP officials stated that they intend to use the results of the needs assessment to influence the development of training and technical assistance activities and curricula and the content of national conferences and workshops. OJJDP’s efforts to conduct evaluations and a needs assessment are consistent with comments we received from our expert interviews. We asked all 26 of the juvenile reentry and substance abuse experts we interviewed to comment on OJJDP’s overall efforts to disseminate information about effective programs to the juvenile justice field. Thirteen experts provided responses, and while they did not comment specifically on NTTAC or the Model Programs Guide, they commented on the utility of the information OJJDP provides in general about effective programs. Ten of 13 experts had negative opinions of how useful the information OJJDP disseminates is to members of the juvenile justice field. For example, 1 expert stated that practitioners often do not have the time to read research data disseminated by OJJDP, which prevents them from being able to effectively use it in their work. The expert added that it would be more useful if OJJDP disseminated information that was practical and could be applied in the field. In addition, 2 of these 10 experts suggested that it would be helpful for OJJDP to obtain feedback from members of the juvenile justice field about what types of information they would find useful. Thus, OJJDP’s needs assessment should help to address this concern. The remaining 3 experts who commented on OJJDP’s information dissemination efforts had generally positive opinions, stating that the information is useful to researchers and practitioners. With respect to the Model Programs Guide, although OJJDP has ad hoc mechanisms in place to solicit feedback about the information it provides, it does not solicit this feedback on a regular basis or use feedback to help ensure that the information disseminated by the Model Programs Guide is useful to the field, in accordance with federal guidelines. For example, the Model Programs Guide’s Program Director gives several presentations about the guide each year at juvenile justice conferences. Officials who operate the Model Programs Guide stated that following these presentations, they request verbal feedback from participants. Officials also stated that they regularly receive unsolicited feedback through the e- mail address that is listed on the Model Programs Guide’s Web site, which they respond to on a case-by-case basis. Additionally, officials said that they collect feedback about the Model Programs Guide through an annual e-mail survey that is sent to the program points of contact listed on the guide to obtain updated program information. Although these efforts to solicit feedback about the Model Programs Guide provide OJJDP with some information from its users, according to OJJDP officials, because the guide does not have a systematic feedback mechanism, information received cannot be analyzed on an aggregate level in order to regularly assess how the juvenile justice field views the utility of the information provided by the Model Programs Guide. Further, while the annual e-mail survey can help OJJDP confirm that the program information featured in the Model Programs Guide is accurate, it does not provide information about whether the guide is useful to the field as a whole since OJJDP sends the survey’s request for comments about the Model Programs Guide to a portion of the juvenile justice field whose programs are already published in the guide, which means that the comments it receives about the Model Programs Guide do not necessarily reflect the opinions of the juvenile justice field as a whole. OJJDP officials agreed that they had not established a systematic mechanism to obtain feedback from the field regarding the usefulness of the Model Programs Guide and recognized that such a mechanism would be useful to have in place. Officials also stated that NTTAC’s needs assessment might be used as a model to build in more consistent mechanisms for feedback for the office’s broader efforts. Because NTTAC uses evaluations and is taking steps to conduct a needs assessment to monitor the information needs of the juvenile justice field, OJJDP is in a better position to help ensure that the information provided by NTTAC is useful to the juvenile justice field. Recognizing that, although there is a cost associated with gathering and analyzing feedback data, establishing a cost-effective mechanism to regularly solicit feedback about the Model Programs Guide should provide OJJDP with the information necessary to assess whether the information provided by this tool is useful to the juvenile justice field. OJJDP has articulated research and evaluation goals to support its mission of promoting effective programs and improving the juvenile justice system. According to OJJDP, one of its three main goals is to promote improvements in juvenile justice and facilitate the most effective allocation of resources by conducting research to understand how the juvenile justice system works in serving children and families. Under the JJDPA, OJJDP is required to publish an annual program plan that describes planned activities that are under accounts authorized for research and evaluation activities and that demonstrate promising initiatives, among other things. This plan is required to be published annually in the Federal Register for public comment, and is to describe the activities the Administrator intends to carry out under parts D and E, the appropriations accounts that in general are available for research and the development of new programs and initiatives, respectively. Specifically, according to the JJDPA, the Administrator must take into account the public comments received during the 45-day period and develop and publish a final plan before December 31 of each fiscal year, describing the particular activities that the Administrator intends to carry out under parts D and E. While OJJDP has not published an annual program plan since 2002, it issued a proposed plan in the Federal Register to solicit public comment in December 2009. OJJDP aims to publish the final version once public comments are incorporated, in accordance with the JJDPA’s requirements. Although the annual program plan is required to describe the particular activities the Administrator intends to carry out under parts D and E of the JJDPA, the proposed program plan includes the office’s priorities with respect to all discretionary funding, including its research and evaluation efforts. According to the Acting Administrator, this will, in part, provide complete transparency for all such funding. According to OJJDP, the development and publication of the annual program plan is a first step that will lead to a comprehensive evaluation plan as the annual program plan outlines the agency’s overall research and evaluation goals. Additionally, the Office of Management and Budget’s (OMB) fiscal year 2006 Program Assessment Rating Tool found that juvenile justice programs would benefit from evaluations of their effectiveness but noted that such evaluations are difficult and expensive to do. As a result, OMB recommended that OJP develop a comprehensive evaluation plan for juvenile justice programs to obtain better information about the programs’ impacts. Although OMB’s recommendation was directed at OJP, OJP and OJJDP officials stated that because OJJDP is the office within OJP required to conduct juvenile justice evaluations, it is that office’s responsibility to develop this evaluation plan. In addition to the above requirement and recommendation, federal guidelines stipulate the importance of developing a plan to achieve agency goals. As established in the standard practices for program management, specific goals of an agency must be conceptualized and defined in a plan. Specifically, this plan is to contain a description or road map of how the goals and objectives are to be achieved, including identifying the needed resources and target milestones or time frames for achieving desired results. We have also reported on the importance of planning research and evaluation efforts, in part to ensure that goals are met and resources are used effectively. OJJDP’s Research Coordinator stated that such a road map or plan for conducting research and evaluation would help better target the agency’s research and evaluation efforts toward achieving their goals. However, from 2006 to 2009, OJJDP had not developed such a plan, primarily because of resource constraints. According to this official, in lieu of having a comprehensive evaluation plan in place to guide its research and evaluation efforts, the office’s efforts are influenced by a number of factors, including whether Congress directs the agency to conduct research in a particular area or whether ideas are generated internally by staff or externally by members of the juvenile justice field. For example, OJJDP staff responsible for the mentoring area may generate ideas about how available research funds could be used, for example, by evaluating a particular type of mentoring program. In addition, the office may receive recommendations from the Federal Advisory Council on Juvenile Justice or feedback from others in the juvenile justice field. While these factors have influenced OJJDP’s research and evaluation efforts, they have not provided a framework for helping the office meet its research and evaluation goals. Therefore, once the program plan is finalized, OJJDP intends to develop a comprehensive evaluation plan in accordance with OMB recommendations to provide direction and priorities for its research and evaluation efforts. According to the Acting Administrator, OJJDP intends to use this comprehensive evaluation plan to better align and target available discretionary funds toward achieving its research and evaluation goals. In addition to having a road map to help ensure it meets its goals, it is important for OJJDP to have a comprehensive plan that lays out how the office will evaluate its juvenile justice programs. Such a plan would help to ensure that its limited resources are being used effectively. This is important because OJJDP does not currently receive dedicated funding for research and instead must make trade-off decisions to balance funding to implement programs with funding to evaluate which programs are effective. The office has not received dedicated research funding since fiscal year 2005 when it received $10 million for its part D appropriations account—the appropriations account specifically available for research and evaluation efforts. Without part D funding, OJJDP has relied on funds it has set aside from its other appropriation accounts to fund its research and evaluation activities. Specifically, as shown in table 4, OJJDP is authorized by the appropriations act to set aside up to 10 percent of certain appropriations accounts for its research and evaluation efforts. In fiscal year 2008, the last year for which set-aside funding data are available, the appropriations act authorized OJJDP to set aside over $23 million for research and evaluation. However, according to OJJDP, the office set aside approximately $11 million. OJJDP officials stated that this was, in part, because the JJDPA requires and the agency wants to ensure that sufficient funds are available to the states for grant programs. In addition, officials explained that some of OJJDP’s accounts are transferred to other program offices, such as the Office of Community Oriented Policing Services, so research funds are not deducted from those accounts. Of the over $11 million that OJJDP did set aside, officials reported that the office used nearly $8 million (or 70 percent) for research and evaluation. Table 4 shows the amounts authorized to be set aside by the annual appropriations act, as well as the amounts actually set aside and used by OJJDP. Additionally, all of the set-asides from these four accounts must be used for research, evaluation, and statistics activities designed to benefit the juvenile justice issues that the accounts specify. For example, set-aside funds from the youth mentoring grant appropriation account must be used to research or evaluate mentoring programs. For other accounts, OJJDP can elect to fund research and evaluation efforts in a number of different areas. For example, under Juvenile Accountability Block Grants, OJJDP provides funds to states and units of local government to strengthen the juvenile justice system. The states can use these funds for 17 different purpose areas, including establishing programs to help the successful reentry of juvenile offenders from state and local custody in the community or for hiring staff or developing training programs for detention and corrections. Consequently, there are limits on the amount of funds OJJDP can divert to research and evaluation and on its discretion over how to use of some of these funds. In fiscal year 2008, the appropriation act allowed OJJDP to set aside more than $23 million that could be dedicated to research and evaluation efforts on numerous eligible programs. Because OJJDP has to decide how to split set aside funds between supporting state and local program implementation and program evaluation, in accordance with federal guidelines, a comprehensive evaluation plan that in part identifies its funding resources could help OJJDP make this determination. According to OJJDP, the office has spent several years considering developing a plan to provide a road map for how it would meet its research and evaluation goals. However, officials stated that it has been difficult to complete a comprehensive evaluation plan to fulfill OMB’s Program Assessment Rating Tool recommendation because they have not had the resources available—that is, funding and staffing—to develop the plan. Specifically, because funds have not been appropriated for part D since fiscal year 2005, OJJDP has not had a dedicated source of funding that could be used to develop a comprehensive evaluation plan or to fund the research identified by such a plan. Additionally, in 2003, OJJDP reorganized its divisions and, as part of this, dissolved its research division, as well as the training and information dissemination units. According to OJJDP, the intention of the former Administrator who implemented this reorganization was to better integrate these functions throughout the agency. OJJDP officials stated that those staff who were dedicated to research and evaluation work were reassigned to other divisions. Although some of these staff retained the research projects they had at the time, they also assumed new grant management duties. Also, over the past 8 fiscal years, OJJDP’s overall authorized staffing level has decreased from 95 to 76. Specifically, those staff dedicated to research and evaluation decreased from 10 in fiscal year 2002 to 3.5 in fiscal year 2009. According to OJJDP officials, the reduction in staff who were dedicated to research and evaluation has strained the staffing resources that could be used for developing a comprehensive evaluation plan. Although OJJDP cited funding and staffing constraints, the Acting Administrator has made developing a comprehensive evaluation plan a priority and the office is committed to moving forward with developing this plan. Following through with its planning efforts will help OJJDP to meet its research and evaluation goals and better ensure that its resources are being used effectively as stipulated by federal guidelines. As the juvenile justice field—including states and local communities— works to implement programs to lower juvenile recidivism rates and address juvenile substance abuse, it is important that the field has information about which programs have been shown to be effective through program evaluations. The importance of OJJDP’s goal to research and evaluate programs to improve juvenile delinquency underscores the need for a comprehensive plan to evaluate juvenile justice programs, one that identifies resources to be committed to its research and evaluation efforts and outlines the details of how OJJDP will accomplish its research and evaluation goals. OJJDP efforts to publish a fiscal year 2010 program plan in December are positive steps in developing the comprehensive evaluation plan that officials have said they are committed to developing. Having such a plan will provide OJJDP with a road map to help ensure that it meets its research and evaluation goals, uses its limited resources effectively, and contributes to identifying effective programs to help support states and localities. With respect to OJJDP’s efforts to disseminate information about effective programs, NTTAC’s efforts to regularly assess the needs for the information it is disseminating through training and technical assistance are important to helping OJJDP assess the utility of its efforts and make appropriate improvements. We also recognize that OJJDP’s efforts to conduct a needs assessment could help provide important information to NTTAC that can be used in conjunction with its evaluation efforts. Consistent with federal guidelines from OJP and prior GAO reports, assessing the utility of the information disseminated through OJJDP’s Model Programs Guide is also critical to ensuring that such information meets the needs of the juvenile justice field so the field can better implement effective programs. Having a mechanism in place to regularly solicit feedback from the field about the usefulness of the Model Programs Guide would better position OJJDP to assess whether the information it is disseminating through the guide on effective programs regularly meets the needs of its users. To help ensure that OJJDP’s Model Programs Guide is regularly meeting user needs and providing the most helpful information on effective programs, consistent with federal guidelines, we recommend the Administrator of OJJDP develop a cost-effective mechanism for regularly soliciting and incorporating feedback from the juvenile justice field on the usefulness of the information provided in its Model Programs Guide. We provided a copy of this report to the Attorney General for review and comment. On December 3, 2009, OJP provided written comments, which are reprinted in appendix VIII. OJP stated that it agreed with our recommendation and intends to develop a mechanism for regularly soliciting and incorporating feedback from the juvenile justice field on the usefulness of the information provided in its Model Programs Guide by March 31, 2010. OJP also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Attorney General, selected congressional committees, and other interested parties. The report also is available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact Eileen Larence at (202) 512-8777 or larencee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IX. For the purposes of our review, we selected a total of 26 experts to interview—13 of whom had expertise related to juvenile reentry programs, 7 of whom had expertise related to juvenile substance abuse programs, and 6 of whom had both juvenile reentry and substance abuse program expertise. See table 5 for a list of these experts. Of the 26 reentry and substance abuse experts we interviewed, 22 experts—as well as research we reviewed—identified several factors that can help programs achieve intended outcomes, that is, be effective. The following factors, while not an exhaustive list of items for programs to consider when implementing juvenile justice intervention programs, were the most frequently cited by the experts we interviewed: maintaining fidelity to the program; selecting, training, and retaining qualified providers; conducting needs-based assessments to provide individualized treatment; and improving juvenile program participation by engaging and motivating juvenile and family involvement. While incorporating these factors into reentry or substance abuse programs does not guarantee that any particular intervention program will be successful, existing programs that have been evaluated and found to be effective have generally included these factors in their designs or implementation. According to 17 of the 22 experts, maintaining fidelity to the program as it was intended to be implemented can help programs achieve their intended objectives. This factor focuses on ensuring that core program services or intervention components are delivered as they were designed, that is, with fidelity. For example, for cognitive behavioral therapy, this would entail that core intervention components, such as cognitive and social skills training, were provided exactly as they were designed to each participant. According to one expert’s research, the degree to which an intervention program is delivered with fidelity is closely related to its effects on recidivism. Another expert concurred, stating that the more closely core program services or intervention components are implemented as they were designed, the more the intervention program will reduce recidivism rates. For example, one expert emphasized the importance of maintaining fidelity to the program when replicating the model within a specific community. In particular, another expert explained that some therapists tend to substitute their own preferred treatment techniques instead of using those prescribed by the intervention program, which can affect how effective a program is at reducing recidivism. This is particularly true if the intervention program being delivered is a program that has been evaluated and found to be effective. Furthermore, one of these experts stated that the specific model chosen has less of an effect on intended outcomes than the manner in which it is delivered. As another expert explained it, a weaker intervention—one that has not been evaluated and proven to be effective—may result in decreased recidivism rates, for example, if it is implemented as designed, while an effective intervention program that is implemented poorly may have little or no effect on intended outcomes. According to 19 of the 22 experts, selecting, training, and retaining qualified providers can help intervention programs achieve intended outcomes. For example, the quality of the services that cognitive behavioral therapy delivers depends, in part, on the provider’s ability and whether the provider has been trained on the specific therapies and components of the intervention program. Three of these experts noted that if providers are not appropriately trained in the therapy or intervention being implemented, they may not provide the program as it was intended, or as one of them noted, may substitute their own preferred treatment techniques for those prescribed by the intervention program. As a result, the providers’ failure to deliver the intervention program as it was designed reduces the ability of the program to achieve intended outcomes. Furthermore, many intervention programs utilize providers who have certain educational or clinical experience, such as having a background in mental health or being a licensed practitioner for the specific therapy being implemented. One of the 19 experts we interviewed also mentioned the importance of gaining the support of the juvenile justice community, as well as agencies’ program management, in the selection and training of providers. According to 18 of the 22 experts, by assessing a juvenile’s specific treatment needs, program providers can better design intervention programs that will be targeted to a juvenile’s individual situation. For example, 4 experts noted that this can help intervention programs achieve intended outcomes because individualized treatment is more likely to affect participants’ individual outcomes since it takes into account differences such as age, gender, culture, environment, and problem severity. One expert noted that individualized treatment ensures that juveniles do not receive unnecessary treatment, which in some instances may produce harmful results. According to this expert, providing juveniles who do not have substance abuse problems some programs, such as cognitive behavioral therapy, may lead to harmful results because these juveniles are exposed to others who have more serious addictions. Four experts also noted that using needs-based assessments to develop individualized treatment plans can be more cost beneficial than using standard treatment plans. Specifically, as one of these experts noted, this is because individualized treatment plans can help ensure that costly interventions are not provided to juveniles who do not need extensive services. In addition, 5 experts stated that conducting a risk-based assessment is important to determining which juveniles are at higher risk of reoffending in order to focus programming efforts on them. One of these experts cited a study that shows that targeting specific treatment needs of offenders is correlated with recidivism outcomes, that is, providing targeted treatment needs is generally related to lower recidivism. According to 16 of the 22 experts, engaging and motivating juvenile and family involvement can help to improve a juvenile’s program participation, thereby helping intervention programs to achieve intended outcomes. For example, 1 expert noted that successful programs rely on staff members to gain the trust of juvenile offenders. These programs also recognize that juveniles may experience program fatigue because they are participating in numerous programs and that motivation may become an issue. In addition, this expert noted that after being released into the community, juveniles and their families may not be motivated to participate in intervention programs. Additionally, research has shown that encouraging families to participate in the juvenile’s treatment program can reduce family risk factors for delinquency. Eleven experts also mentioned that motivating juvenile offenders and their families to participate can assist juveniles in successfully completing an intervention program. Two of these experts noted that by involving family members in treatment, some issues that may contribute to juvenile dropout rates, such as a history of traumatic stress and family members who also abuse substances, can be addressed within an intervention program. To prevent and respond to juvenile delinquency and help states improve their juvenile justice systems, the Office of Juvenile Justice and Delinquency Prevention (OJJDP) administers a wide variety of grants to states, territories, localities, and public and private organizations through formula, block, and discretionary grant programs. The office also provides training and technical assistance, produces and distributes publications and other products containing information about juvenile justice topics, and funds research and evaluation efforts. Table 6 shows funding by fiscal year from 2007 through 2009 for the appropriation accounts for juvenile justice programs. From fiscal years 2007 through 2009, OJJDP allocated approximately $33 million through discretionary grants to four juvenile reentry grant programs and three juvenile substance abuse programs. See table 7 for a description of these reentry and substance abuse programs and the amounts OJJDP awarded to grantees. Of the 19 reentry experts we interviewed, 9 had specific experience or knowledge related to wraparound/case management and 8 experts had positive comments about the effectiveness of this program. In general, wraparound/case management interventions involve making an array of individualized services and support networks available to juveniles, rather than requiring them to enroll in treatment programs that may not address individual needs. According to OJJDP, the goal of wraparound/case management programs is to keep delinquent juveniles at home and out of institutions whenever possible. The basic elements that constitute a wraparound program include, among other things, (1) a collaborative, community-based interagency team responsible for designing, implementing, and overseeing the intervention program in a given jurisdiction; (2) care coordinators who are responsible for helping juveniles create customized treatment programs, among other things; (3) juvenile and family teams consisting of family members and community members who work together to ensure the juvenile’s needs are met at home, at school, and in the community; and (4) a plan of care developed and updated by all members of the juvenile and family teams that identifies the juvenile’s strengths and weaknesses, targets specific goals such as improved performance in school, and outlines how to achieve them. Of these nine experts, eight provided positive opinions of the results of wraparound/case management intervention programs. For example, an expert commented about how in one specific wraparound intervention program, a single case manager is assigned to a juvenile and is responsible for determining the services the juvenile is to receive based on his or her specific needs, instead of enrolling the juvenile into a treatment program that may not be as beneficial for the juvenile. Two of these eight experts noted that there was a lack of evaluations demonstrating effectiveness of these intervention programs but pointed us to a study on a specific wraparound/case management intervention program, Wraparound Milwaukee, that showed potentially promising results related to a reduction in recidivism rates for juvenile offenders. However, these experts stressed that this study alone did not conclusively demonstrate the effectiveness of wraparound/case management programs. The ninth expert stated that in her experience, wraparound/case management interventions are not effective because, for example, a juvenile is placed into this intervention program based on the availability of program staff and resources rather than program services being tailored to the individual needs of the juvenile. Additionally, three of the nine experts cautioned that these intervention programs are difficult to implement because of such issues as a lack of quality services or low retention of juveniles and their families in the intervention being provided. Specifically, one of these experts noted that the quality of wraparound services can vary depending on a community’s resources. In addition, another expert emphasized the importance of obtaining buy-in from diverse service providers who may be used to working on their own, such as within the welfare, foster care, and public school systems. Of 19 reentry experts we interviewed, 15 had specific experience or knowledge related to aftercare programs and 6 cited a lack of conclusive evidence of effectiveness of the program type. Aftercare intervention programs are intended to prepare juvenile offenders to return to the community during the reentry process by focusing on the delivery of services and supervision that start while juveniles are incarcerated and continue after they return to their communities. Specifically, aftercare programs collaborate with the community and marshal its resources to help ensure that juvenile offenders receive services that address their individual needs, such as treatment for a substance abuse problem. These intervention programs focus on changing individual behavior thereby preventing further delinquency. For example, an aftercare program might incorporate the use of techniques from an intervention therapy, such as motivational enhancement therapy, to engage juvenile offenders in treatment and increase their commitment to change. Of these 15 experts, 7 offered positive opinions regarding aftercare intervention programs, based on their own experience or knowledge of the intervention programs. For example, 1 expert noted that if aftercare included intervention programs that were proven to be effective, used assessment tools that identified the individual needs of the juvenile, and implemented the therapies as they were designed, then the aftercare intervention program should be effective at reducing recidivism rates. Although these experts could not provide examples of studies that had been conducted to show evidence of the effectiveness of the intervention programs, all 7 of them agreed based on their own experience or knowledge that aftercare interventions are important reentry programs, in part, because they link the juvenile with his or her community and provide regular contact with a caseworker. Additionally, 3 of these 7 experts stated that aftercare could be effective depending on the intervention programs used and if they were delivered as intended. For example, they said that if aftercare includes intervention programs that have proven to be effective, such as cognitive behavioral therapy, and identifies the individual needs of the juvenile, the programs can reduce recidivism. However, 6 of the 15 reentry experts said there was inconclusive evidence to determine whether these programs can be effective in achieving results. Three of these experts based their opinions on an evaluation of the Intensive Aftercare Program that showed inconclusive results about program effectiveness. Specifically, the study found no evidence that the program had its intended impact of reducing recidivism among juveniles who were released back into the community under supervision in the three states that piloted the program. However, the evaluation did find that the three states that implemented the Intensive Aftercare Program model did successfully incorporate most of its core features, which prepared juveniles to transition back into the community. For instance, these states created new Intensive Aftercare Program–specific treatment programs that among other things, prepared juveniles for increased responsibility in the community, facilitated interaction with the community, and worked with the juveniles’ schools and families. The state programs had a large percentage of juveniles involved in various treatment services. Despite the inconclusive results of the study, one expert credited aftercare programs with addressing the issue of juveniles having to deal with different probation officers throughout the reentry process because, in general, aftercare programs assign one probation officer to a juvenile as a consistent point of contact. The evaluation also stated that in order for the general aftercare model to be effective, it must not only provide supervision and services after a juvenile’s release into the community, but also focus on preparing a juvenile for release. The remaining 2 experts opined that aftercare intervention programs had not been shown to be effective at achieving desired results because, for example, the treatment a juvenile receives depends on what services are actually available in the community. Of the 19 reentry experts we interviewed, 11 had specific experience or knowledge related to vocational or job training programs and indicated potential positive outcomes for these programs. According to OJJDP, providing juveniles with employment opportunities during reentry is a common strategy used to try to reduce future criminal behavior. Vocational or job training intervention programs are intended to improve juveniles’ social and educational functioning by, for example, increasing earnings, raising self-esteem, and instilling a positive work ethic. Juveniles can participate in vocational/job training intervention programs while they are incarcerated and after they return to the community. Of the 11 reentry experts, 10 of them had positive comments based on their experience or knowledge of the program type. Specifically, they said that vocational/job training programs were potentially beneficial, in part, if they were applied to older juveniles and if they led to those juveniles getting jobs. The remaining expert said that there is little evidence to demonstrate the effectiveness of these intervention programs. Reentry courts are specialized courts that manage the return of juvenile offenders to the community after they are released from residential facilities. The court manages reentry by using its authority to direct resources to support the offender’s return to the community and promote positive behavior, among other things. For example, a reentry court would oversee a juvenile’s release into the community by assigning a judge to meet with the juvenile once a month. The judge would actively engage the supervising authority, such as a parole officer, in assessing the juvenile’s progress. The judge would also oversee sanctions for violations as well as rewards, like early release from parole, for successful achievement of goals, such as successfully completing a cognitive behavioral therapy intervention program. Of the 19 reentry experts we interviewed, 2 provided comments related to reentry courts and had differing opinions on their effectiveness. One had a negative impression of the courts, stating that the reentry courts do not provide more to a juvenile than a probation officer would. The other commented that he considers concepts encompassed in reentry courts, such as intensity of supervision, to be a best practice when it comes to reentry programs. However, neither was aware of any evaluations of these types of courts. Of the 13 substance abuse experts we interviewed, 10 had specific experience or knowledge related to drug courts that resulted in mixed views of the effectiveness of this program type. Juvenile drug courts are specialized courts established within and supervised by juvenile courts to provide intervention programs, such as cognitive behavioral therapy or family therapy, for substance-abusing juveniles and their families. Juvenile offenders assigned to drug courts are identified by a juvenile court as having problems with alcohol or drugs. The drug court judge maintains close oversight of each case through frequent—often weekly—status hearings with the individuals involved. The judge both leads and works as a member of a team that can comprise representatives from juvenile justice, social services, school and vocational training programs, law enforcement, probation, the prosecution, and the defense. Together, the team determines how best to address the substance abuse and related problems of the juvenile and his or her family. Specifically, of these 10 experts, 5 experts described drug courts as having insufficient evidence to determine program effectiveness. For example, 2 experts mentioned that while some studies show drug courts reducing substance abuse while juveniles were under court supervision, the results did not last after juveniles were no longer being supervised by the courts. Another expert stated that since drug courts tend to be used for juveniles who have their first or second contact with the juvenile justice system, they are ineffective at achieving desired results because they expose these first-time offenders to peers who have more serious substance abuse addictions and therefore might influence them to continue to abuse substances. By contrast, the remaining 4 experts stated that drug courts can be effective at achieving desired results such as reducing substance abuse if, for example, the juvenile is sent to a community where there are intervention programs offered that have been evaluated and have been shown to be effective, such as cognitive behavioral therapy or family therapy intervention programs. One expert cited a study to support the opinion that drug courts supplemented with multisystemic therapy resulted in a decrease in substance abuse by juvenile offenders. Of the 13 substance abuse experts we interviewed, 8 had specific experience or knowledge related to mentoring intervention programs that resulted in mixed views of their effectiveness. Mentoring programs consist of a relationship between two or more people over a prolonged period of time, where an older, more experienced individual provides support and guidance to a juvenile. The goal of mentoring is for the juvenile to develop positive adult contact, thereby reducing risk factors, such as exposure to juveniles who use substances, while increasing positive factors, such as encouragement for abstaining from substance use. In the substance abuse field, juveniles in need of sobriety are teamed with older sponsors to serve as positive role models in helping them become sober. Of these eight experts, four stated that mentoring programs are ineffective or unsuccessful at achieving desired results, such as reducing substance abuse, and that these intervention programs are more effective at preventing at-risk juveniles from engaging in delinquent behavior. Also, one expert stated that there have been too few evaluations conducted on mentoring programs to make a general statement about the relative benefits of mentoring. Conversely, three experts stated that mentoring programs are effective or can be effective if, for example, mentors are trained or if mentoring is combined with another intervention program that has been evaluated and has been shown to be effective, such as multisystemic therapy. Of the 13 substance abuse experts we interviewed, 11 had specific experience or knowledge related to wraparound/case management intervention programs that resulted in mixed views of the program type. Of these 11 experts, 7 stated that wraparound/case management is effective or can be effective if, for example, it is combined with another intervention program that has been evaluated and has shown to be effective, such as multisystemic therapy. Although these experts had limited evidence demonstrating the effectiveness of wraparound/case management, 2 experts cited two studies that show potentially promising results related to a reduction in recidivism. For example, one study showed that juveniles in wraparound/case management receive a number of individualized services, such as mental health treatment for those juveniles who struggle with emotional issues. However, this study stressed that it is difficult to evaluate wraparound/case management in a controlled way since treatment plans are individualized for each juvenile. The other 4 experts stated that wraparound/case management intervention programs are ineffective because, for example, the intervention programs lack follow-through as there are no consequences if a juvenile does not show up for treatment, or there is not yet sufficient evidence to determine their effectiveness. In addition to the National Training and Technical Assistance Center and the Model Programs Guide, OJJDP disseminates information about effective programs through a variety of other efforts. Specifically, the office has developed mechanisms to disseminate information related to effective programs in specific issue areas, such as youth gang activity, disproportionate minority contact, and girls’ delinquency, as described in table 8. In addition to the contact named above, Mary Catherine Hult, Assistant Director; David Alexander; Elizabeth Blair; Ben Bolitzer; Carissa Bryant; Katherine Davis; Sean DeBlieck; Allyson Goldstein; Rebecca Guerrero; Jared Hermalin; Dawn Locke; Lisa Shibata; Janet Temko; and Delia Zee made key contributions to this report.

Summary: State juvenile justice systems face critical problems when it comes to juvenile delinquency issues such as reentry--when offenders return home from incarceration--and substance abuse. GAO was asked to review juvenile reentry and substance abuse program research and efforts by the Department of Justice's (DOJ) Office of Juvenile Justice and Delinquency Prevention (OJJDP) to provide information on effective programs (i.e., whether a program achieves its intended goal) and cost-beneficial programs (i.e., whether the benefits of programs exceeded their costs). This report addresses (1) expert opinion and available research on these types of reentry and substance abuse programs, (2) the extent to which OJJDP assesses its efforts to disseminate information on effective programs, and (3) OJJDP's plans to accomplish its research and evaluation goals. GAO, among other things, reviewed academic literature, and OJJDP's dissemination efforts and research goals. GAO also interviewed OJJDP officials and a nonprobability sample of 26 juvenile justice experts selected based on their experience with juvenile reentry and substance abuse issues. The majority of the juvenile justice reentry and substance abuse experts GAO interviewed cited evidence that shows cognitive behavioral therapy--programs that help individuals change their beliefs in order to change their behavior--and family therapy--programs that treat juveniles by focusing on improving communication with family members--are effective and cost beneficial when addressing reentry and substance abuse issues. For example, two juvenile reentry experts cited studies showing that 1 year after participating in a cognitive behavioral therapy program, participants were less likely to commit another offense than nonparticipants. Additionally, experts cited a study that reported that a family therapy program provides about $80,000 in savings per participant when accounting for savings from a decline in crime, such as the cost the police would have incurred. Most experts indicated that there was limited evidence on the effectiveness and cost benefits of reentry programs, such as aftercare--programs that assist juvenile offenders in returning to their communities during the reentry process--and substance abuse programs, such as drug courts--specialized courts that provide programs for substance-abusing juveniles and their families. GAO reviewed two OJJDP efforts that provide information on effective programs across the range of juvenile justice issues, the National Training and Technical Assistance Center (NTTAC) and the Model Programs Guide. OJJDP has mechanisms in place to regularly assess the utility of the information provided by NTTAC, but does not have such a mechanism for the guide. OJJDP ensures the utility of NTTAC's information through evaluations in accordance with federal guidelines that highlight the importance of regularly soliciting feedback from users. However, OJJDP could better ensure the utility of the information disseminated by the Model Programs Guide by having a mechanism in place to solicit regular feedback from members of the juvenile justice field--for example, program practitioners--that is specifically related to the guide. OJJDP has articulated research and evaluation goals to support its mission of improving the juvenile justice system and is developing plans to assist in meeting these goals. OJJDP is required under the Juvenile Justice and Delinquency Prevention Act, as amended, to publish an annual program plan that describes planned activities under accounts authorized for research and evaluation activities, among other things. Additionally, the Office of Management and Budget (OMB) recommended that OJJDP develop a comprehensive evaluation plan for juvenile justice programs. While OJJDP has not published an annual program plan since 2002, in December of 2009, it issued a proposed plan for public comment and aims to publish the final program plan once public comments are incorporated. Additionally, although the office has considered developing a comprehensive evaluation plan to address OMB recommendations, it had not previously done so because of a lack of resources. However, OJJDP is committed to developing a comprehensive evaluation plan once the program plan is finalized.