The Justice Assistance Act of 1984 (P.L. 98-473) created OJP to provide federal leadership in developing the nation’s capacity to prevent and control crime, administer justice, and assist crime victims. OJP carries out its responsibilities by providing grants to various organizations, including state and local governments, Indian tribal governments, nonprofit organizations, universities, and private foundations. OJP comprises five bureaus, including BJA, and seven program offices, including VAWO. In fulfilling its mission, BJA provides grants for programs and for training and technical assistance to combat violent and drug-related crime and help improve the criminal justice system. VAWO administers grants to help prevent and stop violence against women, including domestic violence, sexual assault, and stalking. During fiscal years 1995 through 2001, BJA and VAWO awarded about $943 million to fund 700 Byrne and 1,264 VAWO discretionary grants. One of BJA’s major grant programs is the Byrne Program. BJA administers the Byrne program, just as its counterpart, VAWO, administers its programs. Under the Byrne discretionary grants program, BJA provides federal financial assistance to grantees for educational and training programs for criminal justice personnel; for technical assistance to state and local units of government; and for projects that are replicable in more than one jurisdiction nationwide. During fiscal years 1995 through 2001, Byrne discretionary grant programs received appropriations of about $385 million. VAWO was created in 1995 to carry out certain programs created under the Violence Against Women Act of 1994. The Victims of Trafficking and Violence Prevention Act of 2000 reauthorized most of the existing VAWO programs and added new programs. VAWO programs seek to improve criminal justice system responses to domestic violence, sexual assault, and stalking by providing support for law enforcement, prosecution, courts, and victim advocacy programs across the country. During fiscal years 1995 through 2001, VAWO’s five discretionary grant programs that were subject to program evaluation were (1) STOP (Services, Training, Officers, and Prosecutors) Violence Against Indian Women Discretionary Grants, (2) Grants to Encourage Arrest Policies, (3) Rural Domestic Violence and Child Victimization Enforcement Grants, (4) Domestic Violence Victims’ Civil Legal Assistance Grants, and (5) Grants to Combat Violent Crimes Against Women on Campuses. During fiscal years 1995 through 2001, about $505 million was appropriated to these discretionary grant programs. As already mentioned, NIJ is the principal research and development agency within OJP, and its duties include developing, conducting, directing, and supervising Byrne and VAWO discretionary grant program evaluations. Under 42 U.S.C. 3766, NIJ is required to “conduct a reasonable number of comprehensive evaluations” of the Byrne discretionary grant program. In selecting programs for review under section 3766, NIJ is to consider new and innovative approaches, program costs, potential for replication in other areas, and the extent of public awareness and community involvement. According to NIJ officials, the implementation of various types of evaluations, including process and impact evaluations, fulfills this legislative requirement. Although legislation creating VAWO does not require evaluations of the VAWO discretionary grant programs, Justice’s annual appropriations for VAWO during fiscal years 1998 through 2002 included monies for NIJ research and evaluations of violence against women. In addition, Justice has promulgated regulations requiring that NIJ conduct national evaluations of two of VAWO’s discretionary grant programs. As with the Byrne discretionary programs, NIJ is not required by statute or Justice regulation to conduct specific types of program evaluations, such as impact or process evaluations. The Director of NIJ is responsible for making the final decision on which Byrne and VAWO discretionary grant programs to evaluate; this decision is based on the work of NIJ staff in coordination with Byrne or VAWO program officials. Once the decision has been made to evaluate a particular program, NIJ issues a solicitation for proposals for grant funding from potential evaluators. When applications or proposals are received, an external peer review panel comprising members of the research and relevant practitioner communities is convened. Peer review panels identify the strengths, weaknesses, and potential methodologies to be derived from competing proposals. When developing their consensus reviews, peer review panels are to consider the quality and technical merit of the proposal; the likelihood that grant objectives will be met; the capabilities, demonstrated productivity, and experience of the evaluators; and budget constraints. Each written consensus review is reviewed and discussed with partnership agency representatives (e.g., staff from BJA or VAWO). These internal staff reviews and discussions are led by NIJ’s Director of the Office of Research and Evaluation who then presents the peer review consensus reviews, along with agency and partner agency input, to the NIJ Director for consideration and final grant award decisions. The NIJ Director makes the final decision regarding which application to fund. To meet our objectives, we conducted our work at OJP, BJA, VAWO, and NIJ headquarters in Washington, D.C. We reviewed applicable laws and regulations, guidelines, reports, and testimony associated with Byrne and VAWO discretionary grant programs and evaluation activities. In addition, we interviewed responsible OJP, NIJ, BJA, and VAWO officials regarding program evaluations of discretionary grants. As agreed with your offices, we focused on program evaluation activities associated with the Byrne and VAWO discretionary grant programs. In particular, we focused on the program evaluations of discretionary grants that were funded during fiscal years 1995 through 2001. To address our first objective, regarding the number, type, status of completion, and award amount of Byrne and VAWO discretionary grant program evaluations, we interviewed NIJ, BJA, and VAWO officials and obtained information on Byrne and VAWO discretionary grant programs and program evaluations. Because NIJ is responsible for carrying out program evaluations of Byrne and VAWO discretionary grant programs, we also obtained and analyzed NIJ data about specific Byrne and VAWO discretionary grant program evaluations, including information on the number of evaluations as well as the type, cost, source of funding, and stages of implementation of each evaluation for fiscal years 1995 through 2001. We did not independently verify the accuracy or completeness of the data that NIJ provided. To address the second objective, regarding the methodological rigor of the impact evaluation studies of Byrne and VAWO discretionary grant programs during fiscal years 1995 through 2001, we initially identified the impact evaluations from the universe of program evaluations specified by NIJ. We excluded from our analysis any impact evaluations that were in the formative stage of development—that is, the application had been awarded but the methodological design was not yet fully developed. As a result, we reviewed four program evaluations. For the four impact evaluations that we reviewed, we asked NIJ to provide any documentation relevant to the design and implementation of the impact evaluation methodologies, such as the application solicitation, the grantee’s initial and supplemental applications, progress notes, interim reports, requested methodological changes, and any final reports that may have become available during the data collection period. We also provided NIJ with a list of methodological issues to be considered in our review and requested them to submit any additional documentation that addressed these issues. We used a data collection instrument to obtain information systematically about each program being evaluated and about the features of the evaluation methodology. We based our data collection and assessments on generally accepted social science standards. We examined such factors as whether evaluation data were collected before and after program implementation; how program effects were isolated (i.e., the use of nonprogram participant comparison groups or statistical controls); and the appropriateness of sampling and outcome measures. Two of our senior social scientists with training and experience in evaluation research and methodology separately reviewed the evaluation documents and developed their own assessments before meeting jointly to discuss the findings and implications. This was done to promote a grant evaluation review process that was both independent and objective. To obtain information on the approaches that BJA, VAWO, and NIJ used to disseminate program evaluation results, we requested and reviewed, if available, relevant handbooks and guidelines on information dissemination, including, for example, NIJ’s guidelines. We also reviewed BJA, VAWO, and NIJ’s available print and electronic products as related to their proven programs and evaluations, including two NIJ publications about Byrne discretionary programs and their evaluation methodologies and results. We conducted our work between February 2001 and December 2001 in accordance with generally accepted government auditing standards. We requested comments from Justice on a draft of this report in January 2002. The comments are discussed near the end of this letter and are reprinted as appendix III. During fiscal years 1995 through 2001, NIJ awarded about $6 million to carry out five Byrne and five VAWO discretionary grant program evaluations. NIJ awarded evaluation grants using mostly funds transferred from BJA and VAWO. Specifically, of the approximately $1.9 million awarded for one impact and four process evaluations of the Byrne discretionary program, NIJ contributed about $299,000 (16 percent) and BJA contributed about $1.6 million (84 percent). VAWO provided all of the funding (about $4 million) to NIJ for all program evaluations of five VAWO discretionary grant programs. According to NIJ, the five VAWO program evaluations included both impact and process evaluations. Our review of information provided by NIJ showed that 6 of the 10 program evaluations—all 5 VAWO evaluations and 1 Byrne evaluation—included impact evaluations. The remaining four Byrne evaluations were exclusively process evaluations that measured the extent to which the programs were working as intended. As of December 2001, only one of these evaluations, the impact evaluation of the Byrne CAR Program, had been completed. The remaining evaluations were in various stages of implementation.