The mission of the WWC is to be a central source of scientific evidence for what works in education. To accomplish this, the WWC reviews existing education research and posts information based on its reviews on the WWC Web site, http://ies.ed.gov/ncee/wwc/. The types of products currently available on the WWC Web site are described in table 1. In addition to the Clearinghouse, Education provides other technical assistance and research-related resources to assist states, districts, and schools. Examples of research-related resources include the Regional Educational Laboratories (REL) and the Doing What Works (DWW) Web site (http://dww.ed.gov): Regional Educational Laboratories. IES’s Regional Educational Laboratory Program is a network of 10 laboratories that conduct research and provide policymakers and practitioners with expert advice, training, and technical assistance on how to interpret findings from scientifically valid research. Doing What Works. Led by Education’s Office of Planning, Evaluation and Policy Development, DWW is a Web-based resource intended to help teachers, schools, districts, states and technical assistance providers implement research-based instructional practice. In 2002, IES awarded a $27 million 5-year contract to the initial contractors to operate the Clearinghouse. The WWC contractors developed the Clearinghouse’s research review standards with IES and reviewed research related to topic areas considered to be pressing issues in education. One of the goals of the Clearinghouse was to promote informed education decision making through a Web-based dissemination system that featured rigorous reviews of studies on the effectiveness of educational interventions. The WWC experienced a slow start due in part to the amount of work involved in developing a research review and reporting process that was valid, transparent, and replicable, according to the initial contractors. In developing the research review process, the contractors and IES addressed over 60 technical issues, such as determining what constitutes an acceptable level of participant loss (attrition) from a study and what methods should be in place to accommodate common education research techniques. In addition, initial plans for topic areas and reporting formats were modified. For example, IES decided to drop one planned topic area because IES officials determined it to be too broad. The WWC and IES also spent a substantial amount of time developing and refining a reporting format to communicate research results to a lay audience. As a result, the WWC began releasing reports in 2006. By September 2007, the WWC had released 89 intervention reports, six topic reports, and three practice guides. The WWC uses a three-step review process to assess the quality of studies and report on what the research indicates about the effectiveness of interventions. The WWC definition of interventions includes programs (such as whole school reform), products (such as a textbook or curriculum), practices (such as mixed-age grouping), or policies (such as class size reduction). The process begins with an initial screening of published and unpublished studies relevant to the intervention being reviewed. Studies are collected from electronic databases, journals, conference proceedings, and nominations solicited from the general public. The studies that pass initial screens are reviewed to determine whether they provide valid evidence of an intervention’s effectiveness. Using these studies, the WWC then synthesizes the evidence about the effectiveness of an intervention and publishes a report describing its findings. The Clearinghouse categorizes interventions as either having positive effects, potentially positive effects, mixed effects, no discernable effects, potentially negative effects, or negative effects (see fig. 1). The WWC uses evidence standards to assess the strengths and weaknesses of a study’s methodology, such as the type of design it uses, the quality of the study’s data, and the appropriateness of the study’s statistical procedures. Until recently, the WWC accepted two types of study designs—randomized experiments and quasi-experimental studies. Only randomized controlled trials (or randomized experiments) that WWC has determined to be well-designed and well-implemented are considered strong evidence and can receive the highest rating of “meets evidence standards without reservations.” The WWC also considers evidence from quasi-experiments it has determined to be well-designed and well- implemented. The highest rating a study using quasi-experimental design can receive is “meets evidence standards with reservations.” This rating category is intended to inform educators to interpret the study results with caution, as the results may reflect other factors, in addition to the impact of the intervention (see table 2). The WWC is administered by IES through a contract with a private research organization. IES monitors implementation of the specific tasks detailed in the WWC contract by reviewing an annual work plan and monthly performance and expenditure reports submitted by the contractor. IES tracks implementation of the tasks, completion of performance goals, and adherence to the budget outlined in the contractor work plan. The contractor monitors the work of any subcontractors that it uses to perform services such as research reviews, technological support, and communications support. IES is also involved in the development and dissemination of WWC products. IES reviews and approves proposed topics for WWC products, product formats, and the research review procedures. It also coordinates a group of independent researchers to peer review WWC products and reviews and approves all WWC products prior to public release. IES required the contractor to develop a communications plan to inform WWC customers about features of the Web site. We found that the WWC review process follows generally accepted practices for reviewing research. Specifically, GAO’s November 2009 report reviewing federally supported efforts to identify effective interventions found that the WWC determines whether a study provides credible evidence on effectiveness based on several dimensions, including the quality of the research design, how the study was implemented, and other technical considerations. Our 2009 report also noted that WWC follows a common approach to conducting its reviews, and provides information to help educators understand the body of existing research on specific interventions. Additionally, a congressionally mandated panel of experts found in October 2008 that WWC’s research review process was based on appropriate methods for judging the strength of the evidence regarding the effectiveness of interventions. For example, the panel agreed that the minimum qualifications a study must meet in order to be reviewed by the WWC are appropriate. The panel also found that WWC’s reporting process is reasonable and that the WWC provides succinct and relevant evidence on the effectiveness of education interventions. While the panel concluded that the WWC’s processes are generally appropriate, the panel made several recommendations to the WWC for continued improvement. The recommendations primarily related to establishing or clarifying procedures, reviewing statistical methods, and documenting the screening process. The WWC implemented or is considering implementing 14 of the panel’s 17 recommendations. The WWC implemented nine recommendations, in part by modifying some procedures and creating a procedures and standards handbook. For example, in response to the panel’s recommendation that the WWC include a table of study dispositions (e.g., whether studies meet WWC evidence standards) at the front of intervention reports, the WWC is modifying the report template to include a summary table along with the existing listing of dispositions in the reference section. The WWC also addressed panel concerns about technical issues in its review process by making its treatment of study attrition—the rate at which subjects drop out of a study—more consistent across topic areas. The panel noted that the WWC’s practice of determining acceptable attrition levels by topic area led to arbitrary inconsistencies across the topic areas. In response to the panel’s recommendation that the WWC reconsider this practice, the WWC took steps to increase its consistency by developing attrition guidance that applies to all topic areas. (See app. III for a table detailing the recommendations, WWC and IES’s response, and the status of any changes made in response to recommendations.) In addition, the WWC is considering implementing five other panel recommendations. For example, the panel raised concerns that the WWC does not document some potential conflicts of interest for the studies it reviews. In response to this concern, the WWC is considering tracking and publishing whether studies of a program are funded or conducted by the program’s developers. Further, in response to the panel’s concern that the WWC’s screening process may exclude some eligible studies, the WWC is undertaking an evaluation of the reliability of its screening process. According to IES officials, they will postpone decisions about the recommendations until the newly appointed Commissioner for the WWC is on board and actively involved in the decision making. Some researchers claim that the WWC presents potentially misleading information by including brief experiments involving small numbers of students when evaluating interventions. As a result, according to critics, educators may accept the WWC’s rating of the intervention’s effectiveness, even though the evidence behind the rating is limited. One researcher suggested the WWC emphasize larger studies that span significant periods of time and set a minimum sample size requirement. According to WWC staff, such changes would exclude valuable research and prevent the WWC from providing educators with research-based information about some interventions. Instead of changing its treatment of sample size and study duration, the WWC began publishing information on the extent of the evidence supporting its findings in 2007. The WWC’s “extent of evidence” rating alerts educators when the WWC effectiveness ratings are based on a small amount of evidence. As figure 2 shows, 76 percent of interventions with positive or potentially positive ratings of effectiveness are based on a small amount of evidence (see fig. 2). Further, researchers suggested that the WWC presents misleading information by rating interventions based on studies in which measures of student performance closely match the content taught to the intervention group, but not the control group. In such studies, higher test scores among the intervention group may not accurately represent the effectiveness of the intervention more generally. The researchers suggested that the WWC exclude such measures, or at least report on them separately.