WIA sets forth various requirements for the Secretary of Labor relating to research and evaluation of federally funded employment-related programs and activities. The law calls upon the Secretary of Labor to publish in the Federal Register every 2 years a plan that describes its pilot, demonstration, and research priorities for the next 5 years regarding employment and training. Specifically, WIA requires the Secretary to develop the research plan after consulting with states, localities, and send the plan to the appropriate committees of Congress; and take into account such factors as the likelihood that the results of the projects will be useful to policymakers and stakeholders in addressing employment and training problems. Within ETA, OPDR’s Division of Research and Evaluation plans, conducts, and disseminates employment and training-related research and evaluations. Nearly all of the agency’s research and evaluation studies are conducted under contract; these contractors represent a range of research organizations and academic institutions. Furthermore, OPDR plans and conducts its research and evaluation activities in consultation with ETA’s program offices, such as the Office of Workforce Investment and the Office of Trade Adjustment Assistance. ETA’s research and evaluation funding is divided into two separate budget line items: Pilots, demonstrations, and research. Efforts in this category are focused on developing and testing new ways to approach problems and to deliver services. Under WIA, pilots and demonstrations shall be carried out “for the purpose of developing and implementing techniques and approaches, and demonstrating the effectiveness of specialized methods, in addressing employment and training needs.” WIA also states that the Secretary shall “carry out research projects that will contribute to the solution of employment and training problems in the United States.” Evaluations. Efforts in this category are focused on continuing evaluations of certain programs and activities carried out under WIA. These evaluations must address the effectiveness of these programs and activities carried out under WIA in relation to their cost; the effectiveness of the performance measures relating to these programs and activities; the effectiveness of the structure and mechanisms for delivery of services through these programs and activities; the impact of the programs and activities on the community and participants involved, and on related programs and activities; the extent to which such programs and activities meet the needs of various demographic groups; and such other factors as may be appropriate. In program year 2010, ETA’s combined budget appropriation for conducting evaluations and pilots, demonstrations, and research was about $103 million—or nearly $34 million above what the agency requested. (See fig. 1.) About $84 million of the 2010 funds were designated by the Congress for specific projects, including $30 million for Transitional Jobs activities for ex-offenders, and another $5.5 million for competitive grants addressing the employment and training needs of young parents. According to agency documents, in 2008 and 2009, the Congress similarly increased ETA’s requested budget for pilots, demonstrations, and research, at the same time specifically designating how the majority of those funds would be used, including $4.9 million in 2008 and $5 million in 2009 for the young parents’ demonstration. While there is no single or ideal way for government agencies to conduct research, several leading national organizations have developed guidelines that identify key elements that promote a sound research program. These guidelines identify five elements as key: agency resources, professional competence, independence, evaluation policies and procedures, and evaluation plans. Resources. Research should be supported through stable, continuous funding sources and through special one-time funds for evaluation projects of interest to executive branch and congressional policymakers. Professional competence. Research should be performed by professionals with appropriate training and experience for the evaluation activity (such as performing a study, planning an evaluation agenda, reviewing evaluation results, or performing a statistical analysis). Independence. Although the heads of federal agencies and their component organizations should participate in establishing evaluation agendas, budgets, schedules, and priorities, the independence of evaluators must be maintained with respect to the design, conduct, and results of their evaluation studies. Evaluation policy and procedures. Each federal agency and its evaluation centers should publish policies and procedures and adopt quality standards to guide evaluations within its purview. Such policies and procedures should identify the kinds of evaluations to be performed and the criteria and administrative steps for developing evaluation plans and setting priorities, including selecting evaluation approaches to use, consulting experts, ensuring evaluation product quality, and publishing reports. Evaluation plans. Each federal agency should require its major program components to prepare annual and multiyear evaluation plans and to update these plans annually. The planning should take into account the need for evaluation results to inform program budgeting, reauthorization, agency strategic plans, program management, and responses to critical issues concerning program effectiveness. These plans should include an appropriate mix of short- and long-term studies to produce results for short- or long-term policy or management decisions. To the extent practical, the plans should be developed in consultation with program stakeholders. Furthermore, leading organizations, including the American Evaluation Association and the National Academy of Sciences, emphasize the need for research programs to establish specific policies and procedures to guide research activities. Based on several key elements identified by these organizations, we developed a framework comprised of five phases—agenda setting, selecting research, designing research, conducting research, and disseminating research results. (See fig. 2.) Agenda setting. Agencies should establish a structured process for developing their research priorities. The process should identify how agencies set research priority areas and provide for updating the areas on a regular basis. The process should also allow for the consideration of critical issues and state how internal and external stakeholders will be included in developing the plan. Selecting research. At this phase, the process should identify how the research program’s staff identifies and selects studies to fund, including the criteria it uses to make those decisions. Steps might describe how the staff assembles a list of potential studies, works with internal program offices, and makes final decisions. Designing research. During the design phase, the process should identify steps taken to select appropriate research approaches and methods and the safeguards in place to ensure appropriate tradeoffs are made between what is desirable and what is practical and between the relative strengths and weaknesses of different methods. Conducting research. At this stage, the process should include policies and procedures to guide the conduct of research. The process should ensure that key events, activities, and time frames are specified and that knowledgeable staff in the sponsoring agency monitor the implementation of the research. Disseminating research. This process should describe how research findings are made available to the public and disseminated to all potential users. These dissemination methods should include safeguards to ensure research findings are disseminated in a timely manner and are accessible through the Internet with user-friendly search and retrieval technologies. In this report, we use several technical terms in describing ETA’s research designs and study characteristics. (See table 1.) Our expert panel generally considered ETA’s research areas to be the right ones for the period the research plan covered. About three-fourths of the panel members reported that ETA’s 2007 to 2012 research agenda reflected key employment and training issues to at least a moderate extent. However, a few experts commented that some of ETA’s research areas may be too broad and lack specificity. The areas in ETA’s 2007 to 2012 research plan covered a range of issues, from job training to postsecondary education. Table 2 illustrates the scope of ETA’s research areas. With regard to the specific studies within these research areas, ETA invested most of its research and evaluation resources in work that focused on increasing the labor market participation of underutilized workers and on UI. Of the estimated $96 million that supported the 58 research reports published between January 2008 and March 2010, more than half—about $56 million—funded research that addressed these two research areas. Other areas received far less funding. For example, funding for studies addressing the methods of expanding U.S. workforce skills and using state-level administrative data to measure progress and outcomes accounted for about $6.5 million, or about 6.7 percent of the cost of studies published during the period we examined. (See table 3.) Overall, the individual studies that ETA funded addressed a wide variety of issues and ranged in cost from about $15,000 to a high of about $22 million. In addition to the research areas covered in ETA’s 2007 to 2012 research plan, experts from our virtual panel suggested that ETA incorporate additional research areas in its future research agenda. Of the research areas identified, over half of our experts (28 of 39) ranked the identification of employment and training approaches that work, and for whom, as one of the top areas that ETA’s future research should address. (See fig. 3.) Without such focus, experts commented that it will be difficult to know how to improve the nation’s workforce system. Other issues ranked at the top by experts included research on job creation strategies and the impact of long-term and short-term training. (See app. III for more information on issue-area rankings.) In addition to identifying overall employment and training areas, including issues related to UI, experts also identified specific aspects of the UI system that could be examined in ETA’s future research. In particular, most experts (34 of 39 respondents) reported that it would be at least moderately important, in the future, for ETA to research the linkage between UI and employment and safety net programs, such as Temporary Assistance for Needy Families or the Supplemental Nutrition Assistance Program. (See fig. 4.) This area of research may be particularly important given the role that these programs play in supporting individuals during economic downturns. In addition, many experts (24 of 39 respondents) mentioned that ETA should make the examination of the incentives and disincentives in the UI system a research priority, given the challenge of supporting unemployed workers during difficult economic times, while promoting self-sufficiency through employment.