DHS’s OCHCO is responsible, in broad terms, for the strategy, oversight, and planning of DHS employee training and development. At the same time, each DHS component, such as CBP and TSA, also has its own human capital office and training and development functions. In practice, DHS’s OCHCO focuses on department-wide efforts while each component focuses on ensuring its employees are trained and developed to meet its specific mission needs. In addition, FLETC, a component of DHS, offers and delivers law enforcement training to DHS components, including those in our review—CBP, ICE, TSA, and the Coast Guard. FLETC also serves as an interagency law enforcement training organization for more than 90 federal partner organizations, as well as state, local, tribal, and international entities. Table 1 provides a summary of training and development responsibilities at OCHCO and DHS components selected for our review. In 2009, OPM developed and published regulations that require agencies to regularly evaluate training programs. Among other things, these regulations require agencies to evaluate their training programs annually to determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals. The training and development process can loosely be segmented into four broad, interrelated elements: (1) planning/front-end analysis, (2) design/development, (3) implementation, and (4) evaluation. The four elements help to produce a strategic approach to federal agencies’ training and development efforts. One commonly accepted training evaluation model, which is endorsed by OPM in its training evaluation guidance, is known as the Kirkpatrick model. This model is commonly used in the federal government, including at DHS. The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. The following is a description of what each level within the Kirkpatrick model is to accomplish: Level 1: The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training. Level 2: The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course. Level 3: The third level measures how training affects changes in behavior on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on the job. Level 4: The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before and after a training modification was made. Figure 1 highlights the elements of the training development process, from the planning stage through the implementation and evaluation of training, and depicts how the Kirkpatrick model complements the training development process. In addition to utilizing this training development process, agencies may also seek Federal Law Enforcement Training Accreditation (FLETA) for some or all of their training programs and academies. By attaining FLETA accreditation for their training academies or programs, agencies provide assurance that they have voluntarily submitted to a process of self- regulation and have successfully achieved compliance with a set of standards that demonstrate their adherence to quality, effectiveness, and integrity. FLETA accreditation also helps maintain training standards by ensuring that training programs are comprehensively evaluated, using Kirkpatrick levels 1 through 3 or an equivalent approach, within a 5-year period. In October 2010, DHS issued its DHS Learning Evaluation Guide. DHS created the guide to help the department’s learning and development community evaluate the effectiveness of training activities in a diverse organization with varied training needs. Among other things, the guidance gives an overview of best practices and provides components with tools they can use to implement the Kirkpatrick model in their training evaluations. In addition, the guide highlights the need for a training evaluation plan to identify and address (1) what is being evaluated, (2) how it is being evaluated, (3) when it is being evaluated, and (4) the factors involving stakeholder expectations, such as agency policies and procedures. In 2004, the Secretary of Homeland Security announced the “One DHS” policy, which identified the need for a common leadership competency framework across the department, as well as a unified training curriculum for current and future leaders. Accordingly, DHS established the LDP Office in May 2010 under the Office of the Chief Human Capital Officer to design, develop, and execute a department-wide leadership program that would strengthen leadership at all levels of the DHS workforce. Through the LDP, all DHS components are to invest in developing leaders with skills that transfer across the department, yet retain the agility to balance this with their own mission-focused leader development needs. In January 2011, DHS also developed the Leader Development Framework to serve as a 3- to 5-year strategic roadmap for implementing the LDP. This framework consists of five tiers of leader development programs for employees of different levels, such as the executive and supervisory levels. In February 2013, DHS issued a directive—Directive 258-02: Leader Development—formally establishing responsibilities and policies related to leader development at DHS through the LDP, as well as instructions for implementing the directive. This directive specifies that the LDP is, among other things, to delineate requirements and activities to be implemented by components. The LDP is also to develop and manage centrally coordinated and high-potential programs for developing employees to fill future leader positions. DHS components have documented processes in place for evaluating their training programs and have used evaluation feedback to improve their training offerings; however, their documented processes varied on the extent to which selected attributes of an effective training evaluation process were included. Further, DHS identified opportunities for efficiencies and cost savings, but varying approaches for capturing training costs across the department affect DHS’s ability to reliably capture and track its training costs department-wide. The five DHS components in our review have a documented process in place for evaluating their training programs using the Kirkpatrick four-level model. However, their documented evaluation processes varied on the extent to which they included selected attributes of an effective evaluation process. Components use the results of their evaluations to make improvements to the training programs and assess training needs. For example, components used evaluation feedback to improve the delivery of training content, such as through additional hands-on training, and the use of e-learning. Table 2 provides such an example for each component. OPM guidance on training evaluation, DHS’s learning evaluation guidance, and our prior work on effective training and development programs identify various attributes for effective training evaluations. Consistent with these criteria, the attributes of an effective training evaluation process include communicating (1) the goals the training programs are supposed to achieve, (2) which training programs will be evaluated, (3) the methodology for conducting the evaluations, (4) timeframes for conducting the evaluation, (5) roles and responsibilities for evaluation efforts, and (6) how the evaluation results will be used. All DHS components in our review reflected a number of the attributes of an effective training evaluation process in their documentation. For example, all components included information on identifying goals that the training programs are to achieve, identifying which training programs are to be evaluated, and explaining how the evaluation results will be used. Table 3 presents information on the extent to which each component’s documented evaluation process includes these attributes, and additional details about component ratings are explained in appendix II. However, components varied on the extent to which they included information in their documentation about evaluation methodologies, timeframes, and roles and responsibilities for evaluation. For example, Evaluation methodologies: Each component’s documentation indicates that its training programs are to be evaluated using the Kirkpatrick model. However, only ICE’s and the Coast Guard’s documentation specify the methods to be used when performing each Kirkpatrick level of evaluation. The other three components’ documentation does not specify how each Kirkpatrick level of evaluation is to be performed in practice. Timeframes: ICE and FLETC’s documented evaluation processes fully presents timeframes for evaluations, but documentation for the remaining three components does not. For example, all of the components’ documentation identified the timeframes for the initial steps of completing evaluation surveys and collecting data. However, ICE and FLETC’s documentation also communicated timeframes for the subsequent steps for analyzing the results of the evaluation. Roles and responsibilities: We found that three of five components did not consistently outline the roles and responsibilities for the evaluation efforts. For instance, two components, CBP and ICE, communicate information on roles and responsibilities for some parts of the process, but do not present this information for others. TSA’s documentation did not communicate information on the specific roles and responsibilities within the Office of Training and Workforce Engagement (OTWE) for evaluation activities. Officials at one component, CBP, told us that since training at CBP is more decentralized through separate academies that follow their own processes, their documentation did not include some attributes of effective training and development programs, as their training standards were intended to be more of a “how-to guide” rather than a formal step- by-step methodology. According to TSA officials, they did not fully include certain attributes such as explaining the methodology to be used to conduct the evaluations and defining roles and responsibilities because their process is still under development and agreement on this information has not yet been reached internally. TSA officials stated that their documentation is to be finalized by the first quarter of fiscal year 2015. Officials at the Coast Guard told us that their documentation did not include information on the timeframes for analyzing the results, but they plan to rectify this with their current effort in fiscal year 2014 to revise their evaluation processes.