DHS approached the 9/11 Commission Act requirement for a quadrennial homeland security review in three phases. In the first phase, DHS defined the nation’s homeland security interests, identified the critical homeland security missions, and developed a strategic approach to those missions by laying out the principal goals, objectives, and strategic outcomes for the mission areas. DHS reported on the results of this effort in the February 2010 QHSR report in which the department identified 5 homeland security missions, 14 associated goals, and 43 objectives, as shown in figure 1. The QHSR report also identified a strategy for maturing and strengthening the homeland security enterprise, with 18 associated objectives. In the second phase—the BUR—DHS identified its component agencies’ activities, aligned those activities with the QHSR missions and goals, and made recommendations for improving the department’s organizational alignment and business processes. DHS reported on the results of this second phase in the July 2010 BUR report. In the third phase DHS developed its budget plan necessary to execute the QHSR missions. DHS presented this budget plan in the President’s fiscal year 2012 budget request, issued February 14, 2011, and the accompanying Fiscal Year 2012-2016 FYHSP, issued in May 2011. DHS officials stated that together, these three phases and their resulting reports and documents address the 9/11 Commission Act requirement for the quadrennial homeland security review. DHS initiated the QHSR in August 2007. Led by the DHS Office of Policy, in July 2009 the department issued its QHSR terms of reference, outlining the framework for conducting the quadrennial review and identifying threats and assumptions to be used in conducting the review. Through the terms of reference, DHS identified the initial four homeland security missions to be studied, which were refined during the QHSR process— Counterterrorism and Domestic Security Management; Securing Our Borders; Smart and Tough Enforcement of Immigration Laws; and Preparing For, Responding To, and Recovering from Disasters—as well as three other separate, nonmission study areas to be part of the review—DHS Strategic Management, Homeland Security National Risk Assessments, and Homeland Security Planning and Capabilities. The fifth QHSR mission on Safeguarding and Securing Cyberspace was added after DHS issued the terms of reference. A sixth category of DHS activities—Providing Essential Support to National and Economic Security—was added in the fiscal year 2012 budget request but was not included in the 2010 QHSR report. DHS established seven study groups for the QHSR, which were composed of officials from across DHS offices and components. The study groups were each led by a DHS official and facilitated by an independent subject matter expert from the Homeland Security Studies and Analysis Institute. These study groups conducted their analysis over a 5-month period and shared their work products, such as outlines of missions and assumptions, with other stakeholder groups in order to develop goals and objectives for each mission. At the end of the study group period, DHS senior leadership, including the Deputy Secretary of Homeland Security, the General Counsel, and office and component heads, met multiple times to review and discuss the study group recommendations. The DHS Office of Policy consolidated the study groups’ recommendations into a draft QHSR report and obtained and incorporated feedback on the draft report from other federal agencies and stakeholder groups, including the stakeholders listed in the 9/11 Commission Act, with which DHS was to consult in conducting the QHSR. Agreement on the QHSR report’s final content was reached between the Secretary for Homeland Security and senior White House officials. DHS issued the final QHSR report in February 2010. DHS initiated the BUR in November 2009. Each DHS directorate, component, and office created an inventory of its activities and categorized them according to the QHSR missions. For example, U.S. Immigration and Customs Enforcement (ICE) identified one of its activities as investigating human smuggling and trafficking, which it categorized under the Securing and Managing our Borders QHSR mission. The BUR resulted in a catalog of about 1,300 DHS activities organized under each of the five QHSR missions or categorized as mission or business support activities. DHS identified over 300 potential initiatives for increasing mission performance and accountability and improving department management, derived 43 priority initiatives from this list, and highlighted them in the July 2010 BUR report. For example, under the Enforcing and Administering our Immigration Laws mission, DHS identified as priority initiatives improving DHS’s immigration services processes and dismantling human-smuggling organizations (see fig. 2). In addition, DHS categorized its 43 BUR initiatives according to whether they require organizational, programmatic, policy, or legislative activities in order to be implemented. DHS defines these categories as (1) organizational, where implementation requires some type of departmental reorganization (e.g., create a cybersecurity and infrastructure resilience operational component within DHS); (2) programmatic, where implementation requires budgetary activity, such as a funding increase (e.g., increase efforts to detect and counter nuclear and biological weapons and dangerous materials); (3) policy, where implementation requires a policy decision but no additional funding (e.g., enhance the department’s risk management capability); and (4) legislative, where implementation requires a change in legislation or congressional approval because DHS does not have the legislative authority to implement the initiative (e.g., restore the Secretary’s reorganization authority for DHS headquarters). According to DHS officials, some BUR initiatives require one or more of these types of changes to be implemented, such as the initiative to strengthen internal DHS counterintelligence capabilities, which requires policy and programmatic changes. DHS’s fiscal year 2012 budget request highlighted funding requests to support projects and programs within each QHSR mission. For example, for QHSR mission 1, Preventing Terrorism and Enhancing Security, DHS’s fiscal year 2012 budget request includes requests for 18 projects and programs to support that mission. These requests include items such as $273 million for explosives detection systems at airports and $12.4 million for enhanced watchlist vetting of airline passengers. According to DHS officials, DHS intends to include longer-term project and program funding plans for QHSR missions through annual iterations of its FYHSP. For example, the Fiscal Year 2012-2016 FYHSP contains initiatives and planned performance information aligned with the missions of the department. According to the 9/11 Commission Act, DHS is to report on the results of its QHSR every 4 years with the next report due by December 31, 2013. DHS plans to issue its next QHSR report in accordance with the act. DHS solicited input from various stakeholder groups in conducting the first QHSR. The 9/11 Commission Act required DHS to consult with seven federal agencies in conducting the QHSR—the Departments of Agriculture, the Treasury, Justice, State, Defense, and Health and Human Services and the Office of the Director of National Intelligence. DHS consulted with these agencies and also sought input from a range of other stakeholders, including its directorates, offices, and components; other federal agencies; and nonfederal governmental and nongovernmental entities and representatives, such as state and local governmental associations and individuals working in academia. In obtaining input from these stakeholders, DHS used a variety of mechanisms, such as multiagency working groups, solicitation of homeland security research papers, and a web-based forum, referred to as the National Dialogue, as shown in table 1. We obtained comments from 63 stakeholders who DHS consulted with through these mechanisms. The 63 stakeholders who responded to our request for comments on the QHSR process noted that DHS conducted outreach to them to solicit their views and provided opportunities for them to give input on the QHSR. For example, DHS stakeholders, including its directorates, offices, and components, reported participating in the QHSR process by, for example, helping develop strategic outcomes and measurable end states for the QHSR missions, assigning representatives to the various QHSR study groups, and helping to draft QHSR report language. Stakeholders from 21 federal agencies other than DHS and its components that responded to our request for comments noted that they provided input during the QHSR process by, among other things, having representatives attend QHSR meetings, participating in sub- interagency policy committee meetings, and commenting on draft versions of the QHSR report. Additionally, 6 nonfederal stakeholders reported to us that DHS consulted with them by, for example, sending a representative to association meetings, participating in conference calls to discuss the QHSR, and holding stakeholder briefings to discuss QHSR strategic goals, outcomes, and responsibilities. DHS, QHSR stakeholders, and other entities, specifically the QRAC and NAPA, that reviewed aspects of the QHSR identified various benefits from DHS’s consultation efforts throughout the QHSR. For example, the Deputy Assistant Secretary for Policy (Strategic Plans) stated that stakeholder position paper submissions obtained at the beginning of the QHSR process were beneficial in that study groups had stakeholder input at the outset of the work. The Deputy Assistant Secretary also stated that the National Dialogue was beneficial in that it gave DHS the ability to gauge reactions to proposals for including information in the QHSR in real time, as the National Dialogue represented a virtual discussion among stakeholders. Further, 33 respondents to our request for comments on the QHSR reported that one positive aspect of DHS’s consultations during the QHSR was the range of stakeholders DHS contacted. Two DHS stakeholders reported, for example, that DHS made extensive efforts to involve a wide range of stakeholders and that involvement of federal non-DHS agencies was beneficial in helping DHS obtain views on the QHSR outside of the department. One DHS stakeholder noted that the benefit of involving state, local, and private industry in the QHSR study group discussions via the National Dialogue was that the study groups were able to systematically consider viewpoints of the public during the course of developing the QHSR mission goals and objectives. The public perspectives offered different views than those provided by DHS and other federal stakeholders. Similarly, 2 federal stakeholders responded that the interagency meetings and the National Dialogue were positive ways in which DHS involved stakeholders during the QHSR, and that DHS’s consultations provided a mechanism for interagency collaboration to discuss QHSR goal and objective areas. Additionally, one QRAC member noted that DHS involved and coordinated well with federal agencies; reached out reasonably well to state, local and tribal organizations; included a large number of academics and other policy experts; and gave the American public an opportunity to comment through the National Dialogue. Moreover, in its report on the QHSR, the QRAC noted that while not privy to the details of all inputs received, the QHSR report represented a synthesis of stakeholder consultations that was designed to set forth a shared vision of homeland security in order to achieve unity of purpose across the homeland security enterprise. In addition, with regard to the National Dialogue, NAPA reported that by engaging stakeholders at all levels, DHS was able to incorporate ground-level expertise and specialized knowledge into the review. According to NAPA, by conducting a process accessible to all interested parties, the National Dialogue provided the opportunity to strengthen trust among stakeholders and create potential buy-in for later implementation of policies and priorities they helped to shape. DHS consulted with a range of stakeholders through various mechanisms, but DHS officials and stakeholders identified challenges that hindered DHS’s consultation efforts in conducting the QHSR. These challenges were (1) consultation time frames, (2) inclusion of nonfederal stakeholders, and (3) definition of stakeholders’ roles and responsibilities. According to DHS officials, the department consulted with stakeholders primarily over a 5-month period—from May through September 2009— during the QHSR process. In response to our request for comments on the QHSR process, 16 stakeholders noted concerns regarding the time frames they had for providing input into the QHSR or BUR. Nine DHS stakeholders, for example, responded that in their view, the limited time available for development of the QHSR did not allow DHS to have as broad and deep an engagement with stakeholders as DHS could have experienced if more time had been allotted to stakeholder consultations. DHS stakeholders also reported to us that DHS’s time frames for conducting the BUR were short and that the BUR process was hampered by an overly aggressive timeline for deliberation and decision making. Two of the study group facilitators who responded to our request for comments reported that in their view, stakeholders needed more time to review draft work products and hold more discussions. Three federal stakeholders suggested that the process be initiated earlier than it was for the first QHSR to provide more time for DHS to consider and resolve stakeholder comments, draft the report, and provide stakeholders with an opportunity to review the draft report. One of these federal stakeholders stated that more detail and other viewpoints would have been added to the QHSR if DHS had conducted outreach earlier in the QHSR process while another noted that it was difficult to keep up with the changes in the QHSR draft report and therefore to fully participate in providing comments. There were multiple drafts and no dialogue on how the comments were incorporated, according to this stakeholder. This federal stakeholder stated that more lead time in the provision of QHSR materials would have allowed for stakeholders to better consider the information and provide DHS with feedback. Two state and local associations responded that more lead time for the arrangement of meetings and a review of the complete QHSR report prior to its release would have been helpful. In addition, NAPA identified challenges associated with time frames for conducting aspects of the QHSR. Specifically, in its report on the National Dialogue, NAPA stated that the abbreviated turnaround time between phases of the National Dialogue—approximately 3 weeks on average— resulted in very constrained time periods for the study groups to fully review stakeholder feedback, incorporate it into the internal review process, and use it to develop content for subsequent phases. NAPA reported that for DHS to improve online stakeholder engagement it should build sufficient time for internal review and deliberations into its timetable for public engagement on the QHSR, and provide the public an opportunity to see that it is being heard in each QHSR phase. Thus, related to the National Dialogue, NAPA recommended that DHS build a timetable that allows ample time for internal deliberations that feed directly into external transparency. According to the Deputy Assistant Secretary for Policy (Strategic Plans) at DHS, addressing NAPA’s recommendations, in general, is part of the QHSR project planning to begin during summer 2011 for the next QHSR. The official stated that DHS is considering NAPA’s recommendations and is looking for opportunities for additional stakeholder involvement during the next QHSR. DHS identified those stakeholders to be consulted and various consultation mechanisms to be used prior to initiation of stakeholder consultations, but planned the consultation time periods based on the limited time available between when the QHSR process began and when the report was due, contributing to the time frame concerns raised by the 16 QHSR stakeholders and NAPA. Our prior work on strategic studies has shown that when federal agencies are defining missions and outcomes, such as DHS did in developing the QHSR report, involving stakeholders is a key practice. According to program management standards, stakeholder and program time management are recognized practices, among others, for operating programs successfully. Stakeholder management defines stakeholders as those whose interests may be affected by the program outcomes and that play a critical role in the success of any program; it should ensure an active exchange of accurate, consistent, and timely information that reaches all relevant stakeholders. Time management is necessary for program components and entities to keep the overall program on track, within defined constraints, and produce a final product. According to the Deputy Assistant Secretary for Policy (Strategic Plans) at DHS, constrained time periods for stakeholder consultations are part of the challenge of executing a time-limited process with a broad stakeholder base, such as the QHSR. According to the Deputy Assistant Secretary, longer time periods for stakeholder consultations could be beneficial, but a tradeoff to consider is that the review as a whole would be more time consuming. DHS officials determined time periods for consultation by planning from the QHSR issuance date and then building in stakeholder consultation periods for white paper solicitation and receipt, the National Dialogue, and executive committee meetings. Stakeholder consultation time frames were built into the QHSR project plan, with planned time periods such as 23 days between white paper solicitation notifications and the deadline for submissions from stakeholders, which was dictated by the necessities of the December 31, 2009 issuance deadline. The National Security Staff set timelines for report review by other federal agencies, according to the Deputy Assistant Secretary. Moreover, this official said that setting target time frames for stakeholder consultations during the next QHSR is something that DHS plans to address during project planning. By considering ways to build more time for stakeholder consultations into the timeline or target time frames for the next QHSR, DHS could be better positioned to manage stakeholder consultations and feedback received throughout the process, including determining and communicating how much time stakeholders will be given for providing feedback and commenting on draft products. In addition, DHS could be better positioned to ensure that stakeholders have the time needed for reviewing QHSR documents and providing input. DHS consulted with a range of stakeholders, including federal and nonfederal entities, during the QHSR, and these consultations provided DHS with a variety of perspectives for consideration as part of the QHSR process. However, the department faced challenges in obtaining feedback from nonfederal stakeholders. Our prior work on key practices for performance management has shown that stakeholder involvement is important to help agencies ensure that their efforts and resources target the highest priorities. Involving stakeholders in strategic planning efforts can also help create a basic understanding among the stakeholders as to the agency’s programs and results they are intended to achieve. Without this understanding successful implementation can be difficult because nonfederal stakeholders help clarify DHS’s missions, reach agreement on DHS’s goals, and balance the needs of other nonfederal stakeholders who at times may have differing or even competing goals. As we have previously reported, nonfederal entities have significant roles in homeland security efforts. For example, state, local and private sector entities own large portions of critical infrastructure in the United States and have responsibilities for responding to and recovering from homeland security incidents. Thus we have previously reported that it is vital that the public and private sectors work together to protect these assets. Further, we have reported on the need for federal and nonfederal entities to more effectively communicate their emergency preparedness and response roles, responsibilities, and activities. For example, we have reported that effective public warning depends on the expertise, efforts, and cooperation of diverse stakeholders, such as state and local emergency managers and the telecommunications industry. In responding to our request for comments, 9 stakeholders commented that DHS consultations with nonfederal stakeholders, such as state, local, and private sector representatives, could be enhanced. For example, 1 stakeholder noted that state, local and private sector representatives, such as those with responsibility for securing critical infrastructure and key resources, the maritime sector, and overseas interests, should be further consulted during the next QHSR process. One federal stakeholder noted that state and local involvement is critical for homeland security and that a review of state and local readiness would be beneficial to determine the gaps that would need to be filled at the federal level. DHS could map out what state and local officials need in case of an emergency and include the various federal agencies in these discussions. Further, another stakeholder noted that DHS faced challenges in consulting specifically with the private sector during the 2010 QHSR. DHS consulted with private sector entities primarily through (1) the QRAC, whose membership was comprised of individuals from academia, nonprofit research organizations, private consultants, and nonprofit service providers and advocacy organizations; and (2) the National Dialogue. With regard to the QRAC, it met nine times during which it received information from DHS leadership regarding the QHSR design, analysis, and interim conclusions, and its members provided feedback and advice to DHS. However, one QRAC respondent noted that the council’s members were predominately consultants and not representatives of industries affected by homeland security threats, such as critical infrastructure sectors, which resulted in views that were not representative of one of the most affected members of the nonfederal homeland security community. This respondent stated that enhancing participation of private sector representatives is important for the next QHSR, as it would help DHS obtain views from entities that provide homeland security and emergency responses services, such as one corporation providing water to victims after Hurricane Katrina. According to this stakeholder, private sector entities could help offer DHS views on, for example, best practices for how to prepare for and respond to homeland security events or technology enhancements for homeland security. With regard to the National Dialogue—one of the primary mechanisms used for soliciting input from nonfederal stakeholders—17 stakeholders who responded to our request for comments on the QHSR, as well as NAPA, identified challenges. As an example of comments we received from these 17 stakeholders, 1 federal stakeholder reported that the National Dialogue did not appear to have significant impact on the QHSR because in interagency meetings involving this stakeholder, information from the National Dialogue was not discussed. In an additional example, one QRAC member responded that the National Dialogue included a small number of comments from the private sector and did not reflect the significant number of stakeholders around the country with homeland security responsibilities. This respondent stated that the National Dialogue was an important exercise but was not an effective means for obtaining representative views specifically of the private sector. Further, as another example, one state and local association responded that in its view, DHS’s conclusions on QHSR strategy had been reached prior to initiation of the National Dialogue, making it appear to the association that although DHS was soliciting its input, the department did not view the association as playing a consultative role in the QHSR development. In addition, NAPA reported that engaging nonfederal associations, such as the National Association of Counties, did not necessarily equate to reaching out to individual nonfederal entities, such as cities and counties. Therefore, according to NAPA’s report, through the National Dialogue, DHS notified approximately 1,000 contact members of nonfederal associations in an effort to include a range of nonfederal homeland security practitioners. Based on this outreach effort, NAPA’s report recommended continuing efforts to gain significant buy-in from nonfederal associations to ensure that DHS obtains access to the nonfederal stakeholders it wishes to consult regarding the QHSR. DHS faced challenges in obtaining nonfederal input during the QHSR process for two reasons. First, convening state and local government officials for consultation, especially from individual nonfederal stakeholders, on the QHSR was a significant logistical challenge, according to DHS officials. Because of this challenge, DHS opted to consult with national associations that could represent the perspectives of state, local, and tribal homeland security stakeholders. Second, according to the Deputy Assistant Secretary for Policy (Strategic Plans) at DHS, the Federal Advisory Committee Act (FACA), which establishes standards and uniform procedures for the establishment, operation, administration, and duration of advisory committees, affected how DHS was able to consult with private sector stakeholders when developing the QHSR report. Specifically, the Deputy Assistant Secretary noted that the department was limited in its ability to consult with private sector groups on an ongoing basis without forming additional FACA committees specifically for conducting consultations on the QHSR. DHS was also limited in its ability to seek feedback from established FACA committees that had been convened for other purposes. The meeting schedules of those committees did not align well with the QHSR study period, and there were significant logistical challenges to scheduling additional meetings of those groups to address QHSR. In addition, the Deputy Assistant Secretary for Policy (Strategic Plans) stated that under FACA DHS could not invite members of established FACA committees convened for other purposes to join meetings of the QRAC for the purpose of providing advice and feedback. One study group facilitator commented that the FACA consideration significantly reduced the role that nonfederal stakeholders played in the QHSR. According to this respondent, addressing the FACA requirements and including appropriate FACA-compliant groups with a broader range of academics and others could have affected the outcome of the study group’s deliberations. However, according to the Deputy Assistant Secretary, establishing new FACA committees in addition to the QRAC, which DHS established as a FACA-compliant committee specifically for QHSR consultations, was prohibitively time consuming within the time frames DHS had for conducting the 2010 QHSR. Four respondents to our request for comments made suggestions for alternative approaches for obtaining viewpoints of nonfederal stakeholders in future QHSRs. For example, one study group facilitator stated that state and local associations could put together a group of their members to engage in the QHSR process and be part of the study groups. In addition, the National Dialogue could have provided more focused questions and provided to a broad group of state and local experts questions on specific issues, such as housing disaster resiliency. This approach could have allowed more state, local, private sector, academic and nongovernmental organizations into the QHSR process, according to the facilitator. DHS officials noted, though, that the National Dialogue was not intended to address individual initiatives, as the QHSR was intended to focus on broader homeland security issues. Further, one local government association suggested that this association could put together a crosscutting group of local officials who could discuss specific issues, such as national preparedness. In addition to alternative approaches for obtaining viewpoints of nonfederal stakeholders provided by respondents outside of DHS, one DHS stakeholder responded they held in-person or teleconferencing meetings with numerous interest groups and associations, while another DHS stakeholder responded that the component sent emails to its stakeholder groups soliciting the groups’ views on the QHSR. Additionally, in our prior work on the Federal Emergency Management Agency’s (FEMA) process for updating the National Response Framework, we identified examples of ways in which FEMA involved nonfederal stakeholders in the process. For example, FEMA posted a spreadsheet that included the comments made by nonfederal stakeholders and the final disposition DHS assigned to each of those comments to allow stakeholders to see how DHS did or did not incorporate their comments. Further, FEMA had agency leaders appoint advisory council members who represented a geographic and substantive cross section of officials from the nonfederal community. Given the significant role that state and local governments and the private sector play in homeland security efforts, which is acknowledged by DHS in the QHSR report, examining mechanisms, such as those proposed by QHSR stakeholders or used by components, could help DHS include a broader segment of these representatives during the QHSR process and better position DHS to consider and incorporate, as appropriate, nonfederal concerns and capabilities related to homeland security in the next QHSR. DHS identified stakeholders’ roles and responsibilities in the QHSR report primarily by referencing other homeland security-related documents, such as the National Response Framework and National Infrastructure Protection Plan, that describe homeland security roles and responsibilities. With regard to federal agencies, the QHSR report described homeland security roles and responsibilities with brief summaries of federal agencies’ leadership roles for coordinating homeland security–related efforts. For example, the QHSR report listed the Attorney General’s responsibilities as conducting criminal investigations of terrorist acts or threats by individuals or groups, collecting intelligence on terrorist activity within the United States, and leading the Federal Bureau of Investigation, the Drug Enforcement Administration, and the Bureau of Alcohol, Tobacco, Firearms and Explosives in their respective areas of homeland security responsibilities. With regard to nonfederal stakeholders’ roles and responsibilities, the QHSR report provided summaries of roles and responsibilities, based on these and other homeland security–related documents, such as identifying that critical infrastructure owners and operators are responsible for developing protective programs and measures to ensure that systems and assets are secure from and resilient to threats. Our prior work has shown that agencies that work together to define and agree on their respective roles and responsibilities when implementing federal strategies that cross agency boundaries can enhance the effectiveness of interagency collaboration. In doing so, agencies clarify who will do what, organize their joint and individual efforts, and facilitate decision making. Further, our work on key characteristics for effective national strategies identified, among others, one desirable characteristic as defining the roles and responsibilities of the specific federal departments, agencies, or offices involved and, where appropriate, the different sectors, such as state, local, private, or international sectors. Inclusion of stakeholders’ roles and responsibilities in a strategy that crosses agency boundaries is useful to agencies and other stakeholders in clarifying specific roles, particularly where there is overlap, and thus enhancing both implementation and accountability. In addition, we have reported that DHS needs to form effective and sustained partnerships with a range of other entities, including other federal agencies, state and local governments, and the private and nonprofit sectors. Successful partnering involves collaborating and consulting with stakeholders to develop and agree on goals, strategies, and roles to achieve a common purpose. In responding to our request for comments on the QHSR, 10 federal stakeholders noted that the roles and responsibilities listed in the QHSR report, as derived from other documents, such as the National Response Framework and National Infrastructure Protection Plan, reflected their homeland security missions and activities. For example, 1 federal stakeholder responded that the roles and responsibilities listed in the QHSR report were established in previous documents and were accurate, and another federal stakeholder noted that the roles and responsibilities listed in the QHSR report were derived from previously published material for which the stakeholder had provided input. However, DHS and 10 other respondents to our request for comments noted that the department could strengthen its definition of homeland security roles and responsibilities in the next QHSR by better reflecting the range of the stakeholders’ roles and responsibilities. Specifically, in the QHSR report DHS identified the need to better assess stakeholders’ homeland security roles and responsibilities, noting that although the report was not intended to describe stakeholders’ roles and responsibilities, the division of operational roles and responsibilities among federal departments and agencies for various homeland security goals and objectives emerged as a major area requiring further study following the QHSR report. DHS reported that an analysis of roles and responsibilities across the homeland security missions would help resolve gaps or unnecessary redundancies between departments and agencies going forward. Further, 10 stakeholders commented to us that the definitions of roles and responsibilities in the QHSR report could be enhanced to better reflect the range of homeland security stakeholders’ responsibilities. For example, 3 federal stakeholders reported that roles and responsibilities definitions in the QHSR could be enhanced by, for example, recognizing the variety of agency or administration-level responsibilities of the cabinet departments. In particular, one of these federal stakeholders suggested that the next QHSR may want to include a more detailed delineation of the roles and responsibilities of departments to support the homeland security enterprise by (1) reflecting the broad nature of responsibilities across a broad spectrum of threats and (2) identifying readiness and resource requirements to address the stated roles and responsibilities. The brief narrative on roles and responsibilities in the 2010 QHSR report presented a shortened version of the roles and responsibilities that the federal stakeholder has in supporting the homeland security enterprise. Another federal stakeholder noted that formalizing the process to elicit values and judgments from individual agencies would help ensure adequate representation of each agency’s role in the next version of the QHSR report. The formalized process, according to the federal stakeholder, would convene agency officials and facilitate a discussion, resulting in a common understanding of how agency roles and responsibilities are defined for executing the QHSR strategy. In our December 2010 report on the extent to which the QHSR report addressed reporting elements that the 9/11 Commission Act specified for the report, we noted that DHS partially addressed two reporting elements for the QHSR report related to roles and responsibilities for homeland security stakeholders. These elements were for the QHSR report to include a discussion of the status of (1) cooperation among federal agencies in the effort to promote national security and (2) cooperation between the federal government and state, local, and tribal governments in preventing terrorist attacks and preparing for emergency response to threats to national homeland security. With regard to the first element, we reported that although the QHSR and BUR reports discussed homeland security roles and responsibilities for federal agencies, they did not discuss cooperation on homeland security efforts among federal agencies other than DHS. We reported that while the QHSR discussion of roles and responsibilities as found in other documents was helpful for understanding which federal agencies lead particular homeland security efforts, the QHSR report did not provide a description of how federal agencies cooperate with one another in addressing homeland security efforts. With regard to the second element, we reported that although the QHSR and BUR reports provided descriptions of cooperation between DHS and state, local, and tribal governments, they did not discuss the status of cooperation between other federal agencies that have homeland security responsibilities and state, local, and tribal governments. DHS officials stated that DHS solicited comments from other federal departments and state, local, and tribal governments on the role and responsibility descriptions for each of these entities listed in the QHSR report. According to the Deputy Assistant Secretary for Policy (Strategic Plans) at DHS, during the QHSR process the department did not attempt to discuss the status of cooperation among other federal departments and between other federal departments and state, local, and tribal governments. DHS officials stated that the department viewed such a discussion as outside its authority to conduct and that those discussions were conducted in other venues, such as the National Infrastructure Protection Plan and the National Response Framework. Because the National Response Framework and the National Infrastructure Protection Plan were completed during DHS’s launch of the QHSR, in 2008, use of those definitions in the QHSR was appropriate, according to the official. DHS did not obtain comments from all stakeholders on the definitions listed in the QHSR report, but looked at stakeholder comments on roles and responsibilities received during the National Infrastructure Protection Plan and National Response Framework drafting processes. The definitions listed in the QHSR report were also shared with DHS’s Office of Intergovernmental Affairs, which solicited comments from stakeholders, as necessary, based on any roles that may have changed since the National Infrastructure Protection Plan and National Response Framework were published, according to the official. In its May 2010 report on the QHSR, the QRAC noted that the QHSR report included a summary of roles and responsibilities of key stakeholders that was derived from existing statutes, among other documents. However, according to the QRAC report, the QHSR report did not provide a mapping of these roles and responsibilities to the QHSR missions and further work was required to deconflict and potentially supplement existing homeland security stakeholder role and responsibility policies and directives. According to the QRAC report, the QHSR report was designed to create a shared vision of homeland security in order to achieve unity of purpose; a vital next step was the delineation of key roles and responsibilities for individual QHSR goals and objectives to generate unity of effort. A comprehensive mapping of stakeholder roles and responsibilities to QHSR missions, goals, and objectives was needed to (1) enable assessment of the current state of cooperation and coordination between all public and private sector stakeholder communities; (2) identify potential gaps, conflicts, or both in current policies and directives from an enterprise perspective; and (3) underpin follow-on planning efforts. The QRAC recommended that DHS map goals to objectives for each core QHSR mission and key stakeholder communities to delineate the stakeholders’ respective roles and responsibilities. In response to this recommendation, DHS plans to map the existing QHSR mission goals and objectives to stakeholder roles and responsibilities during the pre-execution year for the next QHSR, if possible. This setup work on role and responsibility mapping would allow for work at the end of the QHSR process to map roles and responsibilities to final QHSR goals and objectives developed during the next QHSR. Consistent with the QRAC’s recommendation and DHS’s planned actions, by seeking to further define homeland security stakeholders’ roles and responsibilities in the next QHSR, DHS could be better positioned to identify, understand, and address any potential gaps in roles and responsibilities or areas for additional or enhanced cooperation and coordination. Through the QHSR, DHS identified various threats confronting homeland security but did not conduct a risk assessment for the QHSR. In the 2010 QHSR report, DHS identified six threats confronting homeland security, such as high-consequence weapons of mass destruction and illicit trafficking and related transnational crime, as well as five global challenges, including economic and financial instability and sophisticated and broadly available technology. According to the QHSR report, these threats and challenges were the backdrop against which DHS planned to pursue its homeland security efforts. The threats and global challenges listed in the QHSR report were developed through discussions with federal national security officials and through reviews of intelligence community materials, according to DHS officials. Multiple DHS guidance documents emphasize the importance of considering risk assessment information when engaging in strategic decisions. For example, DHS’s Integrated Risk Management Framework (IRMF), published in January 2009, calls for DHS to use risk assessments to inform DHS-wide decision-making processes. Risk assessments, which include assessing and analyzing risk in terms of threats, vulnerabilities, and consequences of a potential homeland security incident, are the foundation for developing alternative strategies for managing risk, according to the IRMF. Similarly, the QHSR report includes an objective for DHS to establish an approach for national-level homeland security risk assessments, specifically calling for development and implementation of a methodology to conduct national-level homeland security risk assessments. Our prior work on federal strategic studies has also found that establishing an analytic framework to assess risks is a key aspect of developing a strategy to address national problems, such as homeland security. Consistent with the IRMF, we define risk assessment as a qualitative determination, a quantitative determination, or both of the likelihood of an adverse event occurring and the severity, or impact, of its consequences. DHS has called for the use of national risk assessments for homeland security but did not conduct such an assessment as part of the 2010 QHSR. DHS officials stated that at the time DHS conducted the QHSR, DHS did not have a well-developed methodology or the analytical resources to complete a national risk assessment that would include likelihood and consequence assessments. The QHSR terms of reference, which established the QHSR process, also stated that at the time the QHSR was launched, the homeland security enterprise lacked a process and a methodology for consistently and defensibly assessing risk at a national level and using the results of such an assessment to drive strategic prioritization and resource decisions. In recognition of a need to develop a national risk assessment methodology, the QHSR National Risk Assessment Study Group was created as part of the QHSR process. In establishing the study group, the QHSR Terms of Reference stated that assessing national risk was a fundamental and critical element of an overall risk management process, with the ultimate goal of improving the ability of decision makers to make rational judgments about tradeoffs between courses of action to manage homeland security risks. The QHSR National Risk Assessment Study Group consulted with subject matter experts from the federal government, academia, and the private sector and, in October 2009, produced the Homeland Security National Risk Assessment (HSNRA) methodology, which established a process for conducting a national risk assessment in the future. According to DHS officials, because the HSNRA methodology was developed as part of the QHSR process and finalized as the QHSR report was being completed in late 2009, it was not intended to be implemented during the 2010 QHSR. The HSNRA is to use a methodology for assessing risk across a range of hazards for use by DHS in its decisions on strategy and policy development, planning priorities, resource allocation, and capability requirements development. The HSNRA includes definitions or descriptions of the scope of incidents it applies to, risk formula, and likelihood and consequence (see table 2). Outputs from the HSNRA calculations could be expressed in a number of ways, such as plotting scenarios on a two-dimensional graph with scenario frequency estimates on the x axis and scenario consequence estimates on the y axis, as shown in figure 3. In accordance with the QHSR goal of implementing a national risk assessment and with issuance of Presidential Policy Directive 8, which calls for risk analysis across a range of homeland security threats, DHS is planning to conduct a national risk assessment as part of its next QHSR. In determining how to conduct a national risk assessment, DHS is considering various factors, such as how to incorporate and use an assessment’s results, the time frames and costs for conducting an assessment, and what alternatives exist to conducting a national assessment.  Use of national risk assessment results. DHS officials stated that one consideration in determining how to conduct a national risk assessment is the manner in which the department would use the results of such an assessment to inform the QHSR. Specifically, DHS officials told us that a national risk assessment, such as the HSNRA, should be one of multiple inputs considered in conducting the QHSR, with other inputs including such factors as privacy and civil liberties concerns, economic interests, and administration priorities. DHS’s risk assessment guidance makes a similar point, stating that risk information is usually one of multiple factors decision makers consider and is not necessarily the sole factor influencing a decision. There may be times when the strategy selected and implemented does not optimally reduce risk, and decision makers should consider all factors when selecting and implementing strategies. The National Research Council of the National Academies also reported that risk analysis is one input to decision making, designed to inform decisions but not dictate outcomes. DHS officials noted that it would be important to communicate this to its stakeholders, including Congress, the public, and others, to manage any expectations that QHSR decisions would be solely based on risk assessment results.  National risk assessment time frames and costs. According to DHS officials, the HSNRA would require 12 months to complete and would need to be completed before launching the next QHSR, since the assessment would help frame how the QHSR missions are defined. If the next QHSR is conducted during fiscal year 2013 and reported by December 2013, as anticipated by DHS, the HSNRA would need to be completed during fiscal year 2012 to help inform the QHSR, according to DHS officials. With regard to financial costs, DHS officials estimated that conducting the HSNRA over a 12-month period would cost from $3 million to $6 million. DHS’s Deputy Assistant Secretary for Policy (Strategic Plans) stated that the HSNRA is a sound methodology that should be used as part of the next QHSR, and officials within DHS’s unit responsible for developing the HSNRA, the Office of Risk Management and Analysis, stated that the benefits of having risk information available for input into developing the QHSR are worth the costs.  National risk assessment alternatives. In order to identify risks and inform mission areas for the next QHSR, DHS could consider alternatives to conducting a national risk assessment, according to DHS officials. These officials stated that one alternative approach would involve using segments of the HSNRA process to help provide risk information to department decision makers, such as eliciting expert judgments and surveying nonfederal experts about perspectives on the risks DHS should address. The officials stated that this approach would not be as useful as a complete HSNRA because a full HSNRA provides likelihood and consequence estimates for various homeland security incident scenarios, which offers a more complete picture of the risks DHS must address. Another approach, according to the officials, would be to identify risks through existing DHS analyses, such as the Homeland Security Threat Assessment or the National Planning Scenarios. The officials stated that identification of risks through these tools would also be limited and would not be as effective as completing the HSNRA. For example, the HSNRA includes likelihood estimates for scenarios, which these other tools do not include, and therefore provides a more complete picture of risk by addressing threats, likelihoods, and consequences. Consistent with DHS’s plans, a national risk assessment conducted in advance of the next QHSR could assist DHS in developing QHSR missions that target homeland security risks and could allow DHS to demonstrate how it is reducing risk across multiple hazards. DHS considered various factors in identifying high-priority BUR initiatives for implementation in fiscal year 2012 but did not include risk information as one of these factors. Through the BUR, DHS identified 43 initiatives aligned with the QHSR mission areas to help strengthen DHS’s activities and serve as mechanisms for implementing those mission areas. According to DHS officials, the department could not implement all of these initiatives in fiscal year 2012 because of, among other things, resource constraints and organizational or legislative changes that would need to be made to implement some of the initiatives. In identifying which BUR initiatives to prioritize for implementation in fiscal year 2012, DHS leadership considered (1) “importance,” that is, how soon the initiative needed to be implemented; (2) “maturity,” that is, how soon the initiative could be implemented; and (3) “priority,” that is, whether the initiative enhanced secretarial or presidential priorities. Component leadership officials, as subject matter experts, completed a survey instrument indicating their assessment of each BUR initiative based on these criteria. The results were then aggregated and presented to DHS’s Program Review Board—which is the body that oversees DHS program reviews and the budgeting process. With the Deputy Secretary’s leadership, the Program Review Board evaluated the results of the survey and refined the prioritization. The BUR initiative prioritization process resulted in the Secretary and Deputy Secretary of Homeland Security ranking and selecting 14 high-priority BUR initiatives to be implemented in fiscal year 2012, as shown in table 3. We and DHS have called for the use of risk information in making prioritization, resource, and investment decisions. For example, DHS’s IRMF states that DHS is to use risk information to inform strategies, processes, and decisions to enhance security and to work in a unified manner to manage risks to the nation’s homeland security. The IRMF states that one of its objectives is to use an integrated risk management process to inform resource allocations on a departmentwide basis, which is critical to balance resources across the set of DHS strategic objectives. Likewise, our prior work has shown the importance of using risk information to inform resource prioritization decisions. For example, our risk management approach advises using risk information to inform resource allocation decisions so that management can consider which risks should be managed immediately and which risks can be deferred and addressed at a later time. According to DHS officials, using risk information as an input into DHS’s prioritization of the initiatives was difficult for several reasons. For example, the BUR initiatives were highly differentiated, making comparisons based on risks the initiatives address impossible, according to DHS officials. Some of the BUR initiatives focus on organizational changes at DHS; others are extremely broad, addressing multiple and overlapping risks; and others focus on specific risks. For example, comprehensive immigration reform is a broad BUR initiative, addressing broad illegal immigration risks, while promoting safeguards for access to secure areas in critical facilities targets more specific risks. According to the officials, the variance in how the initiatives were defined allowed DHS to align initiatives with the QHSR strategy and consideration of such variance, in addition to risks addressed by QHSR implementation mechanisms, such as BUR initiatives, would be important in defining implementation mechanisms and initiatives for future QHSRs. However, DHS could not apply its existing risk assessment tools to evaluating and prioritizing BUR initiatives for the 2010 QHSR. For future QHSRs, DHS officials described several characteristics of mechanisms for implementing QHSR missions that would enable risk information to be used among prioritization criteria. First, the implementation mechanisms or initiatives to be prioritized based on risk information should be comparable in terms of the nature of the risks addressed. For example, comparing mechanisms to address DHS organizational changes that do not directly reduce homeland security risks with mechanisms that are designed to directly prevent terrorism risks would be an inappropriate comparison. Second, expected outcomes of the mechanisms or initiatives should be defined so that the risks reduced by the mechanisms can be estimated. For example, the BUR initiatives do not indicate the degree to which investments will change DHS’s security capabilities. Knowing the increase (or decrease) in security capabilities associated with an implementation mechanism would allow estimates of risks reduced, which could be compared in prioritization efforts. Third, an implementation mechanism or initiatives should have a “line of sight” directly between the DHS activities associated with the mechanism and the risk reduced by those activities. In other words, according to the officials, DHS operations need to be closely aligned with identified risk reductions in order for risk reduction calculations to be accurately achieved. For example, U.S. Border Patrol efforts to stop illegal border crossings are closely aligned with reducing risks of illegal immigration. DHS officials stated that although existing DHS risk assessment tools could not be used to systematically prioritize the BUR initiatives for the 2010 QHSR, there is utility in thinking qualitatively about risks addressed by the initiatives when making future prioritization decisions. Risk information should not be the sole input but should be considered along with other criteria, according to Office of Risk Management and Analysis officials and the Deputy Assistant Secretary for Policy (Strategic Plans). DHS has various tools that could, with some limitations, provide risk information for consideration when prioritizing implementation of QHSR mission objectives, as shown in table 4. Two tools, the Risk Analysis Process for Informed Decision-making (RAPID) and the methodology for conducting the HSNRA, were created to provide risk information for decision making across DHS mission areas. Risk analyses conducted within DHS components could also provide risk information useful for prioritizing QHSR implementation mechanisms, according to DHS officials. The officials stated that at least five current risk assessments used by DHS components could be useful for prioritizing QHSR implementation efforts within the mission areas relevant to the risk assessment tools. DHS officials stated that there are benefits to considering risk information in resource allocation decisions; however, DHS has not yet examined the extent to which risk information could be used when implementing subsequent QHSRs. Consideration of risk information could help strengthen DHS’s prioritization of mechanisms for implementing the QHSR, including determining which initiatives or programs should be implemented in the short or longer term and the resources required for implementation. Such information could also help the department to more effectively make decisions about implementing initiatives and allocating resources across initiatives that address different levels and types of risks. DHS is managing and monitoring its implementation of BUR initiatives primarily through its budget development and execution process, called the Planning, Programming, Budgeting, and Execution (PPBE) process. The objective of the PPBE process is to articulate DHS’s goals, objectives, and priorities; align DHS programs with those goals; guide the development of the department’s budget request; and set guidelines for implementing the current budget. To manage implementation of the BUR initiatives, beginning with the fiscal year 2012 budget request, DHS officials told us that DHS developed implementation plans for each of the 43 BUR initiatives during the planning phase of the PPBE process (see fig. 4). DHS assigned a directorate, component, or office to lead departmentwide implementation efforts for each initiative, including developing the implementation plans. According to DHS, each implementation plan included what needs to be done to accomplish the BUR initiative, what is currently being done to address identified implementation problems, a description of stakeholders involved in the implementation effort, and a discussion of next steps. DHS initiative leads submitted BUR implementation plans to the department for review and discussion during the fiscal year 2012 budget development process and the fiscal years 2012-2016 budget review process, and these plans served as a basis for components to develop their Resource Allocation Plans (RAP)—components’ descriptions of funding needs for fiscal year 2012. All DHS components execute missions, spend resources, and report on performance. Key activities: DHS is to translate planning priorities into 5-year resource and performance plan (the Future Years Homeland Security Program), and allocate limited resources to best meet the prioritized needs. Key activities: DHS is to develop detailed budget estimates of approved resource plans for budget year justification and presentation, and work with the Office of Management and Budget and Congress to get a budget enacted. Key activities: DHS Chief Financial Officer monitors accountability and execution of budget authority and reports results and makes recommendations on realigning resources. Funds are apportioned to the directorates and components in accordance with apportionment guidelines. Outputs: Secretary’s Resource Allocation Decisions, DHS Future Years Homeland Security Program. Outputs: Budget request sent to the Office of Management and Budget and Congress’ briefing and information sent to Congress. Outputs: Monthly Budget Execution Reports, Midyear Review, congressionally directed reports, Annual Financial Report, and Annual Performance Report. DHS plans to implement the BUR initiatives primarily through components’ existing programs and activities. For example, DHS plans to implement the strengthen aviation security BUR initiative through its existing aviation security programs, such as checking airline passengers against watchlists and screening passengers at airports. Through the PPBE process for fiscal year 2012, DHS requested additional funding for select BUR initiatives, above base funding for programs and activities that support those initiatives. For example, under the Improve Detention and Removal Process BUR initiative, DHS requested about $222 million in increased funding to support its existing Secure Communities program and effort to rightsize detention bed space. In addition to increased funding requested for select BUR initiatives, according to DHS, the department planned to fund existing programs and activities that support the other BUR initiatives through its base funding. For example, the Domestic Nuclear Detection Office stated that it plans to fund its programs that support the BUR initiative to increase efforts to detect and counter nuclear and biological weapons and dangerous materials through its base funding. To monitor implementation of the BUR initiatives, DHS established scorecards as part of its Integrated Planning Guidance—which DHS developed during the planning phase of the fiscal year 2013 PPBE process to provide guidance to DHS components for the programming and budgeting phases. The scorecards depict the status of implementing BUR initiatives, including, among other things, whether DHS requested funding for BUR initiatives in fiscal year 2012 or plans to request funding in future years. The scorecards also allow DHS to periodically assess progress made on implementing individual BUR initiatives and the status of BUR implementation as a whole. For those BUR initiatives for which the department did not identify specific funding needs in future years, DHS officials told us that they have discussions with DHS components and directorates during midyear budget review meetings to discuss progress made toward implementing BUR initiatives. In addition, DHS officials told us that because the BUR initiatives reflect existing DHS priorities, the initiatives are monitored through the Secretary of Homeland Security’s discussions with component and directorate leadership, such as discussions on progress being made on a particular BUR initiative like strengthening aviation security. DHS has taken action to develop and strengthen its performance measures, including linking them to QHSR missions and goals and ensuring limited overlap among measures. While DHS has not developed performance measures for all QHSR missions, goals, and objectives, DHS has efforts under way to develop measures to address those missions, goals, and objectives. Our prior work on key practices for performance measurement has shown that measuring performance allows organizations to track the progress they are making toward their goals and gives managers critical information on which to base decisions for improving their performance. We also have previously reported on attributes of successful performance measures that include ensuring that measures are linked to agencies’ missions and goals. Since issuance of the QHSR report, DHS has undertaken efforts to develop new performance measures and link its existing measures to the QHSR missions and goals. These efforts included DHS providing guidance to components that outlines how to assess QHSR missions, goals, and objectives and achievement of QHSR outcomes. DHS also provided components with performance measure development training and formed working groups to discuss performance measurement best practices. To support these efforts, in 2010, we provided technical assistance to DHS and its components as they developed and revised their performance measures to align with the strategic missions and goals of the QHSR. Our feedback ranged from pointing out components’ limited use of outcome-oriented performance measures to assess the results or effectiveness of programs to raising questions about the steps taken by DHS or its components to ensure the reliability and verification of performance data. While we offered advice on best practices for performance measurement and developing outcome-oriented measures, we did not suggest specific performance measures or targets or recommend methodologies for collecting, analyzing, and reporting performance measure data. Therefore, there was no expectation that we and DHS reached agreement on the performance measures, and thus decisions related to performance measures were fundamentally an executive branch management responsibility. In response to this feedback and its internal review efforts, DHS took action to develop and revise its performance goals and measures to strengthen its ability to assess its outcomes and progress in key mission areas. In DHS’s fiscal years 2010-2012 Annual Performance Report, DHS identified 57 new performance measures for fiscal year 2011, retained 28 measures from the fiscal year 2010 measure set, and is in the process of refining the methodologies for additional measures that the department plans to implement in fiscal year 2012. DHS’s actions to strengthen its performance measures have helped the department link its measures to QHSR missions, goals, and objectives. DHS has not yet developed performance measures for all of the QHSR goals and objectives but has plans to do so. Specifically, DHS has established new performance measures, or linked existing measures, to 13 of 14 QHSR goals, and to 3 of the 4 goals for the sixth category of DHS activities—Providing Essential Support to National and Economic Security. DHS reported these measures in its fiscal years 2010-2012 Annual Performance Report. At the time of issuance of that report, DHS had not yet developed performance measures for QHSR Goal 2.3, Disrupt and Dismantle Transnational Criminal Organizations, or one of the goals for its sixth category of activities—Provide Specialized National Defense Capabilities. However, since then, DHS officials told us that the department has developed performance measures for these goals and plans to publish them in its budget justification to Congress upon approval of the measures by DHS leadership and the Office of Management and Budget. Further, within QHSR Goal 4.2, Promote Cybersecurity Knowledge and Innovation, DHS has not yet developed measures for two of the three objectives—foster a dynamic workforce and invest in innovative technologies, techniques, and procedures. DHS officials told us that the department is collaborating with the Office of Personnel Management on a multiyear effort to identify competencies and more accurately gauge workforce needs for cybersecurity professionals and is working to develop a measure related to innovative technologies that have been developed and deployed. Homeland security includes a vast range of mission areas—from preventing terrorism to securing U.S. borders, safeguarding cyberspace, and ensuring resilience to disasters. It also involves a wide variety of stakeholders and partners, including federal departments and agencies; state, local, and tribal governments; and nongovernmental entities, including the private sector. Given the scope and magnitude of the homeland security enterprise, it is important for the federal government to set clear goals, objectives, and priorities for securing the United States and making resource allocation decisions. DHS’s 2010 QHSR—the department’s first quadrennial review—was a massive undertaking to review the nation’s homeland security strategy and identify homeland security missions and organizational objectives. It involved the input of numerous stakeholders with homeland security roles and responsibilities, including other federal agencies, state and local government entities, and academics. DHS plans to initiate its next QHSR in fiscal year 2013 and to report on that review’s results in fiscal year 2014. In conducting this next review, DHS could leverage lessons learned from the 2010 QHSR to strengthen its planning and risk management efforts. Specifically, given the array of federal and nonfederal stakeholders involved in implementing homeland security missions, building more time for obtaining stakeholders’ feedback and input and examining additional mechanisms to obtain nonfederal stakeholders’ input could strengthen DHS’s planning and management of stakeholder consultations and better position it to obtain, review, and incorporate, as appropriate, stakeholders’ feedback. Risk assessment in the homeland security realm is an evolving field, although DHS has developed methodologies, human capital, and departmental policies for integrating risk information into DHS decision- making processes. Such information can help decision makers identify and assess homeland security threats and vulnerabilities facing the nation and evaluate strategies for mitigating or addressing those threats and vulnerabilities. Using existing risk assessment tools could assist DHS in prioritizing QHSR implementation mechanisms. Specifically, examining the extent to which risk information could be used to help prioritize implementation mechanisms for the next QHSR could help DHS determine how to incorporate and use such information to strengthen prioritization and resource allocation decisions. To strengthen DHS’s planning, management, and execution of the next QHSR, we recommend that the DHS Assistant Secretary for Policy take the following three actions:  Provide more time for consulting with stakeholders during the QHSR process to help ensure that stakeholders are provided the time needed to review QHSR documents and provide input into the review, and build this time into the department’s project planning for the next QHSR.  Examine additional mechanisms for obtaining input from nonfederal stakeholders during the QHSR process, such as whether panels of state, local, and tribal government officials or components’ existing advisory or other groups could be useful, and use them for obtaining nonfederal stakeholders’ input, as appropriate, during the next QHSR.  Examine the extent to which risk information could be used as one input to prioritize QHSR implementing mechanisms, including reviewing the extent to which the mechanisms could include characteristics, such as defined outcomes, to allow for comparisons of the risks addressed by each mechanism. To the extent that DHS determines that risk information could be used, consider such information as one input into the decision-making process for prioritizing the QHSR implementation mechanisms. We requested comments on a draft of this report from DHS. On September 12, 2011, DHS provided written comments, which are reprinted in appendix III. DHS concurred with our three recommendations and described actions planned to address them. With regard to our first recommendation that DHS provide more time for consulting with stakeholders during the QHSR process and to build this time into the department’s project planning for the next QHSR, DHS stated that it would endeavor to incorporate increased opportunities and time for stakeholder engagement during the next QHSR. Regarding our second recommendation that DHS examine additional mechanisms for obtaining input from nonfederal stakeholders during the QHSR process and use them for obtaining nonfederal stakeholders’ input, DHS stated that it will examine using panels of state, local, and tribal government officials and existing advisory groups to obtain input. With regard to our third recommendation that DHS examine the extent to which risk information could be used as one input into prioritizing QHSR implementing mechanisms and to consider such information, if appropriate, when prioritizing QHSR implementation, DHS stated that it intends to conduct risk analysis specific to the QHSR in advance of the next review. DHS stated that it plans to consider the results of such analysis, along with other factors, as an input into decision making related to QHSR implementation. DHS also provided technical comments, which we incorporated as appropriate. We also requested comments on a draft of this report from the Departments of Agriculture, Defense, Health and Human Services, State, the Treasury, and Justice and the Office of the Director of National Intelligence. The Department of Defense provided technical comments which we incorporated as appropriate. In e-mails received from departmental liaisons on September 7, 2011, the Departments of Agriculture, State, and Justice indicated that they had no comments on the report. In e-mails received on September 7, 2011, from the Department of the Treasury’s Director for Emergency Programs and the Department of Health and Human Service’s Office of the Assistant Secretary for Legislation, both departments indicated that they had no comments on the report. In an e-mail received on September 9, 2011, from a departmental liaison, the Office of the Director of National Intelligence indicated that it had no comments on the report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 20 days from the report date. At that time, we will send copies to the Secretaries of Agriculture, Defense, Health and Human Services, Homeland Security, State, and the Treasury; the Attorney General; the Director of National Intelligence; and selected congressional committees. The report also will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions, please contact me at (202) 512- 9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. The Department of Homeland Security’s (DHS) strategic documents, such as component strategic plans and budget requests, align with the Quadrennial Homeland Security Review (QHSR) missions. The May 2010 National Security Strategy (NSS) also identifies strategic elements related to homeland security that are identified in the QHSR report, such as similar listings of homeland security threats. Each of the DHS strategic documents we reviewed includes language explicitly aligning at least some aspects of the strategy with the QHSR report, as shown in table 5. According to DHS officials, DHS does not have an explicit policy that strategic documents, such as component strategic plans, be consistent with the missions, goals, and objectives listed in the QHSR report. However, such consistency is expected by DHS senior management, according to the officials. We also identified 17 references to homeland security within the NSS that relate to DHS responsibilities. While not explicitly linked to the QHSR report in the NSS document, each of the 17 statements link to aspects of the QHSR report, such as homeland security threats identified or specific QHSR goals or objectives. The objectives for this report were to evaluate the extent to which the Department of Homeland Security (DHS) (1) consulted with stakeholders in developing the Quadrennial Homeland Security Review (QHSR) strategy; (2) conducted a national risk assessment to develop the QHSR; and (3) developed priorities, plans, monitoring mechanisms, and performance measures for implementing the QHSR and Bottom-Up Review (BUR) initiatives. To address our objectives, we analyzed DHS documents related to the QHSR, BUR, and budget development processes, including the QHSR report, BUR report, fiscal year 2012 budget request, and Fiscal Years 2012-2016 Future Years Homeland Security Program. We identified criteria for evaluating these processes by analyzing our prior reports on key characteristics of effective national strategies, key practices for effective interagency collaboration, strategic planning, performance measurement, and standards for internal control, among others. For a listing of these prior reports, see the related products listed at the end of this report. Based on these reports, we identified those key practices and characteristics applicable to quadrennial reviews, like the QHSR. The key practices we identified were involving stakeholders in defining QHSR missions and outcomes; defining homeland security problems and assessing risks; including homeland security strategy goals, subordinate objectives, activities, and performance measures; including resources, investments, and risk management; including organizational roles, responsibilities, and coordination across the homeland security enterprise; and establishing a DHS process for managing implementation of BUR initiatives. We vetted the key practices with our subject matter experts—staff with legal and methodological expertise and experience analyzing the Quadrennial Defense Review—and provided them to DHS officials for review and incorporated their comments as appropriate. As we developed our report, we grouped these key practices into three areas—stakeholder involvement, risk assessment, and implementation processes for the QHSR and BUR initiatives. Because respondents volunteered information about their views on the QHSR, we do not know the extent to which other officials within the same organizations shared these views. provided insights into stakeholder perspectives on how QHSR stakeholder consultations were conducted and how they could be improved. Further, we reviewed reports on the QHSR by the National Academy of Public Administration (NAPA) and the QRAC, both of which were based upon each organization’s collaboration experiences with DHS in developing the QHSR report. During the QHSR, NAPA partnered with DHS to conduct three National Dialogues, which allowed any member of the public to review draft QHSR material and provide online suggestions for the QHSR. According to the QRAC’s report, the QRAC served as a forum in which committee members, who were nonfederal representatives, shared independent advice with DHS on the QHSR process. We compared DHS’s stakeholder consultation efforts to our prior work on effective practices for collaboration and consultation. For example, based on a key practice in federal agency collaboration, we analyzed the extent to which DHS worked with stakeholders to establish agency roles and responsibilities when developing the QHSR. To determine the extent to which DHS conducted a national risk assessment to develop the QHSR, we analyzed risk analysis–related documents produced as part of the QHSR process, such as DHS risk assessment tools, and interviewed DHS officials responsible for developing risk analyses for use at DHS. We compared DHS’s risk assessment process in the QHSR to our prior work on key characteristics for risk assessment as well as DHS risk analysis guidance documents. For example, we reviewed our previous reports on key practices in risk management, including risk assessment approaches, and compared them to DHS’s effort to develop a national risk assessment methodology. In addition, we reviewed DHS guidance for use of risk assessment information and compared the guidance with DHS’s QHSR risk assessment process. To determine the extent to which DHS developed priorities, implementation plans, monitoring mechanisms, and performance measures, we analyzed DHS’s BUR implementation priorities and plans, such as DHS’s fiscal year 2012 budget request; monitoring mechanisms, such as BUR initiative scorecards; and DHS’s performance measures. We also interviewed DHS officials responsible for managing and monitoring the implementation of the BUR initiatives. We compared DHS’s processes for prioritizing, monitoring, and measuring implementation efforts to our prior work on key practices for risk management and implementation and monitoring of strategic initiatives. For example, we identified practices in our past reports and DHS guidance for using risk information in resource prioritization decisions and compared DHS’s efforts to prioritize and implement the QHSR strategy with those practices. We also compared DHS’s strategic-level performance measures for fiscal year 2011 to our criteria on key attributes of successful performance measures. Because DHS focused on aligning its performance measures with QHSR missions, we selected three key attributes of successful performance measures that were most relevant—linkage, core program activity, and limited overlap. In applying the attributes, we analyzed documentation, such as the QHSR report and DHS’s fiscal years 2010-2012 Annual Performance Report. We also interviewed DHS officials who are involved in overseeing the development and reporting of DHS performance measures. We conducted this performance audit from January 2011 through September 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Rebecca Gambler, Assistant Director, and Ben Atwater, Analyst-in-Charge, managed this assignment. Jean Orland and Janay Sam made significant contributions to this work. Michele Fejfar assisted with design and methodology, and Tracey King provided legal support. Labony Chakraborty, Jessica Orr, and Robert Robinson assisted with report preparation. Defense Transportation: Additional Information Is Needed for DOD’s Mobility Capabilities and Requirements Study 2016 to Fully Address All of Its Study Objectives. GAO-11-82R. Washington, D.C.: December 8, 2010. Department of Homeland Security: Actions Taken Toward Management Integration, but a Comprehensive Strategy Is Still Needed. GAO-10-131. Washington, D.C.: November 20, 2009. Transportation Security: Comprehensive Risk Assessments and Stronger Internal Controls Needed to Help Inform TSA Resource Allocation. GAO-09-492. Washington, D.C.: March 27, 2009. Quadrennial Defense Review: Future Reviews Could Benefit from Improved Department of Defense Analyses and Changes to Legislative Requirements. GAO-07-709. Washington, D.C.: September 14, 2007. Risk Management: Further Refinements Needed to Assess Risks and Prioritize Protective Measures at Ports and Other Critical Infrastructure. GAO-06-91. Washington, D.C.: December 15, 2005. Results-Oriented Government: Practices That Can Help Enhance and Sustain Collaboration among Federal Agencies. GAO-06-15. Washington, D.C.: October 21, 2005. Results-Oriented Government: Improvements to DHS’s Planning Process Would Enhance Usefulness and Accountability. GAO-05-300. Washington, D.C.: March 31, 2005. Combating Terrorism: Evaluation of Selected Characteristics in National Strategies Related to Terrorism. GAO-04-408T. Washington, D.C.: February 3, 2004. Tax Administration: IRS Needs to Further Refine Its Tax Filing Season Performance Measures. GAO-03-143. Washington, D.C.: November 22, 2002. Homeland Security: Proposal for Cabinet Agency Has Merit, But Implementation Will be Pivotal to Success. GAO-02-886T. Washington, D.C.: June 25, 2002. Standards for Internal Control in the Federal Government. GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999. Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1996.

Summary: The United States continues to face a range of evolving threats, such as the 2010 attempted attack on the nation's air cargo system, that underscore why homeland security planning efforts are crucial to the security of the nation. The Implementing Recommendations of the 9/11 Commission Act of 2007 required the Department of Homeland Security (DHS) to provide a comprehensive examination of the U.S. homeland security strategy every 4 years. In response, DHS issued its first Quadrennial Homeland Security Review (QHSR) report in February 2010 and a Bottom-Up Review (BUR) report in July 2010, to identify initiatives to implement the QHSR. As requested, this report addresses the extent to which DHS (1) consulted with stakeholders in developing the QHSR, (2) conducted a national risk assessment, and (3) developed priorities, plans, monitoring mechanisms, and performance measures for implementing the QHSR and BUR initiatives. GAO analyzed relevant statutes and DHS documents on the QHSR and BUR processes and, in response to a request for comments on the processes, received comments from 63 of the 85 federal and nonfederal stakeholders it contacted. Their responses are not generalizable, but provided perspectives on the processes. DHS solicited input from various stakeholder groups in conducting the first QHSR, but DHS officials, stakeholders GAO contacted, and other reviewers of the QHSR noted concerns with time frames provided for stakeholder consultations and outreach to nonfederal stakeholders. DHS consulted with stakeholders--federal agencies; department and component officials; state, local, and tribal governments; the private sector; academics; and policy experts-- through various mechanisms, such as the solicitation of papers to help frame the QHSR and a web-based discussion forum. DHS and these stakeholders identified benefits from these consultations, such as DHS receiving varied perspectives. However, stakeholders also identified challenges in the consultation process. Sixteen of 63 stakeholders who provided comments to GAO noted concerns about the time frames for providing input into the QHSR or BUR. Nine DHS stakeholders, for example, responded that the limited time available for development of the QHSR did not allow DHS to have as deep an engagement with stakeholders. Further, 9 other stakeholders commented that DHS consultations with nonfederal stakeholders, such as state, local, and private sector entities, could be enhanced by including more of these stakeholders in QHSR consultations. In addition, reports on the QHSR by the National Academy of Public Administration, which administered DHS's web-based discussion forum, and a DHS advisory committee comprised of nonfederal representatives noted that DHS could provide more time and strengthen nonfederal outreach during stakeholder consultations. By providing more time for obtaining feedback and examining mechanisms to obtain nonfederal stakeholders' input, DHS could strengthen its management of stakeholder consultations and be better positioned to review and incorporate, as appropriate, stakeholders' input during future reviews. DHS identified threats confronting homeland security in the 2010 QHSR report, such as high-consequence weapons of mass destruction and illicit trafficking, but did not conduct a national risk assessment for the QHSR. DHS officials stated that at the time DHS conducted the QHSR, DHS did not have a well-developed methodology or the analytical resources to complete a national risk assessment that would include likelihood and consequence assessments--key elements of a national risk assessment. To develop an approach to national risk assessments, DHS created a study group as part of the QHSR process that developed a national risk assessment methodology. DHS officials plan to implement a national risk assessment in advance of the next QHSR, which DHS anticipates conducting in fiscal year 2013. DHS developed priorities, plans, monitoring mechanisms, and performance measures, but did not consider risk information in making its prioritization efforts. DHS considered various factors in identifying high-priority BUR initiatives for implementation in fiscal year 2012 but did not include risk information as one of these factors, as called for in GAO's prior work and DHS's risk management guidance, because of differences among the initiatives that made it difficult to compare risks across them, among other things. Consideration of risk information during future implementation efforts could help strengthen DHS's prioritization of mechanisms for implementing the QHSR, including assisting in determinations of which initiatives should be implemented in the short or longer term. GAO recommends that for future reviews, DHS provide the time needed for stakeholder consultations, explore options for consulting with nonfederal stakeholders, and examine how risk information could be considered in prioritizing QHSR initiatives. DHS concurred with our recommendations.