DHS issued the National Response Framework for federal, state, and local agencies to use in planning for emergencies. It establishes standardized doctrine, terminology, and processes for responding to disasters and other catastrophic events in the United States. The framework is based on a tiered, graduated response; that is, incidents are managed at the lowest jurisdictional levels and supported by additional higher-tiered response capabilities as needed (see fig. 1). Local governments respond to emergencies using their own resources or assistance from neighboring localities. For larger-scale incidents that overwhelm the capabilities of local governments, assistance from the state may be sought. Depending on the circumstances, states have capabilities, such as the National Guard, that can help communities respond and recover. If additional resources beyond what an individual state can provide are required, the state may request assistance from other states through Emergency Management Assistance Compacts or from the federal government. Catastrophic incidents, by definition, result in extraordinary levels of damage or disruption to government functions, and governors have the option to seek federal assistance. The federal government has a wide array of capabilities and resources to assist state and local agencies respond to incidents. In accordance with the National Response Framework and applicable laws including the Robert T. Stafford Disaster Relief and Emergency Assistance Act (Stafford Act), various federal departments and agencies may play primary, coordinating, or supporting roles, based on their authorities and resources and the nature of the incident. In certain instances, national military capabilities may be requested to respond to an incident. Defense resources are committed following approval by the Secretary of Defense or at the direction of the President. According to FEMA planning guidance, plans must be integrated vertically among levels of government (e.g., local, state, and federal) to ensure a common operational focus and horizontally (e.g., health departments and law enforcement) to ensure that individual department and agency operations plans complement the jurisdiction’s plans. This integration enables stakeholders to synchronize the sequence and scope of a planned response in terms of roles and responsibilities, place, and time to a catastrophic incident. Catastrophic incidents include those that are defined in the National Planning Scenarios, which represent examples of the gravest dangers facing the United States, including terrorist attacks and natural disasters, and have been accorded the highest priority for federal planning efforts. FEMA’s primary mission is to reduce the loss of life and property and protect the nation from all hazards, including natural disasters, acts of terrorism, and other man-made disasters, by leading and supporting the nation in a risk-based, comprehensive emergency management system of preparedness, protection, response, recovery, and mitigation. As described in the National Response Framework, planning is one of six activities essential to preparing for an incident. Other activities include training; equipping; exercising; and evaluating and improving to build tribal, local, state, regional, and national capabilities necessary to respond to any type of disaster. As shown in figure 2, FEMA has 10 regional offices nationwide. Each region serves several states, and FEMA regional personnel work generally with the states to help plan for disasters, develop mitigation programs, and meet other needs when major disasters occur. Regional office locations are starred in the figure below, and in some cases are located within a pilot state. Washington, D.C. U.S. Northern Command (NORTHCOM) and U.S. Pacific Command (PACOM) are the combatant commands charged with carrying out DOD’s domestic civil support mission, which involves responding to the needs of 56 separate and often unique state and territorial governments. Additionally, NORTHCOM and PACOM coordinate with numerous federal agencies that also have roles in planning for and responding to a variety of incidents in the homeland. As part of the lessons learned from Hurricane Katrina, both NORTHCOM and PACOM have established Defense Coordinating Officers with staff to serve as DOD representatives to civilian authorities in the 10 FEMA regions. Defense Coordinating Officers are full-time senior-level military officers who provide liaison support and requirements validation, and they serve as single points of contact for state, tribal, local, and other federal authorities that need DOD support. In that way they are able to develop relationships with civilian authorities and gain an understanding of civilian capabilities so that DOD will know what, if anything, it may be called upon to provide in the event of a disaster or other incident. Additional defense resources include Emergency Preparedness Liaison Officers who are senior Reserve officers that are expected to assist the Defense Coordinating Officers in coordinating the provision of military personnel, equipment, and supplies to support the emergency relief and cleanup efforts of civilian authorities. When NORTHCOM and PACOM are called upon to support civilian authorities, in most cases support will be localized, limited, and specific. When the scope of the disaster is reduced to the point where the primary federal agency can again assume full control and management without military assistance, NORTHCOM and PACOM will withdraw. The TFER pilot program was first envisioned by senior DOD leadership to address the National Planning Scenarios through supporting and strengthening the catastrophic disaster preparedness of individual states. However, since DOD is always a supporting agency when responding to catastrophic events in the homeland, DOD leadership did not think it was appropriate for DOD to administer the pilot program, and FEMA administered and funded the pilot program. Although FEMA administered TFER, pilot program guidance stated that the TFER program was to place a special emphasis on coordination and integration between the pilot states, FEMA, and DOD. Further, a key measure of achievement in the pilot program is the degree to which DOD stakeholders are integrated as full partners in the development of states’ plans. FEMA designed TFER to help assess, strengthen, and advance state catastrophic preparedness planning across state and local agencies and the private sector, as well as achieve a fuller integration of state planning efforts with federal agency partners through a deliberate blending of civil- military planning expertise. The TFER pilot program centered on the creation and employment of a dedicated planning team. Specifically, the planning team was to synchronize catastrophic planning efforts at the respective state, regional, and federal levels; prepare planning documents; identify gaps in capabilities; and assist in the overall organization, administration, and improvement of the states’ catastrophic preparedness planning capacity. The three objectives of TFER are to (1) develop plans, (2) build relationships, and (3) document lessons learned. FEMA designed TFER to run 18 months, beginning on September 1, 2008, and ending on March 31, 2010. The pilot program was extended another 6 months to September 30, 2010—for a total time of 2 years—to allow states more time to accomplish their planning goals. As shown in figure 3 , FEMA announced in March 2010 that TFER would not continue past the pilot stage, but allowed the pilot states to draw down the remaining TFER funds and continue to complete initiatives started under TFER through August 2011. FEMA limited the pilot program to five states, each of which received $200,000 in the first year of the pilot program, and an additional $150,000 for the remainder of the program, for overall program funds totaling $1.75 million. It selected applicants to achieve a representative cross-section of geographic differences, FEMA regions, threat of catastrophic events, state emergency management structures, and participation in a separate FEMA catastrophic planning grant. As discussed in appendix II, FEMA provides funds to states and localities for catastrophic planning through other grant programs. For example, FEMA’s State Homeland Security Program provides funding that may be used for planning efforts that enable states to prioritize needs, build capabilities, update preparedness strategies, allocate resources, and deliver preparedness programs across disciplines and levels of government. FEMA administers the pilot program within its National Preparedness Directorate. This directorate is responsible for overseeing the coordination and development of strategies necessary to prepare for all hazards. As part of this mission, the National Preparedness Directorate is to provide policy and planning guidance, training and exercises, and technical assistance that builds prevention, protection, response, and recovery capabilities. In the aftermath of Hurricane Katrina, FEMA began to address the challenges revealed by its response. FEMA’s 2006 Nationwide Plan Review highlighted the need for fundamental planning modernization and observed, among other things, that: catastrophic planning efforts were unsystematic and uneven; planning expertise was insufficient for catastrophic incidents; and collaboration requirements were not well- defined, fostering a tendency to plan internally. In addition, according to FEMA, the 2006 Nationwide Plan Review revealed that 95 percent of state and urban area participants cited the need for federally funded planning support and technical assistance. FEMA also reported that in 2007, states and urban areas ranked planning among their highest grant funding priorities. The Department of Homeland Security’s Office of Inspector General recently reported that while progress has been made, the National Preparedness Directorate needs more effective coordination with state, local, and tribal governments. Our prior work on FEMA, and specifically the National Preparedness Directorate, has highlighted the need for strategic planning that identifies outcomes and performance measures for the Directorate’s programs. In October 2010, we also reported that since April 2009, FEMA has made limited progress in assessing preparedness capabilities and has not yet developed national preparedness capability requirements based on established metrics to provide a framework for these assessments. We reported that until such a framework is in place, FEMA will not have a basis to operationalize and implement its conceptual approach for assessing local, state, and federal preparedness capabilities against capability requirements to identify gaps for prioritizing investments in national preparedness. FEMA responded that it has made much progress since 2009 in meeting its legislative requirements and highlighted some of its specific achievements, such as the establishment of a working group to help consolidate and streamline reporting requirements for state, tribal, and local stakeholders. FEMA developed program objectives and a data collection plan—elements of sound management practices—but did not implement other elements of sound management practices, such as documenting standards for determining the program’s success and did not always follow its stated processes and procedures for TFER. Further, FEMA does not have agencywide policy guidance for FEMA program managers to follow when developing pilot programs. In November 2008, we reported that pilot programs can more effectively inform future program rollout when sound management practices are followed. Consistent with best practices in program management, our guide for designing evaluations, and our prior work, we identified sound management practices to design a pilot, to guide consistent implementation of a pilot, and to conduct analysis of the results. Our assessment of FEMA’s implementation of sound management practices for TFER is shown in table 1. FEMA developed objectives for the pilot states to follow, but did not document standards for determining whether TFER was successful. According to sound management practices, having objectives that reflect the goals of the pilot program can help determine whether the objectives of the pilot have been met. FEMA outlined three objectives for the TFER pilot program—develop plans, build relationships, and document lessons learned—and these objectives reflect TFER’s goals. For example, the “develop plans” and “document lessons learned” objectives reflect the goals of TFER because achieving these objectives may help assess, strengthen, and advance state catastrophic preparedness planning—a stated TFER goal. Further, achieving the “build relationships” objective may enhance the stated TFER goal of integrating state planning efforts with those of federal agency partners. However, FEMA did not define how states’ implementation of the objectives would impact the decision to extend or end TFER. According to sound program management practices, defining standards for determining pilot-program performance is necessary to determine what success level is appropriate for judging the pilot program’s effect. The TFER program manager said that discussions about the expectations for TFER were held at a high level with senior leadership from FEMA, DOD, and the participating states, such as the Adjutants General, before TFER planners were hired. However, FEMA did not include the standards resulting from these conversations in the TFER program guidance. The program manager said that the agency had a limited time frame in which to design the pilot and that if he were to design the TFER pilot now, he would build language into the grant guidance that included standards for measuring performance. By clearly defining standards for determining program performance, FEMA would be better positioned to determine whether TFER enhanced states’ catastrophic planning, as well as achieved a fuller integration of state planning efforts with federal agency partners through a deliberate blending of civil-military expertise. FEMA outlined steps for administering TFER, which aligns with sound management practices, but FEMA did not always adhere to the processes and procedures it developed for the pilot. Sound management practices include developing processes and procedures for approving, reporting, and monitoring and providing program support. Further, according to Standards for Internal Control in the Federal Government, such control activities are an integral part of an entity’s planning, implementing, reviewing, and accountability for stewardship of government resources and achieving effective results. FEMA outlined steps in two key documents, the TFER Guidance and Application Kit and the TFER Pilot Information Package in which FEMA described the support it would provide the participating states and the procedures for monitoring the program. FEMA developed processes and procedures for providing funding, training, holding a meeting with the pilot states, and approving state project management plans that met certain requirements, and establishing a mechanism for sharing lessons learned among the states, but FEMA did not fully follow the steps it outlined. For example, FEMA provided $1.75 million in funds to the state administrative agencies identified in the TFER Pilot Information Package and established a virtual communication tool for the TFER pilot states to communicate and share lessons learned. However, FEMA provided training for the initial cohort of TFER planners in April 2009, which FEMA officials told us consisted of two-thirds of the planners originally hired by the states. All five states found FEMA’s training useful. However, officials in two states told us additional training would have been useful. For example, one TFER planner who was not hired at the time of the training stated that additional training could have provided planners more opportunities to network and gain knowledge about state and federal resources. Further, FEMA conducted a meeting with the pilot states to accomplish tasks such as establishing support requirements and planning deliverables, but FEMA approved three state project management plans that did not contain all the recommended information, such as specific tasks, milestones, and deliverables. The TFER program manager stated that part of the intent of TFER was to design it in real time, working with the jurisdictions, and that the omissions of the recommended information did not negatively impact the states’ implementation of TFER. However, the states whose project management plans did not include specific deliverables have not completed draft plans. Further, FEMA also established processes and procedures for reporting and monitoring, but did not fully follow them. Specifically, FEMA stated in the TFER Pilot Information Package that the project manager would conduct biweekly conference calls with the states, but state officials in all five pilot states noted that FEMA headquarters involvement was inconsistent. According to the program manager, the purpose of these meetings was to ensure the pilot was functioning smoothly, so FEMA conducted biweekly meetings as TFER became operational but the meetings were discontinued once the pilots were underway. However, officials in four states reported that it would have been helpful if FEMA had provided more guidance on plan development. Further, TFER planners in one state reported that the lack of guidance made it more difficult to write plans. Another planner stated that FEMA did not provide the expected level of meetings and technical assistance. The program manager stated he was available to provide the states with assistance as needed throughout the pilot program. In addition, FEMA established reporting requirements and a plan to conduct a 6-month evaluation. For example, the states submitted reports to FEMA—to include the status of milestones and deliverables—but the program manager told us he did not review all of these reports because not all of these reports were submitted to his office. As a result, the program manager could not monitor progress in meeting milestones and completing deliverables. FEMA also conducted a 6-month evaluation of the pilot program, but four pilot states reported that FEMA did not provide feedback on the evaluation. According to sound management practices, continuous monitoring can be a useful tool for facilitating effective management by providing regular feedback about how well a program is performing its functions. This type of feedback allows managers to take corrective action when problems arise and can also provide stakeholders with regular assessments of program performance. By adhering to the steps outlined in TFER documents, FEMA could have more effectively monitored the pilot program to better ensure that the states receive the level of guidance and support they identified as beneficial in order to enhance their TFER catastrophic planning efforts. FEMA developed a data collection plan but did not reliably collect data for the pilot evaluation, and FEMA did not develop a data analysis plan to describe how it would track TFER’s performance and evaluate its effectiveness based on standards to determine program performance. For example, FEMA developed elements of a data collection plan, following sound management practices such as detailing the type of data necessary to monitor and evaluate the pilot and identifying the timing and frequency of data collection. Further, FEMA’s data collection plan followed reporting requirements specified in its program documents for TFER participants, and also included a data collection tool for evaluation of the pilot, presented in appendix III. However, FEMA did not always identify the source that was to collect the data. Sound management practices say that the source of data should be specified in the data collection plan, and according to program evaluation guidance, the reliability of evaluations can vary depending on who responds. During interviews with officials from participating states and FEMA regional offices, we found inconsistencies in who completed the evaluation questions. For example, two pilot states developed their own responses and three FEMA regions developed the responses for their respective pilot states, even though one of the questions explicitly asked for the opinion of state stakeholders. Further, two of the three states were not cognizant of the fact that their FEMA regional offices had completed the evaluation questions; therefore, these states’ views may not have been accurately captured in the evaluation responses. Moreover, officials in one FEMA regional office questioned whether they were the appropriate FEMA personnel to administer the evaluation questions, since FEMA headquarters had collected all prior information from the states. Collecting data from consistent sources across the five pilot states could have better positioned FEMA to ensure the data were reliable and compare responses across states in order to draw conclusions about states’ experiences in the pilot program. Further, FEMA did not collect information to determine whether all of the pilot program objectives were fully met. Specifically, FEMA changed some of the data collection questions it provided the states in the TFER Pilot Information Package when it released updated questions in the TFER Evaluation Criteria document. For example, FEMA used the same three objectives in both its original and updated documents, and its objectives on developing plans and documenting lessons learned objectives were consistent across documents, but FEMA did not collect information to assess all of the relationships it identified as important for its objective on building relationships in the TFER Pilot Information Package. For instance, FEMA did not evaluate coordination with FEMA, Defense Coordinating Officers, or Emergency Preparedness Liaison Officers—all relationships principal to implementing TFER, according to the TFER Pilot Information Package document. FEMA included a question that asked the pilot states which federal agencies they coordinated with, but did not include any question that would allow it to evaluate the extent to which coordination occurred with these stakeholders. As a result, the evaluation responses we reviewed listed the stakeholders they coordinated with but did not discuss the extent of coordination between state officials and stakeholders. FEMA officials said that because the Defense Coordinating Officer and Emergency Preparedness Liaison Officer positions were not the only relationships explored between states and federal agencies, the evaluation criteria was changed to make it more general to determine with which agencies the TFER states engaged. However, by changing the evaluation question to a general and descriptive question on federal relationships, it will be difficult for FEMA to determine if the states achieved a fuller integration of planning efforts with federal agency partners—one of the goals of TFER. FEMA did not develop a data analysis plan to describe how it would track TFER’s performance and evaluate its effectiveness based on criteria to determine program performance. According to sound management practices, a data analysis plan sets out who will do the analysis and when and how data will be analyzed to measure the pilot program’s performance. FEMA did not finalize a data-analysis methodology describing how FEMA would use information collected from the states to evaluate TFER until after it collected responses to FEMA’s evaluation criteria questions. The TFER program manager stated that FEMA plans to look for trends in the states’ responses to the evaluation questions to discern common threads and outliers. According to the program manager, FEMA did not develop a plan that outlined the overall approach it would use to analyze the TFER data during design of the pilot program because the pilot was not intended to produce quantitative results. He further stated that FEMA wanted to see what worked and what did not work as the pilot evolved. However, a data analysis plan could have helped guide the analysis of the qualitative data FEMA collected from the pilot states and FEMA regional offices. Developing data collection and data analysis plans following sound management practices to establish a methodology for collecting and evaluating the pilot’s results could have provided FEMA with reasonable assurance that it possessed the data and performance information needed to draw reasonable conclusions on the impact of TFER. The TFER program manager told us he is analyzing the data collected for TFER to draw conclusions regarding the pilot program, but FEMA officials stated that TFER would not be continued past the pilot stage. The information on the TFER pilot program’s evaluation stated that the results of the evaluation would determine the prospect for the program’s continuation and level of future funding. However, prior to conducting its evaluation of the pilot, FEMA announced in March 2010 that TFER would not be continued as a stand-alone grant program. According to FEMA officials, the decision to discontinue TFER as a stand-alone grant program was made because DOD may begin a similar initiative and states could allocate other FEMA grant funding for catastrophic planning purposes. Specifically, the TFER program manager noted that the National Guard Homeland Response Force, which is a DOD concept that would place National Guard personnel in each FEMA region, could continue the planning efforts associated with TFER. DOD officials stated that these personnel would be located regionally and not in individual states; therefore, the National Guard Homeland Response Force initiative could supplement but not replace a TFER-like initiative. Also, according to DOD, the National Guard Homeland Response Force would have as its mission response to chemical, biological, radiological, nuclear, and high-explosive incidents and would not have a broad purview like TFER. A FEMA assistant administrator also said that FEMA made the decision to not continue TFER due to changing budget conditions and to focus on its legislative and presidential directive mandates. Other FEMA grant programs available for catastrophic planning are discussed in appendix II. According to the TFER program manager, FEMA leadership will make a decision on whether and how the results of the evaluation are disseminated after the program office finishes the evaluation of TFER. According to sound management practices, results and recommendations that emerge from evaluations must be disseminated in ways that meet the needs of stakeholders. The TFER program manager stated it is not his decision whether the results of the pilot’s evaluation will be shared. According to the TFER program manager, FEMA leadership will make a decision on whether and how the results of the evaluation are disseminated after the program office finishes the evaluation of TFER. FEMA administered the TFER pilot program, but other key stakeholders in federal disaster response, such as the pilot states and DOD, could benefit from the results of FEMA’s evaluation of the TFER pilot program. Disseminating the results, recommendations, and lessons learned from the TFER pilot could assist future state catastrophic planning efforts, whether they be led by FEMA, DOD, another federal agency, or a state. FEMA’s Office of Policy and Program Analysis reported that FEMA does not have a directive on how to design, administer, and evaluate a pilot program. In the absence of this, FEMA does not have a systematic approach for designing, administering, and evaluating such programs. The program manager stated there are no formal guidelines for developing FEMA grants. We previously reported that FEMA faced methodological challenges in assessing capabilities and has not generated meaningful preparedness information from data collected from pilot programs. Standards for Internal Controls in the Federal Government state that management is responsible for ensuring that detailed policies, procedures, and practices are developed and built into and are an integral part of operations. These elements, which are an integral part of an agency’s ability to ensure accountability and achieve effective results, need to be clearly documented to help ensure that management directives are carried out as intended. By developing and implementing policy guidance that includes sound management practices, FEMA could be better positioned to ensure its pilot programs meet their intended goals. All five states have taken steps to follow FEMA’s TFER Information Package and TFER Evaluation Criteria document to address the TFER pilot program’s three objectives—develop plans, build relationships, and document lessons learned—and these efforts are ongoing. First, of the four states committed to developing plans, two states have completed draft plans, and TFER planners in all states reported using FEMA’s CPG 101 planning process to develop initiatives that fill gaps in state catastrophic planning. Second, all five states built relationships with stakeholders, including FEMA and state agencies, but coordination with DOD—specifically, with the Defense Coordinating Officers and Emergency Preparedness Liaison Officers—was limited. Third, all states documented lessons learned from the pilot program, which included the benefit of dedicated funding for catastrophic planning, and establishing longer time frames for states to complete the planning process. However, these efforts are ongoing as four states continue to spend TFER funds. Four of the states are developing plans and all five pilot states reported they are using the planning process outlined in FEMA’s CPG 101. However, two of the four states developing plans have not completed them. The fifth state is focusing on other planning initiatives. FEMA articulated the criteria for this objective in the TFER Pilot Information Package and the TFER Evaluation Criteria document, which focused on organizing plans according to CPG 101 and plan development. Based on audit work conducted in the TFER pilot states, we assessed the states’ progress based on these criteria, and concluded that two of the states have made substantial progress in fulfilling the FEMA objective on developing plans, and the other three have made some progress, as shown in figure 4. Figure 4 further explains this assessment, as well as appendix IV, which provides specific information regarding each pilot state’s TFER initiatives and progress made towards meeting FEMA’s objectives for TFER. Developing plans. The status of the states’ TFER plans varies; Washington and South Carolina have completed some but not all of their plans, while Hawaii and West Virginia have not completed any draft plans. Hawaii and West Virginia officials identified reasons for not having completed draft plans, including delays in hiring qualified planners. One of FEMA’s requirements for TFER was that planners have experience in conducting civil-military support planning and operations with the National Guard and/or military experience. In Massachusetts, South Carolina, and Washington, all planners initially hired for TFER had military backgrounds. These states hired planners as early as February 2009—5 months after the pilot program began. One of the planners initially hired had prior military and civilian experience, and the other had a logistics background. Hawaii hired its first TFER planner in April 2009 but did not hire a second planner until August 2009—11 months after the pilot began. Hawaii officials told us that they found it difficult to hire qualified individuals with a military background because better paying options exist in the state for qualified individuals. One of Hawaii’s two current planners does not have military experience, but state officials told us that the individual is qualified because of his information technology background, which was needed to support logistics planning. Additionally, he worked for the state planning agency. West Virginia officials reported that they hired their planners in the July/August 2009 time frame—10 to 11 months after the pilot began. Two applicants originally applied for the position, one of which West Virginia hired. This individual did not have a military background, but officials stated he was qualified based on his decades of experience as a first responder. West Virginia subsequently hired a National Guardsman. The West Virginia TFER supervisor stated that inadequate planning capacity at the local level also delayed TFER efforts because West Virginia TFER focused on building capacity at the local level. FEMA granted all five states a 6-month extension based on delays the states experienced. As previously discussed, planning continues in four states, as they are still spending TFER funds, as of March 2011 (see fig. 5). Massachusetts TFER planners facilitated planning between different levels of government, but other state planners are developing the plans. Massachusetts initially focused on contributing to Boston’s improvised explosive device planning efforts, but differences in time lines, combined with TFER’s brief performance period, caused the team to change its focus to facilitating collaborative emergency planning among federal, state, local, and private stakeholders. For example, the Massachusetts TFER planners facilitated collaborative planning among the Massachusetts Department of Public Health, the U.S. Department of Health and Human Services, the Massachusetts Emergency Management Agency, and FEMA to create a statewide medical support plan. TFER planners developed interagency relationships and helped the planning team select goals and objectives but did not draft the plan, which state officials said other state planners are developing. State officials decided the TFER planners would not actually write the plans because of the pilot program’s short time frame. Filling gaps in state planning. State officials reported that TFER initiatives fill gaps in existing state plans. For example, South Carolina, Washington, and West Virginia chose to work on a catastrophic incident plan as part of their TFER efforts, and this plan is an annex to the states’ basic emergency operations plan. Further, officials in four of five states reported that TFER initiatives focused on gaps not addressed by other planning initiatives. For example, the FEMA Region X TFER contact stated that Washington successfully integrated TFER with the Regional Catastrophic Grant Planning Program. Two of three West Virginia TFER scenarios pertain in part to scenarios addressed by other planning efforts, but officials stated that TFER planners work collaboratively with the planners involved in these other efforts, as described in appendix IV. Additionally, Hawaii officials anticipate the Regional Catastrophic Planning Grant Program will address hurricanes in 2011—before the state exhausts its TFER funds. State officials expect these efforts to be complementary. FEMA’s planning process. All five states reported that they are following FEMA’s planning process, and states that developed draft plans utilized the structure recommended in FEMA’s CPG 101 (see fig. 6). All of the states reported they are following FEMA’s six-step planning process, but they are in different stages of that process, as of March 2011. Washington has exercised three out of seven of its plans—part of step six. However, most of Washington’s plans remain in draft form, which is associated with step five. Since Hawaii, South Carolina, and West Virginia officials told us their states have not exercised any plans, they are in step five—the step where planners write the plan and the plan is approved. Massachusetts state officials decided that their pilot program would coordinate planning efforts, but that other state planners would develop plans; officials stated they did not move beyond step four in the planning process. Officials in the four states that committed to developing plans reported that they are structuring plans as recommended by CPG 101, but some sections are not included in the states’ draft plans. CPG 101 provides a recommended structure for annexes to a state’s basic emergency operations plan. These annexes comprise half of the 32 TFER initiatives spanning all five pilot states. Ten of 13 annexes we reviewed were missing at least one recommended section. For example, none of the three catastrophic incident plans we reviewed contain a section on direction, control, and coordination. South Carolina and Washington officials noted that their plans did not include all of the recommended sections because other state plans like the basic emergency operations plan contained these sections. Washington officials further stated that TFER-supported planning is part of the state emergency operations plan and is not designed to stand alone. Further, Hawaii officials told us that they are following the FEMA guidance, but its plans are incomplete and work is ongoing. West Virginia officials stated that it is following the structure recommended in CPG 101, but since some of the sections rely on information gathered at the county level, it will take about 5 years before the state has enough information to include each recommended section in its catastrophic incident plan. We were unable to further assess the content of the pilot states’ plans because all but three of the plans TFER states are developing remain in draft form. relevant stakeholders. As discussed in figure 7, all five states have made some progress in building relationships with relevant stakeholders. Our assessment—based on audit work conducted in the TFER pilot states—is summarized in figure 7 and further explained below as well as in appendix IV, which provides specific information regarding each pilot state’s TFER initiatives and progress made towards th is objective. All states have documented lessons learned to date, but four states have not exercised TFER plans to determine their effectiveness in the event of an emergency. As shown in figure 8 and further explained in appendix IV, the TFER pilot states have provided a number of lessons learned and recommendations on how to improve a similar program in the future. One state, Washington, has made substantial progress documenting lessons learned because it has exercised three of its plans to evaluate their effectiveness, one of the components FEMA cited in the objective. FEMA articulated the criteria for this objective in the TFER Pilot Information Package and the TFER Evaluation Criteria document, which focused on documenting lessons learned, including exercising plans to evaluate TFER’s effectiveness. All five pilot states documented lessons learned for a report published by the National Guard Bureau in August 2010, and we obtained additional lessons learned during our interviews with state officials. Further, two of the states provided us with the lessons learned they submitted to FEMA for its evaluation, which is not yet complete. Some of the lessons learned were similar across the pilot states. For example, all five states considered the pilot a success; Hawaii, Massachusetts, South Carolina, and West Virginia officials reported that TFER allowed their states to conduct catastrophic planning that otherwise would not have occurred, and Washington reported that TFER advanced its catastrophic and emergency logistics planning by at least 2 years. Washington officials further stated that the $350,000 the state received in TFER funding generated more return on investment than any other $350,000 in the state’s emergency planning budget. Based on their view of TFER’s success, officials in each of the five pilot states recommended FEMA expand TFER to other states. All states reported three principal reasons for considering TFER a success: (1) autonomy to develop their own work plans, (2) provision of dedicated funds to hire planners, and (3) the background of the TFER planners. (1) State officials reported that they worked collaboratively with FEMA to select catastrophic scenarios. For example, South Carolina chose to work on plans related to terrorism attacks because state officials believe they had adequately planned for hurricanes and earthquakes—the two National Planning Scenarios ranked higher than terrorism attacks in the state’s risk assessment. Officials in all five states highlighted the benefit of this flexibility. (2) Officials in all five states reported that the narrow focus of TFER, dedicated solely to enhance catastrophic planning, was key to the pilot program’s success. State officials told us that state planners are often required to assist emergency planning and response efforts during more routine disasters, such as seasonal floods and forest fires, but since the terms of TFER did not allow states to use TFER funds for these efforts, catastrophic planning continued unabated. (3) Officials in all five states reported that the planners’ military background was beneficial. For example, state officials in Massachusetts told us the planners’ military experience provided them with a level of experience and training not often found in other sectors and gave them credibility. However, officials in three states told us that civilian planning or emergency management experience also provided TFER planners the needed skill set. States also reported challenges they faced, including (1) short time frames that limited their ability to complete the planning process, and (2) not enough guidance for the specialized plans they were developing. (1) All five states reported that the short time frames limited their ability to complete the planning process. FEMA structured TFER as an 18-month program, and states noted that this was an insufficient amount of time to complete the full cycle of planning. Officials in one state thought that an adequate time frame would have been 3 years, noting that this is the time frame FEMA established for the Regional Catastrophic Planning Grant Program. The TFER program manager stated that FEMA’s 6- month program extension—bringing the pilot program time frame to 2 years—provided a sufficient amount of time to complete the TFER initiatives. (2) CPG 101 provides general format and content guidance applicable to all plans, but officials in four of the pilot states said that it would have been helpful for FEMA to provide guidance for specialized plans such as catastrophic incident annexes, terrorism plans, and logistics plans. For example, one TFER planner stated that clearer guidance could have accelerated the planning process. Further, officials in a fourth state said that they are using an unpublished FEMA guide to support their planning efforts. FEMA officials told us that states can request technical assistance and examples of well-developed plans to aid in their own plan development. According to the TFER program manager, FEMA is in the process of developing additional Comprehensive Preparedness Guides that will inform plan content for future state planning efforts. The process of documenting lessons learned is ongoing because Hawaii, South Carolina, Washington, and West Virginia are still developing TFER plans and spending TFER funds. Hurricane Katrina highlighted gaps in the nation’s preparedness to respond effectively to catastrophic incidents. By their nature, catastrophic events involve extraordinary levels of casualties, damage, or disruption that will likely immediately overwhelm state and local responders— circumstances that make sound planning for catastrophic events all the more crucial. Planning is a key component of national preparedness. Planning provides a methodology to determine required capabilities and helps stakeholders learn and practice their roles for building and sustaining national preparedness capabilities against terrorist attacks and other hazards. As state and local governments continue to develop and improve plans for catastrophic events and identify potential resource shortfalls, the federal government will be in a better position to understand the nature of the gaps it may be called upon to fill if state and local resources are overwhelmed. The TFER pilot program was one such effort to enhance catastrophic preparedness and provide federal stakeholders with valuable information regarding local, state, or regional response capability. However, by not consistently following sound management practices to design, administer, and evaluate the TFER pilot program and by not following its processes and procedures for administering the pilot, it is unclear whether TFER achieved its intended purpose. As we have reported, FEMA has piloted other efforts that have not generated meaningful preparedness information from the data collected. Future pilot programs at FEMA could benefit from policy guidance that includes sound management practices to design, administer, and evaluate pilot programs. In addition, future state catastrophic planning efforts could benefit from the dissemination of the evaluation results of the TFER pilot program to relevant stakeholders. We recommend the Administrator, Federal Emergency Management Agency, take the following two actions. To help ensure future pilot programs achieve their intended results and provide the performance information needed to make effective management decisions for broader implementation, develop and implement policies and guidance for pilot programs that follow sound management practices. This guidance should include, at a minimum, requirements for a clearly articulated methodology with objectives reflective of overall program goals and standards for determining program performance; procedures for monitoring program performance; a data collection plan; a data analysis plan; and a process to disseminate the results and lessons learned that emerge from the pilot. To help ensure stakeholders receive valuable information regarding catastrophic preparedness from lessons learned during the TFER pilot program, disseminate the evaluation results and recommendations that emerge from the TFER pilot program in ways that meet the needs of current and future stakeholders. We provided a draft of this report to FEMA and DOD for review and comment. FEMA provided oral comments on the draft report on April 5, 2011. FEMA fully concurred with both of our recommendations but did not specify how it planned to address them. DOD did not provide comments on the draft report because the report did not include recommendations to DOD. Both FEMA and DOD provided technical comments, which we incorporated throughout our report as appropriate. As agreed, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we are sending copies of this report to the Administrator, Federal Emergency Management Agency, and Secretary of Defense. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact Bill Jenkins at (202) 512-8777 or by e-mail at jenkinswo@gao.gov, or Davi M. D’Agostino at (202) 512-5431 or by e-mail at dagostinod@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. To identify the extent to which the Federal Emergency Management Agency (FEMA) followed sound management practices in developing, administering, and evaluating the Task Force for Emergency Readiness (TFER) pilot program, we analyzed program guidance and other key documents including the TFER Pilot Information Package and the Guidance and Application Kit. We analyzed prior GAO work on pilot programs and program management as well as social science literature to determine elements of sound management practices. We considered other criteria, consulting checklists on elements of program management, but determined some of the characteristics identified in this literature were not appropriate or suitable to the type of study performed by FEMA We grouped the elements of sound management practices that we identified as relevant into three categories: design, administration, and evaluation. We then analyzed the information obtained from our interviews and site visits and compared the results of our analysis with th identified elements of sound management practices. Furthermore, we analyzed prior GAO reports and Department of Homeland Security (DHS) Office of Inspector General reports on FEMA grants administration and catastrophic preparedness to identify any past challenges. . We reviewed and analyzed FEMA grant program guidance and met with FEMA officials in the Grants Program Directorate to determine whether other grant programs, in addition to TFER, allow for funds to be used for catastrophic planning. We identified four major grant programs states can use for catastrophic planning purposes: the Regional Catastrophic Planning Grant Program, the State Homeland Security Program, the Urban Areas Security Initiative, and the Emergency Management Performance Grants. Other grants, such as the Port Security Grant Program (PSGP) were considered, but discussions with officials in the Grant Programs Directorate helped to clarify and narrow the scope of our review to the most appropriate candidates for the types of catastrophic planning described in the TFER grant. We collected fiscal year 2010 funding levels for these grants, as well as information about how funds could be used for planning, funding restrictions, and eligibility requirements. To identify the extent to which the five states participating in FEMA’s TFER pilot program addressed the pilot program’s stated objectives, we analyzed information that FEMA provided in the Pilot Information Package and further defined in its TFER Evaluation Criteria document. The pilot program’s three stated objectives were building relationships, developing plans, and documenting lessons learned, and FEMA provided several criteria to measure progress towards addressing each of these objectives. Additionally, FEMA identified key components to each objective. We used FEMA’s criteria to assess states’ progress toward addressing the three program objectives. Moreover, we identified additional criteria for the building relationships objective in FEMA’s TFER Pilot Information Package, including whether pilot states coordinated with other states and with FEMA. These components were identified as key components in the TFER Pilot Information Package, but were not identified in the TFER Evaluation Criteria. Therefore, we included the additional criteria in assessing the states’ progress toward addressing the building relationships objective. We assigned a numerical ranking to represent states’ progress for each objective’s components based on interviews we conducted and our review and analysis of documents we obtained during our site visits that included draft plans, progress reports, and status briefings. We gathered information related to the lessoned learned objective from state submissions to a report published by the National Guard Bureau in August 2010, and we obtained additional lessons learned during our interviews with state officials. Further, two of the states provided us with the lessons learned they submitted to FEMA for its evaluation, which is not yet complete. If the state met the criterion for a particular component, we assigned it a ranking of ‘5’, if the state partially met the criterion, we assigned it a ranking of ‘3’, and if the state did not meet the criterion, we assigned it a ranking of ‘0’. Next, we averaged the components for each of the three objectives to obtain an overall objective score. We then assigned designations to the overall percentages of scores as follows: averages greater than 80 percent were designated as ‘substantial progress’; averages from 20 percent to 80 percent were designated as ‘some progress’; and averages less than 20 percent were designated as ‘little or no progress.’ We did not assign designations to four criteria pertaining to developing plans and six criteria pertaining to lessons learned because we determined that these criteria do not inform the extent to which states met the objective and/or there was not enough information to quantify the results. States did not receive a designation of “substantial progress” for each objective unless they met the criteria related to the key component of that objective, that is, (1) developing actual plans—measured by the completion of draft plans; and (2) building relationships with FEMA officials, Defense Coordinating Officers, and Emergency Preparedness Liaison Officers. FEMA did not identify key components of the third objective, documenting lessons learned. In instances where states received an average above 80 percent but did not meet the criteria related to these key components, we assigned a designation of “some progress” for that particular objective. For example, by adding the rankings assigned to South Carolina for building relationships, the state received a score of 31 out of 35, or 89 percent. However, South Carolina did not fully meet the criteria for coordinating with DOD and FEMA; so the state received the designation of ‘some progress.’ Where possible, we used the Comprehensive Preparedness Guide (CPG) 101 to determine the extent to which the states’ plans adhered to FEMA’s suggested guidelines regarding plan structure and content. About one half of the TFER states’ 32 planning initiatives involved developing plans. Specifically, CPG 101 recommends that basic emergency operations plans and their annexes include the following sections: purpose, situation overview, and planning assumptions; concept of operations; organization and assignment of responsibilities; direction, control, and coordination; information collection and dissemination; administration, finance, and logistics; and authorities and references. CPG 101 provides additional guidance for the structure of the concept of operations section for hazard- specific annexes—plans South Carolina developed—and functional annexes—one of which Washington developed. We did not evaluate the content of these plans because the plans remain in draft form. CPG 101 does not offer specific content guidance for the other plans types of plans TFER states developed. We did not assign a progress designation for the criterion that asked the extent to which states’ plans adhered to CPG 101 because all but 3 of the plans TFER states are developing remain in draft form. To address these objectives, we interviewed officials and obtained information and related documents from federal agencies, the five pilot states, and other relevant entities. Within FEMA, we met with officials from the National Preparedness Directorate, the Grants Programs Directorate, the Response Directorate, and officials in FEMA Regions I, III, IV, IX, and X. Within the Department of Defense (DOD), we met with representatives from the Office of the Assistant Secretary for Homeland Defense and America’s Security Affairs and the Joint Chiefs of Staff. We also met with U.S. Northern Command and U.S. Pacific Command officials because they are the combatant commanders whose areas of responsibilities include the United States and its territories. Further, we met with the U.S. Department of Homeland Security, the National Guard Bureau, the Institute for Defense Analyses, and the National Governor’s Association. We conducted site visits to all five participant states from June through September 2010. During our site visits, we met with the planners hired through the TFER pilot program and other state, local, National Guard, and DOD regional officials to discuss FEMA’s management of the TFER pilot program and the states’ implementation of it. We conducted this performance audit from May 2010 to April 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We identified four primary FEMA grant programs that states and localities can use for catastrophic planning purposes, as shown in table 2. Table 3 shows that these FEMA grant programs provide the states and other eligible recipients significantly more funds than TFER provided the pilot states. For example, in fiscal year 2010 two of the TFER pilot states received more than $6 million from the Emergency Management Performance Grant program compared to the $350,000 they each received for TFER over a 24-month period. According to FEMA officials, the TFER grant represents at most 3 percent of the grant money spent on planning by the pilot states, as of March 2010, and all TFER activities are allowable under other grant programs. TFER has unique attributes from other FEMA grants. The primary difference between TFER and these other grant programs is that the TFER pilot states are required to use the grant funds exclusively for planning efforts. In contrast, states may use the other FEMA grant program funds for a full range of activities such as planning, equipment, training, and construction and renovation. Like TFER, the Emergency Management Performance Grants Program provides funding to states with no requirement to pass through funding to local governments. The Regional Catastrophic Planning Grant Program provides funds directly to urban areas, but beginning in fiscal year 2010, states were allowed to withhold up to 20 percent of the funds to implement or integrate the urban areas’ approved project plans throughout the state, or with neighboring states. According to the TFER program manger, FEMA provided the TFER pilot states with the option of continuing their efforts using Regional Catastrophic Planning Grant Program funds; however the four states that received funds under both programs chose not to pursue this option. Similarly, the State Homeland Security Program requires states to pass through at least 80 percent of the funding to local governments. Further, the Urban Area Security Initiative provides the funding to urban areas, and any funds retained by the state must be used to directly support the urban area. Consequently, states cannot use most Regional Catastrophic Planning Grant, Urban Areas Security Initiative, and State Homeland Security Program funds to conduct statewide catastrophic planning. However, officials in one state told us they used 2009 Emergency Management Performance Grant funds to hire planners to continue planning begun under TFER, and officials in two other states told us they may use State Homeland Security Program grant funding to continue the catastrophic planning efforts begun under TFER. The TFER pilot program was to emphasize integration of planning efforts across sectors, jurisdictions, and functional disciplines, as well as integration among state, regional, and federal agencies. The Federal Emergency Management Agency (FEMA) developed the following questions in collaboration with the participating states to evaluate TFER. The evaluation questions were structured around FEMA’s stated objectives for TFER—to develop plans, build relationships, and document lessons learned. Our assessment of the extent to which each state addressed FEMA’s stated objectives for the TFER pilot is presented in appendix IV. We did not assess four criteria pertaining to developing plans and six criteria pertaining to lessons learned because we determined that these criteria do not inform the extent to which states met the objective and/or there was not enough information to quantify the results. Further, we did not assess the administrative elements of the data collection process or administrative elements in appendix IV. Those criteria which we did not assess are italicized below. FEMA’s TFER Pilot Program Evaluation 1. Development of well coordinated and integrated preparedness plans through an effective planning process. a. Please provide a summary of the primary TFER accomplishments. b. Describe efforts to coordinate and integrate plans with state and federal partners. c. Does TFER supported planning reflect the National Preparedness Guidelines and the full cycle of planning steps outlined in CPG 101? d. Is TFER planning responsive to the National Planning Scenarios and/or other threats identified through the state’s own all-hazards risk assessment? e. Are the structure, organization, and content of TFER-developed plans / annexes / appendices consistent with that recommended in CPG 101? f. How are TFER efforts coordinated and integrated with resourcing efforts (e.g., training, exercises, grants, etc.)? g. Are TFER developed plans consistent with current state emergency plans? Are they integrated and not duplicative? h. Have actual plans been produced? If so, what is their impact on state emergency readiness? i. In what areas have TFER supported plans been developed (e.g., logistics, coordination, etc.)? 2. Establishment of cross-sector and interagency relationships and protocols. a. Does TFER supported planning reflect the formation and involvement of the sort of integrated and collaborative planning teams described in CPG 101? b. Have mechanisms been established / augmented and employed to ensure cross-sector, multijurisdictional, and interagency planning coordination and integration? c. Have TFER supported plans been coordinated with local jurisdictions? Which ones? State agencies? Which ones? Federal partners? Which ones? d. Has a blending of TFER civil-military planning expertise been fully exploited to assess, strengthen, and advance state catastrophic preparedness planning? e. How have TFER efforts incorporated the capabilities of the private sector as partners in planning activities? 3. Documentation of lessons learned and recommendations for improvement. a. In the opinion of state stakeholders, has the TFER Program resulted in demonstrable improvement in state planning capability? b. To what degree has state-level TFER planning been synchronized with and improved the overall quality and effectiveness of planning efforts? c. How were TFER planners integrated into the state emergency management structure and process? d. Has the TFER Program resulted in meaningful recommendations for the improvement of catastrophic preparedness planning doctrine? e. Are TFER lessons learned exportable and useful to other states and regions to further the evolution and strengthening of their own planning programs? f. Has the pilot identified any additional planning / protocol / procedure shortfalls due to integration / synchronization efforts? g. Have any exercise After-Action Reports (AARs) or assessment documents been developed by the site in the course of the TFER Pilot implementation to evaluate the effectiveness of the pilot? h. Please provide recommendations for improvements to increase the effectiveness of the catastrophic planning programs. FEMA also requested copies of plans developed and collected feedback on the administrative aspects of the pilot program. 1. Data collection process. a. Please provide copies of completed / draft plans. b. Please provide an update of the program plan, to include the steps necessary to complete ongoing projects. 2. Administrative elements. a. Describe any challenges faced managing the TFER funds. b. Describe the hiring and retention process for your TFER planners. Include time lines, training, and other elements as appropriate. c. Describe additional administrative successes and challenges in managing the pilot. The Federal Emergency Management Agency (FEMA) launched the Task Force for Emergency Readiness Pilot Program (TFER) to improve the capacity and integration of efforts to plan a response to catastrophic disasters. Under the TFER pilot, FEMA was to grant $350,000 to each of five participating states—Hawaii, Massachusetts, South Carolina, Washington, and West Virginia. FEMA designed TFER to run 18 months, beginning on September 1, 2008, and ending on March 31, 2010, but extended the pilot program by an additional 6 months to allow the states more time to accomplish their goals. The TFER pilot was to emphasize horizontal integration of planning efforts across sectors, jurisdictions, and functional disciplines, as well as vertical integration among state, regional, and federal agencies. For example, regarding vertical integration, states were to integrate planning efforts with the Department of Defense (DOD) through its Defense Coordinating Officers and Emergency Preparedness Liaison Officers. To help develop plans to respond to catastrophic disasters, states could hire up to three full-time planners ideally with civilian and military planning expertise. FEMA’s stated objectives for the TFER pilot were to build relationships, develop plans, and document lessons learned, and states were to focus their planning on the National Planning Scenarios, which represent examples of the gravest dangers facing the United States, including terrorist attacks and natural disasters, and have been accorded the highest priority for federal planning efforts. FEMA articulated these objectives in the TFER Pilot Information Package and the TFER Evaluation Criteria document that was ultimately to be used to evaluate the TFER pilot. Both of these documents refer to FEMA’s Comprehensive Preparedness Guide (CPG) 101 as a source states were to use to guide their planning process. In our assessment of the extent to which each state had addressed FEMA’s stated objectives for the TFER pilot, we reviewed those documents, used the criteria in them that we determined were measurable and appropriate to assess the states’ progress towards meeting each objective, assigned a numerical ranking to each objective’s criteria, and then averaged the rankings for each objective’s criteria. Finally, we assigned designations to these averages. When a state addressed more than 80 percent of the criteria for an objective this was designated “substantial progress”; from 20 percent to 80 percent, “some progress”; and below 20 percent, “little or no progress.” We gathered information related to the lessoned learned objective from state submissions to a report published by the National Guard Bureau in August 2010, and we obtained additional lessons learned during our interviews with state officials. Further, two of the states provided us with the lessons learned they submitted to FEMA for its evaluation, which is not yet complete. We did not assign designations to four criteria pertaining to developing plans and six criteria pertaining to lessons learned because we determined that these criteria do not inform the extent to which states met the objective and/or there was not enough information to quantify the results. Four of the five pilot states are still spending TFER funds, and the states’ status in meeting TFER’s stated objectives may change as catastrophic planning continues. We assessed states’ progress as of March 2011. In figure 9, we summarize Hawaii’s TFER initiatives, note their status, and describe the role of the planners working on them. Hawaii is still spending TFER funds and has two TFER planners. As of March 2011, Hawaii had made some progress in addressing FEMA’s stated pilot program objectives. In figure 10, we depict the funding expenditure status of the Hawaii TFER pilot and our assessment of the progress made in the pilot program towards addressing FEMA’s stated pilot program objectives. Ste offici reported tht TFER plnning integrte with the te' exiting hrricne plnning effort, which relte to Ntionl Plnning Scenrio. Ste offici told us tht TFER plnner re following the teptlined in CPG 101, but hve not completed the fll cycle of plnning tep for it prodct, as Hii has not completed it pl. Hii developed plnning te tht inclded plnner from the te civil defene diviion nd other te gencie, as well as relevnt conty, federl, nd privte ector keholder. Ste plnner told us they hd little contct with DOD entitie. According to te offici, the TFER plnner’s prior militry experience was helpfl, but not necessary for civilin plnner to successl. Hii offici provided recommendtion, inclding: continnd expnd the progrm to other te, mintin flexiility to determine work pl, nd dedicte fnding for castrophic plnning. Addition- lly, te offici recommended tht fre castrophic plnning progrve longer time frme thn the TFER pilot progrm, provide more clerly defined gidnce, nd greter coordintion etween FEMA nd DOD. Frther, fnding was not enogh to provide competitive sarie. Hii has not exercied ny of it pl to determine if they will e effective in the event of astrophe. In figure 11, we detailed figure 10’s assessment of the progress made in the Hawaii TFER pilot program towards addressing FEMA’s stated objectives by (1) listing the criteria from the TFER Pilot Information Package and the TFER Evaluation Criteria document used to evaluate the progress of each objective and (2) providing our assessment score for each criterion. We gathered additional information on Hawaii’s efforts to develop plans and document lessons learned for the criteria we were unable to quantify. For example, we analyzed two Hawaii draft plans, which did not contain all of the CPG 101 components. However, we were unable to fully assess the content of the plans because they are incomplete and work is ongoing. Further, Hawaii officials stated that TFER allowed Hawaii to focus on logistics planning, and without TFER, Hawaii would not be able to undertake this planning. Hawaii reported that TFER has clearly contributed to the development of planning task forces and has provided a significant contribution towards state catastrophic planning. Additionally, Hawaii officials stated that the TFER logistics planning can be used by island territories such as Guam and Puerto Rico with similar logistical challenges. In figure 12, we summarize Massachusetts’s TFER initiatives, note their status, and describe the role of the planners working on them. Massachusetts spent all of its TFER funds and therefore does not have any TFER planners employed. As of March 2011, Massachusetts had made some progress in meeting FEMA’s stated pilot program objectives. In figure 13, we depict the funding expenditure status of the Massachusetts TFER pilot and our assessment of the progress made in the pilot program towards addressing FEMA’s stated pilot program objectives. Massachusett focused on inititive pertining to rricne nd other prioritie identified te emergency mgement offici nd TFER plnner. The bance of it effort did not pecificlly pertin to Ntionl Plnning Scenrio. Ste offici said the plnner followed the teptlined in CPG 101, but cold not complete the fll cycle of plnning ecauste offici decided tht the Massachusett TFER plnner wold coordinte plnning effort but pl wold e completed y other te gencie. Rther thn developing pl, TFER plnner erved as lion to help formte plnning te tht worked to improve the te’sastrophic prepredness. Plnner worked within the te Ntionl Guard. According to te offici, plnning te inclded plnner from the te emergency mgement diviion nd other te gencie, as well as relevnt locl, federl, nd privte ector keholder. Ste offici told us the plnners’ militry plnning experience gve them crediility in militry circle. However, te offici told us tht DOD entitie were minimlly involved. Massachusett offici recommended tht FEMA continnd expnd the progrm to other te, mintin flexiility to determine work pl, nd dedicte fnding for castrophic plnning. Offici ted the TFER plnner ll home level of militry experience, which helped them cope with the logiticl chllenge nd hve crediility with certin audience. Additionlly, offici recommended tht fre castrophic plnning progve longer time frme thn the TFER pilot progrnd provide more clerly defined gidnce. According to te offici, they hve not exercied ny pl. In figure 14, we detailed figure 13’s assessment of the progress made in the Massachusetts TFER pilot program towards addressing FEMA’s stated objectives by (1) listing the criteria from the TFER Pilot Information Package and the TFER Evaluation Criteria document used to evaluate the progress of each objective and (2) providing our assessment score for each criterion. We gathered additional information from Massachusetts on developing plans and documenting lessons learned, but were unable to quantify this information using FEMA’s evaluation criteria. For example, Massachusetts TFER planners told us they often attended state and regional training and workshops, which integrated them with resourcing efforts. However, we did not assess progress towards meeting this objective because it is unclear how these training and workshops inform plan development. Further, Massachusetts officials stated that TFER was successful and proved valuable in bringing state, local, and county agencies together with nongovernmental organizations under the umbrella of state emergency planning. Additionally, TFER officials stated it would have been beneficial to have TFER planners in other New England states available to coordinate with for events that would likely impact the entire region. South Carolina focused its TFER pilot on planning in response to terrorist attacks. In figure 15, we summarize South Carolina’s TFER initiatives, note their status, and describe the role of the planners working on them. South Carolina is still spending TFER funds and currently has two TFER planners employed. As of March 2011, South Carolina had made some progress in addressing FEMA’s stated pilot program objectives of building relationships and documenting lessons learned and had made substantial progress in developing plans. In figure 16, we depict the funding expenditure status of the South Carolina TFER pilot program and our assessment of the progress made in TFER towards addressing FEMA’s stated pilot program objectives. Sth Crolin focused on the Ntionl Plnning Scenrio relted to terrorittck. The te has completed reviion of it Castrophic Incident Annex nd it has completed drft of it Terrorim Prevention nd Repone Plnd three other pl relted to terrorittck, which offici told us fill g in te plnning. Plnner reported they re following the teptlined in CPG 101, but hve not finihed the fll cycle of plnning for ny of it prodct, as Sth Crolinas not exercied them. Sth Crolin provided er of recommendtion, inclding: continnd expnd the progrm, mintin flexiility to determine work pl, nd dedicte fnding for castrophic plnning. Additionlly, Sth Crolin offici recommended tht fre castrophic plnning progrve longer time frme thn the TFER pilot progrm, nd tht FEMA hold provide more clerly defined gidnce. Offici identified coordinting ecrity clence cross DHS nd DOD as chllenge thhold e reolved in imilr progr in the fre. The ring of lesson lerned nd techniq/procedre with other te wold hve een helpfl in developing castrophic pl throgh the TFER pilot progrm. Sth Crolin o cited good initil trining opportnitie throgh TFER; however thee opportnitie did not extend throghot the life of the progrm, nd thus were not ilable to ll Sth Crolin TFER ff. Sth Crolin provided er of recommendtion, inclding: continnd expnd the progrm, mintin flexiility to determine work pl, nd dedicte fnding for castrophic plnning. Addition- lly, Sth Crolin offici recommended tht fre castrophic plnning progrve longer time frme thn the TFER pilot progrm, nd tht FEMA hold provide more clerly defined gidnce. Offici identified coordinting ecrity clence cross DHS nd DOD as chllenge thhold e reolved in imilr progr in the fre. The ring of lesson lerned nd techniq/procedre with other te wold hve een helpfl in developing castrophic pl throgh the TFER pilot progrm. Sth Crolin o cited good initil trining opportnitie throgh TFER; however, thee opportnitie did not extend throghot the life of the progrnd thus were not ilable to ll Sth Crolin TFER ff. Offici ted thn exercie i chedled for Mrch 2011 nd thfter-ction report will subseqently e developed. In figure 17, we detailed figure 16’s assessment of the progress made in South Carolina’s TFER pilot program towards addressing FEMA’s stated objectives by (1) listing the criteria from the TFER Pilot Information Package and the TFER Evaluation Criteria document used to evaluate the progress of each objective and (2) providing our assessment score for each criterion. We gathered additional information from South Carolina on developing plans and documenting lessons learned, but were unable to quantify this information using FEMA’s evaluation criteria. For example, we were unable to fully assess South Carolina’s plans because seven of eight TFER- developed plans are in draft form. However, all four of the plans state officials provided are missing at least one of the recommended sections described in CPG 101. State officials told us that omitted sections are included in other state planning documents. Additionally, state officials reported that the state used TFER to continue work on its existing work plan, and South Carolina has developed plans and responses to some of the threats that they would not have otherwise been able to address. Washington focused its TFER pilot on developing logistics plans and on a catastrophic incident plan. In figure 18, we summarize Washington’s TFER initiatives, note their status, and describe the role of the planners working on them. Washington is spending TFER funds and as of March 2011 had one TFER planner employed. As of March 2011, Washington had made some progress in addressing FEMA’s stated pilot program objectives. In figure 19, we depict the funding expenditure status of the Washington TFER pilot and our assessment of the progress made in the pilot towards addressing FEMA’s stated pilot objectives. Washington elected as it primry Ntionl Plnning Scenrio astrophic erthquake nd ccompnying tsumi. Washington integrted TFER logitic pl nd astrophic incident pln with exiting te pl nd filled g in the te pln compendim, ccording to te offici. The te has finlized two of it even pl nd has finl drft for three other. Washington offici told us they re following CPG 101 plnning process nd re trctring pln content ccordingly. Washington offici ted they developed plnning te tht inclded plnner from the te Emergency Mgement Diviion nd other te gencie, as well as relevnt locl, federl, nd privte ector keholder. Ste offici told us the plnners’ militry plnning experience ccelerted the lerning of civilin plnning protocol nd helped the progrdvnce the te logitic nd castrophic plnning. However, ccording to te offici, DOD entitie were minimlly involved. Washington provided recommendtion, inclding: continnd expnd the progrm to other te, mintin the rocope of the project eligile to ddressed throgh TFER, nd dedicte fnding for castrophic plnning. Additionlly, te offici recommended tht fre castrophic plnning progrve longer time frme thn the TFER pilot progrnd FEMA provide more gidnce. Frther, FEMA cold improve in mgement of grnt like TFER. Washington has exercied three ot of even of it TFER pl. In figure 20, we detailed figure 19’s assessment of the progress made in Washington’s TFER pilot program towards addressing FEMA’s stated objectives by (1) listing the criteria from the TFER Pilot Information Package and the TFER Evaluation Criteria document used to evaluate the progress of each objective and (2) providing our assessment score for each criterion. We gathered additional information from Washington on developing plans and documenting lessons learned, but were unable to quantify this information using FEMA’s evaluation criteria. For example, Washington selected projects for TFER which were previously identified as part of the state’s future emergency planning work. Washington’s TFER planning also synchronized with catastrophic planning efforts undertaken through the Regional Catastrophic Preparedness Grant Program as both efforts included evacuation and logistics components. However, we were unable to assess the degree to which TFER efforts were synchronized with other planning efforts because we did not have a basis for measuring how this synchronization improved the effectiveness of planning efforts. Further, Washington’s TFER plans contain some but not all of the recommended sections described in CPG 101, but state and FEMA officials stated that omitted sections are included in other state planning documents. We could not fully assess the content of Washington’s plans because five out of seven of the plans are in draft form. According to state officials, accomplishments included drafting both catastrophic logistics plans and developing a closer working relationship with Hawaii, Oregon, and California by participating in Hawaii’s annual hurricane exercise. West Virginia focused its TFER pilot on developing response plans for three scenarios: a chemical incident, a mass evacuation of the National Capital Region, and a dam failure. In figure 21, we summarize West Virginia’s TFER initiatives, note their status, and describe the role of the planners working on them. West Virginia is still spending TFER funds and currently has one TFER planner employed. As of March 2011, West Virginia had made some progress in meeting FEMA’s stated pilot objectives. In figure 22, we depict the funding expenditure status of the West Virginia TFER pilot program and our assessment of the progress made in the pilot program towards addressing FEMA’s stated pilot program objectives. Wet Virgini identified three possle cenrio: chemicl incident, ass evuation of the Ntionl Cpitl Region, nd m filre. Thee event cold e caused rricne or terrorittck, which pertin to everl Ntionl Plnning Scenrio. According to te offici, Wet Virgini' TFER plnning effort fill g in castrophic plnning nd inclde development of astrophic incident pln. Ste offici said tht plnner re following the teptlined in CPG 101, but hve not finihed the fll cycle of plnning for ny of it prodct, as Wet Virginias not completed drft pl. Wet Virgini offici said they developed plnning te tht inclded TFER plnner, the Deprtment of Militry Affir nd Public Safety, the Ntionl Guard’s plnning office, other te prtner, nd conty prtner, as well as the privte ector. Ste offici told us the militry-civililend of plnning experience was eneficil in Wet Virgini. However, only one of the two plnneras militry backgrond nd te offici told us coordintion with DOD entitieas miniml. Wet Virgini recommended FEMA continnd expnd the progrm, mintin flexiility to determine workpl nd dedicted fnding to hire plnner. Additionlly, offici recommended tht the new progrm hve longer time frme thn the pilot nd provide more clerly defined gidnce. Wet Virgini reported there was ignificnt difficlty in hiring people with the reqite militry/civilin castrophic plnning kill nd experience ecause of the hort dtion of the progrm, which delyed the rt-p of Wet Virginia’s TFER progrm. Wet Virginias not exercied ny of it pl. Subsntil progress Some progress Little to no progress One of the TFER planners is a National Guardsman who was mobilized in September 2010. He is expected to resume his TFER planning position when he returns. West Virginia hired another TFER planner in December 2011. In figure 23, we detailed figure 22’s assessment of the progress made in West Virginia’s TFER pilot program towards addressing FEMA’s stated objectives by (1) listing the criteria from the TFER Pilot Information Package and the TFER Evaluation Criteria document used to evaluate the progress of each objective and (2) providing our assessment score for each criterion. We gathered additional information on West Virginia’s efforts to develop plans and document lessons learned for the criteria we were unable to quantify. For example, we analyzed West Virginia’s Catastrophic Incident Annex, which did not contain all of the CPG 101 components. West Virginia reported that the development of this draft annex fulfilled one of the main objectives of the pilot program. However, we were unable to fully assess the content of the annex because it is incomplete and work is ongoing. Additionally, West Virginia TFER planners determined during their assessments that there were insufficient personnel available at the county level to accomplish the tasks necessary during an incident response. In some cases, key individuals were identified as having multiple roles and/or responsibilities, which would be impossible to fulfill during an actual catastrophic incident. Further, West Virginia reported the “bottom up” approach of the West Virginia TFER pilot program is highly effective in communicating and working with local jurisdictions, but may prove to be less useful in an urban setting. According to state officials, West Virginia’s TFER efforts are synchronized with the Regional Catastrophic Preparedness Grant Program to ensure efforts are not duplicative, particularly for the mass evacuation scenario. In addition to the contacts named above, the following individuals made significant contributions to this report: Marc Schwartz and Leyla Kazaz (Assistant Directors), Courtney Reid, Susanna Kuebler, Paul Hobart, Jena Whitley, Katrina Moss and Cody Loew; Mae Jones, Lara Miklovek, Amie Steele, Cynthia Saunders, Kate Lenane, and Tracey King. Disaster Response: Criteria for Developing and Validating Effective Response Plans. GAO-10-969T. Washington, D.C.: September 22, 2010. Homeland Defense: DOD Can Enhance Efforts to Identify Capabilities to Support Civil Authorities During Disasters. GAO-10-386. Washington, D.C.: March 30, 2010. Homeland Defense: DOD Needs to Take Actions to Enhance Interagency Coordination for Its Homeland Defense and Civil Support Missions. GAO-10-364. Washington, D.C.: March. 30, 2010. Interagency Collaboration: Key Issues for Congressional Oversight of National Security Strategies, Organizations, Workforce, and Information Sharing. GAO-09-904SP. Washington, D.C.: September 25, 2009. Homeland Defense: U.S. Northern Command Has a Strong Exercise Program, but Involvement of Interagency Partners and States Can Be Improved. GAO-09-849. Washington, D.C.: September 9, 2009. National Preparedness: FEMA Has Made Progress, but Needs to Complete and Integrate Planning, Exercise, and Assessment Efforts. GAO-09-369.Washington, D.C.: April 30, 2009. Emergency Management: Observations on DHS’s Preparedness for Catastrophic Disasters. GAO-08-868T. Washington, D.C.: June 11, 2008. National Response Framework: FEMA Needs Policies and Procedures to Better Integrate Non-Federal Stakeholders in the Revision Process. GAO-08-768. Washington, D.C.: June 11, 2008.

Summary: Hurricane Katrina in 2005 highlighted gaps in the nation's preparedness to respond effectively to catastrophic incidents. The Department of Homeland Security's (DHS) Federal Emergency Management Agency (FEMA) designed the Task Force for Emergency Readiness (TFER) pilot program to advance and integrate state and federal catastrophic planning efforts. TFER, first envisioned by the Department of Defense (DOD), ran from September 2008 to September 2010, and FEMA provided the five participating states--Hawaii, Massachusetts, South Carolina, Washington, and West Virginia--with $350,000 each to develop plans, build relationships with stakeholders, and document lessons learned (i.e., TFER's stated objectives). As requested, GAO evaluated the extent to which (1) FEMA followed sound management practices in designing, administering, and evaluating TFER and (2) the five participating states satisfied TFER's stated objectives. GAO analyzed FEMA guidance, such as the TFER Pilot Information Package, conducted site visits to all five participant states, and met with relevant FEMA and DOD officials, to evaluate FEMA's management of TFER and the states' implementation of it. FEMA developed program objectives and procedures for administering the TFER pilot, but did not develop other elements of sound management practices in designing, administering, and evaluating pilot programs that GAO identified from its prior work and social science literature. FEMA developed objectives for the pilot, but did not document standards for determining the pilot's success. FEMA also provided resources such as funding, training, and support, but FEMA did not always follow the procedures it established for TFER. For example, FEMA did not consistently conduct biweekly conference calls with the states, and four states reported that it would have been helpful if FEMA provided more guidance. FEMA did not develop a data analysis plan, which could have better ensured FEMA collected data on the extent to which the pilot states coordinated with key stakeholders and provided reasonable assurance that FEMA conducted a systematic assessment of TFER using comparable data across the five pilot states. In Spring 2010, FEMA announced TFER would not continue past the pilot stage before evaluating TFER on its merits in strengthening and advancing state catastrophic planning, but FEMA allowed the states to draw down the remaining TFER funds and continue to complete initiatives started under TFER through August 2011. GAO previously reported in April 2009 that FEMA faced challenges in assessing pilot program data, and FEMA officials reported the agency does not have pilot program policy guidance. In the absence of this, FEMA lacks a systematic approach to developing, administering, and evaluating pilot programs. FEMA could better ensure other pilot programs meet their intended goals by developing policies and guidance that include sound management practices. All five states have taken steps to follow FEMA's guidance to address TFER's objectives, but no state has fully addressed them all. First, two of the five states have completed draft catastrophic plans, and all five states reported following FEMA's planning process. Second, all five states built relationships with stakeholders such as state agencies and FEMA, but state officials said coordination with DOD--a key federal stakeholder who may be called upon to assist in disaster response--was limited. State officials reported not coordinating with DOD because they did not have draft plans for DOD officials to review. Third, all states have documented lessons learned to date, but four states have not exercised TFER plans to determine their effectiveness in the event of an emergency. GAO recommends that FEMA develop policies and guidance that follow sound management practices for future pilot programs, and share TFER results with stakeholders. FEMA agreed with GAO's recommendations.