The mission of FAA, as a DOT agency, is to provide the safest, most efficient aerospace system in the world. To fulfill its mission, FAA must rely on an extensive use of technology, including many software-intensive systems. FAA constantly relies on the adequacy and reliability of the nation’s ATC system, which comprises a vast network of radars; automated data processing, navigation, and communications equipment; and ATC facilities. Through this system, FAA provides services such as controlling takeoffs and landings and managing the flow of traffic between airports. FAA is organized into several staff support offices and five lines of business, which include Airports, Aviation Safety, Commercial Space Transportation, the Office of Security and Hazardous Materials, and the newly formed ATO. The ATO was formed in February 2004 to, among other things, improve the provision of air traffic services and accelerate modernization efforts. To create the ATO, FAA combined its Research and Acquisition and Air Traffic Services into one performance-based organization, bringing together those who acquire systems and those who use them, respectively. The ATO is led by FAA’s chief operating officer, consists of 10 service units, and has 36,000 of FAA’s 48,000 employees. The ATO is the principal FAA organizational unit responsible for acquiring ATC systems through the use of the agency’s Acquisition Management System (AMS). Because FAA formerly contended that some of its modernization problems were caused by federal acquisition regulations, Congress enacted legislation in November 1995 that exempted the agency from most federal procurement laws and regulations and directed FAA to develop and implement a new acquisition management system that would address the unique needs of the agency. In April 1996, FAA implemented AMS. AMS was intended to reduce the time and cost of fielding new system acquisitions by introducing (1) a new investment system that spans the life cycle of an acquisition, (2) a new procurement system that provides flexibility in selecting and managing contractors, and (3) organizational and human capital reforms that support the new acquisition system. AMS provides high-level acquisition policy and guidance for selecting and controlling ATC system acquisitions through all phases of the acquisition life cycle, which is organized into a series of phases and decision points that include (1) mission analysis, (2) investment analysis, (3) solution implementation, and (4) in-service management. To select system acquisitions, FAA has two processes--mission analysis and investment analysis–that together constitute a set of policies and procedures, as well as guidance, that enhance the agency’s ability to screen system acquisitions submitted for funding. Also through these two processes, FAA assesses and ranks each system acquisition according to its relative costs, benefits, risks, and contribution to FAA’s mission; a senior, corporate-level decision- making group then selects system acquisitions for funding. After a system acquisition has been selected, FAA officials are required to formally establish the life-cycle cost, schedule, benefits, and performance targets— known as acquisition program baselines, which are used to monitor the status of the system acquisition throughout the remaining phases of its life cycle. Through its NAS modernization program, FAA is upgrading and replacing ATC facilities and equipment to help improve the system’s safety, efficiency, and capacity. These systems involve improvement in the areas of automation, communication, navigation and landing, surveillance, and weather to support the following five phases of flight (see fig. 1): Preflight – The pilot performs flight checks and the aircraft is pushed- back from the gate. For preflight, we looked at Collaborative Decision Making (CDM) and OASIS. Airport Surface – The aircraft taxis to the runway for takeoff or, after landing, to the destination gate to park at the terminal. For airport surface, we examined the Airport Surface Detection Equipment – Model X (ASDE-X). Terminal Departure – The aircraft lifts off the ground and climbs to a cruising altitude. For terminal departure, we examined the following systems: Airport Surveillance Radar (ASR-11), Integrated Terminal Weather System (ITWS), Local Area Augmentation System (LAAS), Standard Terminal Automation Replacement System (STARS), and Traffic Management Advisor (TMA). En route/Oceanic -- The aircraft travels through one or more center airspaces and approaches the destination airport. For en route and oceanic, we examined the following systems: Air Traffic Control Radar Beacon Interrogator-Replacement (ATCBI-6), Advanced Technologies and Oceanic Procedures (ATOP), Controller-Pilot Data Link Communications (CPDLC), and User Request Evaluation Tool (URET). Terminal Arrival -- The pilot lowers, maneuvers, aligns, and lands the aircraft on the destination airport’s designated landing runway. For terminal arrival, we looked at the systems already listed under terminal departure: ASR-11, ITWS, LAAS, STARS, and TMA. In addition, for the major ATC systems that support multiple phases of flight, we examined the following systems: En Route Communications Gateway (ECG), En Route Automation Modernization (ERAM), Next- Generation Air-to-Ground Communication (NEXCOM), and Wide Area Augmentation System (WAAS). Furthermore, for major ATC systems that support NAS infrastructure, we examined FAA Telecommunications Infrastructure (FTI) and NAS Infrastructure Management System (NIMS)– Phase Two. (See app. I for additional information on these 16 systems.) For more than two decades, FAA has experienced cost growth, schedule extensions, and/or performance problems in acquiring major systems under its ATC modernization program and has been on our list of high-risk programs since 1995. For example, 13 of the 16 major system acquisitions we reviewed in detail continue to experience cost, schedule, and/or performance shortfalls when assessed against their original baselines. The three other major system acquisitions that we reviewed in detail are currently operating within their original cost, schedule, and performance targets, but are experiencing challenges symptomatic of past problems. Of the remaining 39 system acquisitions within the ATC modernization program, few have had problems meeting cost and schedule targets. However, the ATO made progress during its first year of operation by meeting its acquisition goal for fiscal year 2004. Thirteen of the 16 major system acquisitions that we reviewed in detail for this engagement under the ATC modernization program have continued to experience cost growth, schedule delays, and/or performance problems when assessed against their original performance targets (see table 1). These major system acquisitions had total cost growth ranging from $1.1 million to about $1.5 billion over their original cost targets. In addition, these systems required extensions in their initial deployment schedules ranging from 1 to 13 years. Furthermore, several systems experienced safety-related performance problems. For 12 of the 13 major system acquisitions we reviewed in detail with cost, schedule, and performance shortfalls, one or more of the following four key factors contributed to these shortfalls: (1) The funding level received was less than called for in agency planning documents. Most major ATC system acquisitions have cost, schedule, and performance baselines that are approved by FAA’s Joint Resources Council--the agency’s body responsible for approving and overseeing major system acquisitions. Each baseline includes annual funding levels that the council agrees are needed for a system acquisition to meet its cost, schedule, and/or performance targets. The estimated cost for a given year assumes that the program received all funding for prior fiscal years as described in the baseline. In practice, however, this is not always the case. For example, when FAA’s budget level does not allow all system acquisitions to be fully funded at the levels approved in their baselines, FAA may elect to fully fund higher-priority acquisitions and provide less funding for lower-priority acquisitions than called for in their baselines. When a system acquisition does not receive the annual funding levels called for in its baseline, its ability to meet cost, schedule, and/or performance targets can be jeopardized, for example, by requiring the agency to defer funding for essential development or deployment activities until sufficient funding becomes available, which, in turn, could require FAA to maintain costly legacy systems until a new system is deployed. Receiving less funding than the agency approved for a given acquisition was a factor contributing to the inability of 8 of the 16 major system acquisitions we reviewed in detail to meet their cost, schedule, and/or performance targets. The ASR-11 acquisition, a digital radar system, illustrates how reduced funding has resulted in schedule delays. FAA officials stated that because of funding reductions and reprogramming, the program received $46.45 million less than requested for fiscal years 2004 and 2005 and program officials plan to request that the program’s deployment schedule be extended to 2013. According to FAA officials, in general, schedules for system acquisitions may slip under such circumstances (e.g., the rate of software development may be reduced and planned hardware and software deployments may be delayed). The ATO’s chief operating officer testified in April 2005 that receiving multiyear rather than annual funding from Congress for system acquisitions would help FAA to address this problem by providing funding stability for system acquisitions. In addition, according to a senior DOT official, 50 percent of cost growth is a result of an unstable funding stream. (2) The system acquisition experienced requirements growth and/or unplanned work. Requirements that are inadequate or poorly defined prior to developing a system may contribute to the inability of system acquisitions to meet their original cost, schedule, and/or performance targets. In addition, unplanned development work can occur when the agency misjudges the extent to which commercial-off-the-shelf (COTS)/ nondevelopmental item (NDI) solutions, such as those procured by another agency, will meet FAA’s needs. Requirements growth and/or unplanned work contributed to the inability of 7 of the 16 major system acquisitions we reviewed in detail to meet their cost, schedule, and/or performance targets. (3) Stakeholders were not sufficiently involved in design and development: Insufficient involvement of relevant stakeholders, such as air traffic controllers and maintenance technicians, throughout the development and approval processes for a system acquisition can lead to costly changes in requirements and unplanned work late in the development process. Not involving stakeholders sufficiently contributed to the inability of 4 of the 16 major system acquisitions to meet their cost, schedule, and/or performance targets. (4) The complexity of software development was underestimated. Underestimating the complexity of developing software for system acquisitions or the difficulty of modifying available software to fulfill FAA’s mission needs may contribute to unexpected software development, higher costs, and schedule delays. Underestimation contributed to the inability of 3 of the 16 major system acquisitions we reviewed in detail to meet their cost, schedule, and/or performance targets. (See table 2.) Several of the 16 major systems acquisitions we reviewed in detail effectively illustrate how these four factors can interact to contribute to cost growth, schedule extensions, and performance problems. For example, for WAAS, a precision approach and landing system augmented by satellites, two of the four key factors came into play: underestimation of software complexity and insufficient stakeholder involvement. Specifically, FAA underestimated the complexity of the software that would be needed to support this system when it accelerated the implementation of performance targets, which included moving up the commissioning of WAAS by 3 years. FAA originally planned to commission WAAS by 2000; however, at the urging of government and aviation industry groups in the 1990s, it decided to change the commissioning date to 1997. FAA then tried to develop, test, and deploy WAAS within 28 months, although the software development alone was expected to take 24 to 28 months. In retrospect, FAA acknowledged that the agency’s in-house technical expertise was not sufficient to address WAAS’s technical challenges and that expert stakeholders should have been involved earlier. Although WAAS was being developed by an integrated product team that included representatives from several FAA offices, the team did not effectively resolve problems in meeting a required performance capability—that pilots be warned in a timely manner when a system may be giving them potentially misleading and therefore hazardous information. Consequently, in 2000, FAA convened a panel of expert stakeholders to help it meet this requirement. These actions resulted in unplanned work and contributed to the rise in WAAS’s cost from the original estimate of $509 million in 1994 to $2.036 billion in 2005, and to a 6-year extension in its commissioning date. According to FAA, adding 6 years to the program’s life cycle also contributed to increased costs. Another example involves STARS, a joint program of FAA and DOD that replaced outdated monochromatic controller workstation monitors with multicolor monitors in ATC facilities. While joint FAA and DOD acquisitions offer the opportunity to leverage federal resources, in the case of STARS, the interaction of insufficient stakeholder involvement and subsequent unplanned work contributed to cost growth and schedule extensions. Specifically, FAA and DOD decided to acquire COTS equipment, rather than developing a new system. This strategy envisioned immediately deploying STARS to the highest priority ATC facilities and making further improvements later, thereby avoiding the increasing cost of maintaining the legacy system. However, this strategy provided for only limited evaluation by FAA and DOD controllers and maintenance technicians during the system’s development phase, although these employees were identified as stakeholders in developing the system’s requirements. While DOD controllers adopted and began using the original COTS version of STARS, FAA elected to modify the acquisition strategy and suspended the STARS deployment to address FAA controller and technician concerns with the new system. These concerns included, for example, that many features of the old equipment could be operated with knobs, allowing controllers to focus on the screen. By contrast, STARS was menu-driven and required the controllers to make several keystrokes and use a trackball, diverting their attention from the screen. The maintenance technicians also identified differences between STARS and its backup system that made it difficult to monitor the system. For example, the visual warning alarms and the color codes identifying problems were not the same for the two systems. According to FAA, the original COTS acquisition strategy that limited the involvement of controllers and maintenance technicians to just prior to deployment caused unplanned work for the agency because it had to revise its strategy for acquiring and approving STARS; this contributed to an increase in the overall cost of STARS of $500 million and a schedule extension of 5 years to deploy the system to its first site. The interaction of these factors also contributed to the agency’s ability to deploy STARS at only 47 of the 172 facilities initially planned. As of February 2005, FAA was developing a long-term acquisition plan to modernize or upgrade the highest-priority Terminal Radar Approach Control facilities that direct aircraft in the airspace that extends from the point where the tower’s control ends to about 50 nautical miles from the airport. The plan consists of alternatives to STARS, including the existing Common Automated Radar Terminal System (CARTS), which STARS was designed to replace. Finally, to help avoid similar problems in the future, stemming from the insufficient involvement of stakeholders during critical phases of a system’s design, development, and implementation, FAA has been more proactive in involving the stakeholders that will operate and maintain system acquisitions. A final example of how these factors can interact is FAA’s acquisition of OASIS, which is designed to replace outdated technology in FAA’s automated flight service stations. The new system is intended to improve the ability of air traffic specialists to process flight plans, deliver weather information, and provide search and rescue services to general aviation pilots. In August 1997, FAA awarded a contract to replace the Flight Service Automation System and console workstations. However, unplanned work, insufficient involvement of stakeholders, and lower funding than the agency had determined was needed to meet cost, schedule, and performance targets have together contributed to cost growth and schedules extensions. For example, the agency saw the system acquisition schedule slip because of a larger-than-planned development effort. According to the DOT IG, FAA identified a number of significant concerns, including the inadequate weather graphics capabilities for air traffic specialists. In our view, this indicates that stakeholders were not sufficiently involved throughout the system’s design and development phases. As a result, FAA eliminated the option of COTS procurement. In addition, the OASIS program was rebaselined in March 2000, when the system acquisition received only $10 million of the $21.5 million called for in its baseline for that year. This reduction in funding reduced the rate of software development, delayed and reduced the rate of planned hardware and console deployments, and led to the incremental deployment of operational software. This contributed to a delay in the first-site implementation from July 1998 to July 2002. According to FAA officials, because OASIS received less funding than the agency had approved for fiscal year 2004 and 2005, its deployment to automated flight service stations was postponed. As of February 2005, FAA had deployed 19 OASIS units: 16 at automated flight service stations and 3 at other sites. Software upgrades that are under way will be completed by June 2005. FAA plans neither installations nor software upgrades beyond those at the automated sites, because the agency awarded a contract to a private vendor in February 2005 to operate flight service stations. Until then, FAA has directed the program to remain within its current Capital Investment Plan funding levels for fiscal years 2004 through 2006. According to FAA, since it completed its evaluation of OASIS in February 2005, planning for the program’s implementation and baseline remain unchanged. FAA plans to phase out OASIS between March 2006 and March 2007 in accordance with the new service provider’s transition plan. Three of the 16 major ATC system acquisitions we reviewed in detail are currently operating within their original cost, schedule, and performance targets; however they have experienced challenges, including symptoms of one or more of the four factors cited earlier, such as requirements growth. These system acquisitions include (1) ECG, a communications system gateway that serves as the point of entry and exit for data used by FAA personnel to provide air traffic control at 20 en route facilities; (2) ERAM, a replacement for the primary computer system used to control air traffic; and (3) ATOP, an integrated system for processing flight data for oceanic flights. While ECG has not exceeded its original cost, schedule, and performance targets, it encountered requirements growth when FAA added a new capability to address a security weakness. According to FAA officials, correcting this weakness cost about $25,000, and an additional $480,000 will likely be needed to improve the monitoring capability for this system’s operation. However, these cost increases will not exceed the system’s cost or schedule targets. ERAM and ATOP also have areas that warrant attention. For example, ERAM is a high-risk effort because of its size and the amount of software that needs to be developed—over 1 million lines of code are expected to be written for this effort. In addition, the DOT IG reports that, to date, ERAM has experienced software growth of about 70,000 lines of code. While the DOT IG considers this amount of software growth to be modest, given FAA’s long-standing difficulties with developing this volume of software for system acquisitions while remaining within cost, schedule, and/or performance targets, sustained management attention is warranted. For ATOP, when FAA tried to accelerate the initial deployment of this system by 14 months, it was unable to do so, because of poorly defined requirements, unrealistic schedule estimates, and inadequate evaluation by the contractor. In addition, according to contract provisions, FAA assumed responsibility in February 2005 for the cost of resolving any additional software problems it identifies. Overall, although these system acquisitions are currently operating within their cost, schedule, and performance targets, the challenges they have experienced thus far indicate that they will require the sustained attention of FAA’s senior managers to help ensure that they stay on track. For the 39 system acquisitions that make up the balance of FAA’s ATC modernization program, only 9 are considered “major” or directly comparable to the 16 major ATC system acquisitions we reviewed in detail.(See table 3.) Of these 9 major systems, 2 have required changes in their cost targets. For example, for an automated weather observation system, the Aviation Surface Weather Observation Network,the cost has increased by 15 percent because of system capacity issues, among other things. For another system that will be used on an interim basis for managing air traffic until the new primary computer system is available, the Host and Oceanic Computer System Replacement, the cost has decreased by 13 percent because the agency determined that parts of the existing system could be sustained through fiscal year 2008, which is within the scope of the program.The remaining 30 systems are not directly comparable, because they do not involve acquiring a new system. Instead, they are what FAA terms “buy-it-by-the pound” purchases—systems that are commercially available and ready for FAA to use without modification, such as a landing system purchased to replace one that has reached the end of its useful life. (See app. II for additional information on these 39 systems.) To its credit, FAA has reported that it met its annual acquisition performance goal for fiscal year 2004--to meet 80 percent of designated milestones and maintain 80 percent of critical program costs within 10 percent of the budget as published in its Capital Investment Plan. Specifically, it set annual performance cost goals and schedule milestones for 41 of the 55 system acquisitions under the ATC modernization program. For these 41 system acquisitions, FAA set 51 schedule milestones and met 46 of them—with “meeting the goal” defined as achieving 80 percent of its designated program milestones. It also set and met its annual cost performance goals for each of these 41 system acquisitions. In our opinion, having and meeting such performance goals is commendable, but it is important to note that these goals are updated program milestones and cost targets, not those set at the program’s inception. Consequently, they do not provide a consistent benchmark for assessing progress over time. Moreover, as indicators of annual progress, they cannot be used in isolation to measure progress in meeting cost and schedule targets over the life of an acquisition. Finally, given the problems FAA has had in acquiring major ATC systems for over two decades, it is too soon to tell whether meeting these annual performance goals will ultimately improve the agency’s ability to deliver system acquisitions as promised. FAA has taken a number of positive steps, primarily through the ATO, to address key legacy challenges in acquiring major systems under its ATC modernization program; however, we have identified additional steps that are warranted to reduce risk and strengthen oversight. Some of the steps FAA has taken directly address the four factors we identified as contributing to cost, schedule, and/or performance problems, while others support more general efforts to improve the modernization program’s management. The steps taken and additional steps needed are discussed below by key areas. To address the concern that some system acquisitions have had difficulty meeting performance targets because they have not received annual funding at the levels called for in key planning documents, the ATO has taken several steps. For example, the ATO has demonstrated a willingness to cut major programs that were not meeting their performance targets even after a significant investment of agency resources. The ATO is currently reviewing all of its capital projects to reassess priorities. Both of these actions should help improve the chances that sufficient funding will be available for priority system acquisitions to conduct the annual activities necessary to keep them on track to meet cost, schedule, and performance targets. Specifically, for fiscal year 2005, the appropriation for FAA’s facilities and equipment budget, which funds the ATC modernization program, was $393 million less than the agency had planned to spend. FAA absorbed the $393 million reduction largely by cutting funding for three of the major system acquisitions we reviewed in detail: a digital e-mail-type capability between controllers and pilots was suspended (CPDLC); the next generation air-to- ground communication system had the funding cut for a major component (NEXCOM); and a precision-landing system augmented by satellites for use primarily by commercial airlines (LAAS) was returned to research and development to focus the remaining funding for the system on resolving a key performance shortfall. FAA also plans to defer funding for CPDLC and LAAS for fiscal year 2006. FAA decisions to cut or eliminate funding for system acquisitions in its current ATC modernization system may prove to be positive in the long run. For example, although FAA and National Air Traffic Controllers Association officials say that the cuts the agency made to 3 of its 16 major ATC system acquisitions will delay system benefits until the acquisitions are fully developed and deployed, the cuts demonstrate FAA’s willingness to suspend major ATC system acquisitions, despite large resource investments. In addition, by delaying a system acquisition, FAA may later be able to save time and money by leveraging the experiences that others have had with developing and deploying systems that provide similar capabilities (e.g., the controller-pilot e-mail-type capability for which FAA cut funding is now in use in both Canada and Europe). Furthermore, as FAA continues to reassesses its funding priorities, it could explore cost- saving options including taking steps to systematically (1) evaluate the costs and benefits of continuing to fund system acquisitions across the ATC modernization program at current and planned levels to identify potential areas for savings and (2) identify potentially lower-cost alternatives to current system acquisitions, such as lower-cost controller workstations. FAA has also taken a number of steps to address two other factors—reduce the risk of requirements growth and/or the need to undertake unplanned work—and to improve its ability to better assess and manage the risks associated with acquiring major ATC systems that require complex software development. However, additional steps are needed in these areas. Processes for acquiring software and systems: FAA has made progress in improving its process for acquiring software-intensive systems-- including establishing a framework for improving its system management processes, and performing many of the desired practices for selected FAA projects. The quality of these systems and software, which are essential to FAA’s ATC modernization program, depends on the value and maturity of the processes used to acquire, develop, manage, and maintain them. In response to our previous recommendations, FAA developed an FAA-integrated capability maturity model (iCMM). Since FAA implemented the model, a growing number of system acquisitions have adopted the model, and its use has paid off in enhanced productivity, higher quality, greater ability to predict schedules and resources, better morale, and improved communication and teamwork. However, ATO did not mandate the use of the process improvement model for all software-intensive acquisition projects. In response to our recommendation, the ATO informed us of its plans to establish, by June 30, 2005, an overall policy defining the ATO’s expectations for process improvement, and by September 30, 2005, a process improvement plan to address and coordinate improvement activities throughout the organization. Management of information technology investments: In 2004, we reported that FAA has made considerable progress in managing its information technology investments.However, we also found that FAA’s lack of regular review of investments that are more than 2 years into their operations is a weakness in the agency’s ability to oversee more than $1 billion of its information technology investments as a total package of competing investment options and pursue those that best meet the agency’s goals. FAA recently informed us that it has taken a number of steps aimed at achieving a higher maturity level, including establishing service-level mission need statements and service-level reviews, which address operational systems to ensure that they are achieving the expected level of performance. While these steps could resolve some of the deficiencies that we previously reported, we have not yet performed our own evaluation of these steps. FAA could potentially realize considerable savings or performance improvements if these reviews result in the discontinuation of some investments, since operating systems beyond their second year of service accounted for 37 percent of FAA’s total investment in information technology in fiscal year 2004. Enterprise architecture: FAA has established a project office to develop a NAS enterprise architecture—a blueprint for modernization—and designated a chief architect, and has committed resources to this effort, and issued its latest version of its architecture. However, FAA has not yet taken key steps to improve its architecture development, such as designating a committee or group representing the enterprise to direct, oversee, or approve the architecture; establishing a policy for developing, maintaining, and implementing the architecture; or fully developing architecture products that meet contemporary guidance and describe both the “As Is” and “To Be” environments and developing a sequencing plan for transitioning between the two. To help address concerns that stakeholders have not been sufficiently involved throughout the development of major systems acquisitions, FAA has taken a number of steps. For example, when the ATO was created, it brought together the FAA entities that develop systems and those who will ultimately use them. Specifically, it reorganized FAA’s air traffic services and research and acquisition organizations along functional lines of business to bring stakeholders together and integrate goals. The ATO is also continuing with a phased approach to system acquisitions that it began using under Free Flight Phase 1, through which it has begun to involve stakeholders more actively throughout a system acquisition’s development and deployment. However, as we reported in November 2004, FAA needs to take additional steps to ensure the continued and active involvement of stakeholders in certifying new ATC system acquisitions. In addition, the union that represents the specialists who install, maintain, troubleshoot, and certify NAS systems, recently testified that over the past 2 years, FAA has systematically eliminated the participation of these specialists in all but a few modernization programs. Given the importance of stakeholder involvement in the development and deployment of new ATC systems, their continued involvement in ATC modernization efforts will be important to help avoid the types of problems that led to cost growth and delays for STARS. Reassessment of capital investment to decrease operating costs: Both the FAA Administrator and the ATO’s chief operating officer have committed to basing future funding decisions for system acquisitions on their contribution to reducing the agency’s operating costs while maintaining safety. This is consistent with our 2004 recommendation that FAA consider its total portfolio of investments as a package of competing options. Currently, only 1 of the 55 system acquisitions in FAA’s ATC modernization program—FAA Telecommunications Infrastructure—helps to reduce the agency’s operating costs. Most of FAA’s major system acquisitions are aimed at increasing the capacity of the NAS and delivering benefits to system users. The ATO is in the process of reviewing all of its capital investments, including system acquisitions under the ATC modernization program, to identify areas of cost savings and to focus limited funding on investments that will reduce operating costs. However, because FAA has only recently begun to incorporate this type of analysis of the costs and operational efficiency of system acquisitions into the decision-making and management processes, it is too early to assess the results. Acquisition Management System: The ATO has taken a number of steps to improve its Acquisition Management System (AMS). For example, it has revised AMS to require that acquisition planning documents be prepared in a format consistent with that prescribed by OMB for use in justifying all major capital investments. In addition, the ATO revised AMS in December 2004, in part to respond to recommendations we made about needed changes in its investment management practices for information technology. However, we have not yet independently assessed the sufficiency of these changes. Moreover, additional changes to AMS are warranted. For example, while AMS provides some discipline for acquiring major ATC systems, it does not use a knowledge-based approach to acquisitions, characteristic of best commercial and DOD practices. A knowledge-based approach includes using established criteria to attain specific knowledge at three critical junctures in the acquisition cycle, which we call knowledge points, and requiring oversight at the corporate executive level for each of these knowledge points. Experience has shown that not attaining the level of knowledge called for at each knowledge point increases the risk of cost growth and schedule delays. We recommended, among other things, that FAA take several actions to more closely align its acquisition management system with commercial best practices. FAA said that our recommendations would be helpful to them as they continue to refine this system. Cost accounting and cost estimating practices: FAA has improved its financial management by moving forward with the development of a cost accounting system, which it plans to fully deploy by 2006. Ultimately, FAA plans to use this cost information routinely in its decision-making. When implemented, this cost accounting system will address a long-standing GAO concern that FAA has not had the needed cost accounting practices in place to effectively manage software-intensive investments, which characterize many of agency’s major ATC system acquisitions. This type of information can be used to improve future estimates of cost for these acquisitions. Organizational culture: FAA has also sought to establish an organizational culture that supports sound acquisitions. We have ongoing work to assess FAA’s efforts concerning cultural change. ATO business practices: To improve its investment management decision- making and oversight of major ATC acquisitions, the ATO has informed us that it has initiated the following steps, which we have reported are important to effective oversight: integrated AMS and OMB’s Capital Planning and Investment Control Process to develop a process for analyzing, tracking, and evaluating the risks and results of all major capital investments made by FAA; conducted Executive Council reviews of project breaches of 5 percent in cost, schedule, and/or performance to better manage cost growth; issued monthly variance reports to upper management to keep them apprised of cost and schedule trends; and increased the use of cost monitoring or earned value management systems to improve oversight of programs. However, much work remains before the ATO will have key business practices in place. Specifically, according to the ATO’s chief operating officer, it will be at least 2 years before the ATO has completed the basic management processes needed to use the new financial management systems it has been putting in place. Despite progress to date, until the agency addresses the residual issues cited above, it will continue to risk the project management problems affecting cost, schedule, and/or performance that have hampered its ability to acquire systems for improving air traffic control. The ATO will be further challenged to modernize the ATC system in the current constrained budget environment and remain within the administration’s future budget targets, which are lower than those of recent years. Specifically, for fiscal year 2005, FAA requested $393 million less than it had planned to spend for activities under the facilities and equipment budget account, which funds the ATC modernization program and related modernization activities. In addition, the President’s fiscal year 2006 budget submission calls for an additional cut to this budget account of $77 million from FAA’s planned level, which would bring the fiscal year 2006 funding level to about $470 million below the fiscal year 2004 appropriation. Moreover, FAA officials told us that funding for the facilities and equipment account is likely to hold near fiscal year 2004 levels, or at about $2.5 billion annually, for the next 5 years. In total, FAA plans to spend $4.4 billion during fiscal years 2005 through 2009 on key modernization efforts, despite FAA receiving about $2 billion less than it had planned in appropriations over this 5-year period for its facilities and equipment budget, which funds the ATC modernization program and related modernization activities. To fund its major system acquisitions while remaining within the administration’s budget targets, the ATO has eliminated planned funding to start new projects and substantially reduced planned funding for other areas. These funding decisions are reflected in FAA’s updated Capital Investment Plan. This plan shows substantially reduced funding for two major system acquisitions in fiscal year 2005—CPDLC and LAAS--and defers funding for them in fiscal year 2006. For the remaining 14 major ATC system acquisitions we reviewed in detail, FAA plans to increase funding by $533 million between fiscal year 2005 and fiscal year 2009. In contrast, for the remaining 39 system acquisitions, FAA has reduced funding by $420 million for this period. The planned increases in funding for these 14 major system acquisitions also come at the expense of other modernization activities outside the ATC modernization program, such as capital expenditures to replace aging ATC facilities that will house the system acquisitions. For example, FAA reports that it needs $2.5 billion (2005 dollars) annually to renew its aging physical infrastructure—assuming a $30 billion value of its assets and a 7- to 12-year useful life. According to the ATO, much of its physical infrastructure, including the buildings and towers that house costly ATC systems, is over 30 years old and needs to be refurbished or replaced. However, FAA plans to reduce funding for facilities by nearly $790 million between fiscal year 2005 and fiscal year 2009—a plan that runs counter to its reported need to refurbish or replace its physical infrastructure. Furthermore, FAA also plans to cut $1.4 billion from its spending plans for fiscal years 2005 through 2009 for, among other things, new system acquisitions in the ATC modernization pipeline that do not yet have agency-approved cost, schedule, and performance targets or baselines (e.g., a new technology that would allow pilots to “see” the location of other aircraft on cockpit display). Our work has shown that FAA has taken some important steps to prioritize the 55 system acquisitions under its ATC modernization program. These revised priorities are reflected in its most recent plans, which detail the areas where FAA plans to make cuts within its facilities and equipment budget to live within its expected means during fiscal years 2005 through 2009. However, our work has also shown that these plans do not provide detailed information about the trade-offs that are underlie decisions to fully fund some systems and to defer, reduce, or eliminate funding for others and how these cuts will affect FAA’s modernization efforts, including what impact they will have on interdependent system acquisitions. To convey information to decision-makers on the impact of reduced funding on modernization, the ATO should detail its rationale and explicitly identify the trade-offs it is making to reach the administration’s budget targets, highlighting those programs slated for increased funding and those slated for reduced funding. Key information includes delayed benefits, the impact of cutting one ATC system acquisition on related or interdependent systems, and increased costs for maintaining legacy systems until new systems are deployed. Overall, the ATO needs to explicitly identify the implications of deferring, reducing, or cutting funding for a particular system or activity on the agency’s ability to modernize both the ATC system and related components of the NAS in the near, mid, and longer term. While funding deferrals, reductions, and cuts to ATC system acquisitions and related activities in FAA’s facilities and equipment budget may be beneficial and necessary in the long run, it is important for senior agency, department, OMB, and congressional decision-makers to have complete information to make informed decisions about the trade-offs that are being made when they consider annual budget submissions. As part of our research, we sought the perspective of an international group of experts, who also suggested that the ATO should provide the administration and Congress with detailed information in its budget submissions about the impact of reduced budgets on both ATC and NAS modernization. These experts were a part of an international panel of aviation experts we convened to address, among other issues, how federal budget constraints have affected ATC modernization and what steps the ATO could take in the short term to address these constraints. For example, aviation experts emphasized the need for the ATO—which is now the organizational entity responsible for acquiring ATC systems—to prioritize its capital investments, as well as its investment in operating systems, with affordability in mind. These experts believe that the ATO needs to review all of its spending plans for modernization, determine which programs can realistically be funded, and select programs to cut. Moreover, they indicated that the ATO should have a mechanism to explain to Congress the implications that cutting one system has on other systems. For example, according to one of these experts, the current budget process tears apart a highly layered, interdependent system and does not reveal synergies between projects. Then, when the budget request goes to Congress, he said, “you have no opportunity to try to explain to anybody the interconnections of these programs.” As a result, when the appropriators decide not to fund a project, they may not understand how their decision will affect other projects. The constrained budgetary environment makes it more important than ever for FAA to meet cost, schedule, and performance targets for each of the major ATC systems it continues to fund and to ensure that related activities, such as those to refurbish or replace the buildings that house ATC modernization systems, receive sufficient funding. The need for FAA to accommodate a 25 percent increase in demand for air travel over the next decade underscores the importance of these efforts. FAA has demonstrated a commitment to live within its expected means during fiscal years 2005 through 2009 by setting priorities among its ATC system acquisitions and identifying areas where it plans to cut funding. However, without detailed information about the trade-offs that underlie decisions to fully fund some systems and to defer, reduce, or eliminate funding for others, FAA’s plans do not allow senior agency, department, OMB, and congressional decision-makers to assess the implications of approving annual budget submissions for the ATC modernization program and related modernization activities that support more comprehensive efforts to modernize the NAS. To help ensure that key administration and congressional decision-makers have more complete information to assess the potential impact of annual budget submissions on individual ATC system acquisitions, the overall ATC modernization program, and related larger-scale NAS modernization activities funded through the facilities and equipment budget, we recommend that the Secretary of Transportation direct FAA to identify which activities under the ATC modernization program have had funding deferred, reduced, or eliminated and to provide detailed information about the impact of those decisions on FAA’s ability to modernize the ATC system and related components of the NAS in the near, mid, and longer term. This information should be reported to Congress annually. We provided a copy of our draft report to DOT for review and comment. The draft was reviewed by officials throughout DOT and FAA, including the Vice President for Acquisition and Business Service. These officials provided comments through email. They generally agreed with the report and provided technical comments on specific aspects of the report, which we incorporated as appropriate. The FAA officials said they are continuing to consider our recommendation and indicated they would provide a response to it as required by 31 U.S.C. §720. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees, the Secretary of Transportation, and the Administrator, FAA. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Please call me at (202) 512-2834 if you or your staff have any questions about this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. Modular and expandable, ERAM will replace software and hardware in the host computers at FAA’s 20 en route air traffic control centers, which provide separation, routing, and advisory information. ERAM’s flight data processing capabilities will provide flexible routing around restrictions, such as congestion and weather. It will improve surveillance by increasing the number and types of surveillance sources, such as radars. ERAM will provide safety alerts to prevent aircraft collisions and congestion. ERAM has not breached schedule or cost parameters, but it remains a high- risk program because of its size and its amount of software code (more than 1 million lines). The contractor has reported that engineering costs are rising because of lower productivity than originally planned and an increase in the number of lines of software code. According to FAA officials, the contractor’s management reserve can absorb additional software development costs. We examined (1) FAA’s experience in meeting cost, schedule, and/or performance targets for major system acquisitions under its ATC modernization program, (2) the steps FAA has taken to address long- standing challenges with the ATC modernization program and additional steps that are needed, and (3) the potential effects of the constrained budget environment on FAA’s ability to modernize the ATC system. To address the first objective, we selected 16 of the 55 system acquisitions in the ATC modernization program to review in detail. We selected these 16 systems in July 2004, when this review was still a part of our broader work on FAA’s efforts to modernize the National Airspace System (NAS).Specifically, we selected the 16 ATC system acquisitions with the largest life-cycle costs that met the following criteria: each system had cost, schedule, and/or performance targets, was discussed in our prior and Department of Transportation Inspector General reports, had not been fully implemented or deployed by 2004, and received funding in 2004. We reviewed this list with FAA officials to ensure that we did not exclude any significant system. (See app. I for additional information on these 16 systems.) FAA does not have a formal definition of major systems under its Acquisition Management System; however, agency officials told us that if a system acquisition has a formally approved baseline, we could consider it “major.” Using this definition, we determined that 25 of the 55 system acquisitions under the ATC modernization program are major. The remaining 30 system acquisitions are generally what FAA refers to as buy-it- by-the-pound systems that are commercially available and ready to use without modification, such as those to replace a system that has reached the end of its useful life. For fiscal year 2005, the 55 systems accounted for about 55 percent of FAA’s facilities and equipment (F&E) budget, or $1.38 billion of the $2.52 billion appropriated for the F&E budget. The 16 major systems accounted for 36 percent ($917.3 million), and the other 39 system acquisitions accounted for about 19 percent ($460 million). The remaining 45 percent of the F&E budget will be spent on facilities, mission support, and personnel- related activities ($1.14 billion). To assess the 16 major system acquisitions, we relied largely on data collected from FAA and contracting officials for two engagements we issued in November 2004 on FAA’s acquisition and certification processes. In turn, we updated this information and collected data on the remaining 39 systems under the modernization program, primarily through interviews with FAA officials and analyses of the data they provided, including key acquisition documents. (See app. II for additional information on these 39 system acquisitions.) In addition, we reviewed our past reports and those of the Department of Transportation’s Inspector General. Furthermore, we interviewed FAA officials within the recently created ATO and collected and analyzed the documents they provided. We also interviewed officials with the Aircraft Owners and Pilots Association, Air Transport Association, Department of Defense, National Air Traffic Controllers Association, and RTCA. Furthermore, we convened a panel of international aviation expertsto obtain their views on, among other things, the factors that have affected the cost, schedule, and/or performance of FAA’s ATC modernization program. In addition, we assessed the reliability of FAA’s cost and schedule estimates. Through interviews with FAA officials about their data system and quality controls, we determined that the cost and schedule estimates were appropriate for use in our report. Specifically, the estimates are sufficiently authoritative, appropriate, and reliable to allow us to use them without conducting any further assessment. The estimates appear to be based on reasonable assumptions. Our review did not focus on FAA’s efforts to modernize its facilities. To address the second objective, we interviewed FAA officials, primarily within the recently created ATO, and collected and analyzed the documents they provided. We also interviewed officials with the Aircraft Owners and Pilots Association, Air Transport Association, Department of Defense, National Air Traffic Controllers Association, and RTCA. We also reviewed past GAO reports and those of the Department of Transportation’s Inspector General. In addition, we obtained the views of the international aviation experts who participated in our panel on what steps the ATO could take in the short term to address the factors that have affected the cost, schedule, and/or performance of FAA’s ATC modernization program. To address the third objective, we interviewed officials within FAA’s ATO and obtained and analyzed data on FAA’s capital investments and annual budgets. We also interviewed officials with other organizations cited above. In addition, we obtained the views of the international aviation panelists on how federal budget constraints have affected ATC modernization and what steps the ATO could take in the short term to address these constraints. We conducted our review from November 2004 through May 2005 in accordance with generally accepted government auditing standards. In addition to the person named above, Beverly L. Norwood, Tamera Dorland, Seth Dykes, Elizabeth Eisenstadt, Brandon Haller, Bert Japikse, Maren McAvoy, and Ed Menoche made key contributions to this report.

Summary: The Federal Aviation Administration's (FAA) multibillion-dollar effort to modernize the nation's air traffic control (ATC) system has suffered from cost, schedule, and/or performance shortfalls in its system acquisitions for more than two decades and has been on our list of high risk programs since 1995. FAA's performance-based Air Traffic Organization (ATO) was created in February 2004, in part, to address these legacy challenges. In this report, GAO examined (1) FAA's experience in meeting cost, schedule, and performance targets for major ATC system acquisitions; (2) steps taken to address legacy problems with the program and additional steps needed; and (3) the potential impact of the constrained federal budget on this program. The ATO met its acquisition goal for fiscal year 2004. However, prior to the establishment of the ATO, FAA had experienced more than two decades of cost, schedule, and/or performance shortfalls in acquiring major systems under its ATC modernization program. For example, 13 of the 16 major system acquisitions that we reviewed in detail have experienced cost, schedule, and/or performance shortfalls when assessed against their original milestones. These 13 system acquisitions experienced total cost growth from $1.1 million to about $1.5 billion; schedule extensions ranging from 1 to 13 years; and performance shortfalls, including safety problems. We found that one or more of four factors--funding, requirements growth and/or unplanned work, stakeholder involvement, and software complexity--have contributed to these legacy challenges. While FAA met its recent acquisition goal, it is important to note that this goal is based on updated program milestones and cost targets for system acquisitions, not those set at their inception. Consequently, they do not provide a consistent benchmark for assessing progress over time. Also, as indicators of annual progress, they cannot be used in isolation to measure progress over the life of an acquisition. Although additional steps are warranted, FAA has taken some positive steps to address key legacy challenges it has had with acquiring major systems under the modernization program. For example, the ATO has cut funding for some major systems that were not meeting their goals and is reassessing all capital investments to help ensure that priority systems receive needed funding. The ATO has improved its management of software-intensive acquisitions and information technology investments and begun to more actively involve stakeholders. As we recommended, the ATO plans to establish an overall policy to apply its process improvement model to all software-intensive acquisitions. However, additional steps could be taken to improve its management of system acquisitions. For example, the ATO could use a knowledge-based approach to managing system acquisitions, characteristic of best commercial practices, to help avoid cost, schedule, and performance problems. The ATO will also be challenged to modernize the ATC system under constrained budget targets, which would provide FAA with about $2 billion less than it planned to spend through 2009. To fund its major system acquisitions and remain within these targets, the ATO has eliminated planned funding to start new projects and substantially reduced planned funding for other areas. However, when forwarding its budget submission for review by senior officials at FAA, DOT, the Office of Management and Budget, and Congress, the ATO provides no detail on the impact of the planned funding reductions on ATC modernization and related activities to modernize the NAS. Our work shows that the ATO should provide these decision-makers with detailed information in its budget submissions about the impact of funding decisions on modernization efforts. Without this type of information, decision-makers lack important details when considering FAA's annual budget submissions.