Over the past decade, NASA has experienced significant problems with several of its projects, which GAO and others have reported on (see table 1 for some examples of such problems). In addition, in 2002 we reported on several sources of failures in NASA programs, including underestimating complexity and technology maturity, and inadequate review and systems engineering processes. Further, we reported that the sources of these problems were not new and that NASA failed to consistently apply lessons previously learned. The failures identified in our 2002 report were in part the result of the “faster, better, cheaper” approach to managing its major acquisitions, which NASA adopted in the 1990s. These problems, along with others, highlighted the need for the agency to reevaluate its approach to cost and schedule estimating, risk assessments, technology development, project reviews, and systems engineering. In 1998, NASA adopted a new program and project management policy, which was revised in 2002. The policy provided significant flexibility, including allowing tailoring and projects to opt out of requirements at the discretion of the project manager. In March 2005—following a series of internal and external assessments of NASA that showed that the agency faced significant problems with project management—NASA again revised its program and project management policy, which includes policy requirements for the development of flight systems and ground support projects. According to NASA officials, further changes to the policy are anticipated in light of new agency leadership. The March 2005 policy document, NASA Procedural Requirement (NPR) 7120.5C, differs from previous versions of the policy in that it delineates requirements based on four NASA investment areas: Basic and Applied Research, Advanced Technology Development, Flight Systems and Ground Support, and Institutional Infrastructure. The policy also reinstitutes a life cycle phased approach to product development and institutes a project categorization scheme, based on project cost and priority, which denotes the oversight authorities and the level of detail that is needed to support project planning documents. See figure 2 below. NASA has also attempted to address some of its cost-estimating weaknesses by instituting a cost analysis data requirement (CADRe) and establishing thresholds for the use of earned value management (EVM). Another major change to the policy is the establishment of the Independent Technical Authority (ITA). According to the policy, the purpose of ITA is to establish sound technical requirements and decisions for safe and reliable system operations separate from the project management and reporting chain. Finally, agency officials told us that while the requirements in previous versions of the policy were easy to tailor, projects now must document compliance with the requirements in a “compliance matrix” and must request and have approved any deviations and/or waivers to requirements. Although NPR 7120.5C contains some mandatory requirements with regard to systems engineering, according to agency officials, NASA has never had an agencywide systems engineering policy to inform the development of flight systems and ground support projects. Since 1995, in the absence of a policy on systems engineering, project managers and systems engineers have relied on information contained in NASA’s Systems Engineering Handbook to guide their systems engineering approach on projects. Project managers and systems engineers, however, are not required to follow the handbook. Recognizing the need for a more structured and rigorous approach to systems engineering agencywide, NASA’s Office of the Chief Engineer is currently leading the effort to develop a systems engineering policy. The policy is in draft form. Over the last several years, we have undertaken a body of work on how leading developers in industry and government use a knowledge-based approach to deliver high quality products on time and within budget. A knowledge-based approach to product development efforts enables developers to be reasonably certain, at critical junctures or “knowledge points” in the acquisition life cycle, that their products are more likely to meet established cost, schedule, and performance baselines and, therefore provides them with information needed to make sound investment decisions. See figure 3 for a depiction of a knowledge-based acquisition life cycle. Knowledge point 1 (KP1): Resources and needs match. Knowledge point 1 occurs when a sound business case is made for the product— that is, a match is made between the customer’s requirements and the product developer’s available resources in terms of knowledge, time, workforce, and money. To determine available resources, successful developers rely on current and valid information from predecessor projects, new technologies that have demonstrated a high level of maturity, system engineering data, and experienced people. Successful developers also communicate extensively with customers to match their wants and needs with available resources and with the developers ability to manufacture an appropriate product. Knowledge point 2 (KP2): Product design is stable. Knowledge point 2 occurs when a developer determines that a product’s design is stable—that is, it will meet customer requirements and cost and schedule targets. A best practice is to achieve design stability at the product’s critical design review (CDR), usually held midway through development. Completion of at least 90 percent of engineering drawings at the CDR provides tangible evidence to decision makers that the design is stable. Knowledge point 3 (KP3): Production processes are mature. This level of knowledge is achieved when it has been demonstrated that the product can be manufactured within cost, schedule, and quality targets. A best practice is to ensure that all key manufacturing processes are in statistical control— that is, they are repeatable, sustainable, and capable of consistently producing parts within the product’s quality tolerances and standards—at the start of production. It is important that the product’s reliability be demonstrated before production begins, as investments can increase significantly if defective parts need to be repaired or reworked. If the knowledge attained at each juncture does not confirm the business case on which the initial investment was originally justified, the project should not go forward and additional resources should not be committed. Product development efforts that do not follow a knowledge-based approach can be frequently characterized by poor cost, schedule, and performance outcomes. NASA’s revised acquisition policy for developing flight and ground support systems incorporates some of the elements of a knowledge-based acquisition approach. However, it lacks specific key criteria and decision reviews necessary to fully support such an approach. NASA’s policy defines a phased life cycle approach and requires a major decision review to move from project formulation to implementation. The policy requirements for this review address many of the key elements necessary to match needs to resources, such as requirements to establish project baselines. However, the policy does not require that projects demonstrate technologies at high levels of maturity before launching a project and investing a large amount of resources. In addition, NASA’s policy does not require any further major decision reviews following the formulation phase of the project. Major decision reviews in the implementation phase based on specific evaluation criteria at the final design and fabrication, assembly, and testing milestones—critical decision points in any product development—could help NASA ensure that sufficient knowledge has been gained to warrant moving forward in the development process. The life cycle for all NASA projects is divided into two major phases— formulation and implementation. Because flight systems and ground support projects are particularly complex and have long life cycles, NASA has further divided the formulation and implementation phases for these projects to allow managers to assess management and engineering progress (see fig. 4). Flight systems and ground support projects must successfully complete two major decision reviews: a Preliminary-Non Advocate Review (Pre-NAR) between Phases A and B and a Non-Advocate Review (NAR) between Phases B and C. At these reviews, the Governing Program Management Committee (GPMC) evaluates the cost, schedule, safety, and technical content of the project to ensure that the project is meeting commitments specified in key management documents. Following each of these reviews, the GPMC recommends to the appropriate decision authority whether the project should be authorized to proceed. As with KP1 in a knowledge-based acquisition life cycle, the NAR marks the official project approval point in the life cycle. After approval at the NAR, projects are included as part of implementation reviews of their parent program. These implementation reviews are conducted biennially and are not tied to any design or production milestones. After entering the implementation phase, the GPMC is notified if cost or schedule performance exceeds the baselines established at the NAR by 10 percent or when key performance criteria are not met. Exceeding the NAR baselines can result in the GPMC conducting a termination review to determine whether or not to continue the project. To help ensure project requirements do not outstrip resources, leading developers obtain the right knowledge about a new product’s technology, design, and production at the right time. NASA’s policy emphasizes many elements needed at the NAR (or KP1) to match needs to resources, such as validating requirements, developing realistic cost and schedule estimates and human capital plans, and establishing a preliminary design. The policy does not, however, require projects to demonstrate technologies at high levels of maturity before launching a project. Table 2 compares KP1 criteria and NASA’s policy criteria. While NASA requires projects to develop plans that describe how technologies will be matured and to provide alternative development strategies for technologies that do not mature as expected, it does not establish a minimum threshold for technology maturity. Consequently, projects can enter the implementation phase with immature technologies and embark on a risky path of having to build technology, design, and production knowledge concurrently. Our best practices work has shown that maturing technologies during the preliminary design phase and before entering product development is a key element of matching needs to resources and that there is a direct relationship between the maturity of technologies and the risk of cost and schedule growth. Allowing technology development to carry over into the product development phase increases the risk that significant problems will be discovered late in development. Addressing such problems at this stage may require extensive retrofitting and redesign as well as retesting, which can jeopardize performance and result in more time and money to fix. This approach also makes it more difficult for projects to demonstrate the same level of design stability in later phases of implementation since technology and design activities will be done concurrently. Technology readiness levels (TRL)—a concept developed by NASA—can be used to gauge the maturity of individual technologies. The higher the TRL, the more the technology has been proven and the lower the risk of performance problems and cost and schedule overruns (see fig. 5). TRL 7—demonstrating a technology as a fully integrated prototype in an operational environment—is the level of maturity preferred by product developers to minimize risks when entering product development. Successful developers will not commit to undertaking product development, and more importantly investing resources, unless they have high confidence that they have achieved a match between what the customer wants and what the project can deliver. Technologies that are not mature continue to be developed in an environment that is focused solely on technology development. Once matured, these technologies can be transitioned to projects. This puts developers in a better position to succeed because they can focus on integrating the technologies and testing and proving the product design. Our prior work has shown that successful developers establish specific criteria to ensure that requisite knowledge has been attained before moving forward from final design into the latter stages of development. Before making significant increases in investments to fabricate, assemble, and test the product, these developers conduct a decision review to determine if the design is stable and performs as expected and the project is ready to enter the next phase. To make this determination and reach KP2, successful developers use specific, knowledge-based standards and criteria. (See app. II for more information on the specific knowledge-based standards and criteria successful developers use to judge readiness to proceed beyond detailed design activities.) Successful developers also demand proof that manufacturing processes are in control and product reliability goals are attained before committing to production. To determine whether they have achieved this knowledge point, KP3, successful developers conduct another mandatory decision review in which they use specific, knowledge-based standards and criteria to determine if the product can be produced within cost, schedule, and quality targets. (See app. II for more information on specific knowledge- based standards and criteria used by successful developers to judge readiness to enter into production.) Contrary to these best practices, the NAR at the end of the preliminary design phase—KP1—is the last major decision review in the NASA project life cycle. (See fig. 6.) Although NPR 7120.5C requires that projects document in the project plan a continuum of technical and management reviews, such as a PDR and CDR, it does not require any specific reviews. In addition, NASA’s March 2005 policy does not require a NAR-type decision review to ensure a project has obtained the knowledge needed to proceed beyond the final design phase into the fabrication, assembly, and test phase, which serves as both the demonstration and production phase of the NASA life cycle. According to NASA officials, projects conduct a CDR at the end of the final design phase to ensure adequate information is available about product design and producibility before entering the fabrication, assembly, and test phase. The CDR, however, is a technical review—not a major decision review like the NAR. Furthermore, the policy does not establish criteria as to what constitutes successful completion of a CDR. NASA’s policy also does not require a major decision review before beginning manufacturing. (See fig. 6 above.) Therefore, the transition from final design to fabrication, assembly, and test often marks a de facto production decision. According to NASA officials, the agency rarely enters a formal production phase due to the small quantities of space systems that they build. However, due to the high cost of failure associated with NASA projects and the costs and risks involved in repairing a system in-orbit, a major decision review at production that assesses product reliability is essential even for these limited production systems. In addition, although NASA’s production quantities are typically low, in some instances NASA does produce larger quantities of a system or subsystem, such as the external tanks for the Space Shuttle. Furthermore, NASA’s plans indicate that the agency may be increasing production for elements of their future systems. For example, NASA’s Exploration Systems Architecture Study indicates that NASA plans to build several new crew exploration vehicles, with disposable elements, such as the lunar lander, solid rocket boosters, and space shuttle main engines that will require higher numbers of production runs. Rather than establish specific criteria by which all projects are judged, NASA’s policy requires that projects manage to baselines and plans established in key management documents and approved at the project NAR. The baselines and plans serve as their primary tools for measuring project progress and as the primary basis for judgment at project reviews. While the plans may include some information that addresses knowledge-based criteria for design and production, the instructions for preparing them leaves the establishment of thresholds and success criteria to the discretion of the project manager. For example, NASA policy requires that projects include, as part of the project plan, a Verification and Validation Sub Plan that describes the project’s approach to verifying and validating hardware and software as part of the project plan. The policy, however, includes no instruction as to what constitutes a sufficient approach to testing. In other words, there are no requirements concerning the fidelity of test articles or the realism of the test environment. Similarly, NASA’s policy requires that projects include, as part of the project plan, a Systems Engineering Sub Plan that describes the project’s approach to systems engineering and the technical standards that are applicable, including metrics that verify the processes. The policy, however, does not identify the types of metrics appropriate to verify the process or establish any threshold criteria. The absence of major decision reviews, along with specific criteria in the fabrication, assembly, and test phase, could result in concurrent design and manufacturing activities, a practice our past work has found increases risk in acquisition programs. Furthermore, lacking a major decision review to ensure that projects have gained the appropriate levels of knowledge at KP2 and KP3, NASA decision makers cannot be provided a high level of certainty that the project will meet cost, schedule, and performance requirements and have no assurance that the provisions of the key management documents required by NPR 7120.5C are being executed after the NAR. NASA centers have varying approaches to implementing project management policies and systems engineering guidance for flight systems and ground support projects. Some centers use criteria at key decision points that are similar to the criteria required to ensure a knowledge-based approach is followed, while others lack such criteria. As a result, each center reports a different level and type of knowledge about a project at key decision points. Centers also rely on project managers and systems engineers to employ good project management and systems engineering practices. However, given the loss of experienced project managers and the decline of in-house systems engineering and technical capabilities agencywide, that reliance could be problematic. These situations make it difficult for NASA decision makers to evaluate center projects on a common foundation of knowledge and make sound investment decisions and tradeoffs based on those evaluations. A standardized, knowledge- based approach would prepare NASA to face competing budgetary priorities and make difficult decisions regarding the investment in and termination of projects. While NASA centers are given discretion about how they implement agencywide policies, they are expected to have procedures and guidelines in place for implementing those policies. Some centers have developed center-specific policies and criteria for implementing NASA’s project management policies and system engineering guidance, while others have not. Centers also rely on project managers and systems engineers to implement the requirements of NPR 7120.5C and to use NASA’s Systems Engineering Handbook as guidance for good systems engineering practices. Some NASA centers have also developed criteria in their policies that are similar to the criteria used to ensure a knowledge-based approach is followed; other centers lack such criteria. Because of their varying policies and criteria, each center requires a different level of knowledge at the same point in a project’s development cycle. For example, GSFC requires its projects to mature technologies to TRL 6 by the preliminary design review—before entering the implementation phase. On the other hand, JPL, MSFC, and JSC policies do not require projects to mature technology to a particular level before entering implementation, leaving the determination of needed technology maturity up to the project manager. Requirements for assessing design maturity also vary across the centers. While most of the center policies require a CDR to enter “Phase D— Fabrication, Assembly, and Test”—the criteria used to assess projects at this point vary. For example, both GSFC and MSFC require projects to have completed a percentage of design drawings at this review. MSFC requires that 90 percent of design drawings to be complete by CDR, which is consistent with best practices. While GSFC establishes a minimum threshold of drawings to be complete by CDR—greater than 80 percent— neither JSC nor JPL establish a minimum percentage drawing requirement. Instead, JSC requires the design be complete and drawings ready to begin production, and JPL requires that the design be mature and provide confidence in the integrity of the flight system design. Almost none of the center policies include a requirement to assess the maturity of production processes. According to NASA officials, due to the low quantities of systems generally produced by the agency, most of the center policies do not require a review before beginning manufacturing. Only JSC requires a Production Readiness Review to ensure that production plans, facilities, and personnel are in place and ready to begin production. However, the criteria do not specify quantifiable thresholds to measure production readiness at the review. JPL, MSFC, and GSFC do not have policies that outline requirements for such a review. In addition to individual policy requirements, centers may rely on project managers and systems engineers to employ good project management and systems engineering practices. For example, an experienced project manager at JPL told us that although JPL policy does not require a particular TRL at PDR to enter implementation, he required that all technologies for his project be around a TRL 6 in order to separate technology development from systems development. Reliance on project managers to implement good practices, however, could be problematic given the diminishing number of experienced project managers available to lead projects and the decline of in-house systems engineering and technical capabilities agencywide caused by increasing retirements and outsourcing. In a knowledge-based process, the achievement of each successive knowledge point builds on the preceding one, giving decision makers the information they need, when they need it, to make decisions about whether to invest significant additional funds to move forward with product development. Our work has shown that successful product development efforts are marked by adherence to a disciplined process that establishes and uses common and consistent criteria for decision making at these key points. With varying project management and systems engineering criteria, NASA centers’ technical reviews, such as PDR, provide different levels of knowledge to support NASA’s major decision reviews, such as the NAR, which NASA uses to support its major investment decisions for flight systems and ground support projects. In the near future, NASA will need to determine the resources necessary to develop the systems and supporting technologies to achieve the President’s Vision for Space Exploration and structure its investment strategy accordingly. Initial implementation of the Vision as explained in NASA’s Exploration Systems Architecture Study calls for completing the International Space Station, developing a new crew exploration vehicle, and returning to the moon no later than 2020. NASA estimates that it will cost approximately $104 billion over the next 13 years to accomplish these initial goals. These priorities, along with NASA’s other missions, will be competing within NASA for funding. It will likely be difficult for NASA managers to agree on which projects to invest in and which projects to terminate. The NASA Administrator has acknowledged that NASA faces difficult choices about its missions in the future—for example, between human space flight, science, and aeronautics missions. Using consistent criteria to evaluate all NASA projects would help ensure that the same level and type of knowledge is available about individual projects at key decision points. Analogous information about all flight systems and ground support projects would allow decision makers to make apples-to-apples comparisons across projects and make investment decisions, and trade-offs, based upon these comparisons. Further, policies with consistent criteria can provide inexperienced project managers and systems engineers with the necessary guidance to implement good project management and systems engineering practices and ensure that the right knowledge is available for decision makers. NASA officials within the Chief Engineer’s Office acknowledged the need for more consistency in criteria across the various NASA centers in order to successfully achieve NASA’s Vision for Space Exploration. Some NASA centers, however, are resistant to standardized criteria because they feel it could be overly prescriptive. These centers indicated that because of the unique nature of the work done at each of the 10 NASA centers, it would be unrealistic to hold every project to the same criteria. Nonetheless, our work has shown that the process used by successful product developers to develop leading-edge technology and products does not differ based upon the type of product being developed. Further, these developers adhere to a disciplined process that establishes and uses common and consistent criteria for decision making—regardless of the type of product or technology being developed. Using consistent criteria can allow NASA decision makers to assess the likely return on competing investment priorities and to reevaluate alternatives and make investment decisions across projects to increase the likelihood of attaining the strategic goals of the agency. Several of NASA’s major acquisitions have been marked by cost, schedule, and performance problems. Yet the challenges NASA faces in the future are likely to far exceed those it has faced in the past. The complex technical requirements associated with fulfilling the President’s Vision, the fiscal constraints under which NASA will be required to operate, the diminished number of experienced project managers and systems engineers, and the potential for increased production make following a knowledge-based approach for flight systems and ground support projects all the more critical. While NASA has made improvements to its policies governing project management, the lack of major decision reviews beyond the initial project approval gate leaves decision makers with little knowledge about the progress of the agency’s projects. Further, without a standard set of criteria to measure projects at crucial phases in the development life cycle, NASA cannot be assured that its decisions will result in the best possible return on its investments. Since NASA is currently in the process of revising its program and project management policies and developing an agencywide systems engineering policy, we believe this presents a unique opportunity for the agency to correct some of the problems identified during our review. In order to close the gaps between NASA’s current acquisition environment and best practices on knowledge-based acquisition, we recommend that NASA take steps to ensure NASA projects follow a knowledge-based approach for product development. Specifically, we recommend that the NASA Administrator direct the Office of the Chief Engineer to take the following two actions: In drafting its systems engineering policy, incorporate requirements in the policy for flight systems and ground support projects to capture specific product knowledge by key junctures in project development. The demonstration of this knowledge should be used as exit criteria for decision making at the following key junctures: Before projects are approved to transition from formulation to implementation, the policy should require that projects demonstrate that key technologies have reached a high maturity level. Before projects are approved to transition from final design to fabrication, assembly, and test, the policy should require that projects demonstrate that the design is stable. Before projects are approved to transition into production, the policy should require projects to demonstrate that the design can be manufactured within cost, schedule, and quality targets. Revise NPR 7120.5C to institute additional major decision reviews following the NAR for flight systems and ground support projects, which result in recommendations to the appropriate decision authority. These reviews should be tied to the key junctures during project development mentioned above in order to increase the likelihood that cost, schedule, and performance requirements of the project will be met. We provided a draft of this report to NASA for review and comment. In written comments, NASA indicated that it agreed with our recommendations and outlined specific actions that the agency plans to take to address them. The actions that NASA plans to take to address our recommendations are a positive step toward achieving successful project outcomes and ensuring that decision makers are appropriately investing the agency’s resources. We are pleased to hear that many knowledge-based practices identified in our report are currently being practiced agencywide in the management and development NASA systems. The addition of such practices to NASA’s policies will only strengthen their use agencywide and ensure that these practices continue to be utilized by less experienced project managers and systems engineers as the more experienced workforce retires. The effectiveness of such practices, however, will be limited if project officials are not held accountable for demonstrating a high level of knowledge, consistent with the success criteria that NASA plans to require in its policies, at key junctures in development. It is critical that project officials not only have a high level of knowledge about a project at key junctures, but also that this information is used by decision makers to make decisions on whether to invest additional resources and allow a project to proceed through the development life cycle. NASA’s comments are reprinted in appendix III. NASA also provided technical comments, which we addressed throughout the report as appropriate. As agreed with your offices, unless you announce its contents earlier, we will not distribute this report further until 30 days from its date. At that time, we will send copies to NASA’s Administrator and interested congressional committees. We will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If your or your staff have any questions concerning this report, please contact me at (202) 512-4841 or lia@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix IV. To determine the extent to which NASA’s current policies support an acquisition approach consistent with best practices identified in GAO’s work on system acquisitions, we reviewed and analyzed NASA-wide program and project management policies and systems engineering guidance. Our review and analysis of the policy focused on requirements for flight systems and ground support projects. We interviewed NASA Headquarters officials from the Office of the Chief Engineer who are responsible for the policy and guidance. We compared NASA’s policy on program and project management with criteria contained in GAO best practices work on systems acquisition and space system acquisitions. We concentrated on whether the policy provides a framework for a knowledge-based process and the criteria necessary to carry out this intent. To determine how NASA-wide acquisition policies are implemented across the various NASA centers, we reviewed NASA center-specific program and project management policies and systems engineering policies and interviewed officials responsible for implementing those policies. While we interviewed officials from all centers, we focused on the centers that manage the majority of NASA’s flight systems and ground support projects—GSFC, JPL, JSC, and MSFC. This approach included site visits with GSFC, JPL, JSC, and MSFC and teleconferences with the remaining centers. We compared examples of the centers’ implementation of the policies and specific criteria included in these policies with our best practices work on systems acquisition. We performed our work from May 2005 to November 2005 in accordance with generally accepted government auditing standards. In addition to the individual named above, James L. Morrison, Assistant Director; Valerie Colaiaco, Tom Gordon, Alison Heafitz, Shelby S. Oakley, Ron Schwenn, Karen Sloan, and John S. Warren, Jr., made key contributions to this report. Space Acquisitions: Stronger Development Practices and Investment Planning Need to Address Continuing Problems. GAO-05-891T. Washington, D.C.: July 12, 2005. Defense Acquisitions: Incentives and Pressures That Drive Problems Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington, D.C.: June 23, 2005. Defense Acquisitions: Space-Based Radar Effort Needs Additional Knowledge before Starting Development. GAO-04-759. Washington, D.C.: July 23, 2004. Defense Acquisitions: Risks Posed by DOD’s New Space Systems Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004. Space Acquisitions: Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4, 2003. Defense Acquisitions: Improvements Needed in Space Systems Acquisition Policy to Optimize Growing Investment in Space. GAO-04-253T. Washington, D.C.: November 18, 2003. Defense Acquisitions: Despite Restructuring, SBIRS High Program Remains at Risk of Cost and Schedule Overruns. GAO-04-48. Washington, D.C.: October 31, 2003. Defense Acquisitions: Improvements Needed in Space Systems Acquisition Management Policy. GAO-03-1073. Washington, D.C.: September 15, 2003. Military Space Operations: Common Problems and Their Effects on Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June 2, 2003. Military Space Operations: Planning, Funding, and Acquisition Challenges Facing Efforts to Strengthen Space Control. GAO-02-738. Washington, D.C.: September 23, 2002. Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002. Defense Acquisitions: Space-Based Infrared System-Low at Risk of Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February 28, 2001. Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-05-301. Washington, D.C.: March 31, 2005. Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004. Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-04-248. Washington, D.C.: March 31, 2004. Defense Acquisitions: DOD’s Revised Policy Emphasizes Best Practices, but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10, 2003. Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-03-476. Washington, D.C.: May 15, 2003. Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000. Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. Best Practices: DOD Training Can Do More to Help Weapon System Program Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999. Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998. Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998. Major Acquisitions: Significant Changes Underway in DOD’s Earned Value Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

Summary: The National Aeronautics and Space Administration (NASA) plans to spend over $100 billion on capabilities and technologies to achieve the initial goals of the President's 2004 Vision for Space Exploration. In the past, NASA has had difficulty meeting cost, schedule, and performance objectives for some of its projects because it failed to adequately define project requirements and quantify resources. NASA will be further challenged by a constrained federal budget and a shrinking experienced NASA workforce. To help face these challenges and manage projects with greater efficiency and accountability, NASA recently updated its program and project management policy and is developing an agencywide systems engineering policy. GAO has issued a series of reports on the importance of obtaining critical information and knowledge at key junctures in major system acquisitions to help meet cost and schedule objectives. This report (1) evaluates whether NASA's policy supports a knowledge-based acquisition approach and (2) describes how NASA centers are implementing the agency's acquisition policies and guidance. While NASA's revised policy for developing flight systems and ground support projects incorporates some of the best practices used by successful developers, it lacks certain key criteria and major decision reviews that support a knowledge-based acquisition framework. For example, NASA's policy requires projects to conduct a major decision review before moving from formulation to implementation. Further, before moving from formulation to implementation, projects must validate requirements and develop realistic cost and schedule estimates, human capital plans, a preliminary design, and a technology plan--key elements for matching needs to resources. However, NASA's policies do not require projects to demonstrate technologies at high levels of maturity before program start. By not establishing a minimum threshold for technology maturity, NASA increases the risk that design changes will be required later in development, when such changes are typically more costly to make. In addition, although NASA's policy does require project managers to establish a continuum of technical and management reviews, it does not specify what these reviews should be, nor does it require major decision reviews at other key points in a product's development. Acquiring knowledge at key junctures will become increasingly important as NASA proceeds to implement elements of the Vision. Without a major decision review at key milestones to ensure that the appropriate level of knowledge has been achieved to proceed to the next phase, the risk of cost and schedule overruns, as well as performance shortfalls, increases. NASA centers have varying approaches for implementing the agency's policies and guidance. Some centers have established product development criteria that are similar to the criteria used in a knowledge-based acquisition, while other centers have not. As a result, each center reports a different level and type of knowledge about a project at key decision points. Centers also rely on project managers and systems engineers to employ good project management and systems engineering practices. However, given the loss of experienced project managers and the decline of in-house systems engineering and technical capabilities, that reliance could be problematic. These situations make it difficult for decision makers to evaluate projects on the same basis and make sound investment decisions and tradeoffs based on those evaluations. A standardized, knowledge-based approach would prepare NASA to face competing budgetary priorities and better position the agency to make difficult decisions regarding the investment in and termination of projects.