EVM is a project management tool that, when properly used, can provide accurate assessments of project progress, produce early warning signs of impending schedule delays and cost overruns, and provide unbiased estimates of anticipated costs at completion. Pulling together essential cost, schedule, and technical information in a meaningful, coherent fashion is a challenge for most projects. Without such information, managers can have a distorted view of a project’s status and risks. EVM also allows individuals outside the project to see a standardized metric describing the cost and schedule performance of that particular project and compare it consistently with other projects. EVM measures the value of work accomplished in a given period and compares it with the planned value of work scheduled for that period and with the actual cost of work accomplished. Differences in these values are measured in both cost and schedule variances. Positive variances indicate that activities are costing less or are completed ahead of schedule. Negative variances indicate activities are costing more or are falling behind schedule. For example, if a contractor completed $5 million worth of work and the work actually cost $6.7 million, there would be a $1.7 million negative cost variance. Schedule variances are also measured in dollars, but they compare the earned value of the work completed with the value of work that was expected to be completed. For example, if a contractor completed $5 million worth of work at the end of the month but was budgeted to complete $10 million worth of work, there would be a $5 million negative schedule variance. Earned value provides information that is necessary for understanding the health of a project and an objective view of project status. Cost and schedule variances can also be used in estimating the cost and time needed to complete the project. While some data that an EVM system produces are retrospective and indicate performance to date, EVM data can also be used to predict future performance. For example, estimates at completion for a project can be calculated by using efficiency indices which are based on a project’s past cost and schedule performance. See appendix II for additional information on the importance of EVM. A project’s EVM data comes from multiple sources. For example, each contractor that supports a project will produce and deliver EVM reports to the project for the work they and their subcontractors perform, if the contract so requires. In addition, the project will collect EVM information for the work that it performs in-house at a NASA center. All of this lower level EVM data can then be consolidated at the project level to produce a project level EVM report. Pulling together EVM data from multiple levels into a project level report gives the project a comprehensive outlook of its cost and schedule, and provides the project manager with early warning of potential cost and schedule overruns. EVM has evolved from an industrial engineering tool to a government and industry best practice, providing improved information to conduct oversight of acquisition programs. As such, it is guided by industry best practices and standard, and is required by regulations and requirements at the federal government and the agency level at NASA. These requirements and standards are summarized in table 1 below. Almost two decades of NASA’s past efforts to improve its use of earned value management have had uneven success. An EVM Focal Point Council was created in 1996 and focal points were designated at each NASA center and the Office of Procurement and the Office of the Chief Financial Officer to provide an open forum to share experiences and develop a network of support within the NASA EVM community. In 1997, the agency issued NASA Policy Directive 9501.3 “Earned Value Management Performance” which established the basis for applying EVM to NASA contracts. Prior to this policy, centers used their own individual policies on performance measurement systems. However, in 1999, the NASA Inspector General reported that EVM policy was not consolidated as an overall program management responsibility and that the Agency Program Management Council did not receive comprehensive EVM information.policy from the Office of the Chief Financial Officer to the Office of the Chief Engineer to emphasize EVM as a project management tool, rather than a financial management tool. As a result, in 2003, NASA shifted responsibility for the EVM In 2004, GAO reported that only 2 of 10 NASA projects reviewed used EVM consistently and appropriately. Several barriers to EVM implementation were identified, such as lack of reliable financial data, trained EVM staff, data analysis tools, and incentives. Among other things, GAO recommended that NASA take action to ensure that a true EVM system is used as an organizational management tool to bring cost to the forefront in NASA’s decision-making process and that acquisition and EVM management policies and procedures be enforced. In response to our recommendations, NASA stated that it was updating NASA Procedural Requirements 7120.5, its program and project management processes and requirements policy, to improve its cost estimating and ensure that its cost estimate and earned value analyses were effectively used, and the updated policy was issued in 2005. Similar to other agencies in our 2009 report on governmentwide use of EVM, we reported on weaknesses in NASA’s EVM policies and practices and recommended that the agency modify its policies governing EVM to ensure that they are consistent with best practices. In particular, we found problems with the EVM training requirements for personnel responsible for investment oversight and management responsibilities. In addition, NASA’s policy for revising project cost and schedule baselines did not have adequately defined criteria on the acceptable reasons for permitting a rebaselining. We also found weaknesses in how the NASA projects we reviewed implemented EVM and managed their negative performance trends. NASA acknowledged the identified weaknesses and stated that it was revising its NASA procedural requirements for programs and projects to include expanded and strengthened policies governing EVM application and processes, and revised policies for rebaselining projects. In January 2010, the Agency Program Management Council approved the funding for the EVM capability project to develop an EVM system that complies with the guidelines in ANSI/EIA-748. Another goal of the project was to determine whether it was feasible to implement a single EVM system that would integrate the scope, schedule, and budget of EVM data for NASA’s in-house managed efforts and contractor data across the agency. According to agency officials, NASA invested about $2 million into the capability project to pilot the EVM system through two projects: the Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) and the Constellation Extra-Vehicular Activity project. Through the pilots, the capability project demonstrated that an agency-wide EVM system was feasible. As a result, a finalized set of processes, tools, guidance, and training products that compose NASA’s new EVM system was developed. This new system was peer reviewed and approved by a panel of EVM experts. Our assessment of 10 major spaceflight projects showed that NASA has not yet fully implemented EVM and thus is not taking full advantage of an important tool that could help reduce acquisition risk. GAO found that the projects had shortfalls in two of the three fundamental practices that we assessed. Specifically, we found that half of the projects did not use an EVM system that was certified as compliant with the ANSI/EIA-748 standard. Most of the projects conducted an integrated baseline review (IBR), a practice that ensures the performance measurement baseline reflects all requirements and that resources are adequate to complete the work. Specifically, 9 projects conducted an IBR for their overall efforts; however, the Stratospheric Observatory for Infrared Astronomy (SOFIA) project office only conducted an IBR of one of its major contractors. In addition, we found that only 4 of the 10 projects had established formal independent surveillance reviews to ensure that key elements of the EVM process were maintained over time so that the data produced by the system provided timely indications of actual or potential problems. For the 6 projects that did not have formal independent surveillance in place, each provided evidence that they instituted monthly EVM data reviews, which according to project officials, helps them to continually monitor cost and schedule performance. However, the rigor of both the formal and informal surveillance reviews is questionable given the numerous EVM data anomalies we found in the monthly EVM reports. Specifically, we found many unexplained anomalies, such as the presence of negative numbers or missing data, which caused us to question the reliability of the data. Out of the 10 projects we reviewed, we found that just 3 projects had reliable EVM data while the remaining 7 had only partially reliable data. Overall, the projects are using EVM, but NASA has not consistently implemented EVM across these projects. For example, we found that several projects were not implementing EVM at the project level, which is considered a best practice. Table 2 summarizes the performance of each of the 10 projects in meeting the three fundamental EVM practices and the reliability of the data. Of the 10 projects we reviewed, 4 projects had a certified EVM system, 3 did not, and 3 had a mixture in which some contractors and subcontractors had certified systems and some did not. When an EVM system is certified, the agency has assurance that the implemented system was validated for compliance with the ANSI/EIA-748 standard by independent and qualified staff and therefore can be considered to provide reliable and valid data from which to manage a project. The Global Precipitation Measurement (GPM), Tracking and Data Relay Satellite System (TDRS), Landsat Data Continuity Mission (LDCM), and James Webb Space Telescope (JWST) were the only projects that provided evidence that the contract performance reports provided came from EVM systems that were certified as compliant with the ANSI/EIA-748 standard. The Lunar Atmosphere and Dust Environment Explorer (LADEE), Magnetospheric Multiscale (MMS) and Radiation Belt Storm Probes (RBSP) projects did not have EVM systems that were certified to be compliant with the ASNI/EIA-748 standard. Finally, the Jet Propulsion Laboratory, a federally funded research and development center that the California Institute of Technology manages under a contract with NASA, was the only NASA Center with a certified EVM system. The Jet Propulsion Laboratory is responsible for managing the Orbiting Carbon Observatory 2 (OCO-2) project. The Mars Atmosphere and Volatile Evolution Mission (MAVEN) and SOFIA prime contractors also had certified systems; however, their project offices did not. NASA does not require a certified EVM system for their in-house work. Using the project’s integrated master schedule and contract performance reports, we assessed the EVM data provided by the projects against selected fundamental ANSI/EIA guidelines to determine the extent to which each project’s EVM system, whether certified or not, was meeting them. The guidelines we reviewed included whether the work breakdown structure (WBS)—which provides the basis of the project schedule—was consistent between the EVM report and the schedule, whether the schedule identified significant task interdependencies, and whether the project had identified a time-phased budget baseline for tracking cost and schedule variances. As shown in figure 1, a work breakdown structure breaks down product-oriented elements into a hierarchical structure that shows how elements relate to one another as well as to the overall end product. By subdividing a project into smaller elements, management can more easily plan and schedule the program’s activities and assign responsibility for the work. We found that even for the projects that had certified systems, there were problems with consistency between the WBS and the EVM report and the schedule. For example, we found discrepancies in the hierarchical structure and numbering of WBS elements for JWST, an $8.8 billion project. Specifically, the project’s WBS dictionary showed mission assurance efforts numbered differently than contractor reports for two contractors, each of which had mission assurance labeled with different WBS numbers. NASA officials explained that neither the spacecraft nor the near infrared camera contractor was required to follow the project- level WBS structure or numbering scheme. NASA officials explained that while it is not a requirement for the project and contractor WBSs to be the same, it is recommended that the prime contractor lower-level WBS numbering scheme be consistent with the overall project WBS numbering format. Doing so allows easier total project integration of cost and EVM data for project reporting. Consistency of the WBS element between the cost estimate and the schedule facilitates updating the cost estimate with actual costs and ensures there is correlation between the cost estimate and schedule. Our review of the project schedules also revealed that about half of the schedules were missing predecessor and/or successor dependencies and had constraints that prevented the schedule from responding properly to updates. Since the schedule is the foundation for the EVM baseline, it must be properly sequenced. This means knowing how one activity (the predecessor) affects another (the successor) and how each affects the critical path. When the schedule is not sequenced correctly, the reliability of the EVM data is called into question. Our review found that the MMS project was missing dependencies for 31 percent of its remaining activities for its instrument suite contract and 36 percent of the remaining activities for the instrument suite were constrained. Due to the major sequencing issues in the MMS instrument schedule, we questioned the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. MMS project officials said they believed many of the constrained activities we found were not valid because they reside in another schedule. In addition, officials said some of the constraints found are in the Harness area, and if removed, these constraints would have no effect on the overall schedule. Furthermore, MMS officials said some of the sequencing issues may be caused by manual integration because some instrument provider schedules are in Microsoft Project and others are in Primavera, and therefore it is not possible to ensure all tasks have been linked properly. As part of their schedule health check process, the MMS project scheduler tests the schedule for missing dependencies, logic errors, and reasonable durations and the results are shared with the project office and the contractor so that appropriate action can be taken. However, when we removed the 15 level of effort type activities from the missing dependencies count, the schedule still showed 28 percent of the remaining activities missing dependencies. We also removed the 14 level of effort type activities and 4 Harness from the constraint count, the schedule still showed 33 percent of the remaining activities were constrained. MMS program officials said that they have a process in place to manage float and the critical path, however, the schedule we reviewed still showed significant sequencing issues. Finally, we found that 4 of the 14 the schedules we analyzed were not resource loaded. This means that the schedule did not have the required labor, materials, equipment, and other resources assigned to the appropriate activities. When the schedule is not resource loaded, then costs need to be spread over time using some other method that may not be as straightforward as having the costs integrated directly within the schedule. Having a resource-loaded schedule is a best practice for developing the time phased budget baseline. The time-phased budget baseline represents the plan that performance is measured against for the life of a project. It takes into account that program activities occur in a sequenced order, based on finite resources, with budgets representing those resources spread over time. Deviations from the baseline identify areas where management should focus their attention. In keeping with best practices, 9 of the 10 projects conducted integrated baseline reviews. An IBR is an evaluation of the performance measurement baseline—the foundation for an EVM system—to determine whether all project requirements have been addressed, risks have been identified, mitigation plans are in place, and available and planned resources are sufficient to complete the work. Conducting an IBR increases confidence that the performance measurement baseline provides reliable cost and schedule data for managing the project and that it projects accurate estimated costs at completion. OMB has endorsed the IBR as a critical process for risk management on major investments and requires agencies to conduct IBRs for all contracts that require EVM. Since an IBR’s goal is to verify that the technical baseline’s budget and schedule are adequate for performing the work, it offers many key benefits such as laying a solid foundation for successfully executing the project and enabling better understanding of the risks. Officials for the SOFIA project did not conduct an integrated baseline review at the project level; however, its prime contractor for the engineering and modification of the airborne observatory platform did conduct an integrated baseline review. According to project officials, the lack of a project-level IBR is related to the EVM system being implemented “on the fly” late in the development phase for SOFIA, as a result of an audit recommendation in 2010. However, project officials noted that the EVM baseline was established concurrently with an agency approved re-plan and Joint Cost and Schedule Confidence Level analysis in 2009 and 2010 and was reviewed by a Standing Review Board as part of that process. Four of the 10 projects we assessed had a comprehensive EVM surveillance system in place. Of the remaining 6 projects, 1 had formal surveillance at the project level but its contractor did not, and 2 projects did not have a formal surveillance system at the project level, only their prime contractors did. The remaining 3 projects did not have any formal surveillance, but provided evidence that EVM data, such as cost and schedule variances, were being reviewed during their monthly status reviews. Beyond reviewing cost and schedule variances and variances at completion, formal surveillance reviews ensure that the processes and procedures continue to satisfy the ANSI/EIA EVM guidelines. A formal surveillance plan involves establishing an independent surveillance organization with members who have practical experience using EVM. This organization then conducts periodic surveillance reviews to ensure the integrity of the contractor’s EVM system and where necessary discusses corrective actions to mitigate risks and manage cost and schedule performance. Effective surveillance ensures that the key elements of the EVM process are maintained over time and on subsequent applications. NASA delegates surveillance of contractor EVM systems to the Defense Contract Management Agency (DCMA); however, NASA has no entity to perform independent surveillance reviews to ensure that the ANSI/EIA- 748 standard is being met for EVM efforts performed in-house or by nonprofit organizations. Without an independent surveillance function, an organization’s ability to use EVM as intended may be hampered since surveillance monitors problems with the performance measurement baseline and EVM data. If the kinds of problems that formal surveillance can identify go undetected, EVM data may be distorted and may not be meaningful for decision making. Only 3 of the 10 projects we reviewed, MAVEN, RBSP, and OCO-2, produced fully reliable data for managing the project and reporting status. The other projects only partially met the criterion to have an EVM system in place that produces reliable data. If done correctly, EVM data can provide an objective means for measuring project status and forecasting potential project cost overruns and schedule slippages so that timely action can be taken to minimize their impact. To do so, however, the underlying EVM data must be reliable, meaning that they are complete and accurate and all data anomalies are explained. In our analysis, we found multiple cases of data anomalies that caused us to question the reliability of the EVM data. For example, we found several EVM reports where a contractor reported that no work was planned or accomplished, but actual costs were incurred without an explanation in the variance analysis report to say why this happened. Additionally, we found cases where a contractor reported that work was planned and actual costs were incurred, but a negative amount of work was performed—work that was previously reported as completed was now reported as not completed. Further, we also found several instances where a project reported an estimate at completion but no budget at completion. Finally, we found instances of negative values in the EVM reports. When explanations were provided in the variance analysis reports, the reasons for these anomalies included use of estimated, rather than actual costs, or adjustments from prior periods due to mistakes or errors with the underlying EVM systems. For example, the SOFIA project said that many of the negative values in its EVM reports were due to over-reporting of earlier progress, mischarges by employees, delayed cost postings, inappropriate use of charge codes. When there are data anomalies such as those we identified for SOFIA, the EVM data can become skewed and can distort true performance. Variance thresholds try to quantify an acceptable range of deviation; those that do not exceed a threshold are usually not a concern while those that do are worthy of further inspection to determine the best course of action to minimize any negative impacts to the cost and schedule objectives. should be minimized and the reason for each should be fully explained in the monthly EVM variance analysis reports. To do less limits the completeness and accuracy of the EVM data and makes the resulting variance determinations unreliable. While an industry standard for what constitutes an acceptable volume of anomalies does not exist, EVM experts in the public and private sectors stated that the occurrence of EVM data anomalies should be rare. For four projects that provided subcontractor EVM data, we tried to map EVM data at the project level to lower level EVM data at the subcontractor level. However, we were only able to successfully map the data for one of the projects. This mapping allows project managers to track cost and schedule by defined deliverables to more precisely identify which components are causing cost or schedule overruns and to more effectively mitigate the root cause of the overruns. When the reports do not allow for traceability, project managers are not able to effectively measure progress, use the reports to monitor and control costs based on the original baseline, and/or to track where and why there were differences. For example, when we attempted to map the EVM data in the lower level reports for MAVEN’s spacecraft, science operations center, remote sensing instrument, and Langmuir Probes and Waves instrument efforts to the overall MAVEN project EVM report, we were not able to see how the costs tracked from one report to another and therefore could not reconcile the costs between the reports. Such issues raise the question of which reports contain the true costs for these efforts. However, MAVEN officials walked us through the process they use to ensure that lower-level reports map to the project level reports. Furthermore, MAVEN officials said they do not mandate that their contractors follow a certain reporting format, instead any adjustments necessary to ensure that the lower-level reports map to the project-level reports are made by the project office. Though the MAVEN project does not prescribe to a standard reporting format, attempting to manually resolve incompatible pieces of data can become time-consuming, expensive and can lead to data reliability issues. We also had similar difficulty mapping various levels of EVM data for the MMS project. The MMS project was able to demonstrate how the Southwest Research Institute (SwRI) budget at completion in the lower- level report mapped to the SwRI budget at completion in the MMS project report, but because of the way the contractor submits their data, project officials said that the two reports will never match. Although the project was able to explain how the data tracked, again, attempting to manually resolve incompatible pieces of data can become time-consuming and can lead to data reliability issues. Since our review, MMS officials said the project is working to capture data at lower WBS levels which will allow for a closer tie between the cost and schedule data. All the projects we reviewed were using some EVM to manage their work. However, the extent to which EVM is implemented across NASA’s in- house projects and their contractors varies by project and center. For example, 3 of the 10 projects we reviewed did not report project level EVM data. Implementing EVM at the project level rather than just for the contract is considered a best practice. In addition, OMB policy requires the use of an EVM system for both in-house and contractor work and when there is both government and contractor work, the data from the two EVM systems must be consolidated at the reporting level for total program management and visibility. Integrating government and contractor cost, schedule, and performance status at the project level should result in better project execution through more effective management. In addition, some of the in-house projects we analyzed were only required to meet EVM principles and gather “EVM-like” data. Further, contractors such as the nonprofit organization managing the MMS instrument suite, do not report its EVM in the standard contract performance report format because they are only required to meet the intent of the standard. Other contractors for the JWST and MAVEN projects are required to provide EVM reports that show cost and schedule data by WBS elements. NASA EVM focal points, headquarters officials, project representatives, and program executives cited cultural and technical challenges, as well as other challenges, as impediments to the effective use of EVM at the agency. NASA’s culture traditionally has focused on solving science and engineering challenges and not on monitoring cost and schedule data such as data produced by an effective EVM system. Technical challenges were also cited as an impediment to effective EVM use, but opinions differed within NASA on the extent of their impact. The technical challenges cited involved difficulty in gathering sufficiently detailed data for timely inclusion and analysis in an EVM system. In addition, though NASA has not conducted an EVM skills gap analysis, NASA representatives said it is a challenge for the agency to implement EVM effectively due to a lack of sufficient staff with the skills and experience to analyze EVM data. Almost half of the more than 30 NASA representatives we interviewed, including a large number of those charged with implementing EVM agencywide—known as EVM focal points—and officials from the Chief Engineer’s office, cited NASA’s culture as a challenge to the effective use of EVM at the agency. Specifically, several NASA representatives said that historically, NASA’s culture has not focused on or valued the kind of information that EVM can highlight. For example, a NASA EVM analyst said the culture has been focused on science and engineering and that accomplishment of that work has been the first priority for managers. Discussion of the cost of the work has been a secondary concern. A NASA EVM focal point described overcoming a culture that is based on applied and basic research rather than discrete tasks with discrete deliverables, which are among the requirements for effective EVM implementation. Further, a senior official at NASA headquarters told us that in project reviews, discussion of cost and schedule information, like EVM data and analysis, tends to be pushed to the very end of the review meeting and generally is not discussed in detail. Because EVM data is not universally valued within NASA, in some cases, the data generated to satisfy a project’s EVM requirement may be of limited use. For example, one NASA EVM focal point said some managers are just “checking the box” with respect to using EVM. The task is performed but the requirement to collect EVM data was viewed as a nuisance that ultimately did not provide worthwhile information. Nonetheless, several of those we interviewed said persistent inquiries about EVM data from senior management at headquarters, especially over the last couple of years, are having a positive impact on the culture and forcing projects to pay more attention to the data. The NASA representatives we interviewed also cited technical challenges as having an impact on the effective use of EVM, although their views varied on the extent of these challenges. For example, about half of the focal points we interviewed reported that a challenge to using an EVM system at NASA was aligning it with the agency’s accounting system, the SAP Core Financial System. One of the problems cited was that EVM data collection may require more detailed data than a project has collected for the agency’s accounting system and this may require the use of estimated costs instead of actual costs. Estimating these costs can create additional work for the project, delay the production of EVM data, and limit the reliability of the EVM data that is produced. Nonetheless, a NASA manager said, projects could more effectively plan their work to better accommodate the accounting system. For example, NASA’s accounting system is set up to measure and report on labor in terms of full-time equivalents. A project, however, may have set up its earned value management system with a different measure for labor, such as productive hours. As a result, the accounting system cannot fill in the proper numbers for an earned value analysis, potentially causing more work for the project. Further, the EVM data could be less accurate due to the use of estimates rather the actual figures. However, if the project had planned from the outset to have the same measure of labor as the accounting system, there would not be a problem having this data fit the EVM system. An Office of the Chief Financial Officer representative did not believe that the projects have consistently demonstrated that the accounting system is a problem, but nonetheless agreed that potential work-arounds and slight changes to processes are potential solutions for these issues. The Office of the Chief Financial Officer has started an initiative to address both the level of detail of the data and improve the monitoring of contractor cost performance at levels that may be lower than levels at which obligations are made and costs are reported in the financial system. A report on a NASA EVM pilot project noted that the greatest impediment for implementing EVM is cultural resistance, not technical challenges. Specifically, it noted that “it’s not the EVM Process. It’s not the EVM Tools. It’s not the SAP Accounting System. It is the NASA culture.” NASA’s use of contracts with one entity to provide goods or services to several different NASA projects was also cited as a challenge to use of EVM. For example, NASA centers may have a contract with one firm to provide engineering support services. Multiple projects may seek services using a single task order on this contract. Because of the way the NASA accounting system is configured, this approach can create artificial variances when looking at EVM data on a month-to-month basis. According to a NASA representative, contractual requirements can correct these issues and allow for a closer accounting of the funds for EVM purposes and in fact, one NASA center has already instituted such requirements in a new contract for services. NASA plans to address this issue as current service contracts expire, but it will take time for the new data requirements that would provide the desired data to be implemented. Other challenges cited include the difficulty of gathering sound EVM data from nonprofit subcontractors, such as universities. One project, for example, reported that EVM data from nonprofit subcontractors were immature or non-existent. The nonprofits may be doing a significant amount of work for a center, but are not equipped to collect earned value data at the level of detail needed, Office of the Chief Financial Officer and center representatives reported. EVM focal points said the problem of collecting EVM data from nonprofits to feed into a larger project-level EVM system could be mitigated through contract language that clearly specifies what data are required from contractors. According to these officials, this kind of language has been included in Jet Propulsion Laboratory contracts and has been successfully demonstrated as a result. Furthermore, NASA is concerned that if nonprofits and small businesses are required to have a fully compliant or certified EVM system, they may not be able to bid on the work. However, the Federal Acquisition Regulation is clear that no offeror can be eliminated from consideration for a contract award because the entity does not have a compliant or certified EVM system. NASA representatives we interviewed said there was a need for improved abilities across the agency to analyze EVM data and implement EVM systems. Specifically, several focal points said the challenge for NASA is not as much in obtaining EVM data because most of the information comes from private contractors responsible for much of NASA’s work, but in having a staff that can analyze the data and integrate it at the project level. A senior NASA official also noted that the career civil servants, who typically are the first level of review for EVM data, do not have background or training in EVM and cannot conduct a sound analysis of the data. A project representative echoed this comment and noted a general awareness of EVM within projects but a shortage of in-depth knowledge to understand EVM fundamentals and how to interpret the data it produces. For example, some projects seek to reset EVM baselines to match funding allocations, which thwarts efforts to examine cost and schedule trends. One EVM focal point told us it has been difficult to convince project managers that EVM can predict what will happen in their projects given the highly technical nature of their work. For example, a senior manager of a program that experienced significant schedule delays and cost overruns stated that he is an “EVM skeptic” and that he does not see EVM data as helpful in helping him track the performance of a project. Additionally, the employee skill sets available to analyze and implement EVM vary widely from center to center, headquarters officials said. In recent years, NASA has provided EVM training to a large number of employees; however, the agency has not conducted a skills gaps analysis, which could help to determine the number and extent of the staff’s EVM expertise. NASA centers may have staff skill levels reflective of the level of EVM use at the center. Some centers have many projects producing EVM data while others may only rarely work on a project that uses EVM. Without a sufficient number of trained staff to analyze contractor data and implement in-house EVM efforts, NASA will likely continue to struggle to effectively use EVM as a valuable project management tool. NASA has undertaken several initiatives aimed at improving the agency’s use of EVM. For example, NASA strengthened its spaceflight management policy to require projects to comply with the 32 ANSI/EIA- 748 guidelines and has developed the processes and tools for projects to meet this requirement through its new EVM system. While these are positive steps, the policy continues to lack a requirement for rigorous oversight or surveillance of how projects are implementing EVM and NASA does not require projects to use the new EVM system to implement the EVM requirement of the revised policy. In addition, the issues that have impeded NASA’s ability to effectively implement EVM, such as its culture, are longstanding and, as a result, NASA has not had much success implementing EVM in the past. The agency’s recent revision of NASA Procedural Requirements 7120.5—the policy that governs NASA’s spaceflight projects and contains project EVM requirements—strengthened the EVM requirements over prior versions of the policy. For example, the revised policy requires all spaceflight projects to demonstrate compliance with each of the 32 EVM guidelines as set forth in ANSI/EIA-748, whereas the prior policy only required projects to comply with seven high-level EVM principles. The new requirements took effect through the release of an interim directive on September 28, 2011 and have since been made final in NASA’s most recent update to 7120.5. As a result, projects meeting EVM reporting thresholds that enter implementation after that date are required to comply with the new requirement. According to an agency official, the Office of the Chief Engineer and the mission directorates will determine which projects that began development under the prior policy must comply with the new EVM requirements. At major milestones, Office of the Chief Engineer representatives will review whether the projects have implemented the 32 EVM guidelines. However, the new policy still only contains the minimum requirements for earned value management, such as the thresholds for implementing EVM and the requirement to comply with ANSI/EIA-748 guidelines. The policy does not require projects to implement formal independent surveillance of their EVM systems. Without effective surveillance, agencies cannot ensure they are meeting the ANSI/EIA-748 guidelines because internal management systems are not being reviewed to determine if they are providing reliable cost, schedule and technical performance data. In addition, effective surveillance helps pinpoint problems, and is useful for verifying the effectiveness of corrective action plans used to mitigate EVM system deficiencies. While projects are not required to implement formal independent surveillance, NASA does plan to conduct periodic surveillance of project EVM systems. For example, NASA plans to conduct EVM assessments at two key decision point life cycle reviews and through the Office of the Chief Engineer compliance surveys.these methods will increase the agency’s surveillance efforts, best practices call for project level surveillance to be an ongoing, continuous process conducted by an independent surveillance function. The policy also does not require projects to use NASA’s new EVM system, although the system was designed to help projects meet the ANSI/EIA-748 guidelines. We found that the system meets the intent of the ANSI/EIA-748 guidelines. Examples of how NASA’s EVM system will satisfy three key ANSI/EIA-748 guidelines are summarized in table 3 below. For the projects required to comply with the new policy, use of the agency-developed EVM system would meet the ANSI/EIA guidelines; however, some projects will be permitted to continue using their individual EVM systems as long as the 32 guidelines are met. According to agency officials, while future revisions to the policy may require use of the standardized agency-developed EVM system by all projects, at this time, the agency does not plan to require projects to use the agency-developed system in order to meet the guidelines. Instead, senior managers will determine on a case by case basis whether a project will use the agency’s new EVM system. Currently, only the Space Launch System and ICESat-2 projects have been selected to implement the new EVM system. According to an agency senior official, the Agency Program Management Council approved a phased rollout of the new system because NASA does not have the resources to implement it agency-wide. For example, there are not enough NASA subject matter experts to provide the support needed by the projects when applying the new EVM system and to build the institutional capability at the centers. Their approach aims to incrementally build the capacity to do EVM, and seek increased acceptance of EVM as the requirement for its use is expanded. Over the years, NASA has attempted to address its EVM shortcomings through a series of policy changes, but these efforts have failed to adequately address the cultural resistance to implementing EVM highlighted by many of the NASA officials we interviewed. NASA has made uneven progress since we reported in 2004 that the agency needed to improve its use of EVM as a project management tool. Furthermore, a 2008 NASA internal study noted that projects needed to be educated on the value and approaches for using EVM and to provide support for setting up EVM within the projects early. Also, an internal agency briefing on EVM stated that a change management initiative would be necessary in order to successfully implement EVM at NASA centers. Our work has also shown that implementing a large-scale initiative, such as EVM, requires more than just policy changes. To see real change and, in effect, a cultural shift at NASA, top leadership must provide to employees a succinct and compelling reason that effective implementation of EVM is important. Articulating a compelling reason for implementing EVM enables employees and other stakeholders to understand the expected outcome of the management initiative and engenders not only their cooperation, but also their ownership of the outcome, which our work has shown can take at least 5 to 7 years to fully implement. NASA, by having a policy that is not comprehensive, allowing projects to opt out of using the new EVM system, and not committing resources to adequately train staff, continues to limit progress in the cultural change needed to implement EVM. Without breaking through the cultural resistance to EVM and committing to efforts intended to strengthen the use of EVM, NASA is missing an opportunity to make full use of a key tool that could help it to manage its projects more effectively. Implementing an effective earned value management system and using it across a large federal agency with well-established processes is without doubt a challenging task. However, NASA has had uneven progress to date. NASA acknowledges that EVM can be a valuable tool for monitoring project development and has initiated an effort to implement an agencywide system. Currently, only a few of the 10 major spaceflight projects we reviewed were able to produce reliable EVM data, raising concern that they cannot produce reliable estimates of cost at completion. Moreover, until the data are sufficiently reliable, NASA, as well as external stakeholders, lose valuable insights into project performance that EVM provides. A sound EVM system is not merely an accounting tool; it can alert managers to developing problems so that they can be proactive in reducing the project’s cost and schedule overruns. However, NASA is not making full use of a key tool that could help it address the cost and schedule issues that have kept NASA acquisition management on GAO’s high risk list for more than 20 years. Though NASA’s recent efforts to improve its EVM capability and strengthen its policy are steps in the right direction, implementation—fully integrating EVM into management processes—has been the biggest challenge and remains so today. NASA faces cultural and technical challenges that it must overcome to successfully implement an earned value system and to use this data on a regular basis to inform decision- making. Managing change will be key if NASA’s latest effort to overcome these challenges and implement an agencywide EVM plan is to succeed. To accomplish effective earned value management, strong leadership is required to set an expectation that reliable and credible data are necessary to manage a successful project. This should be buttressed with a sound EVM policy and system that are required, and a commitment of resources to enable staff. Without sustained momentum and commitment, its current efforts could suffer the same consequence as those in the past. To improve NASA management and oversight of its spaceflight projects, we recommend that the NASA Administrator direct the appropriate offices to take the following four actions: Establish a time frame by which all new spaceflight projects will be required to implement NASA’s newly developed EVM system, unless the project is proposing to use a certified system, to ensure that in- house efforts are compliant with ANSI/EIA-748. The time frame selected should take in to account the need to increase NASA’s institutional capability for conducting EVM and analyzing and reporting the data. Conduct an EVM skills gap analysis to identify areas requiring augmented capability across the agency. Based on the results of the assessment, develop a workforce training plan to address any deficiencies. Develop an EVM change management plan to assist managers and employees throughout the agency with accepting and embracing earned value techniques while reducing the operational impact on the agency. The plan should include a strategy for having the agency’s senior leadership communicate their commitment to implementation of EVM. To improve the reliability of project EVM data, NASA Procedural Requirements (NPR) 7120.5 should be modified to require projects to implement a formal surveillance program that: Ensures anomalies in contractor-delivered and in-house monthly earned value management reports are identified and explained, and report periodically to the center and mission directorate’s leadership on relevant trends in the number of unexplained anomalies. Ensures consistent use of WBSs for both the EVM report and the schedule. Ensures that lower level EVM data reconcile to project level EVM data using the same WBS structure. Improves underlying schedules so that they are properly sequenced using predecessor and successor dependencies and are free of constraints to the extent practicable so that the EVM baseline is reliable. We provided a draft of this report to NASA for comment. In its written comments, reproduced in appendix IV, NASA’s Chief Engineer stated that the agency concurred with two recommendations and partially concurred with two other recommendations. In particular, the agency agreed with our recommendation to perform an EVM skills gap analysis and develop a workforce training plan to address any deficiencies identified. To that end, NASA plans to conduct a skills gap assessment and to augment its EVM training program to address the gaps identified. In addition, the agency also concurred with our recommendation to develop an EVM change management plan and is planning to develop a strategy targeted at all levels of the workforce from project team members to the agency’s leadership. The agency partially concurred with our recommendation that NASA establish a time frame by which all new spaceflight projects will be required to implement NASA's newly developed EVM system, stating that they already require projects to perform EVM with an ANSI/EIA 748 compliant system. NASA stated that its phased rollout approach for implementing the agency’s EVM system is based on available resources, budgetary constraints, and institutional and project needs. However, NASA's approach does not include a timeframe for when projects will be required to use the new system. We recommended that NASA establish a timeframe for rolling out the system to all projects because a large number of projects are not in compliance with NASA’s requirement, and very few are implementing the new EVM system. Using the newly developed EVM system could help projects better ensure NASA's projects are using a system that is compliant with the ANSI/EIA standard. The agency also noted its disagreement with the notion that all projects, in particular those that have a skilled EVM workforce and a compliant system in place, should be forced to use the agency's new system. Accordingly, we acknowledged in our report that there may be situations where a project should not be required to use the agency’s EVM system, such as when a project already uses a certified system or for current, ongoing projects. Furthermore, we reported that NASA lacks the appropriate level of surveillance of its projects’ EVM systems to monitor project adherence to the EVM standard; in addition, the extent to which EVM has been effectively implemented across NASA's projects varies. If NASA chooses not to require projects to use its new system it should take steps to ensure that it monitors their compliance with the standard. Finally, while we appreciate that NASA must balance its resources with its needs, the benefits that an effective EVM system can provide, such as allowing project managers to identify cost growth and take actions to stem further growth, warrants prioritization of resources to ensure earlier widespread implementation of NASA's EVM system. The agency also partially concurred with our recommendation that NPR 7120.5 be modified to require projects to implement a formal EVM surveillance program. Citing resource constraints, NASA commented that it does not plan to implement a formal surveillance program, but agreed that the reliability and utility of the EVM data needed to be improved. As a result, the agency plans to establish a surveillance process, expand the workforce’s EVM skills, and provide analytical tools including developing an EVM System Acceptance and Surveillance Guide. Furthermore, NASA said that it was not appropriate to incorporate the surveillance requirement in NPR 7120.5 because of the level of detail associated with requirements in that policy. The most important part of our recommendation is that EVM surveillance should be required to ensure better quality data. We reported that only 4 of the 10 projects we assessed had a comprehensive EVM surveillance system in place and the others had limited or no surveillance being performed and only 3 of the 10 projects had fully reliable data. Without an effective surveillance program, NASA cannot ensure its projects are meeting the ANSI/EIA-748 standard because internal management systems are not being reviewed to determine if they are providing reliable cost, schedule and technical performance data. In its response, NASA also noted that the project data we used in our report is over a year old and does not take in to account progress made over the past year. We disagree and note that in the report we discuss progress the agency has made over the past year, such as strengthening the EVM requirements in its policy and developing its new EVM system. Furthermore, we did not solely rely on project EVM data to develop our findings. For example, interviews with project officials and additional documentation they provided further validated our findings. Finally, it is important to note that NASA Acquisition Management has been on GAO’s High Risk list for many years due to the agency’s cost and schedule performance on its major projects. EVM is an important project management tool that can assist project managers in managing and assessing performance. Not addressing key issues that impact the availability of accurate and reliable data could lessen the usefulness of this key project management tool. NASA also provided technical comments, which have been addressed in the report, as appropriate. We are sending copies of this report to interested congressional committees, NASA’s Administrator, and other interested parties. In addition, this report will also be available at no charge on the GAO website at http://www.gao.gov. Should you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. To determine to what extent NASA’s major spaceflight projects are using earned value management (EVM) to manage acquisitions, we reviewed all NASA major spaceflight projects with a life cycle cost of over $250 million that were in the implementation phase and thus required to report EVM. There were 13 projects that met these criteria. Of these, 2 projects had recently launched and the launch of a 3rd was imminent. These 3 were excluded from our assessment because the work on these projects was nearly complete. Collectively, the 10 projects we reviewed will cost over $14 billion to develop. Our review looked at EVM data for the period of August 2010 to August 2011. While the majority of the 10 projects we reviewed had at least 6 months of EVM data, a few did not because the project had only recently entered the implementation phase. Additionally, some projects were undergoing a re-plan and, therefore, were not required to provide EVM data for certain periods of time. In particular, the Tracking and Data Relay Satellite Sustainment contract had 5 months of data, the Mars Atmosphere Volatile Evolution Mission project had only 3 months of data, and the Orbiting Carbon Observatory 2 Orbital contract had just 4 months of data. The James Webb Space Telescope contract had EVM data for the whole period; however, the contractor underwent a major replan in which all EVM data except for the reporting of actual costs were suspended from January 2011 to April 2011. Although the Global Participation Measurement (GPM) project provided EVM reports for the entire project, we did not conduct an analysis of the project EVM data, because the performance reports did not contain the detailed data we needed for our analysis. However, we were able to assess the performance data for GPM’s Microwave Imager Instrument. To determine cost and schedule performance for the selected major projects based on an evaluation of the earned value data, we analyzed project and contractor data and documentation including contract performance reports; project work breakdown structures; project schedules; integrated baseline review briefings; the extent to which surveillance of the EVM system was occurring; and monthly management briefings for the 10 major spaceflight projects. Specifically, we compared project documentation with EVM and scheduling best practices as identified in GAO’s Cost Estimating and Assessment Guide and Schedule Assessment Guide. To the extent practicable, we assessed how each of the 10 projects’ EVM data adhered to 3 of the American National Standard Institute’s (ANSI) and Electronic Industries Alliance (EIA) 32 guidelines. In addition, we assessed the projects against 3 fundamental EVM practices that we believe are necessary for maintaining a reliable EVM system, as identified in our cost guide. We also analyzed the contract performance reports for each project to determine the level of data reliability. Specifically, we identified instances of the following: (1) negative planned value, earned value, or actual cost; (2) planned value and earned value without actual cost; (3) earned value and actual cost without planned value; (4) actual cost without planned value or earned value; (5) earned value without planned value and actual cost; (6) inconsistencies between the estimated cost at completion and the planned cost at completion; (7) actual cost exceeding estimated cost at completion; and (8) planned or earned values exceeding planned cost at completion. For the contracts that had more than 6 months of data, we used contract performance report data in order to generate our estimated overrun or underrun of the contract cost at completion by using formulas accepted by the EVM community and printed in the GAO Cost Estimating and Assessment Guide. To perform this analysis, we examined contractor performance reports over the period for which we had data to show trends in cost and schedule performances. We generated multiple formulas for the projected contract cost at completion that were based on how much of the contract had been completed up to August 2011 or earlier for some projects. The ranges in the estimates at completion are driven by using different efficiency indices based on the project’s past cost and schedule performance to forecast the cost of the remaining work and adding that cost to the actual costs to date. The efficiency indices capture how the project has performed in the past and can be useful in predicting how it will perform in the future. We also analyzed monthly project management review briefings to support our analysis. Finally, we analyzed the earned value data contained in EVM performance reports obtained from the projects. To perform this analysis, we compared the cost of work completed with budgeted costs for scheduled work to show trends in cost and schedule performances. To assess the reliability of the cost data, in addition to electronically testing the data for anomalies, we also reviewed relevant project documentation and interviewed agency and project officials about the data. We then followed up on these anomalies with the project offices that manage each of the spaceflight projects by sharing our preliminary analysis for each of the 10 projects. When warranted, we updated our analyses based on the agency’s response and additional documentation provided to us. The data that we used were sufficiently reliable for how we portrayed them in our report and we are making recommendations to the agency to improve NASA’s data reliability based on the findings discussed in our report. We did not test the adequacy of the agency or contractor accounting systems. To support and clarify information in our documentation reviews, we interviewed agency officials at NASA headquarters and EVM Focal Point Working Group members—the agency officials that are responsible for developing an integrated, consistent approach for implementing EVM throughout NASA, as well as addressing EVM review and surveillance issues and activities—at each center and the Human Exploration and Operations and Science mission directorates to discuss their roles as well as the extent to which EVM data is used to inform decision making. We interviewed officials at NASA headquarters in Washington, D.C.; and officials from Ames Research Center in Moffett Field, California; Dryden Flight Research Center in Edwards, California; Glenn Research Center in Cleveland, Ohio; Goddard Space Flight Center in Greenbelt, Maryland; Johnson Space Center in Houston, Texas; the Jet Propulsion Laboratory in Pasadena, California; Kennedy Space Center in Florida; Langley Research Center in Hampton, Virginia; Marshall Space Flight Center in Huntsville, Alabama; and Stennis Space Center in Mississippi. Additionally, we received responses to questions regarding the EVM data from each of the 10 projects we selected for review. These questions addressed how EVM practices are implemented at the project level and how the project utilizes EVM data. To determine the challenges that NASA has faced in implementing an effective EVM system, we interviewed NASA headquarters personnel to discuss the status and plans for implementing the agency-wide EVM system. We developed a standard set of questions and interviewed EVM Focal Point Working Group members at each center and the Human Exploration and Operations and Science mission directorates to assess the challenges of implementing EVM at individual centers and across the agency. We also interviewed a selection of senior officials and program executives at NASA headquarters that represent projects from each mission directorate and NASA center included in our engagement to obtain their perspective on the challenges of implementing and using EVM on their projects. We also reviewed prior GAO and NASA Inspector General reports that discuss the agency’s prior efforts to implement EVM. We examined GAO and NASA Inspector General reports that discuss the importance of effective organizational change. Additionally, we received written responses to a standard set of questions regarding the challenges associated with implementing EVM from each of the 10 projects we selected for review. To determine the steps that NASA is taking to improve its use of earned value management, we examined the results of NASA’s EVM capability pilot projects and draft policies and guidance and compared these with best practices in EVM as discussed in GAO’s Cost Estimating and Assessment Guide, the ANSI/EIA-748 standard, and OMB Circular A-11, Preparation, Submission, and Execution of the Budget and the Capital Programming Guide. In addition, we interviewed NASA headquarters personnel and EVM Focal Point Working Group members at each center and the Human Exploration and Operations and Science mission directorates to discuss and obtain information on ongoing initiatives the agency has undertaken. We conducted this performance audit from June 2011 to November 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Pulling together essential cost, schedule, and technical information in a meaningful, coherent fashion is always a challenge for any project. Without this information, management of the project will be fragmented, presenting a distorted view of project status. For several decades, the Department of Defense (DOD) has utilized a tool called earned value management (EVM) to compare the value of work performed to the work’s actual cost. Earned value management goes beyond the two- dimensional approach of comparing budgeted costs to actual costs. It attempts to compare the value of work accomplished during a given period with the work scheduled for that period. By using the value of completed work as a basis for estimating the cost and time needed to complete the project, the earned value concept should alert project and senior managers to the potential problems early in the project. In 1996, DOD adopted 32 criteria for evaluating the quality of earned value management systems. These 32 criteria are organized into 5 basic categories: organization, planning and budgeting, accounting considerations, analysis and management reports, and revisions and data maintenance. In general terms, the criteria require contractors to define the contractual scope of work using a work breakdown measure the progress of work based on objective indicators; structure; identify organizational responsibility for the work; integrate internal management subsystems; schedule and budget authorized work; collect the cost of labor and materials associated with the work performed; analyze any variances from planned cost and schedules; forecast costs at contract completion; and control changes. The criteria have evolved to become an American National Standards Institute (ANSI) and Electronic Industries Alliance (EIA) standard for EVM, which has been adopted by major U.S. government agencies, industry, and the governments of Canada and Australia. The full application of EVM system criteria is appropriate for large-cost reimbursable contracts where the government bears the cost risk. For such contracts, the management discipline described by the criteria is essential. In addition, data from an EVM system have been proven to provide objective reports of contract status, allowing numerous indices and performance measures to be calculated. These can then be used to develop accurate estimates of anticipated costs at completion, providing early warning of impending schedule delays and cost overruns. The standard format for tracking earned value is through a Contract Performance Report (CPR). The CPR is a monthly compilation of cost, schedule and technical data which displays the performance measurement baseline, any cost and schedule variances from that baseline, the amount of management reserve used to date, the portion of the contract that is authorized unpriced work, and the contractor’s latest revised estimate to complete the project. As a result, the CPR can be used as an effective management tool because it provides the project manager with early warning of potential cost and schedule overruns. Using data from the CPR, a project manager can assess trends in cost and schedule performance. This information is useful because trends can be difficult to reverse. Studies have shown that once projects are 15 percent complete, the performance indicators are indicative of the final outcome. For example, a CPR showing a negative trend for schedule status would indicate that the project is behind schedule. By analyzing the CPR, one could determine the cause of the schedule problem such as delayed flight tests, changes in requirements, or test problems because the CPR contains a section that describes the reasons for the negative status. A negative schedule condition is a cause for concern, because it can be a predictor of later cost problems since additional spending is often necessary to resolve problems. For instance, if a project finishes 6 months later than planned, additional costs will be expended to cover the salaries of personnel and their overhead beyond what was originally expected. CPR data provides the basis for independent assessments of a project’s cost and schedule status and can be used to project final costs at completion in addition to determining when a project should be completed. Examining a project’s management reserve is another way that a project can use a CPR to determine potential issues early on. Management reserves, which are funds that may be used as needed, provide flexibility to cope with problems or unexpected events. EVM experts agree that transfers of management reserve should be tracked and reported because they are often problem indicators. An alarming situation arises if the CPR shows that the management reserve is being used at a faster pace than the project is progressing toward completion. For example, a problem would be indicated if a project has used 80 percent of its management reserve but only completed 40 percent of its work. A project’s management reserve should contain at least 10 percent of the cost to complete a project so that funds will always be available to cover future unexpected problems that are more likely to surface as the project moves into the testing and evaluation phase. An Integrated Baseline Review (IBR) is conducted to ensure the reliability of the EVM data and that the performance measurement baseline accurately captures all the work to be accomplished. Data from the CPR can then be used to assess project status—typically, monthly. Cost and schedule variances are examined and various estimates at completion are developed and compared to available funding. The results are shared with management for evaluating contractor performance. Finally, because EVM requires detailed planning for near-term work, as time progresses, planning packages are converted into detailed work packages. This cycle continues until all work has been planned and the project is complete. An IBR is an evaluation of the performance measurement baseline to determine whether all project requirements have been addressed, risks identified, and mitigation plans put in place and all available and planned resources are sufficient to complete the work. Too often, projects overrun because estimates fail to account for the full technical definition, unexpected changes, and risks. Using poor estimates to develop the performance measurement baseline will result in an unrealistic baseline for performance measurement. After the CPRs start being delivered to the government, it is important to oversee the project by conducting surveillance of the EVM system. Surveillance is reviewing a contractor’s EVM system as it is applied to one or more projects. Its purpose is to focus on how well a contractor is using its EVM system to manage cost, schedule, and technical performance. For instance, surveillance checks whether the contractor’s EVM system summarizes timely and reliable cost, schedule, and technical performance information directly from its internal management system; complies with the contractor’s implementation of ANSI/EIA-748 guidelines; provides timely indications of actual or potential problems by performing spot checks, sample data traces, and random interviews; maintains baseline integrity; gives information that depicts actual conditions and trends; and provides comprehensive variance analyses at the appropriate levels, including corrections for cost, schedule, technical, and other problem areas. Effective surveillance ensures that the key elements of the EVM process are maintained over time and on subsequent applications. EVM system surveillance ensures that the contractor is following its own corporate processes and procedures and confirms that the contractor’s processes and procedures continue to satisfy the ANSI guidelines. The surveillance team designated to perform project reviews should consist of a few experienced staff who fully understand the contractor’s EVM system and the processes being reviewed. The surveillance organization should appoint the team leader and ensure that all surveillance team members are independent. This means that they should not be responsible for any part of the projects they assess. Key activities on the surveillance team’s agenda include reviewing documents, addressing government project office concerns, and discussing prior surveillance findings and any open issues. Sufficient time should be allocated to all these activities to complete them. The documents for review should give the team an overview of the project’s implementation of the EVM process. Successful surveillance is predicated on access to objective information that verifies that the project team is using EVM effectively to manage the contract and complies with company EVM procedures. Objective information includes project documentation created in the normal conduct of business. Besides collecting documentation, the surveillance team should interview control account managers and other project staff to see if they can describe how they comply with EVM policies, procedures, or processes. During interviews, the surveillance team should ask them to verify their responses with objective project documentation such as work authorizations, cost and schedule status data, variance analysis reports, and back-up data for any estimates at completion. We conducted case studies of 10 major spaceflight system acquisition projects. This appendix provides a brief description of each project, including an analysis of the project’s earned value data and trends. As part of our analysis, we assessed the projects’ implementation of three fundamental earned value management (EVM) practices that we believe are necessary for maintaining a reliable EVM system—using a certified American National Standards Institute (ANSI) and Electronic Industries Alliance (EIA) compliant system, performing surveillance, and conducting integrated baseline reviews. We also assessed the projects’ EVM data against three ANSI and EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a work breakdown structure (WBS) that has been tailored to the project and that the WBS is the same for the cost estimate, schedule, and EVM. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. As mentioned above, this appendix includes an analysis of each project’s earned value trends from August 2010 to August 2011. These data and trends are often described in terms of cost and schedule variances. Cost variances compare the earned value of the completed work with the actual cost of the work performed. Schedule variances are also measured in dollars, but they compare the earned value of the completed work with the value of the work that was expected to be completed. Positive variances are good—they indicate that activities are costing less than expected or are completed ahead of schedule. Negative variances are bad—they indicate activities are costing more than expected or are falling behind schedule. Variances are merely measures that indicate that work is not being performed according to plan and that it must be assessed further to understand why. Although our EVM cost projections may show that a project is experiencing negative cost variances and schedule slippages, this does not mean that a project has exceeded its agency baseline commitment and will require additional funds to complete the project. These estimates use a project’s EVM baseline, which represents only a portion of the agency baseline commitment for a project. The EVM baseline contains the cost and schedule contained in a project’s management agreement minus unallocated future expenses and schedule margin held by the project and others above the project. As of August 2011, the budget at completion for the 10 projects was estimated to be $6.4 billion. To estimate the project variance at completion, we examined the trends for the earned value data for the entire project, if data was collected at that level, or elements of the project. Table 4 provides a summary of the projects’ implementation of each EVM best practice we assessed and projected costs. In October 2011, the Global Precipitation Measurement mission was approved for a replan. The contract for the Science Operations Center, Remote sensing Package, Langmuir Probe and Waves Instrument does not exceed $50 million. Therefore, the supplier is not required to have a certified system for this contract. Orbiting Carbon Observatory 2 is in the process of being rebaselined due to a change in the launch vehicle. With timely and effective action taken by project and executive management, it is possible to reverse negative performance trends so that the projected negative cost variances at completion may be reduced. To get such results, management needs to obtain reliable EVM data from EVM systems that adhere to the ANSI/EIA-748 standard for informed decision making. Until project offices undertake a rigorous validation of their EVM data, NASA faces an increased risk that managers may not be receiving the information they need to effectively manage their projects. The following information describes the key that we used in tables 5 through 14 to convey the results of our assessment of the 10 case study projects’ implementation of EVM practices. The Global Precipitation Measurement (GPM) mission, a joint NASA and Japan Aerospace Exploration Agency (JAXA) project, seeks to improve the scientific understanding of the global water cycle and the accuracy of precipitation forecasts. The GPM is composed of a core spacecraft carrying two main instruments: a Dual-frequency Precipitation Radar and a GPM Microwave Imager (GMI). GPM builds on the work of the Tropical Rainfall Measuring Mission, and will provide an opportunity to calibrate measurements of global precipitation when it launches in 2014. This analysis focuses only on the GMI-1 effort. Ball Aerospace and Technology Company is the prime contractor for GMI. GMI’s current contract value is $217 million, which represents approximately 23 percent of the total GPM project budget of $932.8 million. The GMI instrument was delivered to Goddard Space Flight Center in February 2012 for integration into NASA’s upcoming Earth science spacecraft. All remaining effort for GMI-1 is post delivery support, which is all level of effort. The GPM project provided EVM reports for the entire project, but we did not conduct an analysis of the project EVM data because the performance reports did not contain the detailed data we needed for our analysis. The GMI-1 contractor met the three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. The Defense Contract Management Agency (DCMA) certified that the GMI-1 contractor’s EVM system complied with the ANSI/EIA standard in April 2008. Though the contractor has a certified system, the implementation of that system is questionable based on our findings below. We assessed GMI-1’s EVM data against three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. Our review found that the WBS in the GMI- 1 schedule did not match the WBS used for the EVM data. Project officials said that the WBS for the project schedule did not match the WBS used in GMI contractor’s EVM reports because GMI-1 is only one element of the total project and the project level schedule has a simplified summary of the GMI-1 schedule that was used for completeness. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review of the GMI-1 schedule found some sequencing issues. For example, about 5 percent of the remaining activities were missing predecessor and successor links, which are necessary for properly sequencing work so that the schedule will update in response to changes. We also found that 19 percent of the remaining activities had date constraints, which also hinder the schedule’s ability to respond dynamically to status updates resulting in an artificial or unrealistic view of the project plan. These sequencing issues and constraint dates within the schedule affect the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. Project officials agreed with our findings but said that these issues are corrected as they are discovered, so there is no impact to the project. Finally, though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that the GPM schedule was not resource loaded, it is GAO’s assessment that the project did not show evidence that it had established and maintained a time-phased budget baseline. The project conducted an integrated baseline review (IBR) in January 2006. From that review, officials believed the results were less than satisfactory due to the contractor’s inability to demonstrate the integration of contract schedule and cost in accordance with their EVM system description. In particular, 64 areas of concern were identified during the IBR, including major concerns with data continuity, cost/schedule risk, and EVM processes. As a result of these issues and changes to the contract another IBR was performed in January 2011. Joint surveillance reviews of the EVM data are being performed by DCMA and the contractor. According to DCMA, although they have found some deficiencies in the contractor’s EVM data, the contractor has responded with acceptable corrective action plans. We reviewed contract performance reports from August 2010 to July 2011. Our review of the GMI-1 EVM data found various data anomalies that call into question the reliability of the data. For example, we found negative values for EVM data without any explanation in three monthly reports. Project officials responded that the negative values all fell within the contract threshold and therefore the GMI-1 contractor did not need to provide an explanation in the variance analysis report. In addition, there were many instances of costs and performances being recorded when no work had been scheduled. In response to this finding, project officials explained that this work had been accounted for in previous months, which explain the missing values. Anomalous EVM data prevents the project from being able to gain meaningful and proactive insight into potential cost and schedule performance shortfalls, and take corrective action to avoid shortfalls in the future. Figure 2 below illustrates that as of July 2011, the project was reporting a negative cumulative cost variance of $13 million and a negative cumulative schedule variance of $3 million. GPM project officials said the cumulative negative cost and schedule variances were due to slips caused by suppliers not delivering flight hardware as planned, which pushed uncompleted work into the future. The negative variances were also caused by tasks being worked that were not included in the baseline. Officials also said the project experienced unfavorable variances in labor costs across all integrated project teams, which were further affected by the unfavorable 2010 year end indirect rate adjustment.officials said the contractor increased staffing and added extra shifts, To address the negative schedule variance, which increased labor costs thereby increasing the contract value. Officials noted, however, that the increase in contract value did not translate into an increase in the baseline, just an increase in the project funding. Due to both the negative cost and schedule variances, we are forecasting a negative variance ranging from $14 million to $22 million at contract completion. According to NASA, the project is not overrunning its approved baseline commitment, because the EVM baseline does not include unallocated future expenses held at the project and headquarters level. The James Webb Space Telescope (JWST) is a large, infrared-optimized space telescope that is designed to find the first galaxies that formed in the early universe. Its focus will include searching for first light, assembly of galaxies, origins of stars and planetary systems, and origins of the elements necessary for life. Scheduled to launch in October 2018, JWST’s instruments will be designed to work primarily in the infrared range of the electromagnetic spectrum, with some capability in the visible range. JWST will have a large primary mirror composed of 18 smaller mirrors, measuring 6.5 meters (21.3 feet) in diameter, and a sunshield that is the size of a tennis court. A successor to the Hubble Space Telescope and the Spitzer Space Telescope, JWST will reside in an orbit about 1 million miles from the Earth. NASA rebaselined JWST in September 2011 and made changes in the project’s management in response to cost and schedule performance issues and the recommendations of the Independent Comprehensive Review Panel report. As part of the rebaseline, NASA took the lead role for systems engineering from the prime contractor. The telescope, along with a segmented primary mirror, will deliver infrared light to the Fine Guidance Sensor and fine pointing updates to the Observatory and four scientific instruments including the Near-Infrared Camera (NIRCam), the Near-Infrared Spectrograph, the Mid-Infrared Instrument and the Fine Guidance Sensor/Near InfraRed Imager and Slitless Spectrograph. For work being performed by its international partners, such as the Near- Infrared Spectrograph, EVM data is not collected. At the time of our review, there was no EVM data for the overall JWST project or for the work done in-house on the Integrated Science Instrument Module. According to project officials, the Integrated Science Instrument Module effort began over a decade ago, and significant parts of the project, particularly those undertaken in-house at GSFC were not structured to enable EVM to be implemented easily. However, the JWST project office is collecting EVM data from Northrop Grumman Aerospace Systems and obtains copies of the Lockheed Martin Space Systems- Advanced Technology EVM data, which is the University of Arizona’s prime contractor. Northrop Grumman Aerospace Systems is responsible for developing and launching the JWST Observatory, which comprises the spacecraft, sunshield, and the optical telescope element, systems integration and test observatory verification, observatory commissioning, and ground and launch support equipment. The University of Arizona is responsible for developing the Near-Infrared Camera science instrument. The observatory contractor met all three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. For the observatory portion of the JWST project, Northrop Grumman Aerospace Systems has a certified EVM System, which it uses to fulfill the earned value reporting requirement. Though the contractor has a certified system, the implementation of that system is questionable based on our findings below. We assessed the observatory contractor’s EVM data against three fundamental ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. We found that while consistent naming conventions existed between the WBS and contract performance reports, there were discrepancies in the hierarchical structure and numbering of the WBS elements. For example, the WBS dictionary shows the Mission Assurance listed as 3.0, while the contract performance report provided by the observatory contractor has Mission Assurance listed as 2.0. NASA officials responded that the observatory contractor is not required to follow the project level WBS hierarchical structure nor the WBS numbering scheme. They further stated that the project and the observatory WBS structures are not identical because procurement of the observatory is only one element of the overall JWST project. NASA did not provide us with a schedule in a format that would allow us to determine if the schedule had the proper sequencing in place. As a result, we were unable to determine if significant task interdependencies such as predecessor and successor links were in place to ensure that the schedule would update in response to changes. In addition, because we could not view the schedule in its native software, we were unable to determine if the schedule was resource loaded, which is a best practice for establishing and maintaining a time-phased budget baseline. An Integrated Baseline Review was conducted in February 2010; however, because of the rebaseline in September 2011, according to the project an additional IBR was held for the observatory in October 2012. Surveillance is being performed by DCMA. In addition, EVM data is reviewed monthly by multiple individuals on the project office as well as at higher levels of NASA headquarters and the Goddard Space Flight Center. We reviewed contract performance reports from August 2010 to July 2011. Our review of the Observatory EVM data found various data anomalies that call into question the reliability of the data. For example, we found actual costs recorded without any work being performed, inconsistencies between the reported estimate at completion and budget at completion, large month to month performance swings, and unexplained variances. NASA officials explained that during this time period the observatory contractor was engaged in re-planning efforts so NASA did not want them to expend resources reporting performance management data to an outdated performance measurement baseline that did not reflect the new rebaseline assumptions. Further, while variances analyses were provided in the variance analysis reports for WBS elements that exceeded contractual thresholds, there was no explanation for the anomalies we found. A variance analysis report provides a detailed, narrative report explaining significant cost and schedule variances and other contract problems and topics. Without this information, management cannot understand the reasons for the variances and the contractor’s plan for fixing them. When information is missing in a variance analysis report, the EVM data will not be meaningful or useful as a management tool. As of July 2011, the observatory portion of the JWST Project was 51 percent complete with a positive cumulative cost variance of $2.4 million. For the same period the project was also experiencing a positive cumulative schedule variance of $0.9 million as seen in figure 3 below. In January 2011, the observatory contractor began replanning the remaining effort in order to meet the October 2018 launch readiness date. As a result, NASA suspended performance measurement reporting during the period of January 2011 through April 2011. However, the observatory contractor was still required to submit contract performance reports depicting actual cost and estimate at completion data. Since there were only 3 months of EVM data after the rebaseline, we were not able to forecast a variance at completion. Since our assessment, JWST project officials said they have made some significant improvements in implementation and use of EVM that includes an EVM approach for the in-house work that will provide EVM metrics to measure progress. In addition, the officials said the project is also doing managerial analysis on its contracts and project components and producing independent estimate of completion each month based on the EVM data. The Near-Infrared Camera (NIRCam) contractor, Lockheed Martin, met one of the three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. The contractor has a certified EVM System that it is using to report EVM data for NIRCam. Our review found similar problems with the WBS hierarchical structure and numbering of the elements. For example, the WBS dictionary shows Mission Assurance listed as 3.0 while the NIRCam contract performance report shows this effort under element 5.5. NASA officials explained that the NIRCam contractor is not required to follow the project level WBS hierarchical structure or numbering of elements. As stated above, we did not receive a schedule in its native software, so we were unable to confirm whether the schedule was sequenced using predecessor and successor links or if it was resource loaded, necessary for establishing the time-phased budget baseline. As stated above, while the JWST project conducted an IBR in February 2010, the rebaseline necessitated a new IBR, which occurred in March 2012. While formal surveillance is not occurring for the Lockheed Martin EVM data, monthly reviews of the EVM data are performed by both the project and program offices and by independent groups. As part of these reviews, trending metrics are prepared and presented to management as part of the internal project reviews and monthly status reviews. We reviewed contract performance reports from August 2010 through July 2011 and found various data anomalies that call into question the reliability of the data. For example, we found EVM data with negative values, no work scheduled but work performed, and actual costs being incurred without any work being performed. NASA officials stated that variances that exceeded contractual thresholds should be reflected in the variance analysis reports, however many of these anomalies did not breach a variance threshold so the report provided no explanation. In addition, NASA officials explained that many of these anomalies occurred during the project replan, which was formally approved in September 2011, when the project was rebaselined. As a result of the replan, NASA suspended EVM data reporting, which resulted in many of the anomalies we found. As of July 2011, the NIRCam portion of the JWST project was 98 percent complete. In July 2011, there was a negative cumulative cost variance of $33 million and a negative cumulative schedule variance of $4.4 million as seen in figure 4 below. The reasons for the downward trend reflected in the graph below were not explained because variance analysis reporting was suspended during the replan period. Based on the downward trend, we are forecasting a negative variance at completion ranging anywhere from $34 million to $48 million. This analysis is based on information prior to the project’s 2011 replan. The variances projected do not take in to account the establishment of a new EVM baseline as a result of the replan. The Landsat Data Continuity Mission (LDCM) is a joint mission between NASA and the United States Geological Survey (USGS) that seeks to extend the ability to detect and quantitatively characterize changes on the global land surface at a scale where natural and man-made causes of change can be detected and differentiated. It is the successor mission to Landsat 7. The Landsat data series, begun in 1972, has provided the longest continuous record of changes in the Earth’s surface as seen from space. Landsat data is a resource for people who work in agriculture, geology, forestry, regional planning, education, mapping, and global change research. The LDCM provides remotely sensed, highly calibrated, moderate resolution, multispectral imagery affording systematic global coverage of the Earth’s land surfaces on a seasonal basis and makes the data readily available for large-scale and long-term Earth System Science and Land use/land cover change research and management. The project plans to launch early in February 2013. LDCM consists of an Operational Land Imager (OLI) and a Thermal Infrared Sensor (TIRS) science instrument, a spacecraft, and a mission operations element. LDCM does not collect EVM data at the project level. The decision not to perform EVM at the project level was reviewed extensively prior to proceeding into the design and development phase, according to project officials. Also, there is no EVM data for the spacecraft effort because this work is being done under a firm fixed price contract and NASA regulations do not require EVM for firm fixed price contracts. The TIRS instrument, built in-house at NASA’s Goddard Space Flight Center, was added late in the formulation phase with an aggressive delivery schedule and delivered in February 2012. The Ground System is being built and delivered by the USGS. The developmental part of the OLI contract with Ball Aerospace and Technology Corporation was completed with delivery of the instrument in early October 2011. Following on-orbit checkout, the contract will transfer to USGS for management. The OLI instrument is the only part of the project that performed EVM. The OLI contractor met two of the three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. In 2007, the Defense Contract Management Agency (DCMA) certified Ball Aerospace and Technology Corporation’s EVM system. Though the contractor has a certified system, the implementation of that system is questionable based on our findings below. We assessed the contractor EVM data against three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. Our review found that the WBS in the OLI schedule was consistent with the WBS used for the EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our analysis found 11 percent of the remaining activities were missing dependencies, 13 percent had lags, and 24 percent had constraints, among other things. When schedules are not sequenced properly, float values and the calculated critical path will not be valid. Project officials said the schedule sequencing is driven by external forces such as facilities availability, spacecraft and ground system interfaces, DCMA inspections, and so forth. The effects of these forces on schedule sequencing and critical path are reviewed extensively and the validity of the critical path is not typically an issue for project management. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guideline, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. Our analysis found that the OLI schedule was resource loaded. The project conducted an IBR in 2007. The IBR identified 80 areas of concern and as a result the LDCM Project did not accept the Performance Measurement Baseline at the IBR. However since 2007, Ball Aerospace and Technology Corporation has addressed the areas of concern. Joint surveillance reviews are being conducted on Ball Aerospace and Technology Corporation’s EVM system by DCMA and the Defense Contract Audit Agency. We reviewed contract performance reports from August 2010 to August 2011. While the EVM data reflected several data anomalies, Ball Aerospace provided detailed explanations for each of them. For example, negative values were attributed to year end rate savings, labor corrections, or material transfers that were greater than the current month actual costs which resulted in a negative number. The EVM data assessed below reflects all work associated with the LDCM OLI instrument. Figure 5 below illustrates that as of August 2011, the project was a reporting a negative cumulative cost variance of approximately $46 million and a negative cumulative schedule variance of $1.3 million. The negative cumulative cost and schedule variances were due to various technical challenges experienced during instrument development, including detector fabrication issues, coatings issues that necessitated the build of a second calibration subassembly, and instrument integration onto the baseplate taking two weeks longer than expected to complete fabrication. Due to the negative variances, we are forecasting a negative variance at completion ranging from $49 million to $52 million. According to a project official, the project is not overrunning its agency baseline commitment, because the EVM baseline does not include unallocated future expenses held by the project or NASA headquarters. The developmental part of the OLI instrument contract with Ball Aerospace and Technology Corporation was completed with delivery of the instrument in early October 2011. The Lunar Atmosphere and Dust Environment Explorer (LADEE) will determine the global density, composition, and time variability of the lunar atmosphere. LADEE’s measurements will determine the size, charge, and spatial distribution of electrostatically transported dust grains. Additionally, it will carry an optical laser communications demonstrator that will test high-bandwidth communication from lunar orbit. Finally, it will broaden the scientific understanding of other planetary bodies regarding exospheres or very thin atmospheres, like the moon. LADEE met only one of three key fundamental ANSI/EIA-748 practices for reliable EVM system. NASA’s EVM guidance says that projects must start reporting EVM data once the project enters the project implementation phase, if the project’s life-cycle cost is at or greater than $20 million. While LADEE may not have a contract that exceeds $20 million, the overall LADEE project cost is about $262.9 million. Nevertheless, project officials said LADEE is responsible only for gathering “EVM-like” data at the project level. The “EVM-like” data is collected using the “EVM Lite” process, which attempts to meet the ANSI/EIA-748 standard where applicable. Project officials said that when LADEE was initiated in February 2008, NPR 7120.5D was still in effect which required application of EVM principles. This is, in effect, “EVM-Lite” or “EVM-Like.” Officials further stated that prior to August 2011, the LADEE project evaluated candidate EVM techniques using sample data from January to March 2011. Based on that evaluation period, LADEE decided to use the “EVM Lite” technique to collect the necessary data to manage the project. From April to June 2011, additional evaluations of this technique continued. Therefore, standard reporting of the LADEE “EVM-like” project level data did not begin until August 1, 2011. We reviewed all available EVM data from September 2010 to June 2011, which was before the standard EVM reporting period began. Though LADEE does not have a certified system, we assessed how well LADEE project was meeting three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS tailored to the project and the WBS should be the same for the schedule, cost estimate, and EVM. Our analysis found that the WBS in the LADEE schedule matched the WBS used for EVM data. The ANSI/EIA guidelines also states that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifies significant task interdependencies required to meet project requirements. Our review of the LADEE schedule found 3 percent of the remaining activities were missing predecessor or successor links, which cause the schedule to not properly update in response to changes. We also found that about 6 percent of the remaining activities had date constraints and/or lags, which also hinder the schedule from responding dynamically to changes and can portray an artificial or unrealistic view of the project plan. While these issues may be relatively small, any missing dependencies, constraints, and lags may disrupt the reliability of the overall network. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that the LADEE schedule was not resource-loaded, therefore, it is GAO’s assessment that the project did not show evidence that it had established and maintained a time-phased budget baseline. LADEE conducted an IBR in December 2010. From that review, a list of concerns about the EVM data and schedule were identified. Specifically, the project was not using a consistent approach to collect EVM data, calling into question the credibility of the data. In addition, it was unclear if an objective assessment of cost and schedule performance could be made using data generated by the “EVM Lite” approach. Since that review, officials said all issues and actions have been addressed and were formally closed by the IBR panel in July 2011. While formal surveillance is not occurring, EVM data assurance reviews are being performed during monthly management reviews with the Lunar Quest Program office. We reviewed contract performance reports from September 2010 to June 2011. Our review of LADEE’s EVM data found various data anomalies that call into question the reliability of the data. For example, from October 2010 to December 2010 there were some instances of negative values in the EVM reports that were unexplained. Since a variance analysis report provides a detailed narrative explaining significant cost and schedule variances, when this information is missing, management cannot understand the reasons for variances and the plan for fixing them. Also, the EVM data provided by the project office was not presented in a standard EVM format. This could be attributed to the fact that LADEE is required to provide only “EVM-like” data. Figure 6 below illustrates that as of June 2011, the LADEE project was reporting a positive cumulative cost variance of $3 million and a negative cumulative schedule variance of $10 million. LADEE project officials provided no information regarding positive cost and negative schedule variance drivers. As such, we have no insight into what could be causing deviations from the plan. Based on the positive cost variance and negative schedule variance thus far, we are forecasting a negative variance at completion ranging from $0.1 million to $13 million dollars. According to NASA, the project is not overrunning its commitment because the EVM baseline does not include unallocated future expenses held at the project and NASA headquarters level. The Magnetospheric Multiscale (MMS) is made up of four identically instrumented spacecraft. The mission will use the Earth’s magnetosphere as a laboratory to study the microphysics of magnetic reconnection, energetic particle acceleration, and turbulence. Magnetic reconnection is the primary process by which energy is transferred from solar wind to Earth’s magnetosphere and is the physical process determining the size of a space weather storm. The four spacecraft will fly in a pyramid formation, adjustable over a range of 10 to 400 kilometers. The data from MMS will be used as a basis for predictive models of space weather in support of exploration. The MMS spacecraft is being designed, developed, and tested in-house at Goddard Spaceflight Center (GSFC) while instrument development activities are under contract with Southwest Research Institute (SwRI). The Mission Operations Center and the Flight Dynamics Operations Area will be developed and operated at GSFC. The Science Operations Center for the instruments will be developed and operated at the Laboratory for Atmospheric and Space Physics at the University of Colorado and is under contract to SwRI. The MMS project office is collecting EVM data both at the project level as well as from SwRI, which is responsible for the entire instrument suite. Therefore, the SwRI Instrument Suite effort is a subset of the overall MMS project level EVM report. At the project level, MMS met one of three fundamental ANSI/EIA-748 practices for a reliable EVM system. MMS does not have a certified EVM system that complies with the ANSI/EIA-748 standard. NASA project officials said in-house projects are required only to be ANSI/EIA compliant, and are not required to have a certified system. Although the MMS project does not have a certified system, we assessed how well the MMS project was meeting three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. Our review found that the WBS in the MMS schedule did not match the WBS used for the EVM data, which is not in line with best practices. According to project officials, the MMS project was started before the requirements for earned value management were developed. As a result, the schedule and WBS were created without significant consideration of a one-to-one correlation between the two. A project official stated that MMS has retrofitted its EVM system to provide as close a correlation as possible without having to rebuild the WBS. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review of the MMS schedule found some sequencing issues within the schedule. For example, 9 percent of predecessor and successor tasks were not linked to one another, which is necessary for properly sequencing work so that the schedule will update properly once changes are made. This number accounts for removing all external tasks and level of effort (LOE) type activities. We also found that 21 percent of the remaining activities were constrained. This number also does not include LOE type activities. In fact, the majority of these constraints were hard constraints. Hard constraints can sometimes be impossible to meet, given the network characteristics, and can thereby result in schedules that are logically impossible to carry out. The presence of constraints also impacts the schedule’s ability to respond dynamically to changes and may portray an unrealistic view of the project plan. As a result, these sequencing issues and date constraints within the schedule affect the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guideline, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that that the MMS schedule was resource loaded. The MMS project office conducted an IBR in June 2010. This resulted in positive comments, general observations, and constructive recommendations by the review team. While formal surveillance is not occurring at the project level, EVM data assurance reviews are being performed by the Explorers and Heliophysics Program Office, GSFC Flight Projects Directorate managers, GSFC Chief Financial Officer’s Office, Standing Review Board, and NASA headquarters. We reviewed contract performance reports from August 2010 to July 2011. Our review of the MMS project level EVM data found some minor data reliability issues. The 12 months of project level EVM data reflect all of the performed in-house effort at GSFC for the project and the instrument suite contractor’s summary level effort. Also, the data provided was not reported in the standard contract performance report format. Project officials said a defined contract performance report was not dictated to the MMS project, but that all reporting has the same information as a “standard” contract performance report even though the formatting may be different. However, beginning in October 2011, MMS began reporting with the standard format 1 contract performance report, which provides cost and schedule data for each element in the project’s product-oriented WBS. We tried to map the EVM data in the lower level report for the instrument suite to the MMS project level report and we were not able to see how the costs tracked from one report to another. For example, the project level EVM data showed that the instrument suite contractor’s July 2011 budget at completion was $296 million, whereas the lower level EVM data in the instrument suite contractor’s report showed the budget at completion to be $217 million. The MMS project was able to demonstrate how the SwRI budget at completion in the lower-level report mapped to the SwRI budget at completion in the MMS project report. However, officials said because of the way the contractor submits their data the two reports will never match. Though we acknowledge that the project was able to explain how the data tracked, attempting to manually resolve incompatible pieces of data can become time-consuming, expensive and can lead to data reliability issues. Project officials said that MMS is working to capture data at lower WBS levels, which will allow for a closer tie between the cost and schedule data. In addition, MMS said that in addition to receiving the SwRI EVM reports, the project now internally calculates earned value metrics on the contractor provided instrument suite EVM data, which gives MMS completely internally derived earned value performance reports, based on the project team’s assessment without bias from contractor data. Figure 7 below illustrates that as of July 2011, the project was reporting a negative cumulative cost variance of $18 million and a negative cumulative schedule variance of $25 million. In October 2010, the project was in the midst of a replan and not all data was available at the time of report submission to generate detailed variance explanations. The replan was conducted so that the earned value baseline was the same as the cost plan required by the agency for monthly plan versus actual reporting. The goal of the replan was to prevent the project from having to report variances against two different plans. Since the replan, however, the project has experienced a downward trend in both cost and schedule performance. As a result, we are forecasting a negative variance at completion ranging from $47 million to $80 million dollars. NASA stated that the project is not overrunning its approved baseline commitment, because the EVM baseline does not include unallocated future expenses held at the project and headquarters level. SwRI, the contractor responsible for the entire instrument suite for MMS, met one of the three fundamental ANSI/EIA-748 practices for a reliable EVM system. SwRI does not have a certified EVM system that complies with the ANSI/EIA-748 standard. According to a project official, the SwRI contract does not require SwRI to have a certified system but only to be compliant with the ANSI/EIA-748. NASA convened an independent team to review the contractor’s readiness for EVM system certification and concluded that while the contractor has qualified people to support implementation of EVM, a single point of failure exists without a documented process. Not documenting the process is a problem because if the people who know the process leave, new staff will not know what to do. In addition, the team found that even though the right software tools are in place to support EVM, more integration is needed to reduce manual inputs. Finally, the team reported that compliance with the ANSI/EIA-748 standard would not be achievable without management support and resources. Despite these findings, the project office believed that contractor’s EVM data is useful in examining trends and overall performance of the instrument suite effort. Though SwRI does not have a certified system, we assessed how well the contractor was meeting three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. Our review found that the WBS used in the schedule was not consistent with the WBS used for the EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review of the schedule found some sequencing issues. For example, the Primavera schedule provided showed 31 percent of the remaining activities were missing dependencies and 36 percent were constrained. Officials said majority of the constrained tasks were due to external dependencies. Due to the major sequencing issues in the MMS SwRI schedule, we question the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. MMS project officials said they believed many of the constrained activities are not valid because they reside in another schedule. Also, some constraints, in the Harness area for example, if removed, have no effect on the overall schedule. In addition, officials also said some of the sequencing issues may be caused by the manual integration because since some instrument provider schedules are in Microsoft Project and others are in Primavera, and therefore it is not possible to ensure all tasks have been linked properly. However, the MMS project scheduler tests the schedule for missing dependencies, logic errors, and reasonable durations and the results are shared with the project office and the contractor so that appropriate action can be taken. Lastly, officials said several of the activities identified in our analysis are not really schedule items but level of effort type activities. When we removed the 15 LOE type activities from the missing dependencies count, the schedule still showed 28 percent of the remaining activities were missing dependencies. When we removed the 14 LOE type activities and the 4 Harness activities from the constraint count, the schedule still showed 33 percent of remaining activities were constrained. Because the schedule is the foundation for the EVM baseline, we question the reliability of the Instrument Suite EVM data. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that the MMS schedule was resource-loaded. The MMS project conducted an IBR of the MMS instrument suite effort in January 2010 and all action items have been closed out. While the instrument suite contractor does not have a formal surveillance program, the MMS project office has an EVM analyst, schedule team, resources team, instrument management team, and project management team who all review the instrument suite EVM data on a monthly basis. In addition, the Solar Terrestrial Probes project office, Science Mission Directorate management, and The Aerospace Corporation review the instrument suite EVM data monthly. We reviewed contract performance reports from August 2010 to August 2011. Our review of the instrument suite EVM data found various data anomalies that call into question the reliability of the data. For example, there were negative numbers reported for EVM data in four of the months that we reviewed. For some of the negative numbers, there was no explanation for the cause. For others, the negative values were due to correcting several months of translation errors brought on by a known issue with importing data from the schedule into the EVM system software. There were also errors such as inflated EVM data that once corrected, resulted in negative values. Figure 8 below illustrates that as August 2011, the project was reporting a negative cumulative cost variance of $4 million and negative cumulative schedule variance of $6 million. The cost variance in August 2011 dramatically improved from the downward trend during the previous months due to the project applying almost $13 million from its management reserve to the instrument suite contract. However, due to the negative cumulative cost and schedule variances, we are forecasting a negative variance at completion ranging from $10 million to $24 million. The Mars Atmosphere and Volatile EvolutioN (MAVEN) mission is part of NASA’s Mars Scout program, a robotic orbiter mission that will provide a comprehensive picture of the Mars upper atmosphere, ionosphere, solar energetic drivers, and atmospheric losses. Set to launch in 2013, MAVEN will deliver comprehensive answers to long-standing questions regarding the loss of Mars’ atmosphere, climate history, liquid water, and habitability. MAVEN will provide the first direct measurements ever taken to address key scientific questions about Mars’ evolution. Lockheed Martin is building the MAVEN spacecraft and will carry out mission operations for MAVEN. NASA’s Jet Propulsion Laboratory will navigate the spacecraft. The Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado at Boulder will coordinate the science team and science operations and lead the education and public outreach activities. NASA’s Goddard Spaceflight Center will provide management and technical oversight for the mission and will also provide mission systems engineering, mission design, and safety and mission assurance. The MAVEN project office is using EVM at the project level as well as collecting EVM data from Lockheed Martin, the spacecraft contractor, and LASP, which is responsible for the Science Operations Center, Remote Sensing, and Langmuir Probe and Waves instrument efforts. Both the Lockheed Martin and LASP contracts are subsets of the overall MAVEN Project EVM report. MAVEN fully met one of the three key practices for implementing EVM at the project level. Specifically, the project did not have a certified EVM system and is not required to have a certified system. Nevertheless, we assessed how well the MAVEN Project was meeting three ANSI/EIA guidelines. As part of our analysis, we assessed MAVEN’s EVM data against three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and that the WBS should be the same for the cost estimate, schedule, and EVM. We found that the project’s WBS was consistent between the schedule and EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review found some sequencing issues in the schedule. For example, 5 percent of the activities were missing dependencies, 6 percent had open ended logic links, and 20 percent had constraints, among other things. When schedules are not sequenced properly, float values and the calculated critical path will not be valid. In addition, the project conducted an integrated baseline review in July 2011. The MAVEN project office provided a May 2012 Schedule Health Check Report that showed the number of missing dependencies and constraints had been reduced considerably. Though we cannot validate the improvement in the schedule without performing our own assessment, we believe that MAVEN is working towards producing a more reliable schedule. This review found that the project schedule and technical design were in good shape, but noted concerns that more resources were needed to implement and maintain EVM, there were cost and schedule integration issues that caused the budgets for some work packages to not be in sync with the schedule, and reliable critical path analysis was at risk because of missing schedule links and constraints. Finally, a project should establish and maintain a time-phased budget baseline at the control account level, against which performance can be measured. Though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that the schedule was resource-loaded. Project conducted an IBR in July 2011 and all nine areas of concern were addressed and closed. While formal surveillance is not occurring at the project level, EVM data assurance reviews are being performed by the Mars Program Office representatives, MAVEN standing Review Board representatives, and Aerospace Corporation representatives at both the project level and for LASP efforts. We reviewed contract performance reports from June 2011 to August 2011. EVM data prior to spring 2011 was not available because MAVEN had not been confirmed into the implementation phase. Our review of the MAVEN project EVM data found that there was a mistake causing the costs to be overstated by twice their actual amount. Other than this mistake, which affected most of the data in the report, no other errors were found. The project EVM data reflects all work on this project including effort related to the project office, two in-house instruments, and the Space Sciences Laboratory, as well as major efforts from Lockheed Martin and LASP. Consolidated reporting of all components at a summary level began with the May 2011 data, with the first full summary report delivered on July 15, 2011. Therefore, we had only 3 months of data to review. Figure 9 below illustrates that as of August 2011, the project was reporting a positive cumulative cost variance of $5 million while also experiencing a negative cumulative schedule variance of $5 million. The positive cumulative cost variance was due to a decrease in labor charges, delayed material costs, a reduction in re-work, and the leveraging of common engineering products from other projects. The negative schedule variance was being driven by the Neutral Gas and Ion Mass Spectrometer instrument, which experienced technical issues such as vendor machines not being manufactured to specifications. Because we had only 3 months of data, we did not forecast an estimate at completion. In addition, we tried to map the EVM data in the lower level reports for spacecraft, Science Operations Center, Remote Sensing, and Langmuir Probe and Waves efforts to the overall MAVEN project EVM report and in some cases we were not able to see how the costs tracked from one report to another. For example, while we could easily trace the costs for the Science Operations Center effort from the lower level EVM report to the overall MAVEN project report, we could not clearly map the costs for the spacecraft, Remote Sensing or Langmuir Probe and Waves efforts. In particular, for Remote Sensing and Langmuir Probe and Waves efforts, the lower level EVM report cost elements did not have their costs burdened at the WBS level, which could account for some of the differences between the lower level report costs and the overall MAVEN project costs for those elements. MAVEN project officials walked us through their process of how they ensure lower-level reports map to the project level reports. In addition, MAVEN project provided supporting documentation that validated this assertion. Though MAVEN project officials helped explain the mapping, officials said they do not mandate that their contractors follow a certain reporting format, instead any adjustments necessary to ensure that the lower-level reports map to the project-level reports are made manually by the project office. Though MAVEN project does not prescribe to a standard reporting format, attempting to manually resolve incompatible pieces of data can become time-consuming, expensive and can lead to data reliability issues. Although the agency provided explanations for the mapping issues, the ability to reconcile the costs between the reports should be evident, without additional explanations. Lockheed Martin, the spacecraft contractor, met the three fundamental ANSI/EIA practices necessary for a reliable EVM system. In August 2008, the Defense Contract Management Agency (DCMA) certified that Lockheed Martin’s EVM system is compliant with the ANSI/EIA standard. However, the implementation of that EVM system is questionable based on our findings. We assessed the contractor’s data against the three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. We found that the project’s WBS in the schedule was consistent with the WBS used in the EVM data. However, we found issues with the schedule. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review found some sequencing issues in the schedule. For example, 2 percent of the activities remaining were missing predecessor and successor links, which are necessary for properly sequencing work so that the schedule will update in response to changes. We also found that 37 percent of the remaining activities had constraints, which also hinder the schedule’s ability to respond dynamically to status updates resulting in an artificial or unrealistic view of the project plan. These issues with the schedule affected the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. Project officials further explained that some of the “start no earlier than” constraints were due to resource availability and “finish no later than” constraints were used intentionally to plan task activities to occur as late as possible. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that the contractor schedule was resource loaded. The project conducted the spacecraft’s IBR in April 2011. During the review, several areas of concern regarding the schedule were identified. The project office said that during the integrated baseline review, the review team identified many of the same observations with the schedule as our findings. As a result, the project office directed the contractor to eliminate the constraints, lags, and missing logic links in their integrated master schedule. Since the November 2011 schedule submittal, the contractor has decreased the number of sequencing issues in the schedule, according to project officials. The project office also said that the contractor continues to conduct schedule health checks to uncover any sequencing issues and provide the project office with schedule variance reports and critical path analysis, which are discussed during monthly management meetings. Joint surveillance reviews of the EVM data are being performed by DCMA, MAVEN project officials, and the contractor. We reviewed contract management reports from January 2011 to August 2011. Our review of the EVM data found no major issues with data reliability. However, from January to May 2011, there were no variance analysis reports produced to explain significant cost and schedule variances and other contract problems and topics because they did not meet reporting thresholds. Without this information, however, management cannot understand the reasons for variances and the contractor’s plan for fixing them. Figure 10 below illustrates that as of August 2011, the project was reporting a positive cumulative cost variance of $6 million and a negative cumulative schedule variance of $3 million. One reason for the positive cost variance was due to the ability to leverage a lower subcontractor rate than planned, which resulted in a cost savings. The negative schedule variance was also being driven by the mechanism subsystem falling behind schedule due to the shop being overloaded with work and the mechanism designers supporting other NASA efforts, among other things. Due to the positive cost variance, we are forecasting a positive variance at completion ranging from $1 million to $14 million. LASP, the contractor responsible for the Science Operations Center, Remote Sensing package, and Langmuir Probe and Waves instrument efforts met one of three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. To date, LASP does not have a certified EVM system. Though the contractor is not required to have a certified system, we assessed how well the contractor was meeting three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored for effective internal management control of the project and the WBS is the same for the cost estimate, schedule, and EVM. We found some slight inconsistencies in the WBS numbering between the EVM report and the schedule for the Remote Sensing package, which calls into question the reliability of their overall schedule network. Moreover, the Langmuir Probe and Waves Instrument had issues with consistency between the WBS and the schedule. In particular, there was varying levels of information between the two WBSs, making it hard to use the WBS as a common thread between the EVM data and the schedule. Since the WBS is a critical component of EVM, it should be the same for developing the EVM performance measurement baseline and the schedule. Without a common link between these two features, project managers cannot fully understand project cost and schedule variances. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. We found some sequencing issues with the Langmuir Probe and Waves Instrument schedule. For example, 47 percent of the remaining activities had constraints, which defeated the purpose of a using a dynamic schedule. The quality of the schedule was also hampered by the presence of schedule lags on 18 percent of the remaining activities. Schedule lags must be justified because they cannot be easily monitored or included in risk assessments. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that all three components showed evidence of a time-phased budget baseline. The project conducted an IBR in March 2011 and several areas of concern were noted. MAVEN project officials stated that many of the schedule issues we found were also discovered during the IBR and have now been corrected. While formal surveillance is not occurring, EVM data assurance reviews are being performed by the, project office, Mars Program Office, and MAVEN Standing Review Board representatives. Also, The Aerospace Corporation is working as an advisor to Science Mission Directorate’s Planetary Systems Division. Our review of MAVEN’s Science Operations Center, Remote Sensing package, and Langmuir Probe and Waves instrument EVM data found no major issues with data reliability. However, there was a lack of variance analysis reports for these efforts. For example, Science Operations Center had no variance analysis reports with the explanations for any of the months reported. In addition, the Remote Sensing package, and Langmuir Probe and Waves instrument variance analysis reports did not provide any explanation for major performance swings from one month to another. Figure 11 below illustrates that as of August 2011, the Science Operations Center portion of the MAVEN project was reporting a positive cumulative cost variance of $0.01 million and a slightly negative cumulative schedule variance of $0.01 million. However, since the variance analysis reports provided no information regarding what is driving the positive cost and slightly negative schedule variances, we have no insight into the causes for deviations from the plan. Based on the positive cost variance thus far, we are forecasting a positive variance at completion of less than $0.5 million at contract completion. Figure 12 below illustrates that as of August 2011 the Remote Sensing package portion of the MAVEN project was reporting a positive cumulative cost variance of $0.8 million and a slightly negative cumulative schedule variance of $0.6 million. Factors behind the positive cost variance include an accounting lag on the invoicing and payment process associated with the procurements, which results in the appearance of cost efficiency. This issue has been occurring for several months and is causing the EVM metrics to be skewed to reflect false positive cost variances. The variance analysis reports do not give any explanation for why there is a negative schedule variance situation as of August 2011. As a result of the positive cost variance, we are forecasting a positive variance at completion ranging from $1 million to $7 million. Figure 13 below illustrates that as of August 2011 the Langmuir Probe and Waves portion of the MAVEN project was reporting a negative cumulative cost variance of $0.6 million and negative cumulative schedule variance of $0.3 million. Reasons for the negative cost and schedule variances are due to costs for outside services and materials being more than planned as well as additional work required to troubleshoot problems and mitigate risks. Due to these problems, we are forecasting a negative variance at completion ranging from $2 million and $3 million. According to NASA, the project is not overrunning its commitment because the EVM baseline does not include unallocated future expenses held at the project and headquarters level. NASA’s Orbiting Carbon Observatory 2 (OCO-2) is designed to enable more reliable predictions of climate change and is based on the original OCO mission that failed to reach orbit in 2009. It will make precise, time- dependent global measurements of atmospheric carbon dioxide. These measurements will be combined with data from a ground-based network to provide scientists with the information needed to better understand the processes that regulate atmospheric carbon dioxide and its role in the carbon cycle. NASA expects enhanced understanding of the carbon cycle will improve predictions of future atmospheric carbon dioxide increases and the potential impact on the climate. The OCO-2 mission consists of a dedicated spacecraft with a single instrument, flying in a near-polar, sun-synchronous orbit. The Jet Propulsion Laboratory (JPL) has overall responsibility for project management. The OCO-2 spacecraft is being built by Orbital Sciences Corporation while the instrument is being built in-house at JPL. Orbital Sciences Corporation submits spacecraft effort EVM data monthly and the project incorporates that data into the overall project EVM report. The project is facing a launch delay because the Taurus XL launch vehicle failed on the Glory Mission, and the contract was terminated. The project will be rebaselined as a result of NASA having to select a new launch vehicle. The current $477.2 million total project cost is a preliminary amount pending the outcome of the rebaseline process. The OCO-2 project met all three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. JPL has a certified EVM system that complies with the ANSI/EIA standard. However, the implementation of its EVM system is questionable based on our findings below. We assessed the JPL EVM data against three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. We found that the project had a work breakdown structure that was consistent with the WBS used for the EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review found some sequencing issues in the schedule. In particular, 14 percent of predecessor and successor tasks were not linked to one another, which is necessary for properly sequencing work so that it will update in response to changes. We also found 15 percent of the remaining activities were constrained, which also hinders the schedule from responding dynamically to changes and can portray an artificial or unrealistic view of the project plan. These sequencing issues and constraint dates within the schedule affect the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. Project officials said the missing dependencies are mainly handoffs produced by level of effort (LOE) activities. Officials said these activities are not necessary for valid schedule network logic, and under no circumstances do these activities drive the critical path. They also said the constrained activities are largely composed of mandated delivery dates, which JPL uses as control points to manage subsystem schedule performance prior to assembly, test, and launch operations delivery, and coordinate major meeting logistics. Finally, the ANSI/EIA guideline states that a project should establish and maintain a time- phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. To develop the baseline, resources must be accounted for in the schedule and our review found that the schedule was resource loaded. The project conducted an integrated baseline review in March 2011. JPL also has a formal surveillance plan in place for monitoring the EVM data. In particular, project officials said each month detailed earned value data analysis is performed on each cost account and each work package so that conclusive understanding of the performance status can be reached and communicated within the project team. In addition, during the project monthly management reviews, both cost and schedule variances and reasons causing them are presented. We reviewed contractor management reports from December 2010 to July 2011. Though we found no major data reliability issues when we tried to map the EVM data in the lower level spacecraft report to the JPL project level report, we could not understand how the costs tracked from one report to another. In a subsequent interview, officials explained, with supporting documentation, how the lower level Orbital Sciences budget at completion mapped to the budget at completion found in the JPL Project level EVM report. Though we appreciate the explanations provided by project officials regarding the differences between the two reports, the issue remains that without additional documentation and explanations by project officials, GAO or another independent party could not have reconciled the data. Figure 14 below illustrates that as of July 2011, the project was reporting a positive cumulative cost variance of $0.6 million and a negative schedule variance of $8 million as seen in the graph below. According to project officials, the schedule variance was being caused by spectrometer slit instability and an incompatible memory chip in the Remote Electronics Module. Due to the positive cost variance and negative schedule variance we are forecasting a negative variance at completion ranging from $1 million to $26 million. Project officials said the project is not overrunning its commitment because the EVM baseline provided to GAO does not include unallocated future expenses held at the project and headquarters level. The spacecraft contractor, Orbital Sciences Corporation, met two of the three fundamental ANSI/EIA-748 practices necessary for a reliable EVM System. At the time of our review, the Orbital Sciences did not have a certified EVM system that complied with the ANSI/EIA-748 standard. In January 2012, DCMA certified the Orbital Sciences EVM System. Though the contractor now has a certified system, the implementation of that system is questionable based on our findings below. We assessed how well the contractor’s EVM system was meeting three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. We found that the spacecraft’s WBS in the schedule was not consistent with the WBS used for the EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. We found some sequencing issues in the spacecraft contractor schedule. We found that 27 percent of the remaining activities in the spacecraft’s schedule were missing predecessor and successor links, which are necessary for properly sequencing work so that the schedule will update in response to changes. In addition, 24 percent of the remaining activities had date constraints, which also hinder the schedule’s ability to respond dynamically to status updates resulting in an artificial or unrealistic view of the project plan. These sequencing issues and constraint dates within the schedule affect the reliability of the overall network and the schedule’s ability to correctly calculate float values and the critical path. Orbital Sciences conducted an integrated baseline review of the spacecraft effort in March 2011. While formal surveillance is not occurring, project officials stated that EVM performance data is reviewed during the monthly status reviews. We reviewed contract performance reports from April 2011 to July 2011. Because we had only 4 months of data to review, we were unable to forecast estimate variance at completion. As of July 2011, the project was reporting a positive cumulative cost variance of $0.4 million and a negative cumulative schedule variance of $2 million as seen in the graph below. The positive cost variance was being driven by lower than expected burden rates and less staff supporting project management, flight assurance, and systems engineering efforts. The positive cost variance was being driven by lower than expected burden rates and less staff supporting project management, flight assurance, and systems engineering efforts. The Radiation Belt Storm Probes (RBSP) mission will explore the Sun’s influence on the Earth and near-Earth space by studying the planet’s radiation belts at various scales of space and time. This insight into the physical dynamics of the Earth’s radiation belts will provide scientists with data to make predictions of changes in this little understood region of space. Understanding the radiation belt environment has practical applications in the areas of spacecraft system design, mission planning, spacecraft operations, and astronaut safety. The RBSP project built two spacecraft that will be used to measure the particles, magnetic and electric fields, and waves that reside in the Van Allen radiation belts. RBSP launched on August 30, 2012 on a two-year prime mission. The RBSP spacecrafts and ground system are being designed, developed, and tested by the John Hopkins University’s Applied Physics Laboratory.level. At the project level, the Applied Physics Laboratory only fully met one and partially met another of the three fundamental practices necessary for a reliable EVM system. RBSP is the first full NASA mission to use EVM at the Applied Physics Laboratory. According to the RBSP project manager, the RBSP project implemented a limited earned value management system in Phase B as a risk mitigation activity for Phase C/D. This early implementation was a risk mitigation activity, which allowed the project’s control account manager, instrument provider, and project office to better understand the reporting process and the use of the EVM system. The use of EVM during this phase was also intended to allow for timely, accurate, and useful EVM reporting during the formal reporting in later phases of the project. Since then Applied Physics Laboratory has made good progress and is in the process of meeting the intent of being compliant with the 32 ANSI/EIA-748 guidelines. We assessed how well the Applied Physics Laboratory was meeting the three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. Our review, we found that that the WBS used in the schedule was not consistent with the WBS used for the EVM data. Project officials said that the project utilizes the Applied Physics Laboratory WBS for all earned value management activity, which is then mapped to the NASA WBS for reporting to the sponsor. Also, the Applied Physics Laboratory WBS is uniformly utilized and consistent across all control accounts in the EVM system. This internal WBS ties into both the contract performance report and the integrated master schedule utilized on the project. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review found some sequencing issues in the schedule. For example, 23 percent of the remaining activities were missing predecessor and successor links, which are necessary for properly sequencing work so that the schedule will update in response to changes. We also found that 29 percent of the remaining activities had date constraints, which also hinder the schedule’s ability to respond dynamically to status updates resulting in an artificial or unrealistic view of the project plan. When schedules are not sequenced properly, float values and the calculated critical path will not be valid. Project officials explained that sequencing issues were a result of the project consciously including constrained instrument deliveries and deliverables, level of effort activities and material and subcontractor expenditures in the integrated master schedule, and though the RBSP integrated master schedule had these issues it was able to monitor the critical and near critical paths of all spacecraft systems and subsystems. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. Our review found that the schedule was resource loaded. NASA conducted an integrated baseline review in August 2009. Of the seven IBR objectives identified, two were partially met and five were met. In December 2010, the IBR deputy chief notified the Applied Physics Laboratory that all areas of concern had been closed. As a result, the overall consensus of the government review team was that the integrated baseline review was successful. While formal surveillance is not being performed, the project office reviews the EVM data on a monthly basis. Although they do not perform formal surveillance, an Applied Physics Laboratory official said that they performed additional monthly independent reviews of the RBSP EVM system throughout Phase C/D. We reviewed contract performance reports from August 2010 to April 2011 and July 2011 to August 2011. We were not provided reports for May and June 2011. This was due to a re-plan determined necessary by the Living with a Star Program Office and NASA headquarters in May 2011 due to changes in the launch manifest. Figure 16 below illustrates that as of August 2011, the project was reporting a negative cumulative cost variance of approximately $32 million and a negative cumulative schedule variance of $3 million. The negative cumulative cost variance was caused by sustained effort on the radio frequency communications, as well as by work on the avionics equipment, and ground system software launch and post launch components being behind schedule. The schedule variance is minimal since the project is nearing completion. Due to the negative cost variance we are forecasting a negative variance at completion from $40 million to $41 million. Project officials noted the forecasted variance at completion is below the revised contract value of $351.1 million, although above the project’s estimated budget at completion. In addition, officials said this is a project-level variance, and does not account for the application of unallocated future expenses to fund the movement of the launch date and to keep the project on track. When the project launched in August 2012, its estimate at completion was below the $351.1 million budget at completion. The Stratospheric Observatory for Infrared Astronomy (SOFIA) is a joint project between NASA and Deutsche Zentrum für Luft-und Raumfahrt (DLR), the German space agency, to install a 2.5 meter telescope, as well as other scientific instruments capable of infrared and sub-millimeter observations, in a specially modified Boeing 747SP aircraft. This airborne observatory is designed to provide routine access to the visual, infrared, far-infrared, and sub-millimeter parts of the electromagnetic spectrum. Its mission objectives include studying many different kinds of astronomical objects and phenomena, including star birth and death; the formation of new solar systems; planets, comets, and asteroids in our solar system; and black holes at the center of galaxies. Currently, five U.S. and two German funded interchangeable instruments for the observatory are being developed to allow a range of scientific measurement to be taken by SOFIA. The SOFIA project office is using EVM at the project level as well as collecting EVM data from the L-3 Communications Integrated Systems L.P. (L-3), which is responsible for the airborne system platform effort. The EVM data provided to the project for the airborne observatory platform effort is a subset of the overall SOFIA project level EVM report. The German component of the SOFIA project does not generate earned value data and are not part of the project’s budget baseline. The University Space Research Association (USRA) has a support contract to help the Ames Research Center manage SOFIA’s science and mission operations in cooperation with the Deutsches SOFIA Institut. The USRA contract was established before NASA began requiring earned value management compliance. As a result, they are not required to generate earned value data. However, all of these components are subsets of the overall SOFIA project. SOFIA project did not meet any of the three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. Project officials said in- house projects are not required to have certified EVM systems. Though the SOFIA project does not have a certified system, we assessed how well the project was meeting three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. We found that the WBS used in the SOFIA schedule was consistent with the WBS used for the EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. Our review found some sequencing issues in the schedule. For example, 16 percent of the remaining activities had open ended logic links, and 1 percent had date constraints, among other things. SOFIA project officials explained that the majority of the missing links were due to using hammock activities to resource load the schedule, which resulted in missing successors. Out of all of the activities missing successors, only 11 were not hammocked activities, which accounted for a very small amount. In addition, SOFIA project officials said that constraints in the schedule were justified since they represented external deliveries or fiscal year funding availability. They added that the one hard constraint, Must Start On, was used to represent a fixed date for an international visit. While the majority of these explanations seem reasonable, when schedules are not sequenced properly, the slack values and the calculated critical path will not be valid. An IBR was not conducted at the project level. Project officials said SOFIA, an in-house project, did not begin collecting EVM data until very late in the development phase, and because of this an IBR was not conducted. Officials added although SOFIA did not conduct a project- level IBR, the EVM baseline was established concurrently with an Agency approved re-plan and joint confidence level analysis in the 2009/2010 time frame, and was reviewed by a Standing Review Board. Though the Standing Review Board review satisfied some of the objectives of an IBR, including confirmation of the schedule and budget baselines (e.g., schedule review, risk review, key milestones identified), it did not address the full IBR checklist (e.g. work authorizations, control account plans, EVM system description). While formal surveillance is not occurring at the project level, EVM data assurance reviews are being performed monthly and quarterly by the SOFIA project office representatives. We reviewed contract performance reports from August 2010 to August 2011. We found various issues that bring into question the reliability of the SOFIA Project EVM data. For example, we found negative values due to an “over-reporting” of progress in previous months that was caused by a problem with translating percent complete progress from the University Space Research Association schedule to the SOFIA integrated master schedule as well as corrections/modifications in costs posted for support service contracts. In addition, we found other anomalies in the data that SOFIA project officials explained were most likely due to mischarges by employees, delayed cost postings, or employees continuing to use charge codes inappropriately. While the cost impact of these problems was not that large for any one WBS element, each of these issues causes us to question the reliability of the data. According to project officials, the variances that caused these anomalies did not meet the reporting threshold. A variance analysis report provides a detailed, narrative report explaining significant cost and schedule variances and other contract problems and topics. Without this information, management cannot understand the reasons for variances and the contractor’s plan for fixing them. When variance analysis reports are not produced, the EVM data will not be meaningful or useful as a management tool. Figure 17 below illustrates that as of August 2011, the SOFIA project was reporting a positive cumulative cost variance of $3.5 million and a negative cumulative schedule variance of $6.6 million. Due to the cost and schedule variances we are forecasting a negative variance at completion ranging from $1.4 million to $76 million. The contractor met all three fundamental ANSI/EIA-748 practices necessary for a reliable EVM system. In 2002, DCMA certified that the contractor has an EVM system compliant with the ANSI/EIA standard. The project conducted an IBR in November 2007. It is unclear if EVM surveillance is being performed on the L-3 EVM data. We reviewed contract performance reports from August 2010 to July 2011. We found various data anomalies that call into question the reliability of the contractor’s EVM data for the contract performance reports we reviewed. For example, we found negative values, actual cost of work performed being reported without work being scheduled and/or performed. Project officials stated that these anomalies were caused by performance being taken in later months and also because these issues did not trip the threshold reporting. Figure 18 below illustrates that as of July 2011, the airborne system platform contractor was reporting a positive cumulative cost variance of $3.2 million and a negative schedule variance of $0.6 million. The favorable cost variance is due to efficiencies in project oversight and engineering. The unfavorable cumulative schedule variance is being driven by a lack of government furnished equipment supposed to be received from NASA that is causing a backlog of tasks. Due to the cost and schedule variances we are forecasting a positive variance at completion ranging from $3 million to $4 million. The Tracking and Data Relay Satellite (TDRS) System consists of in-orbit communication satellites stationed at geosynchronous altitude coupled with two ground stations located in New Mexico and Guam. The satellite network and ground stations provide mission services for near-Earth user satellites and orbiting vehicles. TDRS-K and L are the 11th and 12th satellites, respectively, to be built for the TDRS system. They will contribute to the existing network by providing continuous high bandwidth digital voice, video, and mission payload data, as well as health and safety data relay services to Earth-orbiting spacecraft such as the International Space Station and the Hubble Space Telescope. NASA is planning to launch TDRS-K in December 2012 followed by the TDRS-L launch in February 2014. NASA is collecting EVM data from both the spacecraft and sustainment efforts. In December 2007, NASA awarded a fixed price incentive contract to design, develop, fabricate, integrate, test, ship, provide launch support, conduct on-orbit checkout operations and provide sustaining engineering support for two spacecraft, TDRS-K and TDRS-L, to Boeing Satellite Systems, Inc (Boeing). The spacecraft contractor, Boeing, met the three fundamental ANSI/EIA- 748 practices necessary for a reliable EVM system. Boeing has a certified EVM system that complies with the ANSI/EIA EVM standard. Though the contractor has a certified system, the implementation of that system is questionable based on our findings below. We also assessed whether the spacecraft contractor’s EVM data against three ANSI/EIA guidelines. These guidelines state that the authorized work elements for the project should be defined typically using a WBS that has been tailored to the project and the WBS is the same for the cost estimate, schedule, and EVM. We found that the WBS used in the spacecraft integrated master schedule was consistent with the WBS used for the EVM data. The ANSI/EIA guidelines also state that projects should have a schedule that describes the sequence of work by listing activities in the order in which they are to be carried out and identifying significant task interdependencies required to meet project requirements. We found some sequencing issues in the contractor’s schedule. For example, 13 percent of the remaining activities were constrained. When schedules are not sequenced properly, float values and the calculated critical path will not be valid. Project officials acknowledged the constraints and said they are a result of having to adjust support activities due to spacecraft integration and test delays as well as alignments to the current/actual manifest dates, which differ from the Boeing contractual and launch readiness dates. In addition, critical path metrics are generated and analyzed monthly to track the Boeing performance against the critical path activities. Finally, the ANSI/EIA guidelines state that a project should establish and maintain a time-phased budget baseline to track cost and schedule variances in an EVM system. Though resource loading the schedule is not required to meet the ANSI/EIA guidelines, it is a best practice and therefore resources should be accounted for in the schedule in order to develop this baseline, according to the GAO cost guide. We found that the schedule was not resource loaded. Project officials said the integrated master schedule is resource loaded but not inside the Microsoft Project schedule because it does not directly interface with the Boeing financial/EVM system. They stated that the integrated master schedule is produced using Microsoft Project and imported into a planning software tool. After the resource loading effort is performed, the planning data is transferred into the EVM system. The tool integrates the Microsoft Project schedule with the Boeing financial system and the resource allocations for each task. When adjustments are required Boeing again utilizes the planning tool. An integrated baseline review was conducted in 2008 where 211 issues were raised. All of these issues have since been resolved and closed. DCMA prepares a monthly EVM analysis report and performs continuous surveillance of Boeing’s EVM implementation by sampling various control account managers for interviews about the process. We examined contract performance reports from August 2010 through August 2011. Figure 19 illustrates that as of August 2011, the project was reporting a negative cumulative cost variance of approximately $131 million and a negative cumulative schedule variance of $7 million. The negative cumulative cost variance was being driven by higher staffing levels to support integration, the Preliminary Design Review and Critical Design Review, as well as an incorrect assessment of project requirements and the inability to use heritage specifications. Labor costs were also higher than expected due to part failures and the late completion of component qualifications. Extended test activities also contributed to the cumulative negative cost variance. Finally, more than expected resources were needed to complete board and slice designs, generate drawings, and assemble and test components because of the complexity of the design. Due to the negative cost and schedule variance we are forecasting a variance at completion ranging from $152 million to $185 million. Project officials said GAO’s independent variance at completion gives the impression that NASA may request additional funding to complete the TDRS K and L. Because this is a fixed price, incentive fee contract, NASA officials said the agency is obligated only to pay up to the price ceiling of the contract. Boeing is the contractor for both the spacecraft and sustainment efforts. As noted above, Boeing met all three fundamental practices for a reliable EVM system. In addition, the 2008 integrated baseline review was conducted for both the spacecraft and sustainment effort. As well, the formal surveillance performed by Boeing and the Defense Contract Management Agency applies to the sustainment effort We reviewed contract performance reports from April 2011 to August 2011. Because we had only 5 months of data, we were not able to forecast an independent estimate at completion. Figure 20 below illustrates that as of August 2011, the project was reporting a positive cumulative cost variance of $0.08 million. Because there were no variance analysis reports accompanying the sustaining effort, we were unable to determine what was causing the positive cost variance. Project officials stated that the Performance Measurement Baseline for this effort was almost entirely level of effort, so minimal variances would be occurring. They added that during the August 2011 time period, the contract reflected an April 2012 launch date even though the launch was being delayed. Consequently, Boeing’s reports were reflecting work scheduled to occur in support of the earlier launch date when in fact very little effort was being done. As a result, since minimal costs were incurred, this resulted in a positive cumulative cost variance. In addition to the contact named above, Shelby S. Oakley and Karen Richey (Assistant Directors), Greg Campbell; Jennifer K. Echard; Tisha D. Derricotte; Laura Greifner; Kristine R. Hassinger; Ben Jaskiewicz; William Laing; Richard Lee; Eric Lofgren; Kenneth E. Patton; Jose A. Ramos; Carrie W. Rogers; Stacey L. Steele; Roxanna T. Sun; and Umesh Thakkar made key contributions to this report.

Summary: NASA historically has experienced cost growth and schedule slippage in its portfolio of major projects and has taken actions to improve in this area, including adopting the use of EVM. EVM is a tool developed to help project managers monitor risks. GAO was asked to examine (1) the extent to which NASA is using EVM to manage its major space flight acquisitions, (2) the challenges that NASA has faced in implementing an effective EVM system, and (3) NASA's efforts to improve its use of EVM. To address these questions, GAO obtained contractor and project EVM data and used established formulas and tools to analyze the data and assess NASA's implementation of EVM on 10 major spaceflight projects; interviewed relevant NASA headquarters, center and mission directorate officials on their views on EVM; and reviewed prior reports on EVM and organizational transformations. GAO compared NASA policies and guidance on EVM to best practices contained in GAO's cost estimating best practices guide. The National Aeronautics and Space Administration's (NASA) 10 major spaceflight projects discussed in this report have not yet fully implemented earned value management (EVM). As a result, NASA is not taking full advantage of opportunities to use an important tool that could help reduce acquisition risk. GAO assessed the 10 projects against three fundamental EVM practices that, according to GAO's best practices cost guide, are necessary for maintaining a reliable EVM system. GAO found shortfalls in two of three fundamental practices. Specifically, we found that More than half of the projects did not use an EVM system that was fully certified as compliant with the industry EVM standard. Only 4 of the 10 projects established formal surveillance reviews, which ensure that key data produced by the system was reliable. The remaining 6 projects provided evidence of monthly EVM data reviews; however, the rigor of both the formal and informal surveillance reviews is questionable given the numerous data anomalies GAO found. GAO also found that 3 projects had reliable EVM data while 7 had only partially reliable data. For the EVM data to be considered reliable per best practices it must be complete and accurate with all data anomalies explained. NASA EVM focal points, headquarters officials, project representatives, and program executives cited cultural and other challenges as impediments to the effective use of EVM at the agency. Traditionally, NASA's culture has focused on managing science and engineering challenges and not on monitoring cost and schedule data, like an effective EVM system produces. As a result, several representatives said this information traditionally has not been valued across the agency. This sentiment was also echoed in a NASA study of EVM implementation. Also cited as a challenge to the effective use of EVM was NASA's insufficient number of staff with the skills to analyze EVM data. Without a sufficient number of staff with such skills, NASA's ability to conduct a sound analysis of the EVM data is limited. However, NASA has not conducted an EVM skills gap analysis to determine the extent of its workforce needs. NASA has undertaken several initiatives aimed at improving the agency's use of EVM. For example, NASA strengthened its spaceflight management policy to reflect the industry EVM standard and has developed the processes and tools for projects to meet these standards through its new EVM system. While these are positive steps, the revised policy contains only the minimum requirements for earned value management. For example, it lacks a requirement for rigorous surveillance of how projects are implementing EVM and also does not require use of the agency's newly developed EVM system to help meet the new requirements. NASA has attempted to address EVM shortcomings through policy changes over the years, but these efforts have failed to adequately address the cultural resistance to implementing EVM. GAO recommends that NASA establish a time frame for requiring new spaceflight projects to implement its new EVM system; conduct an EVM skills gap assessment; develop a change management plan for EVM; and strengthen its EVM requirements by requiring projects to implement formal EVM surveillance. NASA concurred with two recommendations and partially concurred with two others citing resource constraints. Despite NASA's plans to address some issues GAO identified, not addressing all key issues lessens the usefulness of EVM at NASA.