In May 2009, Congress passed the Weapon Systems Acquisition Reform Act of 2009 (Reform Act) in an effort to improve the way weapon systems are acquired and avoid further cost overruns on such programs. When signing the Reform Act into law, the President stated that its purpose was to limit weapon system cost overruns and that it would strengthen oversight and accountability by appointing officials who will closely monitor the weapons systems acquisition process to ensure that costs are controlled. Four offices were established as a result of the Reform Act: SE, DT&E, CAPE and PARCA. The SE and CAPE offices existed under other organizational titles prior to the Reform Act. Staffing levels, following the Reform Act, remained relatively stable for both of these offices, but some reorganization was necessary to reflect new Reform Act responsibilities. The DT&E and PARCA offices were newly established. The key roles and responsibilities of these four offices as outlined in the Reform Act are explained below: Each of the offices has varying levels of interaction with defense acquisition programs. For example, the SE and DT&E offices have ongoing interaction with acquisition programs throughout development in the form of program reviews, working group activities, and other program meetings. They also coordinate program documents in preparation for major milestone reviews. CAPE issues guidance to programs on how to conduct an analysis of alternatives at the beginning of the acquisition process. The office approves the analysis of alternative study plan that is developed based on its guidance. It then develops independent cost estimates for major milestone reviews and in the event that an acquisition program experiences a Nunn-McCurdy breach. According to PARCA, it assesses all major defense acquisition programs at least once per quarter or when requested by the Under Secretary of Defense for Acquisition, Technology and Logistics, and disseminates this information to senior leaders. The office also interacts with specific programs if they experience a Nunn-McCurdy breach. In these cases, the office assesses program performance not less than semi-annually until 1 year after it receives a new milestone approval. In addition to the new organizational requirements, the Reform Act requires DOD to ensure that the acquisition strategy for major defense acquisition programs includes measures to ensure competition or the option of competition throughout the program life cycle. This could include strategies such as maintaining two sources for a system (dual-sourcing) and breaking requirements for supplies or services previously provided or performed under a single contract into separate smaller contracts (unbundling of contracts). Major defense acquisition programs are also required to provide for competitive prototyping—where two or more competing teams produce prototypes before a design is selected for further development—prior to Milestone B unless a waiver is properly granted by the milestone decision authority, and to meet the following Milestone B certification requirements, including: Appropriate trade-offs among cost, schedule, and performance objectives have been made to ensure the program is affordable; A preliminary design review and formal post-preliminary design review assessment have been conducted and on the basis of such assessment the program demonstrates a high likelihood of accomplishing its intended mission; Technology has been demonstrated in a relevant environment on the basis of an independent review and assessment by the Assistant Secretary of Defense for Research and Engineering; Reasonable cost and schedule estimates have been developed to execute, with concurrence of the Director of CAPE, the program’s product development and production plan; Funding is available to execute the program’s product development DOD has completed an analysis of alternatives with respect to the The Joint Requirements Oversight Council has approved program requirements, including an analysis of the operational requirements. The Reform Act also requires the Joint Requirements Oversight Council to ensure trade-offs among cost, schedule, and performance objectives are considered for joint military requirements. GAO previously reported that the Council considered trade-offs made by the military services before validating requirements, but the military services did not consistently provide high-quality resource estimates to the Council for proposed programs in fiscal year 2010. We also found that the Council did not prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. DOD has implemented most of the fundamental Reform Act provisions as required and is taking additional steps to strengthen acquisition reviews, policies, and capabilities. Offices established as a result of the Reform Act are continuing to issue policies, review and approve relevant acquisition documents, monitor weapon acquisition program activities, and develop performance measures. In addition, all four of the major defense acquisition programs we reviewed that had not started development when we selected our case studies plan to implement Reform Act provisions regarding preliminary design reviews, competitive prototyping, and competition. Also, some provisions, such as issuance of guidance on estimating operating and support costs, by the CAPE, are still in the process of being completed. Finally, we found that the Under Secretary of Defense for Acquisition, Technology and Logistics has revised the defense acquisition review process to consider additional knowledge collected on programs earlier and efforts are being made to strengthen acquisition policies and capabilities. The offices established as a result of the Reform Act—SE, DT&E, CAPE, and PARCA—are continuing to make progress in implementing four fundamental Reform Act provisions aimed at strengthening acquisition outcomes and oversight of weapon acquisition programs. Specifically, the offices are (1) developing policy and guidance to the military services for conducting work in their respective areas, (2) approving acquisition documents prior to milestone reviews, (3) monitoring and assessing weapon acquisition program activities on a consistent basis, and (4) developing performance measures to assess acquisition program activities. Figure 1 provides the status of DOD efforts to implement the four fundamental provisions. Some offices are still in the process of completing a few of these provisions. For example, CAPE and PARCA are in the process of developing policies and guidance for their respective areas and DT&E is in the process of establishing performance measures that can be used to assess weapon acquisition program activities. The office piloted the performance measures on two major defense acquisition programs and reported that they are currently applying them to over 40 programs. Note that some activities related to approving documents and monitoring or assessing programs require on-going efforts on the part of some of the offices. We also found evidence that major defense acquisition programs are integrating Reform Act provisions in their acquisition strategies. The four weapon acquisition programs we reviewed that had not started development activities when we began our review plan to implement Reform Act provisions related to preliminary design reviews, competitive prototyping, and competition. For example, the Ground Combat Vehicle has two contractors developing competitive prototypes of two key subsystems to support technology development. The program intends to conduct preliminary design reviews on both contractors’ designs prior to Milestone B and to conduct full and open competition through Milestone C. Similarly, according to program officials, the Joint Light Tactical Vehicle program had three contractors develop full-system prototypes during the technology development phase and held preliminary design reviews on each contractor’s design prior to Milestone B. The program plans to continue competition throughout engineering and manufacturing development. None of the four programs in our review received a waiver from Reform Act provisions. OSD is taking additional steps to strengthen the department’s oversight of weapon acquisition programs and guidance for developing the programs. In June 2011, for example, the Under Secretary for Acquisition, Technology and Logistics revised the weapons acquisition review process to consider acquired knowledge on weapon acquisition programs earlier than before. The revised review process includes two new review points. The first new review—the pre-engineering and manufacturing development review—occurs before the release of a final request for proposal for the engineering and manufacturing development phase. The purpose of this new review is to assess each program’s acquisition strategy, request for proposal, and key related planning documents earlier in the process, and to determine whether program plans are affordable, executable, and reflect sound business arrangements. The second new review—the acquisition strategy and request for proposals review and approval—occurs prior to Milestone C, the production decision. The review provides the milestone decision authority an opportunity to review the acquisition strategy and request for proposals for the production and deployment phase prior to Milestone C. Figure 2 illustrates the revised review process. According to the Under Secretary of Defense for Acquisition, Technology and Logistics, who is the authority for making milestone decisions for most major weapon acquisitions, the prior review process did not provide an adequate opportunity for review of program plans prior to release of the final request for proposals—the point at which DOD’s requirements, schedule, planned program content, and available funding should be firm and available for review. Further, the Under Secretary stated that making changes to acquisition strategies and program plans after all bidding activities, proposal evaluation, and source selection are complete is difficult and highly disruptive. DOD is also rewriting the DOD Instruction 5000.02 to include an extensive restructure of acquisition policies according to the Under Secretary of Defense for Acquisition, Technology and Logistics. This update will implement Section 832 of the National Defense Authorization Act for Fiscal Year 2012, which requires DOD to issue guidance on actions to be taken to improve its processes for estimating, managing, and reducing operation and support cost, as well as ensure competition in maintenance and sustainment of subsystems of major weapon systems, among other things. In addition to current policies implementing the Reform Act, officials stated that key provisions from the Reform Act will also be included in the updated instruction. In addition to implementing the provisions of the Reform Act, DOD offices have taken other steps to strengthen acquisition capabilities throughout the department. For example: The SE office, according to DOD officials, led efforts to establish working groups to help the services address systemic reliability issues across the unmanned aircraft and rotary wing portfolios earlier in the process. The office also led several workforce development initiatives to attract and retain a qualified engineering workforce and supported the implementation of legislation requiring each acquisition program office to name a key technical advisor who is responsible for all engineering activities. The DT&E office, according to DOD officials, championed updates to DOD Instructions that will require weapon acquisition programs to consider using DOD test capabilities before paying contractors to develop similar capabilities. In addition, the office supported legislation requiring major defense acquisition program offices to have a government test agency serving as the lead developmental test and evaluation organization for the program and a chief developmental tester. The chief developmental tester position, as included in the National Defense Authorization Act for Fiscal Year 2012, is tasked with coordinating the planning, management, and oversight of all developmental testing activities, among other things. The CAPE office, according to DOD officials, established an operating and support cost directorate to build its expertise and place more emphasis on developing better operating and support cost estimates throughout the acquisition life cycle. This directorate will coordinate the development of an operating and support cost estimating guidebook. The PARCA office, according to DOD officials, is providing additional insights to the Under Secretary of Defense for Acquisition, Technology and Logistics on systemic acquisition problems. Specifically, the office is examining a wide range of acquisition-related information from the past 40 years, such as contract type, stability of key performance parameters, and program manager tenure to determine if there is any statistical correlation between these factors and good or poor acquisition outcomes. We identified four key areas where the Reform Act had a significant influence on programs in the 3 years since it was enacted: (1) requirements, (2) cost and schedule, (3) testing, and (4) reliability. These four areas have been common sources of problems in the past. For example, the services typically started new weapon acquisition programs with requirements that were both demanding and inflexible and planned to use relatively unproven technologies to meet the requirements—all of which increased program risks. In addition, cost and schedule estimates were frequently too optimistic based on the proposed requirements and technologies. Design problems stemming from rigid requirements and the use of immature technologies to meet them were often discovered during testing and fixed late in the development cycle and resulted in cost increases, performance shortfalls, and schedule delays. Finally, DOD’s inattention to reliability has resulted in a dramatic increase in the number of systems that have not met suitability requirements during operational testing. Deficiencies—such as high failure rates and disappointing improvements in the reliability, availability, and maintainability of weapon systems—have limited program performance and increased operation and support costs. We examined 11 programs at various stages of the acquisition process to determine how the offices and policies established as a result of the Reform Act impacted their acquisition strategies and decision-making process. Four programs had not yet passed Milestone B, development start, at the time we began our review. Of the remaining seven programs, three had breached Nunn-McCurdy cost thresholds since the act was passed and have had to satisfy the Reform Act’s new requirements with regards to certification. The other programs had significant interaction with one or more of the OSD offices established by the Reform Act. Table 3 indicates in which of the four areas each program has been affected by the Reform Act. In some cases, programs have made changes based on input from OSD offices like systems engineering or developmental test and evaluation; in other cases, programs have integrated Reform Act policies, such as preliminary design reviews and competitive prototyping, into their acquisition planning. Programs that were already in development or production when the Reform Act was passed were less likely to have interactions with the OSD offices on requirements trades because these discussions typically occur prior to Milestone B. A discussion of how individual programs have been affected in the areas of requirements, cost and schedule realism, testing, and reliability follows. The Reform Act places significant emphasis on early problem solving and requires programs to put much more effort toward considering trade-offs among cost, schedule, and performance requirements prior to Milestone B. As part of this effort, it requires the Secretary of Defense to ensure that acquisition, budget, and cost estimating officials have the opportunity to raise cost and schedule matters before performance objectives are established. The Reform Act also charges the Joint Requirements Oversight Council with the responsibility to ensure that cost, schedule, and performance trade-offs for joint military requirements are considered, and to include combatant commanders in the process to ensure the user’s needs are adequately satisfied. The offices established as a result of the Reform Act have helped programs, such as the Joint Light Tactical Vehicle and Ground Combat Vehicle, make trade-offs among cost, schedule, and technical performance requirements. As a result, these programs have developed a more realistic acquisition strategy from a cost, schedule, and technical standpoint. Joint Light Tactical Vehicle: The program held several reviews prior to Milestone B to identify, modify, or eliminate requirements that were unachievable or unaffordable, thus establishing a more technically realistic program. Officials from DT&E and SE participated in these reviews. By involving both the requirements and acquisition communities in the reviews, the Army was able to reduce the required capability to cut costs while ensuring that trade-off decisions would not impair the system’s ability to meet the warfighter’s operational needs. Examples of requirements changes that helped to cut costs as well as reduce risk include: allowing the active suspension system, crew displays, and integrated starter-generator to be tradable design features. These changes resulted in a 30 percent reduction in the average unit manufacturing cost from the initial target of $475,000 to $331,000, while at the same time reducing technical and weight risk. According to program officials, this makes the $250,000 unit manufacturing goal more achievable. reducing the reliability requirement and changing the Army helicopter lift requirement based on the results of technology development prototype testing. This mitigated technical risks going into development. The program recently moved into engineering and manufacturing development, but not all requirements issues have been resolved and future trade-offs may be necessary. For example, early testing showed that none of the three prototype variants met the program’s soft soil or sand slope requirement. This requirement has not been changed. Program and OSD officials are monitoring this issue closely and plan to actively manage it during engineering and manufacturing development. Ground Combat Vehicle: The Ground Combat Vehicle program exhibited some of the same problems experienced by previous DOD programs prior to Milestone B—demanding and inflexible requirements. The SE office and the Under Secretary of Defense for Acquisition, Technology and Logistics are helping the program set achievable requirements. Following its materiel development decision in February 2010, the program issued a request for proposals that contained nearly 1,000 requirements and a challenging 7-year schedule for the delivery of the first production vehicle. At the request of the Under Secretary for Acquisition, Technology and Logistics, the Army established an independent review team to assess the risks associated with the program’s schedule. The team, which included an SE official, raised concerns about the program’s high number of mandatory requirements and the risks associated with the 7-year schedule. To mitigate program risks, the Army reduced the number of performance requirements by about 25 percent and prioritized the others, giving competing contractors flexibility in addressing some performance requirements. The Army issued a revised request for proposal in November 2010. In August 2011, the Under Secretary of Acquisition, Technology and Logistics approved the program’s entry into technology development, but expressed concern about the cost and schedule risks associated with delivering a production vehicle in 7 years. Because of these concerns, the Under Secretary directed the Army to consider other alternatives, such as existing vehicles, that could meet warfighter needs. The analysis is currently planned to be completed in March 2013 to inform the Milestone B decision. By establishing a new cost assessment and program evaluation office and requiring this office to scrutinize program cost and schedule estimates beginning at Milestone A, CAPE officials believe that the Reform Act has helped infuse more realism in cost estimates and promote earlier discussions about affordability. CAPE officials also believe that because their independent cost and schedule estimates have become more visible within DOD and Congress, the military services are developing more realistic estimates. We saw evidence of these benefits in the programs we reviewed, including the Ohio Class Replacement, Littoral Combat Ship Seaframe, and F-35 Joint Strike Fighter programs. Ohio Class Replacement: The CAPE office was involved in the decision-making process to ensure program affordability. The office prepared an independent cost estimate and reviewed the program’s affordability goals prior to Milestone A. The service and independent estimates were within 2 percent of each other. However, the Under Secretary of Defense for Acquisition, Technology and Logistics directed the Navy to do a rigorous cost comparison of a 16 missile tube design versus a 20 missile tube design. The Navy determined that a 16 missile tube configuration would meet warfighter requirements and users’ needs while reducing program costs by about $200 million per submarine, or approximately $3 billion for the total program. It would also simplify the ships’ design and integration effort. The CAPE office validated the savings associated with the 16 missile tube design. As a result, the Navy incorporated the 16 missile configuration as the program baseline. Littoral Combat Ship Seaframe: The CAPE office helped make program costs more visible. Prior to the program’s Milestone B decision, CAPE completed an independent cost estimate of the seaframe program and found that the resources in the future years’ defense plan budget were lower than the projected program costs for the same time period. Navy officials attributed this problem to the overlap between the timing of the milestone decision and the president’s budget submission. The office further noted that the resources in the Navy’s budget did not include the additional development activities required to support two full ships. Without this information, decision makers would not have had visibility into the expected costs of the seaframe program or be able to make more fully informed decisions. As a result, the Navy re-phased its funding in the budget, adding approximately $397 million to fully fund the development program. Joint Strike Fighter: The SE and CAPE offices helped the program develop more realistic cost and schedule estimates. CAPE officials have been involved in reviews of the F-35 Joint Strike Fighter program even prior to the passage of the Reform Act and have continued to be heavily involved in subsequent programs reviews. For example, the cost analysis improvement group, which was the predecessor to CAPE, led a multi-functional joint estimating team review of the program in 2008. This review found problems with the program’s funding and schedule. In 2010, the Joint Strike Fighter program notified Congress that its estimated unit costs had increased by more than 80 percent since the original Milestone B baseline in 2001. This increase triggered a Nunn-McCurdy unit cost breach and later prompted the program executive officer to commission a technical baseline review of the program to help determine the resources needed to complete development. Officials from the SE office participated in this technical review. The CAPE office also did an independent cost estimate of the program as part of the Nunn- McCurdy certification process. Based on information from these efforts, DOD developed a more realistic program plan by adding $4.6 billion to the development program, reducing near-term procurement quantities by 125 aircraft, and extending the development test period by 4 years to accommodate developmental testing, address the increased program scope, and fix software issues. The Reform Act significantly strengthened the role of developmental testing in the department. In the 2 decades prior to the Reform Act, the prominence of developmental testing had declined within OSD. In the early 1990’s, developmental testing was part of an all-encompassing test organization that reported directly to the Under Secretary of Defense for Acquisition. According to a former senior developmental testing official, by 2004, two people worked on developmental testing activities within the systems engineering organization. In establishing a separate office for DT&E, the Reform Act reinforced the need for robust developmental testing early in the acquisition process. The Reform Act gave the Deputy Assistant Secretary for Developmental Test and Evaluation formal approval authority for the test and evaluation master plans of major defense acquisition programs. This authority enables the Deputy Assistant Secretary to help ensure that programs have robust test and evaluation plans. Our case study reviews illustrate the efforts that the DT&E office has made to help programs such as the Small Diameter Bomb II and KC-46 Tanker obtain more design and performance knowledge early in a program’s acquisition life cycle. Small Diameter Bomb II: Air Force program officials acknowledged that developmental and operational testing officials worked closely with them as they prepared the test and evaluation master plan for the Milestone B decision. After reviewing the plan, developmental and operational test officials concluded that the program would benefit from adding a 28-shot test program prior to entering operational testing. According to the program office, the purpose of the additional testing is to further establish the performance of the weapon in realistic scenarios and to increase the likelihood of completing operational testing without a failure. DT&E officials stated this testing would provide more complete knowledge about the bomb’s functionality and help reduce risk of a major redesign moving forward. Program officials stated that they allocated an additional $41 million to its developmental test program to conduct the 28 additional flight tests prior to operational testing. KC-46 Tanker: The program office acknowledged that the DT&E office, as part of an integrated test team comprised of government and industry officials, helped identify options that could add time to the test plan for important testing if unexpected delays are encountered. In its fiscal year 2011 annual report, the operational test and evaluation office reported that the KC-46’s planned flight test program was not executable, determining that more time would be needed for military flight-testing. It based this conclusion on the historical flight test experience of similar programs. Program officials stated that they were initially reluctant to change the test plan because they had awarded a fixed price contract and any changes could result in reopening the contract, leading to potential cost increases. However, the integrated test team identified a recovery period that may be applied to the KC-46 aerial refueling certification if delays are encountered. The contractor now has a plan that could allocate an additional 1.5 months for two test aircraft to complete this testing, if deemed necessary. This testing would provide more knowledge about the program’s aerial refueling performance prior to operational testing. DT&E officials stated that they plan to continue working with the program to address overall flight test challenges. While testing remains one of the program’s risk areas, this change may lessen that risk. The Reform Act emphasizes the need for designing more reliable weapon systems. It charges the Deputy Assistant Secretary for Systems Engineering with the responsibility to ensure the systems engineering approach used by major acquisition programs includes a robust plan for improving reliability. The DT&E office reviews programs’ reliability growth test plans. This testing provides visibility over how reliability is improving and uncovers design problems so fixes can be incorporated before production begins. A reliability growth curve is used to track projected and actual improvements in reliability over time. The Reform Act further requires that the Deputy Assistant Secretary for Systems Engineering develop policies and guidance for the inclusion of provisions relating to systems engineering and reliability growth in requests for proposals. We observed evidence of this increased emphasis in the Joint Light Tactical Vehicle, Remote Minehunting System, Gray Eagle, and Global Hawk programs. Joint Light Tactical Vehicle: The DT&E office helped this program develop a more realistic reliability growth plan prior to Milestone B. Based on the performance of prototype vehicles, developmental test officials determined that the program’s reliability growth curve was unrealistic. For example, officials reported that the program’s initial reliability growth plan assumed a starting reliability that was almost 60 percent higher than what had actually been demonstrated during technology development. It also assumed commonality between the two vehicle variants, a large reliability increase in a short test time, and two corrective action periods. The DT&E office recommended that the program eliminate the vehicle commonality assumption, add more test miles, and add another corrective action period to its test plan. It also recommended that the program consider lowering the vehicle’s reliability requirement. Based on this input, the Army revised its plan by adding two vehicles and 40,000 more test miles to ensure reliability is adequately addressed for both variants. With approval of the user, it also reduced the reliability requirement from 3,600 to 2,400 miles mean time between operational mission failures. Remote Minehunting System: The SE office worked with program officials to improve reliability growth planning, which was found to be one of the key factors leading to the program’s Nunn McCurdy unit cost breach in 2009. Before the breach, program officials had not funded a reliability growth program or established a design for reliability process. The program had a reliability goal of 150 hours mean time between failures, but program officials stated that testing demonstrated a reliability of only about 45 hours. Since the breach, the program has worked closely with the SE office to establish a reliability program plan and a growth curve to track reliability improvements. During the Nunn-McCurdy review, the program developed a three phase reliability growth program to improve the program’s subsystems, components, and manufacturing processes that contributed to poor reliability. According to program officials, phase one of the reliability growth program was completed in 2011, and reliability has improved by 40 percent, going from 45 hours mean time between operational mission failures to 63 hours. Although this improvement is still below the minimum requirement of 75 hours, program officials stated that phase two of the reliability growth program is scheduled to be completed in April 2013 and is projected to achieve the program’s 75 hour minimum requirement. Gray Eagle and Global Hawk: The SE office has worked to improve reliability across the unmanned aircraft portfolio, including the Gray Eagle and Global Hawk. Prior to the Gray Eagle’s second low rate initial production decision in 2011, SE officials raised concerns about the system’s poor reliability. As a result, the Army was directed to undertake a reliability improvement program. The Under Secretary of Defense for Acquisition, Technology and Logistics approved the program for low rate initial production, but stressed the need to improve the operational reliability as quickly as possible. SE officials worked with program officials to establish a reliability working group, develop reliability growth curves, and develop a reliability enhancement management plan. According to SE officials, the Gray Eagle initially improved the reliability of the aircraft by 15 percent and the ground control station by 30 percent. According to program officials, the initial reliability goals were overstated and not needed to meet the program’s overall operational availability requirement. Based on initial operational test results in August 2012, the program office is working with the user to redefine the reliability goals without impacting the system’s ability to meet its overall operational availability requirement. According to the PARCA office, these efforts have been informed by a detailed reliability model that they built in consultation with the Army. This model showed the relationship between the aircraft’s reliability and its availability to perform operational missions. SE officials also found similar reliability problems on the Global Hawk program and worked with program officials to establish a reliability growth and improvement plan and reliability growth curves. According to SE officials, the time between unscheduled maintenance on the Global Hawk has improved on the order of 50 to 80 percent. While DOD has taken steps to implement most of the fundamental Reform Act provisions, some key efforts to date have been primarily focused on DOD’s largest major defense acquisition programs. Expanding the reach of the Reform Act to bring about systemic change to DOD’s weapons acquisition process so that it influences all programs, however, still has challenges. Although senior leaders were receptive to the Reform Act principles, they identified several challenges that currently limit DOD’s ability to broaden the Reform Act’s influence. We grouped these challenges into five general categories: (1) organizational capability constraints; (2) need for additional guidance on cost estimating and Reform Act implementation; (3) uncertainty about the sufficiency of service level systems engineering and developmental testing resources; (4) limited dissemination of lessons learned; and (5) cultural barriers. Leaders of two of the offices established as a result of the Reform Act told us that even though they have implemented most of the fundamental Reform Act provisions, they have had to limit their activities to a portion of acquisition programs in their portfolios due to resource constraints. Thus, it is doubtful that they could expand the scope of their activities to include more weapon acquisition programs at current staffing levels. For example, the DT&E office has had to be selective in its level of oversight of acquisition programs because the current staff of around 70 government and contractor personnel cannot adequately cover a portfolio of over 200 acquisition programs, according to its Deputy Assistant Secretary. The office has dropped virtually all but the largest programs from its oversight list and eliminated oversight of some major automated information systems. CAPE officials estimated that its cost assessment division would need to double in size in order to meet the Reform Act’s requirements. However, soon after the Reform Act was enacted, budgetary constraints limited the expansion of the cost estimating workforce to about 25 percent of the necessary growth. According to CAPE officials, its current cost analysis staffing is not adequate to meet its mission of improving the analytical skills of the defense cost estimating workforce, issuing policy, and providing sound and unbiased cost and schedule estimates. The office has delegated its independent cost estimating responsibility for most major automated information systems to the military services and some guidance has yet to be issued. The SE and PARCA offices are also struggling in some regards. For example, according to its Deputy Assistant Secretary, the SE office is continuously challenged to maintain the high caliber, qualified personnel required to provide assistance to and oversight of its portfolio of over 200 acquisition programs. Further, PARCA officials stated that the availability of government positions, particularly at the senior executive service level, continues to be a critical issue for the office. The two divisions within the PARCA office, the performance assessments division and the root cause division, do not currently have permanent government personnel at the senior executive level. Officials also stated current proprietary information rules limit the ability of PARCA contractor personnel to handle and maintain some weapon system information, severely impeding operations. Offices within OSD have not yet issued more detailed guidance that could help institutionalize better cost estimating practices and steer program decisions related to competitive prototyping and preliminary design reviews. The CAPE office has not issued guidance for operating and support costs estimates, such as fuel and maintenance costs, that have been estimated to account for two-thirds or more of a system’s total life cycle cost. In addition, although not specifically required by the Reform Act, the CAPE office has not issued guidance for the services to use when developing Milestone A program cost estimates. As a result, senior leaders may not have access to realistic cost estimates prior to Milestone B for decision making purposes. Military service officials told us they are particularly interested in getting guidance on data that should be included in the cost analysis requirements description, which forms the basis for their cost estimates at Milestones B and C. CAPE officials recognize that, while some progress has been made, they need to complete the guidance, but have not been able to dedicate resources to do so. Some officials also told us that they find the competitive prototyping and preliminary design review requirements confusing and would like guidance on how to implement these requirements. DOD policy requires the technology development strategy for major defense acquisition programs to provide for prototypes of the system or, if a system prototype is not feasible, for prototypes of critical subsystems before Milestone B approval. However, officials from the Ground Combat Vehicle program were unclear as to when and what type of prototype to use. From a broader perspective, other military officials questioned the value of competitive prototyping as a blanket requirement for all programs, especially for programs that are using mature technologies, given the cost. For example, senior acquisition officials questioned the necessity of spending $400 million on competitive prototyping for the Small Diameter Bomb II program since the program was aware of problems with one contractor’s design. However, program officials indicated that competitive prototyping enabled them to identify design issues early in development and realize a savings of $1 billion. Officials from the Ground Combat Vehicle program we spoke with also indicated that they struggled with the timing of when to hold the program’s preliminary design review and what type of knowledge was required, since better guidance is needed. The program plans to hold multiple design reviews prior to Milestone B to consider contractor and government designs of the weapon system and then hold another review after Milestone B in order to resolve differences between the government’s and selected contractor’s preliminary designs. We spoke with OSD officials to determine which office should be providing guidance or assistance to program managers on competitive prototyping and preliminary design review issues. None of the offices have official responsibility for these efforts. OSD officials stated that these are program decisions and should be discussed with their respective military service level acquisition officials. OSD officials believe that the services may lack resources in key positions that could help strengthen systems engineering and developmental testing activities on weapon acquisition programs. For example, according to the Deputy Assistant Secretary of Defense for Systems Engineering, the Navy and Air Force have reassigned the duties and responsibilities of their service-level chief engineers, thereby de- emphasizing the importance of systems engineering. The Deputy Assistant Secretary believes maintaining strong systems engineering leadership at the service level is essential for tying the systems engineering community together and promoting good systems engineering practices throughout each respective service. According to the DT&E and SE office’s March 2012 joint annual report to the Congress, the Navy abolished its chief engineer position and while the Air Force recently began to take steps to relocate the systems engineering function to the headquarters level, the impact of a recent reorganization on systems engineering activities is not yet known. In addition, the Deputy Assistant Secretary for Developmental Test and Evaluation expressed concern that the military services may not be implementing new legislation that requires each major defense acquisition program be supported by a chief developmental tester that oversees developmental test and evaluation activities. He stated that in some cases one person is serving as the chief developmental tester across multiple programs instead of having one person dedicated specifically to each program. The Deputy Assistant Secretary is trying to determine the extent to which this practice is occurring and then plans to work with the services to get more focused leadership for each program. It is also unclear whether the services have a sufficient number of qualified personnel to conduct systems engineering and test and evaluation activities. The services planned to grow these workforces by over a combined 5,000 people between fiscal years 2009 and 2015 and had made progress in growing each of these workforces through fiscal year 2010. However, budget cuts have resulted in DOD canceling some of its weapon acquisition programs and reassessing its decision to increase the acquisition workforce. Last year, we recommended that the Secretary of Defense report the impact budget cuts were having on the military service workforce and their ability to meet weapon acquisition program needs in the areas of developmental testing and systems engineering. In the DT&E and SE offices’ March 2012 joint annual report to the Congress, the Deputy Assistant Secretary for Systems Engineering reported that the Army has reduced its systems engineering workforce growth plan as compared to the plan reported in March 2011 joint annual report, and that contractor-to-civilian conversions have been suspended. In addition, the Deputy Assistant Secretary believes a prolonged hiring freeze in the Air Force could potentially create new experience gaps in the workforce. The Deputy Assistant Secretary for Developmental Test and Evaluation did not discuss the impact of budget cuts on the services’ test and evaluation workforce growth plan in the March 2012 joint annual report to the Congress and neither office reported on whether the services had an adequate workforce to meet the needs of the current portfolio of weapon acquisition programs. DOD has not taken full advantage of sharing lessons learned obtained through root cause analyses of programs that experience Nunn-McCurdy cost and schedule breaches with the acquisition workforce, particularly program managers. According to the Defense Acquisition Guidebook, which provides best practices the acquisition workforce can use on programs, lessons learned are a tool that the program manager may use to help identify potential areas of risk associated with a weapon acquisition system by reviewing the experiences encountered in past programs. Lessons learned databases document what worked and what did not work in past programs, in the hopes that future programs can avoid the same pitfalls. Further, if the right best practices are applied, they help to avoid common problems and improve quality, cost, or both. The PARCA office has made some effort to educate program managers on how to avoid acquisition problems through classes taught at the Defense Acquisition University. However, these courses are geared towards educating new program managers and may not be reaching a wide range of program officials. Nevertheless, some officials indicated that this information would be helpful for program officials to understand and avoid problems that have affected weapon acquisition programs in the past. Other officials also stated that it would be helpful if root cause analysis assessments contained more detailed information so acquisition officials could better understand problems and apply lessons learned. For example, when cost estimating was determined to be a root cause of a problem, officials stated they would have found it more beneficial to know if immature technologies or unrealistic requirements were the basis for the poor cost estimate. Perhaps the most difficult challenge the department faces in making systemic changes to the acquisition process is changing the cultural relationship between the military services, which fund and develop new weapon acquisition programs, and OSD offices, which provide advice to and oversee the programs. Senior military service officials have told us they believe they understand and can manage the risks of specific weapon acquisition programs without much assistance from OSD. On the other hand, OSD officials believe more assistance is needed, as evidenced by the high number of programs that have experienced Nunn- McCurdy breaches and poor operational testing results. For example, since it was established in 2009, the DT&E office has assessed whether 15 programs were ready to begin operational testing. The office recommended that 5 of the programs—Global Hawk Block 20/30, Standard Missile 6, Joint Tactical Radio System Handheld Manpack Small Form (HMS) Rifleman Radio, Joint Tactical Radio System HMS Manpack, and MQ-1C Gray Eagle—not proceed into operational testing. However, military service acquisition chiefs decided to allow all 5 of these programs to proceed anyway. Four of the programs—Global Hawk Block 20/30, Standard Missile 6, Joint Tactical Radio System HMS Rifleman Radio and Joint Tactical Radio System HMS Manpack—demonstrated poor performance in operational testing, in areas such as reliability, effectiveness, or suitability. Operational testing results for the MQ-1C Gray Eagle have not yet been reported. On the other hand, a few service officials we met with were reluctant to accept some recommendations made by OSD offices because they believed the recommendations were overly burdensome and could significantly impact weapon acquisition programs’ cost and schedule outcomes without a lot of benefit. This was the case for the KC-46 Tanker program, where program officials were concerned that additional testing recommended by the developmental test and evaluation, and operational testing offices, as part of the integrated test team, could have significant contractual implications during development. In this case, officials identified additional flight test opportunities without having to renegotiate the fixed price contract. However, the additional allotted test time is not equivalent to the 6 to 8 months the developmental testing office felt should be added. A similar situation occurred on the Ship to Shore Connector program. The Navy disagreed with a DT&E office recommendation to conduct full system testing prior to procuring additional craft during initial production. The DT&E office believed the program was high risk because the Ship to Shore Connector was a complete redesign of a previous system with no reuse of any major component (engines, gearboxes, hydraulics, command and control software). Navy officials, however, believe the program is low risk since it is an evolutionary program and has one critical technology, a fire suppression system, which has already been sufficiently demonstrated and qualified through test and evaluation. In addition, the Navy estimated that it would cost $15 million to revise the existing production schedule to accommodate the full system testing as recommended by the DT&E office. The DT&E office and the Navy reached a compromise whereby OSD would review available system test results before more craft are authorized. Current fiscal pressures, along with the threat of more to come, have DOD officials looking for ways to increase buying power by controlling cost and schedule overruns on weapon acquisition programs. The offices established as a result of the Reform Act, as well as policy provisions have helped DOD make inroads towards putting weapon acquisition programs on more solid footing. Together, the offices and policy provisions place more attention on requirements, costs, testing, and reliability as early as Milestone A. The provisions of the act, when specifically focused on newer programs, are having a positive impact on the programs and the acquisition process. They show that expert attention to the cost and achievability of capability requirements, the assumptions made for cost and funding of programs, and the amount of systems engineering knowledge that is brought to bear early make programs more executable. Although senior officials we spoke with throughout the department are receptive to the broad principles of the Reform Act, it is too early to tell if the Reform Act is going to result in systemic change to DOD’s weapon acquisition process. DOD faces several challenges that must be addressed to get lasting change—organizational capability constraints, the need for additional cost estimating and implementation guidance, the possibility of insufficient systems engineering and developmental testing resources, limited dissemination of lessons learned, and cultural barriers between OSD and the services. Some challenges appear to be straight forward to address, such as providing guidance for estimating operating and support costs, providing additional guidance for conducting preliminary design reviews and competitive prototyping activities, and disseminating lessons learned to the broader acquisition community. However, they may require more resources, which have been difficult to obtain. For Reform Act policies and practices to have a systemic effect across the entire portfolio of weapon system acquisition programs, the department must also address challenges related to systems engineering and developmental testing resources and cultural barriers between OSD and the services. This begins with the services identifying key leaders at the headquarters level and within program offices to guide systems engineering and developmental testing efforts and then ensuring that there are enough trained staff to carry out these activities. OSD will need to continue monitoring the services’ efforts. It will also require an environment where the services stop proposing new weapon systems with inflexible requirements, immature technologies, and cost, schedule, and funding assumptions that are too optimistic at the start of a program. Breaking down cultural resistance to change will take more cooperation between the Under Secretary for Acquisition, Technology and Logistics and other OSD offices, and service acquisition executives to address, as well as continuity of leadership. Efforts by the PARCA office to identify factors that correlate to good or poor acquisition outcomes, particularly as it relates to program manager tenure, will be beneficial. The services’ ability to demonstrate that the Reform Act is influencing all weapon acquisition programs, not just the biggest, will be a key indicator for determining whether the Reform Act has had a positive effect on DOD’s culture. We recommend that the Secretary of Defense take the following four actions to enable systemic change across the entire portfolio of weapon acquisition programs: direct the Director of Cost Assessment and Program Evaluation to issue guidance for estimating weapon acquisition program costs at Milestone A and operating and support costs throughout the acquisition life cycle by the end of fiscal year 2013 and ensure that the office prioritizes its resources accordingly to accomplish this task; designate responsibility for providing advice and guidance to program offices on competitive prototyping and preliminary design reviews to the appropriate organization within the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics and ensure that the guidance is developed. The office(s) designated would be the focal point for addressing program office issues related to the practical implementation of these Reform Act provisions, such as the type of competitive prototyping to use, the timing and benefits of holding preliminary design reviews prior to milestone B, and if a preliminary design review should be held after milestone B; direct the Deputy Assistant Secretaries of Defense for Systems Engineering and Developmental Test and Evaluation to assess and include in their annual report to the Congress beginning with the report on fiscal year 2012 activities: the extent to which the office can perform their required activities with allocated resources; the impact budget cuts are having on the military services total workforce (civilians, military, and contractors) and ability to meet program office needs; and progress the services have made filling leadership positions, such as chief engineers at the service level and technical leads for systems engineering and developmental testing at the program office level; direct the Director of Performance Assessments and Root Cause Analyses to make lessons learned collected during its root cause analysis evaluations available to the acquisition workforce and ensure that the office prioritizes its resources accordingly. DOD provided us written comments on a draft of this report. DOD concurred with two recommendations and partially concurred with two others. DOD’s comments appear in appendix III. DOD also provided technical comments, which we incorporated as appropriate in the report. DOD agreed with the intent of our first recommendation, but noted that due to resource constraints, the Cost Assessment and Program Evaluation office could not guarantee that it would be able to issue guidance for estimating major defense acquisition program costs at Milestone A and operating and support costs throughout the acquisition lifecycle by the end of fiscal year 2013. We continue to believe that the Cost Assessment and Program Evaluation office should issue the guidance by the end of fiscal year 2013. However, if that is not possible from a resource standpoint, the office should commit to a date and devote the resources to meeting that date. We will continue to monitor DOD’s efforts to develop the guidance. Although DOD concurred with our second recommendation, we revised this recommendation based upon discussions with DOD officials during the agency comment period. Our revision clarified the intent of this recommendation, which is to have the Secretary of Defense designate a specific organization within the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics to provide advice and guidance on competitive prototyping and preliminary design reviews. We understand that the department has issued Reform Act implementation guidance and has incorporated aspects of competitive prototyping and preliminary design reviews in the Defense Acquisition Guidebook. Further, we recognize that program offices we visited are taking steps to implement the guidance that has already been issued. However, based on our discussions with senior level officials, we believe one or more offices need to be designated with the responsibility of developing additional guidance and answering program specific questions related to the practical implementation of the requirements. As noted earlier in our report, some officials questioned when to use prototyping or what type of prototyping should be used. In addition, there were questions about the timing of the preliminary design reviews. DOD partially concurred with our third recommendation. DOD noted that the type of information we recommended be assessed and reported on should be included as part of DOD’s human capital strategic planning process and as such, be reported in DOD’s annual Acquisition Workforce Strategic Plan. We agree that the impact of budget cuts on the workforce and the status of leadership positions could be addressed in the annual strategic plan. However, we continue to believe that the Deputy Assistant Secretaries for Systems Engineering and Developmental Test and Evaluation should include an assessment in their joint annual report to the Congress on the respective offices’ ability to perform activities specified in the Reform Act with available resources. DOD concurred with our fourth recommendation, which would make lessons learned from root cause analyses available to the acquisition workforce. We are sending copies of this report to the Secretary of Defense and appropriate Congressional Committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you have any questions about this report or need additional information, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were Cheryl Andrew, Assistant Director; Laura Greifner, Julie Hadley, Megan Porter, Rae Ann Sapp, and Marie Ahearn. This report examines DOD’s continued implementation of the Weapon Systems Acquisition Reform Act of 2009 (Reform Act). Specifically, we examined (1) DOD’s progress in implementing Reform Act provisions; (2) the impact the Reform Act has had on specific acquisition programs; and (3) challenges remaining in improving the weapons acquisition process. To assess DOD’s progress in implementing Reform Act provisions, we interviewed officials and analyzed documents, such as reports to the Congress and guidance issued from the Office of the Secretary of Defense (OSD) offices of the (1) Deputy Assistant Secretary of Defense for Systems Engineering (SE), (2) Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DT&E), (3) Cost Assessment and Program Evaluation (CAPE), and (4) Performance Assessments and Root Cause Analyses (PARCA) to determine the extent to which provisions have been implemented. We focused our review on the offices’ implementation of four fundamental Reform Act provisions: developing policy and guidance; approving acquisition documents; monitoring programs and conducting program assessments; and developing performance measures. In cases where provisions had not been implemented, we asked officials about the reasons for the delay and the expected time frame for completion. We also interviewed officials and analyzed documents from the office of the Assistant Secretary of Defense for Research and Engineering and the Defense Procurement and Acquisition Policy office, as well as four weapon acquisition programs, which had not yet started development to determine the progress DOD has made implementing Reform Act provisions related to preliminary design reviews, competitive prototyping, and competition. We believe these programs offer the best glimpse at how the OSD offices and Reform Act policies are influencing acquisition strategies. The weapon acquisition programs we chose for this analysis were part of a larger case study review that is described below. To determine the impact the Reform Act has had on specific weapon acquisition programs, we selected 11 weapon system programs to use as case studies. For each program we reviewed relevant program documentation such as the test and evaluation master plans, assessments of operational test readiness, systems engineering plans, program support reviews, root cause analyses, analysis of alternatives reports and cost estimates as applicable. We also interviewed appropriate program officials and officials from the OSD offices for SE, DT&E, and CAPE to obtain their perspectives about (1) the level of interaction between the programs and OSD offices; (2) changes made to program acquisition strategies as a result of interactions with the OSD offices; and (3) benefits and challenges with implementing Reform Act provisions on each of the programs. We also reviewed the performance assessments and root cause analyses office’s root cause analysis documentation for programs that incurred Nunn-McCurdy cost or schedule breaches. We selected our case studies based on input from the officials in the OSD offices for SE, DT&E, CAPE, PARCA, and operational test and evaluation. We also discussed possible case studies with GAO employees who monitor and report on weapon acquisition programs on an annual basis. The programs we selected for review represent a variety of platforms, including sea vessels, manned and unmanned aircraft, and land systems. Specifically, we examined 11 programs at various stages of the acquisition process. Four programs had not yet passed Milestone B, development start, at the time we began our review. The remaining seven programs had completed their Milestone B review and were in development at the time of our case study selection. Of the seven programs, three have breached Nunn-McCurdy cost thresholds since the act was passed and have had to satisfy the act’s new requirements with regards to certification. The other programs had significant interaction with one or more of the OSD offices established by the Reform Act. A complete list of programs is provided below. While our sample of 11 case studies allowed us to learn about the impact the Reform Act offices have had on DOD acquisitions, it was designed to provide anecdotal information, not findings that would be representative of all the department’s weapon acquisition programs. To determine challenges remaining in improving defense acquisitions we relied on information we collected and analyzed during our case study review of 11 weapon acquisition programs. We also solicited the opinions of the Under Secretary of Defense for Acquisition, Technology and Logistics; other senior level officials in the Office of the Secretary of Defense including the leaders of each of the offices created as a result of the Reform Act; as well as the military services’ Senior Acquisition Executives. We conducted this performance audit from January 2012 to December 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Progress of Reform Act Offices in Implementing Weapon System Acquisition Reform Act Provisions Status Complete: Issued a reliability, availability, and maintainability Directive-Type Memorandum, a development planning Directive-Type Memorandum, and a DOD Instruction for DASD (Systems Engineering), participated in JCIDS revisions, developed guidance for incorporating systems engineering into development contracts, streamlined the Systems Engineering Plan and Program Protection Plan, and released an update to the Defense Acquisition Guidebook, chapter 4. Continue to refine policies and guidance as necessary. Completing on annual basis: Approved 52 Systems Engineering Plans since 2009, including 15 in fiscal year 2011. Completing on annual basis: Review portfolio of 234 programs. In fiscal year 2011, participated in 73 overarching integrated product team meetings, 6 peer reviews of acquisition contracts, and conducted 15 Program Support Reviews. Complete: Developed a set of time-based metrics to assess each program’s ability to execute its system engineering plans and address risks the office had identified in prior reviews. The metrics measure program cost, schedule, staffing, reliability, availability and maintainability, software, integration, performance and manufacturing, and are to be incorporated into each program’s systems engineering plan and evaluated at various points in the development process. Status Complete: Updated guide for incorporating test and evaluation into acquisition contracts. Championed updates to DOD Instruction assigning responsibilities and authorities to Developmental Test and Evaluation office, which is in the process of being updated. Updated guidance to include reliability factors in the Test and Evaluation Master Plan. Continue to refine policies and guidance as necessary. Completing on annual basis: Reviewed and approved 186 Test and Evaluation Master Plans since 2009, including 44 in fiscal year 2011. Monitor programs / conduct assessments Completing on annual basis: Review portfolio of nearly 250 programs. In fiscal year 2011, participated in 22 defense acquisition board meetings and 59 overarching integrated product team meetings. The office has also conducted 16 Assessment of Operational Test Readiness reviews since 2009. In process: Piloted performance measures on two programs. The measures were then updated and are being applied to over 40 programs that were selected for reporting in the fiscal year 2012 joint annual report. The assessments are being used to support the write- up of the program engagement section. Status In process: Issued first policy document in May 2012, which is the basis for additional policy documents. Updating its Operating and Support Cost Estimating Guidebook, which will address the Reform Act requirement for DOD to issue guidance related to full consideration of life cycle management and sustainability costs in major defense acquisition programs. Not applicable: The Reform Act does not require that the office approve acquisition documents. Monitor programs / conduct assessments Completing on annual basis: Conducted independent cost assessments for Milestone A and B certification on 30 future and current major defense acquisition programs, since 2009, including 8 in fiscal year 2011. The office conducted 3 Milestone C and 3 Nunn- McCurdy certification reviews in fiscal year 2011. Not applicable: The Reform Act does not require that the office develop performance measures. Status In process: Developing guidance to assist offices in conducting root cause analyses. The guidance for conducting performance assessments is expected to be released in early fiscal year 2013. Not applicable: The Reform Act does not require that the office approve acquisition documents. Monitor programs / conduct assessments Completing on annual basis: Completed 14 Root Cause Analyses for programs which have undergone a Nunn-McCurdy breach or were requested by OSD and has completed 26 semi-annual follow-up reports on these programs. Providing OSD with the execution status of DOD’s portfolio of acquisition programs through the Defense Acquisition Executive Summary process. In process: Utilize Defense Acquisition Executive Summary information to identify cost performance, schedule, funding, and technical performance issues on Major Defense Acquisition Programs. Continuing to develop performance measures.

Summary: For the past 3 years, DOD has been implementing the Reform Act requirements which are aimed at helping weapon acquisition programs establish a solid foundation from the start. This helps to prevent cost growth, thus helping the Defense dollar go further. This is the third in a series of GAO reports on the Reform Act. GAO was asked to determine (1) DOD's progress in implementing Reform Act provisions; (2) the impact the Reform Act has had on specific acquisition programs; and (3) challenges remaining. To do this, GAO analyzed documents and interviewed officials from the four OSD offices created as a result of the Reform Act, other DOD offices, the military services, and 11 weapon acquisition programs we chose as case studies. Case study programs were selected based on their development status and interaction with the four OSD offices. Results cannot be generalized to all DOD weapon acquisition programs. The Department of Defense (DOD) has taken steps to implement fundamental Weapon Systems Acquisition Reform Act of 2009 (Reform Act) provisions, including those for approving acquisition strategies and better monitoring weapon acquisition programs. DOD is also continuing to take additional steps to strengthen policies and capabilities. Some provisions, such as issuing guidance for estimating operating and support costs, are being implemented. GAO's analysis of 11 weapon acquisition programs showed the Reform Act has reinforced early attention to requirements, cost and schedule estimates, testing, and reliability. For example, prior to starting development, an independent review team raised concerns about the Ground Combat Vehicle program's many requirements and the risks associated with its 7-year schedule. Subsequently, the Army reduced the number of requirements by about 25 percent and prioritized them, giving contractors more flexibility in designing solutions. In addition, the developmental test and evaluation office--resulting from the Reform Act--used test results to help the Joint Light Tactical Vehicle program develop a more realistic reliability goal and a better approach to reach it. While DOD has taken steps to implement most of the fundamental Reform Act provisions, some key efforts to date have been primarily focused on DOD's largest weapon acquisition programs. DOD faces five challenges--organizational capability constraints, the need for additional guidance on cost estimating and Reform Act implementation, the uncertainty about the sufficiency of systems engineering and developmental testing resources, limited dissemination of lessons learned, and cultural barriers between the Office of the Secretary of Defense (OSD) and the military services--that limit its ability to broaden the Reform Act's influence to more programs. Service officials believe additional guidance is needed to improve their cost estimates and other implementation efforts. They also believe that lessons learned from programs that experience significant cost and schedule increases should be shared more broadly within the acquisition community. These challenges seem straightforward to address, but they may require more resources, which have been difficult to obtain. Ensuring the services have key leaders and staff dedicated to systems engineering and developmental testing activities, such as chief engineers at the service level and technical leads on programs, as well as breaking down cultural barriers are more difficult to address. They will require continued monitoring and attention by the Under Secretary for Acquisition, Technology and Logistics, service acquisition executives, and offices established as a result of the Reform Act to address. GAO recommends DOD develop additional cost estimating and Reform Act implementation guidance; make lessons learned available to the acquisition community; and assess the adequacy of the military services' systems engineering and developmental testing workforce. DOD generally concurred with the recommendations. GAO clarified one recommendation to make it clear that DOD needs to designate an office(s) within the Acquisition, Technology and Logistics organization to provide practical Reform Act implementation guidance to program offices.