The Department of the Navy’s (DON) primary mission is to organize, train, maintain, and equip combat-ready naval forces capable of winning wars, deterring aggression by would-be foes, preserving freedom of the seas, and promoting peace and security. Its operating forces, known as the fleet, are supported by four systems commands. Table 1 provides a brief description of each command’s responsibilities. To support the department’s mission, these commands perform a variety of interrelated and interdependent business functions (e.g., acquisition and financial management), relying heavily on business systems to do so. In fiscal year 2009, DON’s budget for business systems and associated infrastructure was about $2.7 billion, of which about $2.2 billion was allocated to operations and maintenance of existing systems and about $500 million to systems in development and modernization. Of the approximately 2,480 business systems that DOD reports having, DON accounts for 569, or about 23 percent, of the total. Navy ERP is one such system investment. In July 2003, the Assistant Secretary of the Navy for Research, Development, and Acquisition established Navy ERP to converge the functionality of four pilot systems that were under way at the four commands into one system. According to DOD, Navy ERP is to address the Navy’s long-standing problems related to financial transparency and asset visibility. Specifically, the program is intended to standardize the Navy’s acquisition, financial, program management, plant and wholesale supply, and workforce management business processes across its dispersed organizational components, and support about 86,000 users when fully implemented. Navy ERP is being developed in a series of increments using the Systems Applications and Products (SAP) commercial software package, augmented as needed by customized software. SAP consists of multiple, integrated functional modules that perform a variety of business-related tasks, such as finance and acquisition. The first increment, called Template 1, is currently the only funded portion of the program and consists of three releases (1.0, 1.1, and 1.2). Release1.0, Financial and Acquisition, is the largest of the three releases in terms of Template 1 functional requirements. See table 2 for a description of these releases. DON estimates the life-cycle cost for Template 1 to be about $2.4 billion, including about $1 billion for acquisition and $1.4 billion for operations and maintenance. The program office reported that approximately $600 million was spent from fiscal year 2004 through fiscal year 2008. For fiscal year 2009, about $190 million is planned to be spent. To acquire and deploy Navy ERP, DON established a program management office within the Program Executive Office for Executive Information Systems. The program office manages the program’s scope and funding and is responsible for ensuring that the program meets its key objectives. To accomplish this, the program office performs program management functions, including testing, change control, and IV&V. In addition, various DOD and DON organizations share program oversight and review activities. A listing of key entities and their roles and responsibilities is provided in table 3. To deliver system and other program capabilities and to provide program management support services, Navy ERP relies on multiple contractors, as described in table 4. Template 1 of Navy ERP was originally planned to reach full operational capability (FOC) in fiscal year 2011, and its original estimated life-cycle cost was about $1.87 billion. The estimate was later baselined in August 2004 at about $2.0 billion. In December 2006 and again in September 2007, the program was rebaselined. FOC is now planned for fiscal year 2013, and the estimated life-cycle cost is about $2.4 billion (a 31 percent increase over the original estimate). The program is currently in the production and deployment phase of the defense acquisition system, having completed the system development and demonstration phase in September 2007. This was 17 months later than the program’s original schedule set in August 2004, but on time according to the revised schedule set in December 2006. Changes in the program’s acquisition phase timeline are depicted in figure 1, and life-cycle cost estimates are depicted in figure 2. Release 1.0 was deployed at NAVAIR in October 2007, after passing developmental testing and evaluation. Initial operational capability (IOC) was achieved in May 2008, 22 months later than the baseline established in August 2004, and 4 months later than the new baseline established in September 2007. According to program documentation, these delays were due, in part, to challenges experienced at NAVAIR in converting data from legacy systems to run on the new system and implementing new business procedures associated with the system. In light of the delays at NAVAIR in achieving IOC, the deployment schedules for the other commands were revised in 2008. Release 1.0 was deployed at NAVSUP in October 2008 as scheduled, but deployment at SPAWAR was rescheduled for October 2009, 18 months later than planned, and at NAVSEA General Fund in October 2010, and at Navy Working Capital Fund in October 2011, each 12 months later than planned. Release 1.1 is currently being developed and tested, and is planned to be deployed at NAVSUP in February 2010, 7 months later than planned, and at the Navy’s Fleet and Industrial Supply Centers (FISC) starting in February 2011. Changes in the deployment schedule are depicted in figure 3. We have previously reported that DOD has not effectively managed key aspects of a number of business system investments, including Navy ERP. Among other things, our reviews have identified weaknesses in such areas as architectural alignment and informed investment decision making, which are the focus of the Fiscal Year 2005 Defense Authorization Act business system provisions. Our reviews have also identified weaknesses in other system acquisition and investment management areas, such as earned value management, economic justification, risk management, requirements management, test management, and IV&V practices. In September 2008, we reported that DOD had implemented key information technology (IT) management controls on Navy ERP to varying degrees of effectiveness. For example, the control associated with managing system requirements had been effectively implemented, and important aspects of other controls had been at least partially implemented, including those associated with economically justifying investment in the program and proactively managing program risks. However, other aspects of these controls, as well as the bulk of what was needed to effectively implement earned value management, had not been effectively implemented. As a result, the controls that were not effectively implemented had, in part, contributed to sizable cost and schedule shortfalls. Accordingly, we made recommendations aimed at improving cost and schedule estimating, earned value management, and risk management. DOD largely agreed with our recommendations. In July 2008, we reported that DOD had not implemented key aspects of its IT acquisition policies and related guidance on its Global Combat Support System–Marine Corps (GCSS-MC) program. For example, we reported that it had not economically justified its investment in GCSS-MC on the basis of reliable estimates of both benefits and costs and had not effectively implemented earned value management. Moreover, the program office had not adequately managed all program risks and had not used key system quality measures. We concluded that by not effectively implementing these IT management controls, the program was at risk of not delivering a system solution that optimally supports corporate mission needs, maximizes capability mission performance, and is delivered on time and within budget. Accordingly, we made recommendations aimed at strengthening cost estimating, schedule estimating, risk management, and system quality measurement. The department largely agreed with our recommendations. In July 2007, we reported that the Army’s approach for investing about $5 billion in three related programs—the General Fund Enterprise Business System, Global Combat Support System-Army Field/Tactical, and Logistics Modernization Program—did not include alignment with the Army enterprise architecture or use of a portfolio-based business system investment review process. Further, the Logistics Modernization Program’s testing was not adequate and had contributed to the Army’s inability to resolve operational problems. In addition, the Army had not established an IV&V function for any of the three programs. Accordingly, we recommended, among other things, use of an independent test team and establishment of an IV&V function. DOD agreed with the recommendations. In December 2005, we reported that DON had not, among other things, economically justified its ongoing and planned investment in the Naval Tactical Command Support System (NTCSS) and had not adequately conducted requirements management and testing activities. Specifically, requirements were not traceable and developmental testing had not identified problems that, subsequently, twice prevented the system from passing operational testing. Moreover, DON had not effectively performed key measurement, reporting, budgeting, and oversight activities. We concluded that DON could not determine whether NTCSS, as defined and as being developed, was the right solution to meet its strategic business and technological needs. Accordingly, we recommended developing the analytical basis necessary to know if continued investment in NTCSS represented a prudent use of limited resources, and strengthening program management, conditional upon a decision to proceed with further investment in the program. The department largely agreed with our recommendations. In September 2005, we reported that while Navy ERP had the potential to address some of DON’s financial management weaknesses, it faced significant challenges and risks, including developing and implementing system interfaces with other systems and converting data from legacy systems. Also, we reported that the program was not capturing quantitative data to assess effectiveness, and had not established an IV&V function. We made recommendations to address these areas, including having the IV&V agent report directly to program oversight bodies, as well as the program manager. DOD generally agreed with our recommendations, including that an IV&V function should be established. However, it stated that the IV&V team would report directly to program management who in turn would inform program oversight officials of any significant IV&V results. In response, we reiterated the need for the IV&V to be independent of the program and stated that performing IV&V activities independently of the development and management functions helps to ensure that the results are unbiased and based on objective evidence. We also reiterated our support for the recommendation that the IV&V reports be provided to the appropriate oversight body so that it can determine whether any of the IV&V results are significant. We noted that doing so would give added assurance that the results were objective and that those responsible for authorizing future investments in Navy ERP have the information needed to make informed decisions. To be effectively managed, testing should be planned and conducted in a structured and disciplined fashion. According to DOD and industry guidance, system testing should be progressive, meaning that it should consist of a series of test events that first focus on the performance of individual system components, then on the performance of integrated system components, followed by system-level tests that focus on whether the entire system (or major system increments) is acceptable, interoperable with related systems, and operationally suitable to users. For this series of related test events to be conducted effectively, all test events need to be, among other things, governed by a well-defined test management structure and adequately planned. Further, the results of each test event need to be captured and used to ensure that problems discovered are disclosed and corrected. Key aspects of Navy ERP testing have been effectively managed. Specifically, the program has established an effective test management structure, key development events were based on well-defined plans, the results of all executed test events were documented, and problems found during testing (i.e., test defects) were captured in a test management tool and subsequently analyzed, resolved, and disclosed to decision makers. Further, while we identified instances in which the tool did not contain key data about defects that are needed to ensure that unauthorized changes to the status of defects do not occur, the number of instances found are not sufficient to conclude that the controls were not operating effectively. Notwithstanding the missing data, this means that Navy ERP testing has been performed in a manner that increases the chances that the system will meet operational needs and perform as intended. The program office has established a test management structure that satisfies key elements of DOD and industry guidance. For example, the program has developed a Test and Evaluation Master Plan (TEMP) that defines the program’s test strategy. As provided for in the guidance, this strategy consists of a sequence of tests in a simulated environment to verify first that individual system parts meet specified requirements (i.e., development testing) and then verify that these combined parts perform as intended in an operational environment (i.e., operational testing). As we have previously reported, such a sequencing of test events is an effective approach because it permits the source of defects to be isolated sooner, before it is more difficult and expensive to address. More specifically, the strategy includes a sequence of developmental tests for each release consisting of three cycles of integrated system testing (IST) followed by user acceptance testing (UAT). Following development testing, the sequence of operational tests includes the Navy’s independent operational test agency conducting initial operational test and evaluation (IOT&E) and then follow-on operational test and evaluation (FOT&E), as needed, to validate the resolution of deficiencies found during IOT&E. See table 5 for a brief description of the purpose of each test activity, and figure 4 for the schedule of Release 1.0 and 1.1 test activities. According to relevant guidance, test activities should be governed by well-defined and approved plans. Among other things, such plans are to include a defect triage process, metrics for measuring progress in resolving defects, test entrance and exit criteria, and test readiness reviews. Each developmental test event for Release 1.0 (i.e., each cycle of integrated systems testing and user acceptance testing) was based on a well-defined test plan. For example, each plan provided for conducting daily triage meetings to (1) assign new defe documented criteria, (2) defects in the test management tool, and (3) address other de testing issues. Further, each plan included defect metrics, such as t number of plan specified that testing was not complete until all major defects found during the cycle were resolved, and all unresolved defects’ impact on the olding next test event were understood. Further, the plans provided for h test readiness reviews to review test results as a condition for proceeding to the next activities in activities no increasing t and perform as inten cts a criticality level using record new defects and update the status of old defects found and corrected and their age. In addition, each event. By ensuring that plans for key clude these aspects of effective test planning, the risk of test t being effectively and efficiently p erformed is reduced, thus he chances that the system will ded. According to industry guidance, effective system testing includes capturing, analyzing, resolving, and disclosing to decision makers the status of problems found during testing (i.e., test defects). Further, this guidance states that these results should be collected and stored accordin to defined procedures and placed under appropriate levels of control to ensure that any changes to the results are fully documented. To the program’s credit, the relevant testing organizations have documented test defects in accordance with defined plans. For example , daily triage meetings involving the test team lead, testers, and functional experts were held to review each new defect, assign it a criticality level, and designate someone responsible for resolving it and for monitoring and updating its resolution in the test management tool. Further, test readines reviews were conducted at which entrance and exit criteria for each key test event were evaluated before proceeding to the next event. As p art of these reviews, the program office and oversight officials, command representatives, and test officials reviewed the results of test events to ensure, among other things, that significant defects were closed and that there were no unresolved defects that could affect execution of the next test event. However, the test management tool did not always contain key data recorded defects that are needed to ensure that unauthorized changes to the status of defects do not occur. According to information systems auditing guidelines, audit tools should be in place to monitor user access to systems to detect possible errors or unauthorized changes. For Navy ERP, this was not always the case. Specifically, while the tool capability to track changes to test defects in a history log, our analysis of 80 randomly selected defects in the tool disclosed two instances the tool did not record when a change in the defect’s status was made or who made the change. In addition, our analysis of 12 additional defects that were potential anomalies disclosed two additional instances where the tool did not record when a change was made and who made it. While our sample size and results do not support any conclusions as to the overall effectiveness of the controls in place for recording and tracking test defect status changes, they do show that it is possible that changes es. can be made without a complete audit trail surrounding those chang After we shared our results with program officials, they stated that they provided each instance for resolution to the vendor responsible for the tracking tool. These officials attributed these instances to vendor updates to the tool t hat caused the history settings to default to “off.” To address this weakness, they added that they are now ensuring that the history log are set correctly after any update to the tool. This addition is a positive step because without an effective information system access audit tool, the probability of test defect status errors or unauthorized changes is increased. Industry best practices and DOD guidance recognize the importance of system change control when developing and maintaining a system. Once the composition of a system is sufficiently defined, a baseline configuration is normally established, and changes to that baseline are placed under a disciplined change control process to ensure that unjustified and unauthorized changes are not introduced. Elements of disciplined change control include (1) formally documenting a change control process, (2) rigorously adhering to the documented process, and (3) adopting objective criteria for considering a proposed change, including its estimated cost and schedule impact. To its credit, the Navy ERP program has formally documented a change control process. Specifically, it has a plan and related procedures that include the made to the system are properly identified, developed, and implemented in a defined and controlled environment. It also is using an automated tool to capture and track the disposition of each change request. Further, it has defined roles and responsibilities and a related decision-making structure for reviewing and approving system changes. In this regard, the program has established a hierarchy of review and approval boards, including a Configuration Control Board to review all changes and a Configuration Management Board to further review changes estimated to require more than 100 hours or $25,000 to implement. Furthermore, a Navy ERP Senior Integration Board was recently established to review and approve requests to add, delete, or change the program’s requirements. In addition, the change control process states that the decisions are to be based on, amon others, the system engineering and earned value management (i.e., co and schedule) impacts the change w number of work hours that will be required to effect the change. Table 7 purpose and scope of the process—to ensure that any changes st ill introduce, such as the estimated provides a brief description of the decision-making authorities and boards and their respective roles and responsibilities. Navy ERP is largely adhering to its documented change control process. Specifically, our review of a random sample of 60 change requests and minutes of related board meetings held between May 2006 and April 20 showed that the change requests were captured and tracked using an automated tool, and they were reviewed and approved by the designated decision-making authorities and boards, in accordance with the program documented process. However, the program has not sufficiently or co cost and schedule impacts of proposed changes. Our analysis of the random sample of 60 change requests, including our review of related board meeting minutes, showed no evidence that cost and schedule impacts were identified or that they were considered. Specifically, we did not see evidence that the cost and schedule impacts of these change requests were assessed. According to program officials, the cost and schedule impacts of each change were discussed at control board meetings. In addition, they provided two change requests to demonstrate this. However, while these change requests did include schedule impact, they did not include the anticipated cost impact of proposed changes. Rather, these two, as well as those in our random sample, included the estimated number of work hours required to implement the change. nsistently considered the Because the cost of any proposed change depends on other factors besides work hours, such as labor rates, the estimated number of work hours is not sufficient for considering the cost impact of a change. I absence of verifiable evidence that cost and schedule impacts were consistently considered, approval authorities do not appear to have b provided key information needed to fully inform their decisions on whether or not to approve a change. System changes that are approved without a full understanding of their cost and schedule impacts could result in unwarranted cost increases and schedule delays. The purpose of IV&V is to independently ensure that program processes and products meet quality standards. The use of an IV&V function is recognized as an effective practice for large and complex system development and acquisition programs, like Navy ERP, as it provides objective insight into the program’s processes and associated work p To be effective, verification and validation activities should be roducts.performed by an entity that is managerially independent of the system development and management processes and products that are being reviewed. Among other things, such independence helps to ensure that the results are unbiased and based on objective evidence. The Navy has not effectively managed its IV&V function because it has not ensured that the contra ctor performing this function is independent of the products and processes that this contractor is reviewing and because it has not ensured that the contractor is meeting contractual requirements. In June 2006, DON awarded a professional support services contract General Dynamics Information Technology (GDIT), to include responsibilities for, among other things, IV&V, program management support, and delivery of releases according to cost and schedule constraints. According to the program manager, the contractor’s IV&Vfunction is organizationally separate from, and thus independent of, th e contractor’s Navy ERP system development function. However, the subcontractor performing the IV&V function is also performing release management. According to the GDIT contract, the release manager is responsible for developing and deploying a system release that meets operational requirements within the program’s cost and schedule constraints, but it also states that the IV&V function is resp supporting the government in its review, approval, and acceptance of Navy RP products (e.g., releases). The contract also states that GDIT is eligible E for an optional award fee payment based on its performance in meeting, among other things, these cost and schedule constraints. Because performance of the system development and management role ma contractor potentially unable to render impartial assistance to the government in performing the IV&V function, the contractor has an inherent conflict of interest relative to meeting cost and schedule commitments and disclosing the results of verification and validation reviews that may affect its ability to do so. The IV&V function’s lack of independence is amplified by t reports directly and solely to the program manager. As we have previously reported, the IV&V function should report the issues or weaknesses that increase the risks associated with the project to program oversight officials, as well as to program management, to better ensure that the verification and validation results are objective and that the officials responsible for making program investment decisions are fully informed . Furthermore, these officials, once informed, can ensure that the issues or weaknesses reported are promptly addressed. Without ensuring sufficient managerial independence, valuable information may not reach decision makers, potentially leading to the release of a system that does not adequately meet users’ needs and o as intended. Beyond the IV&V function’s lack of independence, the program office h not ensured that the subcontractor has produced the range of deliv that were contractually required and defined in the IV&V plan. For example, the contract and plan call for weekly and monthly reports identifying weaknesses in program processes and recommendations for improvement, a work plan for accomplishing IV&V tasks, and associated assessment reports that follow the System Engineering Plan and program schedule. However, the IV&V contractor has largely not delivered these products. Specifically, until recently, it did not produce a work plan and erables only monthly reports were delivered, and these reports only list meetings that the IV&V contactor attended and documents that it reviewed. They d not, for example, identify program weaknesses or provide recommendations for improvement. According to program officials, they have relied on oral reports from the subcontractor at weekly meetings, and these lessons learned have been incorporated into program guid According to the contractor, the Navy has expended about $1.8 million between June 2006 and September 2008 for IV&V activities, with an additional $249,000 planned to be spent in fiscal year 2009. ance. Following our inquiries about an IV&V work plan, the IV&V contractor developed such a plan in October 2008, more than 2 years after the contract was awarded, that lists program activities and processes to be assessed, such as configuration management and testing. While this does not include time frames for starting and completing these assessments, meeting minutes show that the status of assessments ha been discussed with the program manager during IV&V review meetings. The first planned assessment was delivered to the program in March 2009 he program’s configuration and provides recommendations for improving t management process, such as using the automated tool to produce certain reports and enhancing training to understand how the tool is use Further, program officials stated that they have also recently begun requiring the contractor to provide formal quarterly reports, the first of which was de of this quarterly report shows that it provides recommendations for improving the program’s risk management process and organizational change management strategy. d. livered to the program manager in January 2009. Our review Notwithstanding the recent steps that the program office has taken, nevertheless lacks an independent perspective on the program’s products and management processes. DOD’s successes in delivering large-scale business systems, such as Navy ERP, are in large part determined by the extent to which it employs the kind of rigorous and disciplined IT management controls that are reflected in department policies and related guidance. While implementing these controls does not guarantee a successful program, it does minimize a program’s exposure to risk and thus the likelihood that it will fall short of expectations. In the case of Navy ERP, living up to expectations is important because the program is large, complex, and critical to addressing the department’s long-standing problems related to financial transparency and asset visibility. The Navy ERP program office has largely implemented a range of effective controls associated with system testing and change control, including acting quickly to address issues with the audit log for its test manage tool, but more can be done to ensure that the cost and sched proposed changes are explicitly documented and considered when decisions are reached. Moreover, while the program office has contract for IV&V activities, it has not ensured that the contractor is indepen the products and processes that it is to review and has not held the contractor accountable for producing the full range of IV&V deliverables required under the contract. Moreover, it has not ensured that it s IV&V contractor is accountable to a level of management above the program office, as we previously recommended. Notwithstanding the program office’s considerable effectiveness in how it has managed both system testing and change control, these weaknesses increase the risk of investing in system changes that are not economically justified and unnecessarily limit the value that an IV&V agent can bring to a program like Navy E By addressing these weaknesses, the department can better ensure t taxpayer dollars are wisely and prudently invested. RP. To strengthen the management of Navy ERP’s change control process, recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to (1) revise the Navy ERP procedures for controlling system changes to explicitly require that a proposed change’s life-cycle cost impact be estimated and considered in making change request decisions and (2) capture the cost and schedule impacts of each proposed change in the Navy ERP automated change c ontrol tracking tool. To increase the value of Navy ERP IV&V, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to (1) stop performance of the IV&V function under the existing contract and (2) engage the services of a n IV&V agent that is independent of all Navy ERP management, development, testing, and deployment activities that it may review. In addition, we reiterate our prior recommendation relative to ensur the Navy ERP IV&V agent report directly to program oversight officials, while concurrently sharing IV&V results with the program office. In written comments on a draft of this report, signed by the Assistan Deputy Chief Management Officer and reprinted in appendix II, the department concurred with our recommendations, and stated that it will take the appropriate corrective actions within the next 7 months. We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; the Congressional Budget Office; and the Secretary of Defense. The report also is available at no charge on our Web site at http://www.gao.gov. If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of Congressional Relations and be found on the last page of this report. GAO staff who made major c ontributions to this report are listed in appendix III. Our objectives were to determine whether (1) system testing is being effectively managed, (2) system changes are being effectively con and (3) independent verification and validation (IV&V) activities a effectively managed for the Navy Enterprise Resource Planning (ERP) program. To determine if Navy ERP testing is being effectively managed, we reviewed relevant documentation, such as the Test and Evaluation Mas Plan and test reports and compared them with relevant federal and related res guidance. Further, we reviewed development test plans and procedu for each test event and compared them with best practices to determine whether well-defined plans were developed. We also examined test results and reports, including test readiness review documentation and compared them against plans to determine whether they had been executed in accordance with the plans. Moreover, to determine the extent to which test defect data were being captured, analyzed, and reported, we inspected 80 randomly selected defects from a sample of 2,258 def program’s test management system. In addition, we re logs associated with each appropriate levels the results were fully documented. Th percent tolerable error rate at the 95 pe we found 0 problems in the error rate was less than 4 percent. In addition, we interviewed cognizant officials, including the program’s test lead and the Navy’s independent operational testers, about their roles and responsibilities for test management. of these 80 defects to determine whether of control were in place to ensure that any changes to is sample was designed with a 5 rcent level of confidence, so that, if our sample, we could conclude statistically that To determine if Navy ERP changes are being effectively controlled, we reviewed relevant program documentation, such as the change control policies, plans, and procedures, and compared them with relevant federal and industry guidance. Further, to determine the extent to which the program is reviewing and approving change requests according to its documented plans and procedures, we inspected 60 randomly selected change requests in the program’s configuration management system. In addition, we reviewed the change request forms associated with these 60 change requests and related control board meeting minutes to determine whether objective criteria for considering a proposed change, including estimated cost or schedule impacts, were adopted. In addition, we interviewed cognizant officials, including the program manager and systems engineer, about their roles and responsibilities for reviewing, approving, and tracking change requests. To determine if IV&V activities are being effectively managed we revie Navy ERP’s IV&V contract, strategy, and plans and compared them with relevant industry guidance. We also analyzed the contractual relationships relative to legal standards that govern organizational conflict of interest . In addition, w assessment report, and a quarterly report, to determine the extent to which contract requirements were met. We interviewed contractor a nd program officials about their roles and responsibilities for IV&V and to determine the extent to which the program’s IV&V function is independent. e examined IV&V monthly status reports, work plans, an We conducted this performance audit at Department of Defense offices in the Washington, D.C., metropolitan area; Annapolis, Maryland; and Norfolk, Virginia; from August 2008 to September 2009, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the individual named above, key contributors to this report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Carl Barden; Neil Doherty; Cheryl Dottermusch; Lee McCracken; Karl Seifert; Adam Vodraska; Shaunyce Wallace; and Jeffrey Woodward.

Summary: The Department of Defense (DOD) has long been challenged in effectively implementing key acquisition management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk since 1995. One major business system investment is the Navy's Enterprise Resource Planning (ERP) system. Initiated in 2003, it is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its 20-year useful life and be fully deployed by fiscal year 2013. To date, the program has experienced about $570 million in cost overruns and a 2-year schedule delay. GAO was asked to determine whether (1) system testing is being effectively managed, (2) system changes are being effectively controlled, and (3) independent verification and validation (IV&V) activities are being effectively managed. To do this, GAO analyzed relevant program documentation, traced random samples of test defects and change requests, and interviewed cognizant officials. The Navy has largely implemented effective controls on Navy ERP associated with system testing and change control. For example, it has established a well-defined structure for managing tests, including providing for a logical sequence of test events, adequately planning key test events, and documenting and reporting test results. In addition, it has documented, and is largely following, its change request review and approval process, which reflects key aspects of relevant guidance, such as having defined roles and responsibilities and a hierarchy of control boards. However, important aspects of test management and change control have not been fully implemented. Specifically, the program's tool for auditing defect management did not always record key data about changes made to the status of identified defects. To its credit, the program office recently took steps to address this, thereby reducing the risk of defect status errors or unauthorized changes. Also, while the program office's change review and approval procedures include important steps, such as considering the impact of a change, and program officials told GAO that cost and schedule impacts of a change are discussed at control board meetings, GAO's analysis of 60 randomly selected change requests showed no evidence that cost and schedule impacts were in fact considered. Without such key information, decision-making authorities lack an adequate basis for making informed investment decisions, which could result in cost overruns and schedule delays. The Navy has not effectively managed its IV&V activities, which are designed to obtain an unbiased position on whether product and process standards are being met. In particular, the Navy has not ensured that the IV&V contractor is independent of the products and processes that it is reviewing. Specifically, the same contractor responsible for performing IV&V of Navy ERP products (e.g., system releases) is also responsible for ensuring that system releases are delivered within cost and schedule constraints. Because performance of this system development and management role makes the contractor potentially unable to render impartial assistance to the government in performing the IV&V function, there is an inherent conflict of interest. In addition, the IV&V agent reports directly and solely to the program manager and not to program oversight officials. As GAO has previously reported, the IV&V agent should report the findings and associated risks to program oversight officials, as well as program management, in order to better ensure that the IV&V results are objective and that the officials responsible for making program investment decisions are fully informed. Furthermore, the contractor has largely not produced the range of IV&V deliverables that were contractually required between 2006 and 2008. To its credit, the program office recently began requiring the contractor to provide assessment reports, as required under the contract, as well as formal quarterly reports; the contractor delivered the results of the first planned assessment in March 2009. Notwithstanding the recent steps that the program office has taken, it nevertheless lacks an independent perspective on the program's products and management processes