Since 1986, VBA has been trying to modernize its old, inefficient information systems. It reportedly spent an estimated $294 million on these activities between October 1, 1986, and February 29, 1996. The modernization program can have a major impact on the efficiency and accuracy with which about $20 billion in benefits and other services is paid annually to our nation’s veterans and their dependents. Software development is a critical component of this modernization effort. Also, a mature software development capability will provide added assurance that software developers will be able to effectively make changes to the software needed to address the Year 2000 computing problem. To evaluate VBA’s software development processes, in 1996, we applied the Software Engineering Institute’s (SEI) software capability evaluation methodology to those projects identified by VBA as using the best development processes. This evaluation compares agencies’ and contractors’ software development processes against SEI’s five-level software capability maturity model, with 5 being the highest level of maturity and 1 being the lowest. In June 1996, we reported that VBA was operating at a level 1 capability. At this level, VBA cannot reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints, which places VBA software development projects at significant risk. Accordingly, VBA must rely on the various capabilities of individuals rather than on an institutional process that will yield repeatable, or level 2, results. VBA did not satisfy any of the criteria for a repeatable or level 2 capability, the minimum level necessary to significantly improve productivity and return on investment. For example, VBA is extremely weak in the requirements management, software project planning, and software subcontract management areas, with no identifiable strengths or improvement activities. Because of VBA’s software development weaknesses, we recommended that the Secretary of Veterans Affairs obtain expert advice to improve VBA’s ability to develop high-quality develop and expeditiously implement an action plan that describes a strategy for reaching the repeatable (level 2) level of process maturity; ensure that any future contracts for software development require the contractor to have a software development capability of at least level 2; and delay any major investment in new software development—beyond what is needed to sustain critical day-to-day operations—until the repeatable level of process maturity is attained. In commenting on a draft of the June 1996 report, VBA agreed with three of our recommendations but disagreed with delaying major investments in software development. VBA stated that while it agreed that a repeatable level of process maturity is a goal that must be attained, it disagreed that “all software development beyond that which is day-to-day critical must be curtailed.” VBA stated that the payment system replacement projects, the migration of legacy systems, and other activities to address the change of century must continue. In our response to VBA’s comments, we agreed that the change of century and changes to legislation must be continued, and we characterized these changes as sustaining day-to-day operations. However, for those projects that do not meet this criterion, we continue to believe that VBA should delay software development investments until a maturity of level 2 is reached. To assess actions taken by VBA to improve its software capability, we reviewed VBA documents, such as its “Software Process Improvement Initiative Strategic Plan,” dated March 1997; the Best Practices Round Up Method; and the Interagency Agreement with the Air Force, dated September 1996, to obtain expert software process improvement assistance. We also reviewed SEI’s IDEALSM: A User’s Guide for Software Process Improvement, dated February 1996, and technical report on Best Training Practices Within the Software Engineering Industry, dated November 1996. In addition, we reviewed VBA contracts, correspondence to contractors, and supporting documents to determine what VBA has done to ensure that VBA’s software development contractors are at the repeatable level. We interviewed VBA officials and contractor personnel involved with the software process improvement effort to determine what actions VBA has taken to improve its software capability. We also interviewed selected VBA project managers involved in new systems development on their knowledge of the software process improvement initiative. We performed our work from October 1996 through August 1997 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Secretary-Designate of Veterans Affairs. The Secretary-Designate provided us with written comments, which are discussed in the “Agency Comments and Our Evaluation” section and reprinted in appendix I. VBA has initiated several actions to improve its software development capability. For example, in response to our recommendation that it obtain expert assistance, VBA has hired a contractor—SEI—to (1) assist it in developing an integrated set of software practices that will position VBA for successful, lasting improvements, (2) help formulate a software improvement program, and (3) provide expertise in executing software improvement activities. SEI is also expected to provide expertise in strategic and tactical planning, training, policy preparation, and action planning. In addition, in response to our recommendation that it develop and implement an action plan describing a strategy for reaching the repeatable level, VBA launched a software process improvement initiative in June 1996 to lay the foundation and build the context for sustainable, measurable improvements to its software development capability. It has developed a strategic plan that describes the purpose and goals of this improvement initiative. One of the plan’s goals is to establish organization policies and guidelines for the management, planning, and tracking of software projects that will enable VBA to repeat earlier successes on projects with similar applications. VBA has also recently initiated two software process improvement projects. The first project, called “Best Practices Round Up,” will identify software development practices that VBA software development teams are doing correctly. VBA believes that there are “pockets of excellence” within its organization not identified in our June 1996 report that it can build upon. The Best Practices Round Up project team started its review in April 1997 and briefed VBA’s chief information officer on its results in August 1997. The second project, called “Standards, Policies, and Procedures,” will assess whether VBA’s software development teams are following current software development policies, procedures, and standards. This project was initiated in June 1997 and is expected to be completed by the end of September 1997. The project team is expected to make recommendations to ensure compliance with standards. Although VBA has launched a software process initiative and an accompanying strategic plan, VBA has not yet clearly presented how it intends to move from an ad hoc and chaotic level of software development capability to a repeatable level. SEI’s IDEALSM: A User’s Guide for Software Process Improvement requires that a plan be developed that includes a schedule for initial activities, basic resource requirements, and benefits to the organization. VBA’s current plan contains no milestones—beginning, interim, or completion dates—by which to measure the agency’s progress and to identify problems. The plan also contains no analysis or information on costs, benefits, or risks. VBA officials stated that they recognize that the agency’s strategic plan for software process improvement lacks this specificity. At the conclusion of our review, these officials said that VBA intends to address this area in an upcoming action plan for the software process improvement initiative. VBA has also not yet established a baseline from which to measure its software process improvements. According to the SEI IDEALSM: A User’s Guide for Software Process Improvement, an organization needs to understand its current software process baseline so that it can develop a plan to achieve the business changes necessary to reach its software process improvement goals. At the conclusion of our review, VBA officials told us that they plan to use as their baseline the results of our June 1996 report, along with the results from their Best Practices Round Up and Standards, Policies, and Procedures projects. They stated that the baseline should be established by September 1997. Training of key staff is critical to achieving level 2 repeatability. According to SEI’s technical report entitled, Best Practices Within the Software Engineering Industry, best training practices include defining a process for software engineering education. Although VBA has provided process improvement training to many of the managers in its software engineering process group and management steering group, key software personnel—software developers, project managers, and line managers—have not been trained in the process improvement methodology, the principles behind it, and the key process areas. VBA’s software process improvement project manager explained that these key people had not yet been trained because VBA did not want to train them too long before implementing the process improvement projects. The project manager said that VBA plans to train these staff during fiscal years 1998 and 1999. However, VBA does not have a documented training plan to help ensure that these personnel receive training. Unless these individuals are trained in the process improvement methodology, its principles, and the key process areas, it will be difficult for them to implement the new policies and procedures required to reach the repeatable level. At the conclusion of our review, VBA officials stated that a training plan is now under development and will be made part of the software process improvement initiative action plan. In responding to our recommendation that it ensure that contractors have a repeatable software development capability, VBA intends to use a new provision in future software development contracts. This provision, however, does not require potential contractors to submit supporting documentation to VBA certifying their level of maturity. Validation of potential contractors’ software development capability maturity level should be a key factor in VBA’s software contracting decisions. The Internal Revenue Service, for example, recently started requiring that all current, in-process, and future contract solicitations for software development services require that contractors submit documentation to verify how their software development practices and processes satisfy the repeatable key process areas specified by SEI’s capability maturity model. The Internal Revenue Service plans to use this information when selecting software development services. Also, the Department of the Air Force’s acquisition policy states that software capability evaluations should be used for selecting software contractors. Although VBA has asserted that two of its current contractors are at the repeatable level, VBA could not provide documentation to support this. VBA subsequently requested the documentation from the contractors, but the information the contractors provided did not clearly show that they were at the repeatable level. In one case, the contractor presented information on how it assisted federal agencies in achieving the repeatable and/or higher levels of software development capability but did not provide documentation that the contractor was certified. In the second case, a component of the contractor’s organization asserted that it was at the repeatable level but did not provide documentation supporting this assertion. Recognizing the importance of a mature software development capability, VBA has initiated actions to address the weaknesses identified in our June 1996 report. These actions will help it move toward a repeatable software capability maturity level, but additional efforts are needed. Specifically, VBA has not (1) developed a detailed strategy for how VBA plans to achieve a repeatable level of software development capability, (2) established a baseline to measure performance improvements, (3) trained its software development teams in the process improvement methodology, and (4) established a process for ensuring that its software development contractors are at the repeatable level. Recognizing that these deficiencies need to be addressed, VBA has efforts underway to do so. If these deficiencies are not sufficiently addressed, VBA’s software development capability will remain ad hoc and chaotic, subjecting the agency to continuing risk of cost overruns, poor quality software, and schedule delays. We recommend that the Secretary of Veterans Affairs direct the Under Secretary for Benefits, in conjunction with VBA’s chief information officer, to define the milestones, costs, tasks, and risks of the software process improvement initiative in order to provide a clear strategy for how VBA plans to improve its software development capability to a repeatable level; develop and use a baseline showing VBA’s current software development capability from which to measure VBA’s software improvement effort; ensure that a training plan is developed and implemented that will provide key software development staff training in the software process improvement methodology, its principles, and key process areas; and establish a source selection process to ensure that VBA’s software development contractors have the mature processes necessary for timely, high-quality software development, including evaluating and validating documentation provided by potential contractors establishing that they are at the repeatable level or higher. In comments on a draft of this report, VA concurred with our recommendations. VA also agreed that a repeatable level of process maturity is a goal that VBA must attain and described a number of activities underway to improve its software development capability. For example, VBA has developed a draft action plan to define a strategy to reach the repeatable level and specify the activities/tasks, milestones, costs, and timeliness associated with the process improvement effort. VBA also was reviewing and revising the draft plan to fully address the issues raised in our report. VA added that a significant amount of work still remains before this plan is finalized. We are encouraged by VBA’s response and will continue to monitor the agency’s progress in implementing its software improvement effort. We are sending copies of this report to the Ranking Minority Member of the Subcommittee on Oversight and Investigations and the Chairman and Ranking Minority Member of the Subcommittee on Benefits, House Committee on Veterans’ Affairs. We will also provide copies to the Chairmen and Ranking Minority Members of the House and Senate Committees on Veterans’ Affairs and the House and Senate Committees on Appropriations; the Secretary-Designate of Veterans Affairs; and the Director of the Office of Management and Budget. Copies will also be made available to other parties upon request. Please contact me at (202) 512-6253 or by e-mail at willemssenj.aimd@gao.gov if you have any questions concerning this report. Major contributors to this report are listed in appendix II. The following is GAO’s comment on the Department of Veterans Affairs letter dated September 4, 1997. 1. Enclosure (2) has not been included. Helen Lew, Assistant Director Leonard J. Latham, Technical Assistant Director K. Alan Merrill, Technical Assistant Director David Chao, Senior Technical Advisor Tonia L. Johnson, Senior Information Systems Analyst The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary: Pursuant to a congressional request, GAO conducted a follow-up review to determine the actions taken by the Veterans Benefits Administration (VBA) to address management and technical weaknesses identified in the June 19, 1996, hearing on the agency's modernization effort, focusing on the agency's actions to improve its software development capability. GAO noted that: (1) VBA has taken action to improve its software development capability; (2) among other things, it has launched a software process improvement initiative, chartered a software engineering process group, and obtained the services of an experienced contractor to assist in developing and implementing a software process improvement effort; (3) although it has made progress, VBA has not yet fully addressed needed software development improvements; (4) these include a need for: (a) a defined strategy to reach the repeatable level and a baseline to measure improvements; (b) a process improvement training program for its software developers; and (c) a process to ensure that VBA's software development contractors are at the repeatable level; (5) VBA generally agrees that these issues need to be addressed and has efforts under way to do so; and (6) until these deficiencies are sufficiently addressed, VBA's software development capability remains ad hoc and chaotic, subjecting the agency to continuing risk of cost overruns, poor quality software, and schedule delays in software development.