Since 1986, VBA has been trying to modernize its old, inefficient information systems. It reportedly spent an estimated $294 million on these activities between October 1, 1986, and February 29, 1996. The modernization program can have a major impact on the efficiency and accuracy with which about $20 billion in benefits and other services is paid annually to our nation’s veterans and their dependents. Software development is a critical component of this modernization effort. Also, a mature software development capability will provide added assurance that software developers will be able to effectively make changes to the software needed to address the Year 2000 computing problem. To evaluate VBA’s software development processes, in 1996, we applied the Software Engineering Institute’s (SEI) software capability evaluation methodology to those projects identified by VBA as using the best development processes. This evaluation compares agencies’ and contractors’ software development processes against SEI’s five-level software capability maturity model, with 5 being the highest level of maturity and 1 being the lowest. In June 1996, we reported that VBA was operating at a level 1 capability. At this level, VBA cannot reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints, which places VBA software development projects at significant risk. Accordingly, VBA must rely on the various capabilities of individuals rather than on an institutional process that will yield repeatable, or level 2, results. VBA did not satisfy any of the criteria for a repeatable or level 2 capability, the minimum level necessary to significantly improve productivity and return on investment. For example, VBA is extremely weak in the requirements management, software project planning, and software subcontract management areas, with no identifiable strengths or improvement activities. Because of VBA’s software development weaknesses, we recommended that the Secretary of Veterans Affairs obtain expert advice to improve VBA’s ability to develop high-quality develop and expeditiously implement an action plan that describes a strategy for reaching the repeatable (level 2) level of process maturity; ensure that any future contracts for software development require the contractor to have a software development capability of at least level 2; and delay any major investment in new software development—beyond what is needed to sustain critical day-to-day operations—until the repeatable level of process maturity is attained. In commenting on a draft of the June 1996 report, VBA agreed with three of our recommendations but disagreed with delaying major investments in software development. VBA stated that while it agreed that a repeatable level of process maturity is a goal that must be attained, it disagreed that “all software development beyond that which is day-to-day critical must be curtailed.” VBA stated that the payment system replacement projects, the migration of legacy systems, and other activities to address the change of century must continue. In our response to VBA’s comments, we agreed that the change of century and changes to legislation must be continued, and we characterized these changes as sustaining day-to-day operations. However, for those projects that do not meet this criterion, we continue to believe that VBA should delay software development investments until a maturity of level 2 is reached. To assess actions taken by VBA to improve its software capability, we reviewed VBA documents, such as its “Software Process Improvement Initiative Strategic Plan,” dated March 1997; the Best Practices Round Up Method; and the Interagency Agreement with the Air Force, dated September 1996, to obtain expert software process improvement assistance. We also reviewed SEI’s IDEALSM: A User’s Guide for Software Process Improvement, dated February 1996, and technical report on Best Training Practices Within the Software Engineering Industry, dated November 1996. In addition, we reviewed VBA contracts, correspondence to contractors, and supporting documents to determine what VBA has done to ensure that VBA’s software development contractors are at the repeatable level. We interviewed VBA officials and contractor personnel involved with the software process improvement effort to determine what actions VBA has taken to improve its software capability. We also interviewed selected VBA project managers involved in new systems development on their knowledge of the software process improvement initiative. We performed our work from October 1996 through August 1997 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Secretary-Designate of Veterans Affairs. The Secretary-Designate provided us with written comments, which are discussed in the “Agency Comments and Our Evaluation” section and reprinted in appendix I. VBA has initiated several actions to improve its software development capability. For example, in response to our recommendation that it obtain expert assistance, VBA has hired a contractor—SEI—to (1) assist it in developing an integrated set of software practices that will position VBA for successful, lasting improvements, (2) help formulate a software improvement program, and (3) provide expertise in executing software improvement activities. SEI is also expected to provide expertise in strategic and tactical planning, training, policy preparation, and action planning. In addition, in response to our recommendation that it develop and implement an action plan describing a strategy for reaching the repeatable level, VBA launched a software process improvement initiative in June 1996 to lay the foundation and build the context for sustainable, measurable improvements to its software development capability. It has developed a strategic plan that describes the purpose and goals of this improvement initiative. One of the plan’s goals is to establish organization policies and guidelines for the management, planning, and tracking of software projects that will enable VBA to repeat earlier successes on projects with similar applications. VBA has also recently initiated two software process improvement projects. The first project, called “Best Practices Round Up,” will identify software development practices that VBA software development teams are doing correctly. VBA believes that there are “pockets of excellence” within its organization not identified in our June 1996 report that it can build upon. The Best Practices Round Up project team started its review in April 1997 and briefed VBA’s chief information officer on its results in August 1997. The second project, called “Standards, Policies, and Procedures,” will assess whether VBA’s software development teams are following current software development policies, procedures, and standards. This project was initiated in June 1997 and is expected to be completed by the end of September 1997. The project team is expected to make recommendations to ensure compliance with standards. Although VBA has launched a software process initiative and an accompanying strategic plan, VBA has not yet clearly presented how it intends to move from an ad hoc and chaotic level of software development capability to a repeatable level. SEI’s IDEALSM: A User’s Guide for Software Process Improvement requires that a plan be developed that includes a schedule for initial activities, basic resource requirements, and benefits to the organization. VBA’s current plan contains no milestones—beginning, interim, or completion dates—by which to measure the agency’s progress and to identify problems. The plan also contains no analysis or information on costs, benefits, or risks. VBA officials stated that they recognize that the agency’s strategic plan for software process improvement lacks this specificity. At the conclusion of our review, these officials said that VBA intends to address this area in an upcoming action plan for the software process improvement initiative. VBA has also not yet established a baseline from which to measure its software process improvements. According to the SEI IDEALSM: A User’s Guide for Software Process Improvement, an organization needs to understand its current software process baseline so that it can develop a plan to achieve the business changes necessary to reach its software process improvement goals. At the conclusion of our review, VBA officials told us that they plan to use as their baseline the results of our June 1996 report, along with the results from their Best Practices Round Up and Standards, Policies, and Procedures projects. They stated that the baseline should be established by September 1997. Training of key staff is critical to achieving level 2 repeatability. According to SEI’s technical report entitled, Best Practices Within the Software Engineering Industry, best training practices include defining a process for software engineering education. Although VBA has provided process improvement training to many of the managers in its software engineering process group and management steering group, key software personnel—software developers, project managers, and line managers—have not been trained in the process improvement methodology, the principles behind it, and the key process areas. VBA’s software process improvement project manager explained that these key people had not yet been trained because VBA did not want to train them too long before implementing the process improvement projects. The project manager said that VBA plans to train these staff during fiscal years 1998 and 1999. However, VBA does not have a documented training plan to help ensure that these personnel receive training. Unless these individuals are trained in the process improvement methodology, its principles, and the key process areas, it will be difficult for them to implement the new policies and procedures required to reach the repeatable level. At the conclusion of our review, VBA officials stated that a training plan is now under development and will be made part of the software process improvement initiative action plan. In responding to our recommendation that it ensure that contractors have a repeatable software development capability, VBA intends to use a new provision in future software development contracts. This provision, however, does not require potential contractors to submit supporting documentation to VBA certifying their level of maturity. Validation of potential contractors’ software development capability maturity level should be a key factor in VBA’s software contracting decisions. The Internal Revenue Service, for example, recently started requiring that all current, in-process, and future contract solicitations for software development services require that contractors submit documentation to verify how their software development practices and processes satisfy the repeatable key process areas specified by SEI’s capability maturity model. The Internal Revenue Service plans to use this information when selecting software development services.