VBA is in the process of modernizing many of its older, inefficient systems and has reportedly spent an estimated $294 million on these activities between October 1, 1986 and February 29, 1996. The modernization program can have a major impact on the efficiency and accuracy with which over $20 billion in benefits and other services is paid to our nation’s veterans and their dependents. However, in the last 6 years some aspects of VBA’s service to the veterans have not improved. For example, in the past 6 years, VBA’s reported processing time for an original compensation claim rose from 151 days in fiscal year 1990 to 212 days in fiscal year 1994. In March 1996 the average time was 156 days. Software development is a critical component of this major modernization initiative. VBA, with the assistance of contractors, will be developing software for the Veterans Services Network (VETSNET) initiative, a replacement for the existing Benefit Delivery Network. For efforts like VETSNET to succeed, it is crucial that VBA have in place a disciplined set of software development processes to produce high quality software within budget and on schedule. VBA relies upon its own staff and contractors to develop and maintain software that is crucial to its overall operations. In fiscal year 1995, VBA had 314 full-time equivalents, with payroll expenses of $20.8 million, devoted to developing and maintaining software throughout the organization. It also spent $17.7 million in contract services in these areas. To evaluate VA’s software development capability, version 2.0 of the Software Engineering Institute’s (SEI) software capability evaluation (SCE) method was used by an SEI-trained team of GAO specialists. The SCE is a method for evaluating agencies’ and contractors’ software development processes against SEI’s five-level software Capability Maturity Model (CMM), as shown in table 1. These levels and the key process areas (KPAs) described within each level define an organization’s ability to develop software, and can be used to improve its software development processes. The findings generated from an SCE identify (1) process strengths that mitigate risks, (2) process weaknesses that increase risks, and (3) improvement activities that indicate potential mitigation of risks. We requested that VA identify for our evaluation those projects using the best software development processes implemented within VBA and AAC. VBA and AAC identified the following sites and projects. —Compensation & Pension/Financial Management System —Claims Processing System We evaluated the software development processes used on these projects, focusing on KPAs necessary to achieve a repeatable capability. Organizations that have a repeatable software development process have been able to significantly improve their productivity and return on investment. In contrast, organizations that have not developed the process discipline necessary to better manage and control their projects at the repeatable level incur greater risk of schedule delay, cost overruns, and poor quality software. These organizations rely solely upon the variable capabilities of individuals, rather than on institutionalized processes considered basic to software development. According to SEI, processes for a repeatable capability (i.e., CMM level 2) are considered the most basic in establishing discipline and control in software development and are crucial steps for any project to mitigate risks associated with cost, schedule, and quality. As shown in table 2, these processes include (1) requirements management, (2) software project planning, (3) software project tracking and oversight, (4) software subcontract management, (5) software quality assurance, and (6) software configuration management. We conducted our review between August 1995 and February 1996, in accordance with generally accepted government auditing standards. Highlights of our evaluation of VBA’s software practices using the SEI criteria outlined in appendix II follow. Requirements Management - The purpose of requirements management is to establish a common understanding between the customer and the software project of the customer’s requirements that will be addressed by the software project. The first goal within this KPA states that, “system requirements allocated to software are controlled to establish a baseline for software engineering and management use.” VBA does not manage and control system requirements as required by this goal. Moreover, members of software-related groups are not trained in requirements management activities. Also, changes made to software plans, work products, and activities resulting from changes to the software requirements are not assessed for risk. Software Project Planning - The purpose of software project planning is to establish reasonable plans for performing the software engineering and for managing the software project. VBA projects do not have software development plans, estimates for software project costs are not derived using conventional industry methods and tools, and VBA is unable to show the derivation of the estimates for the size (or changes to the size) of the software work products. Also, individuals involved in the software project planning are not trained in estimating and planning procedures applicable to their area of responsibility. Software Project Tracking and Oversight - The purpose of software project tracking and oversight is to provide adequate visibility into actual progress so that management can take effective actions when the software project’s performance deviates significantly from software plans. VBA does track software project schedules against major milestones; however, as mentioned previously, these schedules and milestones are not derived using conventional industry methods nor is there a comprehensive software plan against which to track activities. Moreover, the size of software work products (or the size of changes to software work products) are not tracked, and the software risks associated with cost, resource, schedule, and technical aspects of the project are not tracked. Software Subcontract Management - The purpose of software subcontract management is to select qualified software subcontractors and manage them effectively. VBA does not have a written organizational policy that describes the process for managing software contracts. Additionally, the software work to be contracted is neither defined nor planned according to a documented procedure. Finally, software managers and other individuals who are involved in developing, negotiating, and managing a software contract are not trained to perform these activities. Software Quality Assurance - The purpose of software quality assurance is to provide management with appropriate visibility into the process being used by the software project and of the products being built. VBA has a software quality and control (SQ&C) group that has a reporting channel to senior management, independent of the project managers. The SQ&C group also performs testing of the software code. However, the SQ&C group does not participate in other software quality assurance (SQA) functions, such as the preparation, review, and audit of projects’ software development plans, standards, procedures, and other work products. Also, projects do not have SQA plans. Software Configuration Management - The purpose of software configuration management is to establish and maintain the integrity of products of the software project throughout the project’s software life cycle. VBA has provided formal training to its staff in defining software processes. However, VBA cannot effectively control the integrity of its software work products because it has no software configuration control board, it does not identify software work products to be placed under configuration management, and it has no configuration management library system to serve as a repository for software work products. VBA has begun improvement activities in this area by (1) establishing a software configuration management group and (2) drafting a software configuration management procedure. Following a presentation of GAO’s SCE results to the Chief Information Officer of VBA, the Director of VBA’s Office of Information Systems forwarded a letter to GAO citing a number of initiatives that are currently underway to address some of the stated deficiencies. Initiatives cited by the VBA include: development and distribution of interim configuration management procedures; identification of a library structure to hold all of the work products from the development process; and initiation of several meetings with SEI to discuss the Software CMM. Similar to VBA, we compared the CMM criteria in appendix II to the software development practices at AAC. Summary results of this evaluation follow. Requirements Management - AAC does not create or control a requirements baseline for software engineering. Also, AAC does not manage or control requirements. AAC does have a process for negotiating periodic contractual arrangements with customers, but this process does not include baselining and controlling software requirements. Software Project Planning - Although AAC documents its schedule estimates for software development projects, there is (1) no defined methodology in use for estimating software costs, size, or schedule, (2) no derivation of estimates for the size (or changes to the size) of software products, and (3) no derivation of the estimates for software project costs. Similarly, AAC uses a project planning tool called “MultiTrak”. However, projects do not have software development plans. Software Project Tracking and Oversight - AAC performs schedule tracking at major milestones. However, the goals for this KPA call for (1) the tracking of actual results and performances against software plans, (2) the management of corrective actions when deviations from the software plan occur, and (3) the affected parties to mutually agree to changes in commitments. AAC does not conform to these goals. For example, AAC does not track (1) the software risks associated with cost, resource, schedule, and technical aspects of the project and (2) the size of software work products (or size of changes to software work products). Software Subcontract Management - Although the goals for this KPA emphasize the selection of qualified software subcontractors and managing them effectively, AAC does not (1) have a documented procedure that explains how the work to be contracted should be defined and planned and (2) ensure that software managers and other individuals who are involved in establishing a software contract are trained to perform this activity. Software Quality Assurance - The goals within this KPA emphasize (1) the verification of the adherence of software products and activities to applicable standards, procedures, and requirements and (2) the reporting of noncompliance issues that cannot be resolved within the project to senior management.