SSA is currently undertaking a multiyear, multibillion dollar systems modernization effort that is intended to replace aging equipment, support current and future redesigned work processes, and improve productivity. The cornerstone of this modernization effort is the agency’s transition from its current centralized, mainframe-based computer processing environment to a highly distributed client/server processing environment. The IWS/LAN infrastructure—consisting of networks of intelligent workstations connected to each other and to SSA’s mainframe computers—is intended to provide SSA with the initial computing framework for using client/server technology to achieve cost savings and improve customer service by distributing selected processes and information closer to where they are needed. Through fiscal year 1997, SSA had reported spending approximately $565 million on acquiring workstations, local area networks, and other services to support the IWS/LAN infrastructure. Software development is a critical component of the modernization initiative. SSA’s Office of Systems, with contractors’ assistance, is designing and developing a new generation of software that is scheduled to operate on the IWS/LAN to support redesigned work processes in a client/server environment. It has selected the disability claims process as the first major redesign effort and is currently developing the software—referred to as the Reengineered Disability System (RDS)—that is intended to automate this redesigned process. RDS is scheduled for national implementation on the IWS/LAN from July 1999 through May 2001. To help SSA’s software development efforts succeed, however, it is important that the agency have disciplined and consistent software development practices that produce high-quality software within budget and on schedule. In September 1996, we reported that SSA had experienced problems in developing RDS, which contributed to a delay of more than 2 years in its scheduled implementation. These problems included (1) using programmers with insufficient experience, (2) using software development tools that did not perform effectively, and (3) establishing initial software development schedules that were too optimistic. During that same month, a contractor’s preliminary risk assessment of SSA’s client/server transition strategy identified various risks associated with the existing software development processes, including ineffective requirements definition and inadequate configuration management. In addition, SSA is currently facing the critical challenge of ensuring that its information systems are Year 2000 compliant. By the end of this century, SSA must review all of its computer software and make the changes needed to ensure that its systems can correctly process information relating to dates. These changes affect not only its new network but computer programs operating on both its mainframe and personal computers. We recently reported that while SSA has made significant progress in its Year 2000 efforts, it faces the risk that not all of its mission-critical systems will be corrected by the turn of the century. At particular risk are the systems used by state DDSs to help SSA process disability claims. Making software process improvements to address problems such as those SSA faces is considered to be a challenging undertaking for any organization. To guide agencies in assessing the strengths and weaknesses of their software development processes, SEI, in the late 1980s, developed the Capability Maturity Model (CMM). CMM is organized into five levels that characterize an organization’s software process maturity. As shown in table 1, these levels range from initial (level 1), characterized by ad hoc and chaotic processes, to optimizing (level 5), characterized by continuous process improvement based upon analysis and quantitative data. Further, to assist agencies in implementing effective software process improvement programs, SEI developed the IDEALSM model, which defines five phases of process improvement activity. These phases are: Initiating. Management determines that there is a business reason for improving their processes, sets general process improvement goals, and sponsors a process improvement program. Diagnosing. Using CMM, the current practices of the organization are appraised and characterized. Results of the assessment are documented and recommendations are made regarding areas in which to focus improvement efforts. Establishing. Based on the results of the diagnosing phase and the general goals that were defined in the initiating phase, the organization develops a strategy for improvement, prioritizes activities, and formulates measurable goals. Process action teams are established to develop action plans for improvement. Acting. Action plans are implemented through pilots. The results of pilots are evaluated and action plans are modified as appropriate. When proven effective, action plans are implemented throughout the organization. Learning. After the new processes have been in place for some time, their effectiveness is evaluated, communicated throughout the organization, and, as appropriate, used to formulate new action plans to ensure that goals are achieved. To determine the status of SSA’s efforts to improve its software development processes, we analyzed key documents, including SSA’s Software Process Improvement Program Management Plan, dated April 1997, Client/Server Transition Strategy: Preliminary Risk Assessment, dated September 1996, and relevant systems and strategic planning documents, such as the Information Systems Plan. In addition, to determine the status of specific projects being undertaken by contractors in support of the improvement initiatives, we reviewed the statements of work for contractor services and final deliverables, such as baseline assessments and software process improvement reports. We did not independently verify the accuracy of information reported in the contractors’ assessments of SSA’s software development processes. We analyzed SEI’s IDEALSM: A User’s Guide for Software Process Improvement, dated February 1996, which SSA is using to implement and manage its process improvement program, to determine whether SSA’s current and planned software development practices are consistent with this guidance. We reviewed additional SEI reports, including Moving On Up: Data and Experience Doing CMM-Based Process Improvement, dated August 1995, to identify successful practices of organizations applying CMM-based process improvements. In addition, we reviewed documents discussing the implementation schedule, technical strategies, and risks associated with SSA’s development of the RDS software application to obtain information on the agency’s experiences in software development. However, we did not specifically evaluate the progress of SSA’s ongoing effort to develop RDS. To further support our assessment of the actions that SSA is taking to improve its software development capability, we interviewed the Deputy Commissioner for Systems and other systems officials directly involved in implementing the improvement initiative, the General Services Administration official responsible for administering the support contract for SSA’s client/server software development assessment, and representatives of the contractors involved in this initiative. We performed our work from March 1997 through November 1997 in accordance with generally accepted government auditing standards. SSA provided written comments on a draft of this report. These comments are discussed in the “Agency Comments and Our Evaluation” section and are reprinted in appendix I. Recognizing the need to improve its software development capability, SSA has launched a formal software process improvement program and initiated pilot projects to test improved software development processes. In doing so, it is seeking to achieve a repeatable (level 2) level of software capability maturity. SSA acquired the assistance of SEI to help formulate and implement the improvement program. It adopted SEI’s CMM as the framework for assessing the current state of its software development capability and establishing improvement priorities and is following SEI’s IDEALSM model as the methodology for implementing and managing its software process improvement actions. Further, SSA has developed a software process improvement schedule identifying the specific phases and tasks that it plans to undertake to complete the implementation of its improved software development processes by June 2000. Consistent with SEI’s IDEALSM model, SSA has performed a number of the steps recommended for the initiating, diagnosing, and establishing phases of its software process improvement program and is beginning to implement steps under the acting phase. It has put in place the initial management infrastructure to support and facilitate its software process improvement initiatives by establishing a management steering committee to raise organizational awareness of the improvement program and a software engineering process group to oversee the development and implementation of the software process activities. It has also established various work groups to carry out the specific process improvement initiatives. SSA also has undertaken two assessments of the maturity of its existing software development processes, focusing on identifying (1) effective software development policies and procedures already being used within the agency and (2) key software development process areas needing improvement. SEI’s CMM specifies key process areas and criteria that must be addressed to achieve a particular software development maturity level. The key process areas for level 2 (repeatable) maturity—the level that SSA is currently seeking to achieve—are (1) requirements management, (2) software project planning, (3) software project tracking and oversight, (4) software subcontract management, (5) software quality assurance, and (6) software configuration management. With SEI’s assistance, SSA conducted a self-assessment to determine the strengths and weaknesses of its current processes for developing and maintaining software, which are primarily mainframe-oriented. This self-assessment identified 22 weaknesses in the level 2 key process areas. For example, the assessment found within the area of software project planning that risks had not been identified, assessed, or documented for some projects, and within the area of software project tracking and oversight, that the results and performance of some projects had not been tracked against key elements—such as costs, schedules, and risks—of SSA’s software project plans. In addition, a support contractor hired to identify client/server best practices and assist SSA in transitioning to a client/server environment performed an independent assessment of the agency’s software development processes. This assessment focused specifically on identifying the strengths and weaknesses in SSA’s ability to develop client/server software. It, too, identified weaknesses in SSA’s software development practices. For example, the assessment identified within the area of requirements management a need to improve practices for defining system requirements and specifications, and in the area of project planning, a need to improve practices for scheduling and estimating the cost of software development efforts. Based on the findings identified in both assessments, SSA developed a software process improvement program implementation plan. This plan will be used during the acting phase to, among other things, direct the software development activities of three pilot projects that the Office of Systems intends to undertake to help institutionalize the software process improvements. According to the implementation plan, project teams for the three pilots are expected to test and evaluate improved software development processes to address the identified weaknesses. SSA initiated its pilot activities in September 1997 and, at the conclusion of our review, had begun developing the policies and procedures that it will use to test each of the key process areas. SSA expects to complete all of the pilots by March 1999, after which it will finalize its strategy for implementing the improved software development processes. Although SSA has made important progress in its efforts to improve its software development processes, its improvement program does not yet include specific, measurable goals and baseline data that are essential to helping it achieve a repeatable (level 2) software development capability. Without measurable goals and baseline data, SSA does not yet have critical information needed to guide its improvement efforts and to provide evidence that the efforts are resulting in more consistent, cost-effective, and timely production of higher quality products. According to SEI’s IDEALSM: A User’s Guide for Software Process Improvement, clearly defined and measurable goals are necessary to provide guidance and to assist in developing tactics for improving the software development process. They also allow for objective measurement of the improvement results. SEI prescribes that general goals of the improvement program be defined during the initiating phase based on the business needs of the organization. These general goals are used, in conjunction with baseline data on the agency’s existing processes (such as software size estimates, defects identified, and calendar time for project completion), to develop specific short- and long-term measurable goals during the establishing phase. For example, one general goal could be to make software projects more predictable in terms of cost and schedule. If the measurement baseline established that 80 percent of the organization’s current projects exceed their original cost and schedule estimates by more than 25 percent, then the specific, measurable goal could be to improve that measure such that 80 percent of all projects are completed within 10 percent of their original cost and schedule estimates within 2 years. At the conclusion of our review, SSA had established general goals for its improvement program that included (1) achieving a repeatable (level 2) software capability maturity and (2) creating a software development environment that encourages continuous improvements in quality, productivity, and timeliness. However, it had not yet established specific, measurable improvement goals for its software development processes, nor had it defined the actual baseline data that it will use to monitor progress in achieving its goals. SSA’s Deputy Commissioner for Systems and the director of the software process improvement program told us that they recognized the importance of and need for establishing specific, measurable goals for the agency’s improvement program. However, they stated that the agency has not yet been able to define such goals for the improvement initiatives because it has not traditionally maintained baseline data on its software development projects that would be required to make such determinations. These officials told us that they intend to develop the necessary measures based on data compiled during the pilots being conducted to test improved software development processes. While SSA officials stated that they intend to develop the necessary measures during the pilots, at the time of our review they did not have a detailed strategy explaining how the pilots will be used to generate these measures. SSA contracted with the Gartner Group to develop a strategic measurement plan which recommends a general framework and steps for establishing consistent, repeatable processes to collect and track measurements; it also has outlined within its implementation plan the general strategy it intends to use for the three pilots. However, neither of these documents provided detailed information on how and in what time frames baseline data and specific, measurable goals will be derived from the pilots. Without an explicit strategy and time frames for generating baseline data and measurable goals, SSA cannot ensure that it will have essential information to monitor the progress of its improvement efforts. Recognizing the need to reassess its software development processes in light of transitioning to a client/server processing environment, SSA is taking important steps to improve its software development capability. If successfully completed, these actions should help position the agency to strengthen its processes for developing quality software products. However, until SSA develops baseline data and establishes specific measurable goals for its improvement initiatives as part of its pilot projects, it will not have necessary information to monitor its progress toward achieving its intended software process improvements. To strengthen SSA’s software process improvement program, we recommend that, as part of its recently initiated pilot projects, the Commissioner of Social Security direct the Deputy Commissioner for Systems to develop and implement plans that explicitly articulate SSA’s strategy and time frames for (1) developing baseline data, (2) identifying specific, measurable goals for its improvement initiative, and (3) monitoring and measuring progress in achieving these goals. In commenting on a draft of this report, SSA agreed with our recommendation and described actions that it is undertaking to develop a plan for its measurement activities. These actions include obtaining support for its pilot projects from the Gartner Group and working with SEI to formulate a plan that will include (1) tasks and time frames for developing baseline data, (2) measurable goals for the implementation of CMM-compliant processes, and (3) methods for measuring progress against established goals. We are encouraged by SSA’s response and will continue to monitor the agency’s progress in implementing its software improvement effort. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from its date. At that time, we will send copies to the Commissioner of Social Security; the Director of the Office of Management and Budget; appropriate congressional committees; and other interested parties. Copies will also be made available to others upon request. Please contact me at (202) 512-6253 or by e:mail at willemssenj.aimd@gao.gov if you have any questions concerning this report. Major contributors to this report are listed in appendix II. Valerie C. Melvin, Assistant Director Leonard J. Latham, Technical Assistant Director Michael A. Alexander, Senior Information Systems Analyst Gwendolyn M. Adelekun, Business Process Analyst The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary: Pursuant to a congressional request, GAO reviewed the status of the Social Security Administration's (SSA) software development process improvement efforts. GAO noted that: (1) SSA has initiated a number of actions to improve its software development capability; (2) among other things, it has: (a) launched a formal software process improvement program and initiated pilot projects to test improved software development processes; (b) acquired the assistance of the Software Engineering Institute (SEI) to help it assess the strengths and weaknesses in its current software development processes and to assist in implementing the improvement program; and (c) established a management steering committee and a software engineering process group within the Office of Systems to oversee software process improvement activities; (3) these are positive steps that should help position SSA to improve its software development capability; (4) although these initiatives are under way, SSA has not yet established key elements of its software process improvement program that are needed to measure the progress and success of its improvement efforts; (5) in particular, SSA has not yet defined specific, measurable goals for its software process improvement program or established the baseline data that it will use to assess its progress in achieving these goals; and (6) without this essential information, SSA cannot be assured of the extent to which its improvement efforts will result in the consistent and cost-effective production of high-quality products.