IDEALSM is a service mark of Carnegie Mellon University and stands for initiating, diagnosing, establishing, acting, and leveraging. weapons systems, command and control systems, satellite systems, inventory management systems, financial systems, personnel systems, payment systems, and others. Many of these systems in turn are connected with systems operated by private contractors, other government agencies, and international organizations. DOD’s ability to effectively manage information technology is critical to its ability to accomplish its mission. Its reliance on software-intensive systems to support operations related to intelligence, surveillance, security, and sophisticated weaponry—along with financial management and other business functions—will only increase as the department modernizes and responds to changes in traditional concepts of warfighting. The scope of DOD’s information technology inventory is vast: over 1.5 million computers, 28,000 systems, and 10,000 computer networks. Further, many of DOD’s most important technology projects continue to cost more than projected, take longer to produce, and deliver less than promised. As a result, we have designated DOD systems development and modernization efforts as a high-risk area. The quality of the processes involved in developing, acquiring, and engineering software and systems has a significant effect on the quality of the resulting products. Accordingly, process improvement programs can increase product quality and decrease product costs. Public and private organizations have reported significant returns on investment through such process improvement programs. SEI has published reports of benefits realized through process improvement programs. For example, SEI reported in 1995 that a major defense contractor implemented a process improvement program in 1988 and by 1995 had reduced its rework costs from about 40 percent of project cost to about 10 percent, increased staff productivity by about 170 percent, and reduced defects by about 75 percent. According to a 1999 SEI report, a software development contractor reduced its average deviation from estimated schedule time from 112 percent to 5 percent between 1988 and 1996. During the same period, SEI reported that this contractor reduced its average deviation from estimated cost from 87 percent to minus 4 percent. To aid organizations attempting to initiate and manage SPI programs, SEI has published a best practices model called IDEAL,SM which defines a systematic, five-phase, continuous process improvement approach, with a concurrent sixth element addressing the program management tasks spanning the five phases (see figure 1). Initiating: During this phase, an organization establishes the management structure of the process improvement program, defines and assigns roles and responsibilities, allocates initial resources, develops a plan to guide the organization through the first three phases of the program, and obtains management approval and funding. Two key organizational components of the program management structure established during this phase are a management steering group and a software engineering process group (SEPG). Responsibility for this phase rests with senior management. Diagnosing: During this phase, the SEPG appraises the current level of process maturity to establish a baseline capability against which to measure progress and identifies any existing process improvement initiatives. The SEPG then uses the baseline to identify weaknesses and target process improvement activities. It also compares these targeted activities with any ongoing process improvement activities and reconciles any differences. Responsibility for this phase rests primarily with line managers and practitioners. Establishing: During this phase, the SEPG prioritizes the process improvement activities and develops strategies for pursuing them. It then develops a process improvement action plan that details the activities and strategies and includes measurable goals for the activities and metrics for monitoring progress against goals. Also during this phase, the resources needed to implement the plan are committed and training is provided for technical working groups, who will be responsible for developing and testing new or improved processes. Responsibility for this phase resides primarily with line managers and practitioners. Acting: During this phase, the technical working groups, formed under the establishing phase, create and evaluate new and improved processes. Evaluation of the processes is based on pilot tests that are formally planned and executed. If the tests are successful, the working groups develop plans for organization-wide adoption and institutionalization, and once approved, execute them. Responsibility for this phase resides primarily with line managers and practitioners. Leveraging: During this phase, results and lessons learned from earlier phases are assessed and applied, as appropriate, to enhance the structures and plans of process improvement programs. Responsibility for this phase rests primarily with senior management. The model’s sixth element, continuous program management, specifies management structures and tasks for planning, organizing, directing, staffing, and monitoring the program. Responsibility for this element rests with senior management. Each phase of the IDEALSM model contains several recommended tasks. Appendix I, which describes our objectives, scope, and methodology, identifies all tasks for each phase. The Army and Air Force units that we reviewed, as well as DFAS and two of the four Navy units, have long-standing SPI programs that satisfy almost every task recommended in the IDEALSM model (see table 1 for a summary of how each component and its units, if applicable, compared to the model). For example, in 1996 the Secretary of the Army mandated that all software development, acquisition, and maintenance activities establish SPI programs. Further, the Army requires that its software activities continually improve their process maturity and has set maturity goals for all of its units. Army regulations also mandate that contractors be evaluated for software process maturity. Moreover, the two specific units within the Army that we reviewed have SPI management structures, plans, and dedicated resources. In addition, these units have continuously evolved in software and system process maturity through many years of assessing their baseline process capabilities, implementing new and improved process initiatives, reassessing process maturity, and implementing lessons learned. Both Army units satisfy all IDEALSM tasks. In contrast, DLA, the Marine Corps, and two of the Navy’s four units that we reviewed do not perform important IDEALSM model tasks. In particular, DLA currently does not satisfy any of the model’s recommended tasks. According to DLA officials, it had an SPI program prior to 1998, but at that time the program was terminated to reduce costs. During our review, DLA’s CIO stated that the agency plans to begin a new SPI program and has taken a first step by assigning organizational responsibility. The Marine Corps has many SPI activities under way that could form the foundation of a program. However, it is not performing several key SPI tasks that are fundamental to SPI program success. For example, the Marine Corps has assigned responsibility for process improvement, and it has begun assessing its software process maturity to establish baseline capability. However, it is not using this baseline as a basis for implementing recommended improvements, nor does it have an SPI plan or dedicated resources for these activities. As such, the likelihood of the Marine Corps’ process improvement initiatives producing desired results is diminished. Two of the four Navy software/systems units that we reviewed also do not have SPI programs that are aligned with the IDEALSM model. To their credit, however, one has recently taken the first step toward initiating a program and the other has activities under way that could form the beginnings of a program. (See appendix IV for more detailed results on each of the components that we reviewed.) The four components that have SPI programs—Army, Air Force, DFAS, and parts of the Navy—have different approaches for directing and controlling their respective programs, ranging from centralized to highly decentralized; each, however, reports positive results. For example, DFAS has a centralized approach, with its headquarters office directing and controlling all SPI activities. In contrast, the Army, Air Force, and Navy have decentralized approaches to SPI program management. The Army, which began its SPI program centrally, has since delegated SPI responsibility to its commands, which—in the case of the two commands we reviewed—have further delegated SPI program management to their respective software/systems units. Similarly, the Air Force units that we reviewed further delegated SPI management to their respective software/systems units. The Navy commands follow different approaches—one manages its program centrally and the other has delegated SPI management to its software/systems units. Despite different approaches, each DOD component/unit with an SPI program reports positive effects on software/systems quality. DFAS, for example, reports that its SPI program has reduced its cost to deliver software to about one-third less than organizations of similar size. One Navy software activity reports reduced costs, improved product quality, and a 7:1 return on its SPI investment. An Army activity reports that it has almost doubled its productivity in writing software for new systems because of improvements made under its SPI program. (See appendix IV for more detailed information on the approaches and reported benefits of the components that we reviewed.) Within OSD, the Assistant Secretary for Command, Control, Communications, and Intelligence is responsible for establishing and implementing DOD’s policies, processes, programs, and standards governing the development, acquisition, and operation of nonweapons systems software and information systems. Similarly, the Under Secretary for Acquisition, Technology, and Logistics is responsible for establishing DOD acquisition policies and procedures. Accordingly, OSD has an important leadership role to play in ensuring that DOD components reap the maximum possible benefits of effective SPI programs. Such leadership can include dissemination of policies and guidance promoting SPI programs and activities, knowledge of the nature and extent of components’ SPI programs and activities, associated lessons learned and best practices, and facilitation of SPI knowledge-sharing across DOD components. Both OSD organizational units have efforts under way aimed at improving some aspects of DOD’s ability to develop and acquire software and systems. For example, they have established teams to conduct software acquisition maturity assessments and established a software collaborators group. They also are collecting software metrics and establishing training for managers.