In September 1997, the District of Columbia Financial Responsibility and Management Assistance Authority (Authority) awarded a contract to acquire a new FMS. The overall objective of the FMS project is to improve the District's financial systems through faster, more efficient, and accurate processing providing increased functionality, flexibility, and reduced cost of operations. According to the Chair of the Authority, the new FMS is intended to (1) eliminate the principal problems that exist with the current system and ensure that all financial management guidelines are adhered to, (2) enable managers to more effectively and efficiently monitor and control financial resources, and (3) produce timely, accurate, and reliable information, providing decisionmakers with the basic financial information needed to make more informed decisions. The Authority awarded a contract for the new FMS in September 1997 and committed to an aggressive implementation schedule. The schedule anticipates (1) pilots in five agencies beginning in February 1998, (2) the accounting system to be implemented by October 1998, and (3) District-wide implementation by February 1999. We were asked to review the District's efforts to acquire a new financial management system. Our objective was to determine whether the District had implemented disciplined software acquisition processes for acquiring its new financial management system. To accomplish this, we applied the Software Engineering Institute's Software Acquisition Capability Maturity Model (SA-CMM) and its Software Capability Evaluation (SCE) method. SEI's expertise in, and methods for, software process assessment are recognized and accepted throughout the industry. Our evaluators were all SEI-trained software specialists. SA-CMM ranks organizational maturity according to five levels (see figure 1). Maturity levels 2 through 5 require the verifiable existence and use of certain software acquisition processes, known as key process areas (KPA). According to SEI, an agency that has these acquisition processes in place is in a much better position to successfully acquire software than an organization that does not have these processes in place. We evaluated the District's software acquisition processes against six of the seven level 2 KPAs (the transition to support KPA was not evaluated because the District does not plan to support FMS in-house) and one level 3 KPA (see table 1). We selected level 2 because it is the minimum level at which any assurance exists that software acquisition processes are mature enough to consistently deliver promised software capabilities on time and within budget. We included one level 3 KPA--acquisition risk management--because it is considered by software experts to be a very important process area. Basic project management processes are established to track performance, cost, and schedule. The necessary process discipline is in place to repeat earlier successes on projects in similar domains. The software acquisiton process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined and success depends on individual effort. The purpose of software acquisition planning is to ensure that reasonable planning for the software acquisition is conducted and that all aspects of the total software acquisition effort are included in these plans at the proper level of detail. The software acquisition planning process, among other things, includes (1) addressing software life-cycle support in acquisition plans, (2) preparing life-cycle software cost estimates, (3) having a written software acquisition policy, (4) measuring and reporting on the status of software acquisition planning activities, and (5) having guidance on software training and experience requirements for project personnel. The FMS project had many strengths in this KPA. The District received pro bono assistance from several companies to help define the acquisition strategy and conduct the activities for software acquisition planning. An acquisition strategy was developed and the acquisition planning team was staffed with personnel with software and systems experience. The team developed a cost estimate and the District management was briefed by the team on a periodic basis. This enabled the District management to be informed on the progress of the acquisition planning and the various activities through the solicitation phase. However, the FMS project also had many weaknesses in this KPA. Weaknesses observed included a lack of policy on acquisition planning and no specific assignment of responsibility for acquisition planning. Furthermore, the FMS project did not always document significant project decisions or update the planning document to reflect these decisions. For example, when the District decided to not pursue a single contract to both acquire FMS and outsource data center operations, the capability assessment (a software acquisition planning document) was not updated to reflect this decision. Decisions should be documented and the planning documents updated to ensure that large acquisitions such as FMS can be effectively managed. Table 2 shows the strengths and weaknesses for the software acquisition planning KPA and the specific findings supporting these ratings. The purpose of solicitation is to prepare a request for proposal that delineates a project's software-related requirements and select a contractor that can most cost-effectively satisfy these requirements while complying with relevant solicitation laws and regulations. Specific requirements for a solicitation process include, among other things (1) having and following a solicitation plan, (2) assigning responsibility and ensuring sufficient resources for coordinating and conducting solicitation activities, (3) preparing and reviewing cost and schedule estimates for the software products and services being acquired, and (4) periodically measuring solicitation work completed and effort and funds expended, comparing these measures to plans, and reporting the results to management. The FMS project exhibited many process strengths during the solicitation. The District has a policy on solicitation and the FMS project followed this policy. The project had experienced personnel on the source selection team and these personnel briefed the team members on the objectives of the solicitation. However, the District did not measure either time or funds expended to conduct the solicitation. Specifically, no evidence was provided to show that the FMS project tracked personnel hours or costs during the conduct of the solicitation. Addressing this weakness would enable the District to better estimate the resources needed to conduct similar acquisitions in the future. For example, if these data were collected and made available to other projects, such as the tax systems upgrade, the District would be in a better position to understand its own capability to effectively conduct solicitation, to estimate how long such a solicitation was likely to take, and to eliminate problems that may have hampered the FMS solicitation. Table 3 shows the strengths, weaknesses, and observations for the solicitation KPA and the specific findings supporting these ratings. The purpose of requirements development and management is to establish and maintain a common and unambiguous definition of software requirements among the acquisition team, system users, and software development contractor. This KPA involves two subprocesses: (1) developing a baseline set of software-related contractual requirements and (2) managing these requirements and changes to these requirements for the duration of the acquisition. A number of requirements development and management practices are necessary to satisfy this key process area. These include (1) having a written organizational policy for establishing and managing requirements allocated to software, (2) documenting plans for the development and management of requirements, (3) having documented processes for requirements development, including elicitation, analysis, and verification, (4) measuring and reporting on the status of requirements development and management activities to management, (5) appraising the impact on software of system-level requirements changes and (6) having a mechanism to ensure that contractor-delivered work products meet specified requirements. The FMS project has some process strengths in the conduct of requirements development and management. The project team is performing requirements management activities in accordance with its documented plan and software-related contractual requirements have been baselined. In addition, District management periodically reviews the status of requirements development and management activities with the project team. However, in acquiring FMS, the District did not perform many of the requirements development and management practices necessary to satisfy this KPA. For example, the District does not have an organizational policy for establishing and managing software-related requirements, there is no clear assignment of responsibility for requirements development and management and no documented evidence exists to show either resource requirements or resources expended for requirements development activities. Currently, the FMS project has begun to hold "requirements confirmation meetings" with the users to validate the requirements already specified in the FMS contract. Although requirements should be validated, this should have been done prior to releasing the request for proposal to ensure that the proposal accurately reflects the District's requirements. Changing requirements after contract award may adversely impact project cost, schedule, and/or performance. Table 4 shows the strengths, weaknesses, and observations for the requirements development and management KPA and the specific findings supporting these ratings. The purpose of project management is to manage the activities of the project office and supporting contractors to ensure a timely, efficient, and effective software acquisition. Effective project management requires, among other things, that project teams (1) be organized to accomplish the project's objective, (2) have a written policy for the management of the software project, (3) document their plans for the activities of the project team, (4) have the authority to alter either the project's performance, cost, or schedule baseline while maintaining the other two, and (5) periodically brief management on the status of project management activities. The FMS project has many process strengths in project management. For example, a team was assigned responsibility for managing the project and staffed with experienced individuals whose roles and responsibilities were defined. The program management plan was written and a corrective action system to track issues and problems was implemented. However, the District has no written policy for the execution of the software project. As a result, the District has no assurance that FMS or any other software acquisition project it undertakes will be conducted in a disciplined manner. Table 5 shows the strengths, weaknesses, and observations for the project management KPA and the specific findings supporting these ratings. The purpose of contract tracking and oversight is to ensure that (1) the software development contractor performs according to the terms of the contract, (2) needed contract changes are identified, negotiated, and incorporated into the contract, and (3) contractor performance issues are identified early, when they are easier and less costly to address. An effective contract tracking and oversight process, among other things, includes (1) having a written organizational policy for contract tracking and oversight, (2) having a documented plan for contract tracking and oversight, (3) conducting tracking and oversight activities in accordance with the plan, and (4) ensuring that individuals performing contract tracking and oversight are suitably experienced or trained. The FMS project had many strengths in this KPA. The project has a designated project manager, a group is responsible for managing contract tracking and oversight activities, and the team is meeting periodically with the contractor and tracking issues in a corrective action system. However, at the time of our review, there was no contracting specialist supporting the team in the execution of the contract. In addition, the District has no documented policy for contract tracking and oversight activities. Table 6 shows the strengths, weaknesses, and observations for the contract tracking and oversight KPA and the specific findings supporting these ratings. The purpose of evaluation (testing) is to determine that the acquired software products and services satisfy contract requirements prior to acceptance. The evaluation process includes (1) documenting evaluation plans and conducting evaluation activities in accordance with the plan, (2) developing and managing evaluation requirements in conjunction with developing software technical requirements, (3) incorporating evaluation requirements into the solicitation and the resulting contract, (4) tracking contractor performance of evaluation activities for compliance with the contract, (5) ensuring that adequate resources are provided for evaluation activities, and (6) measuring and reporting on the status of evaluation activities to management. The FMS project has some process strengths in this KPA. For example, responsibility for evaluation activities has been designated to the project manager, individuals designated to perform evaluation activities have experience, and members of the evaluation team received briefings on the objectives of the evaluation. However, there is no documented evaluation policy or plan, no evidence that evaluation requirements have been developed, and neither the Authority nor the project manager reviews the status of evaluation activities. Table 7 shows the strengths, weaknesses, and observations for the evaluation KPA and the specific findings supporting these ratings. SEI defines risk as the possibility of suffering a loss. The purpose of acquisition risk management is to formally identify risks as early as possible and adjust the acquisition to mitigate those risks. An effective risk management process, among other things, includes (1) having a written policy on acquisition risk management, (2) developing a software acquisition risk management plan, (3) conducting software risk management activities in accordance with the plan (e.g., identifying risks, taking mitigation actions, and tracking risk mitigation actions to completion), and (4) measuring and reporting on the status of acquisition risk management activities to management. The FMS project had one strength for this KPA. The project has designated responsibility for risk management to the project management team. However, the District is not performing any of the other practices to satisfy this KPA. For example, there is no written policy or plan for acquisition risk management, resource requirements for risk management have not been identified, and at the time of this audit, neither the Authority nor the project manager were reviewing the activities for risk management. Table 8 shows the strengths, weaknesses, and observations for the acquisition risk management KPA and the specific findings supporting these ratings. Leading software acquisition organizations rely on defined and disciplined software acquisition processes to deliver promised software capabilities on time and within budget, first on a project-by-project basis, and later, as the organization's processes become more mature, consistently across the institution. While the District has many strengths in its acquisition processes for FMS, it also has many weaknesses that, overall, make its processes undisciplined and immature. As a result, the District's success or failure in acquiring FMS depends largely on specific individuals rather than on well-defined software acquisition management practices. This greatly reduces the probability that the system will consistently perform as intended and be delivered on schedule and within budget. To satisfy the intent of all the software acquisition key process areas and thereby have a reasonable assurance that acquisition efforts are effectively planned, managed, evaluated, and tracked, the District must address the many weaknesses identified in this report. This would entail the District formulating and implementing a written policy for software acquisition planning, requirements development and management, project management, contract tracking and oversight, evaluation, and acquisition risk management. In addition, it is important for the District to track the various activities for each KPA to ensure that they are being performed and that evaluation and risk management activities are being planned and effectively conducted. We recommend that the Chairman of the District of Columbia Financial Responsibility and Management Assistance Authority direct the District's Chief Financial Officer to (1) take the following actions for the six KPAs we reviewed to ensure that the current FMS acquisition and implementation is satisfactorily completed and (2) apply these actions to any future software acquisitions. Document decisions and update the planning documents to ensure that large acquisitions such as FMS can be effectively managed. Designate responsibility for software acquisition planning activities. Determine required resources for acquisition planning. Ensure that measurements of software acquisition activities are taken. Ensure that the software acquisition planning documentation is updated as well as make program changes regarding outsourcing of the data center and upgrading the current system versus buying off-the-shelf. Ensure that the software acquisition planning documentation addresses life-cycle support of the software. Develop a written policy for software acquisition planning. Requirements Development and Management: Develop an organizational policy for establishing and managing software-related requirements. Clearly assign responsibility for requirements development and management. Document either resource requirements or resources expended for requirements development activities. Develop the capability to trace between contractual requirements and the contractor's work products. Develop measurements to determine the status of the requirements development and management activities. Develop a written policy for the execution of the software project. Authorize the project manager to independently alter either the performance, cost, or schedule. Require that measurements be taken to determine the status of project management activities. Contract Tracking and Oversight: Develop written policy for contract tracking and oversight activities for the financial management system project. Support the project team with contracting specialists. Require that the project team review the contractor's planning documents (for example, the project management plan, software risk management plan, software engineering plan, configuration management plan). Assign someone responsibility for maintaining the integrity of the contract. Take measurements to determine the status of contract tracking and oversight activities. Develop written policy for managing the evaluation of acquired software products and services. Develop a documented evaluation plan. Develop evaluation requirements in conjunction with system requirements. Assess the contractor's performance for compliance with evaluation requirements. Develop measurements to determine the status of evaluation activities. Ensure that the Authority and the project manager review the status of evaluation activities. Develop written policy for software acquisition risk management. Designate a group to be responsible for coordinating software acquisition risk management activities. Define resource requirements for acquisition risk management. Ensure that individuals designated to perform software acquisition risk management have adequate experience and training. Integrate software acquisition risk management activities into software acquisition planning. Develop a software acquisition risk management plan in accordance with a defined software acquisition process. Develop a documented acquisition risk management plan and conduct risk management as an integral part of the solicitation, project performance management, and contract performance management processes. Track and control software acquisition risk handling actions until the risks are mitigated. Ensure that risk management activities are reviewed by the Authority and the project manager. GAO requested comments on a draft of this report from the Chairman, District of Columbia Financial Responsibility and Management Assistance Authority, and the District's Chief Financial Officer. They provided us with written comments that are reprinted in appendixes I and II. In their comments, the District of Columbia Financial Responsibility and Management Assistance Authority's Executive Director and the District of Columbia's Chief Financial Officer acknowledged that the software acquisition project for the new financial management system was a high risk initiative and that the District's processes were not sufficiently mature. The District Chief Financial Officer identified initiatives in each of the key process areas. Both cited ongoing corrective actions, which, if properly implemented, will address several of our recommendations. For example, the Chief Financial Officer stated that the District is developing a Risk Management Plan and is evaluating various strategies to identify and manage risks, and that the Chief Technology Officer for the District of Columbia is developing policies and procedures for information resource management which will include software acquisition. However, the District Chief Financial Officer also added that their efforts to date have achieved a sound acquisition state consistent with the intent of the SA-CMM. As discussed in the report, significant improvements would be necessary to achieve the minimally acceptable level of maturity as defined by the Software Engineering Institute's Software Acquisition Maturity Model to satisfy the intent of all the software acquisition key process areas. Accordingly, the District has not yet achieved a sound acquisition state consistent with the intent of the SA-CMM. If the District is to instill the needed discipline into its systems acquisition processes consistent with the intent of SA-CMM, it will need to effectively implement all of our recommendations. We are sending copies of this report to the Ranking Minority Member of your Subcommittee and to the Chairmen and Ranking Minority Members of the Subcommittee on Oversight of Government Management, Restructuring, and the District of Columbia, Senate Committee on Governmental Affairs, the Subcommittee on the District of Columbia, Senate Committee on Appropriations, and the Subcommittee on the District of Columbia, House Committee on Government Reform and Oversight. We are also sending copies to the Director of the Office of Management and Budget, the Chairman of the District of Columbia Financial Responsibility and Management Assistance Authority, and the Chief Financial Officer of the District of Columbia. Copies will be made available to others upon request. If you have questions or wish to discuss the issues in this report, please contact me at (202) 512-6412. Major contributors to this report are listed in appendix III. Richard Cambosos, Senior Attorney The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary:
Pursuant to a congressional request, GAO reviewed whether the District of Columbia had implemented disciplined software acquisition processes for its new financial management system (FMS). GAO noted that: (1) while the District has many strengths in its acquisition processes for FMS, it also has many weaknesses; (2) when compared to standards established by the Software Engineering Institute, the District's processes for software acquisitions are not mature; (3) of the six key process areas (KPA) evaluated for the repeatable level, the District fully satisfied only one--solicitation; (4) severe weaknesses were found in other critical key processes, including requirements development and management and evaluation; (5) for example, the District does not have a policy for establishing and managing software-related requirements, does not at present have adequate resources for requirements development, and has not formally designated responsibility for requirements development and management; (6) similarly, the District does not have an effective evaluation process, and is currently unable to objectively determine if the acquired systems will satisfy the contract requirement; (7) finally, the District has not satisfied the one KPA evaluated for the defined level of maturity, acquisition risk management; and (8) the FMS project does not have a risk management plan and does not track project risk.