The Department of Education’s FSA manages and administers student financial assistance programs authorized under title IV of the Higher Education Act of 1965, as amended. These postsecondary programs include the William D. Ford Federal Direct Loan Program (often called the Direct Loan program), the Federal Family Education Loan Program (often called the Guaranteed Loan program), the Federal Pell Grant Program, and campus-based programs. Annually, these programs together provide about $50 billion in student aid to approximately 8 million students and their families. During the past three decades, the Department of Education has created many disparate information systems to support these various student financial aid programs. In many cases, these systems—run on multiple operating platforms using different network protocols and maintained and operated by a host of different contractors—were unable to easily exchange the timely, accurate, and useful data needed to ensure the proper management and oversight of the various student financial aid programs. For example, as we reported in 1997, neither the National Student Loan Data System nor other systems were designed for efficient access to reliable student financial aid information, since many systems were incompatible and lacked data standards and common identifiers. In addition, because FSA used three separate systems to originate and/or disburse title IV funds, access to student and school data was fragmented and unreliable. As a result, FSA found it increasingly difficult to quickly access data to support day-to-day operational and management decisions, and schools could not easily access data to obtain a clear picture of the title IV student aid that had been disbursed. In September 1999, FSA issued its initial modernization blueprint, which was subsequently updated in July 2000, to transform the title IV student financial aid systems using technology. COD is one of four school service business processes in FSA’s blueprint and is intended to implement a simplified process for the operation of the Direct Loan and Pell Grant programs. According to FSA’s modernization blueprint, the common origination and disbursement process is composed of seven steps involving students, the Department of Education, and schools: (1) obtain applicant data, (2) determine eligibility, (3) determine award, (4) notify the Department of Education of the intent to disburse, (5) obtain funds from Education, (6) disburse funds to student, and (7) close out. A common process to support origination and disbursement is considered critical to FSA’s goal of achieving an enterprisewide solution that provides real-time data to students, schools, and financial partners via Web portals. To implement COD, FSA is using middleware and XML technologies. Specifically, middleware is being used to integrate FSA systems that support the COD process. Traditionally, systems integration would require building separate point-to-point interfaces between every two applications. Although this approach can be effective, it creates several problems, such as (1) every connection between two applications requires custom programming; (2) a lot of connections have to be developed when there are multiple data sources; and (3) whenever the logic or data in one application changes, the accompanying interface often also needs to be altered. Middleware represents an alternative means to the traditional approach, and it can provide a quicker and more robust solution to systems integration. In essence, middleware separates the business application from the technical details of interapplication communications. Thus, middleware can simplify and reduce the number of interfaces for multiple systems because it can handle differences in data formats and record layouts. As part of the COD process, XML is being used to consolidate multiple legacy record formats previously used by schools to submit data on the Pell Grant and Direct Loan programs. By using an XML-based common record, schools can transmit one file with all of the student’s data instead of submitting separate legacy records with redundant student and school information. Appendix I provides a high-level depiction of the systems and technologies supporting the COD process as of November 2002. As depicted, the COD system can translate or convert legacy records by using middleware. In addition, middleware has been built into several existing systems so that they can establish connectivity and exchange data with the COD system through a common IT infrastructure. This IT infrastructure, called the Enterprise Application Integration (EAI) bus, is also implemented using middleware to route data between systems in a correct format. In addition, as part of the COD process, some schools have begun submitting Pell Grant and Direct Loan data using the XML-based common record. FSA hired Accenture as its “modernization partner” to help carry out its modernization blueprint, including the implementation of the COD process. Accenture is the prime contractor providing leadership of critical planning activities that are essential to the success of FSA’s modernization. Regarding the COD system part of FSA’s modernization, FSA also hired an independent verification and validation contractor to review the initial release of this system, which was completed earlier this year. FSA has made progress in implementing the new COD process. In particular, it has begun implementing (1) its middleware solution in its IT infrastructure and various existing systems, (2) the COD system, and (3) an XML-based common record. However, FSA’s implementation of COD is behind schedule, and critical work remains to be completed. For example, the basic COD system was to be completed by mid-October 2002; however, only about three-quarters of the COD basic system requirements had been implemented as of October 23, 2002. In addition, FSA is not tracking whether it is achieving certain benefits because it is still in the process of defining applicable metrics to measure progress. Without such tracking processes, FSA lacks critical information about whether it is achieving expected benefits. Finally, FSA lacks assurance that it has captured and disseminated important lessons learned related to schools’ implementation of the common record because it believes that its current ad hoc process is adequate. Accordingly, the thousands of schools that have not yet implemented the common record may not benefit from the experience of those that have. FSA has made progress in implementing COD. The following are significant elements of the COD process that have been implemented: Deployment of the EAI bus. As a prerequisite to implementing COD, in late October 2001, FSA deployed its middleware solution in an EAI “bus”—an IT infrastructure that uses middleware to access data from disparate systems, transform the data formats as necessary, and route the data to the appropriate requesting systems, thus enabling data exchange among disparate systems. The EAI bus provides the set of technical capabilities necessary to integrate FSA’s disparate systems. Initial implementation of basic COD system (release 1.x). On April 29, 2002, FSA went live with version 1.0 of the basic COD system. As of mid- November 2002, FSA had released an additional five sub-versions of the COD system (e.g., version 1.1). The COD system replaces the Direct Loan Origination System and the Recipient Financial Management System, and it currently processes files for all schools participating in the Pell Grant and Direct Loan programs. According to FSA, in the first 6 months of its operation, the COD system processed just under 16 million transactions, representing Pell Grant and Direct Loan awards totaling almost $10 billion to over 5 million recipients. Implementation of middleware in selected systems. As of mid-November 2002, FSA had built middleware into seven systems so that these systems can interact with the COD system through the EAI bus. These systems include (1) the Central Processing System, which determines students’ eligibility and award levels, and (2) the National Student Loan Data System, which contains loan- and grant-level information and is used by schools to screen student aid applicants to identify ineligible borrowers. Development and implementation of the common record. Using XML, FSA developed and began implementing a common record that schools can use to submit student financial aid data to the COD system. The common record, designed with assistance from members of the National Council of Higher Education Loan Programs and the Postsecondary Electronic Standards Council, consolidates multiple legacy file formats previously used by the Pell Grant and Direct Loan programs. Although FSA has made progress in implementing the COD process, critical work remains to be completed. First, FSA is behind schedule in implementing the basic COD system. Although FSA had planned to complete the basic COD system by mid-October 2002, only about three- quarters of the COD basic system requirements had been implemented as of October 23, 2002. For example, as of early November 2002, one of the basic business functions that remains to be implemented is to enable FSA to make automated adjustments in batches to school current funding levels. FSA now estimates that most of the remaining functionality will be completed by the end of September 2003. According to FSA IT and program officials, the implementation of the basic COD functionality was delayed to allow adequate time for testing to ensure the quality of the system. Second, as of November 19, 2002, Accenture reported several operational problems that needed to be addressed. For example, in some cases, the COD system was incorrectly processing school batch data that contained multiple change records for an individual student. According to COD and contractor officials, the causes of operational problems included unclear requirements and software design defects. An independent verification and validation contractor also found problems with the requirements and design aspects of release 1.0. The COD Contracting Officer’s Representative characterized these operational problems as very serious and stated that they could impede operations and the delivery of future COD releases. This same official noted that FSA and Accenture are currently undertaking efforts to address these problems. For example, FSA has established production teams composed of agency and contractor staff to address problems in specific areas. In addition, FSA has established a continuous improvement process to more rigorously manage its relationship with Accenture. Third, fewer postsecondary schools than planned have implemented the common record. FSA had estimated that 50 schools (out of about 5,500) would implement the common record in fiscal year 2002. However, as of November 26, 2002, only 22 schools had implemented and tested the common record with FSA. FSA COD officials attributed the fewer-than- expected number of schools using the common record to schools and vendors not being ready to implement it. FSA expects that the number of schools using the common record will be considerably higher during the next award year (2003–2004) because, by April 2003, it plans to implement and test the common record with EDExpress, a software application FSA distributes free of charge to about 3,000 schools for use in submitting data. In addition, FSA expects that all schools will be using the common record format by March 2004, in time for the 2004–2005 award year. In its COD business case, FSA outlined five expected benefits: (1) reduced cost, (2) increased customer satisfaction, (3) increased employee satisfaction, (4) increased financial integrity, and (5) the integration and modernization of legacy systems. An important aspect of implementing an IT investment cited by the Office of Management and Budget and our IT investment management guide is evaluating the results of the investment by determining whether such expected benefits are being achieved. However, as illustrated in table 1, at this time FSA has only some of the data necessary to determine whether it is achieving all expected benefits. In particular, for the increased customer satisfaction and financial integrity benefits, FSA (1) has not fully defined the performance metrics to be used, (2) does not have all baseline data, and/or (3) is not fully tracking whether the benefits are being achieved. In these cases, FSA COD officials stated that they were in the process of developing relevant metrics, which would be tracked to measure the project’s performance against expected benefits. However, until FSA develops these data and begins tracking actual benefits and comparing them with expected benefits, it will lack vital data with which to demonstrate actual investment results. FSA IT officials also stated that they plan to have a contractor conduct a postimplementation review of the COD basic system in fiscal year 2003, which is expected to look at the achievement of expected benefits. While this is an important initiative that could provide FSA with valuable information, it does not take the place of a continuing and systematic process of tracking actual benefits. According to our IT investment management guide, another critical activity is establishing a process for developing and capturing lessons learned in a written product or knowledge base and disseminating them to decision-makers. Lessons-learned mechanisms serve to communicate acquired knowledge more effectively and to ensure that beneficial information is factored into planning, work processes, and activities. Lessons learned can be based on positive experiences that save money or on negative experiences that result in undesirable outcomes. FSA has recognized the importance of generating lessons learned in certain areas. For example, it has implemented a process for developing lessons related to managing the relationship between the agency and its prime contractor. However, FSA lacks such a process for capturing or disseminating lessons related to school migration to the common record. FSA COD officials stated that lessons learned pertaining to school migration to the common record are addressed through periodic discussions during biweekly conference calls with schools undergoing testing with FSA and during portions of various FSA-sponsored conferences. FSA COD officials stated that they believed this process for capturing and disseminating lessons learned was adequate. However, by relying on such an ad hoc process, FSA lacks assurance that it has captured and disseminated all key lessons learned related to schools’ implementation of the common record and could overlook important improvements that could be made. In addition, schools that do not attend the conferences may not receive and benefit from the lessons identified in the initial phase of implementation. As a result, schools may encounter problems that could have been avoided or mitigated had they known of other schools’ experiences. This could hamper FSA’s ability to facilitate the transition of schools to the new common record and thus the agency’s ability to fully implement the new COD process and achieve the expected benefits. In commenting on a draft of this report, FSA stated that it plans to provide lessons learned as part of a planned update to its school testing guide. While this is a positive step, it does not replace the need for mechanisms to continuously capture and disseminate acquired knowledge as schools implement the common record. Table 2 includes examples of lessons learned provided by FSA at our request that were drawn from schools’ initial implementation of the common record for the 2002–2003 award year. Such information would be important for the thousands of schools that have not yet implemented the common record so that they can avoid problems during the common record implementation and testing processes. FSA has taken important steps toward achieving full implementation of the new COD process. However, critical actions, such as completing the basic functionality of the COD system and the implementation of the common record at thousands of affected schools, must still be undertaken. In addition, FSA has not yet fully established the metrics and processes to track actual benefits related to all of its expected benefits or the lessons that have been generated by the few schools that have implemented the common record thus far. By not tracking actual benefits, FSA lacks information that is critical to determining whether it is meeting all of its goals. Further, not capturing and disseminating information to schools regarding lessons learned could make achieving these goals more difficult. To determine the extent to which the new COD process is achieving expected results related to customer satisfaction and financial integrity, we recommend that you direct FSA’s Chief Operating Officer to expeditiously develop metrics and baseline data to measure these benefits and develop a tracking process to assess the extent to which the expected results are being achieved. To ensure that the schools that have not yet implemented the common record benefit from the experiences of those that have, we recommend that you direct FSA’s Chief Operating Officer to establish a process for capturing lessons learned in a written product or knowledge base and for disseminating them to these schools. In providing written comments on our draft report, FSA’s Chief Operating Officer provided technical comments and updated information, but did not comment on our recommendations. Specifically, The Chief Operating Officer did not believe the report adequately portrayed the level of COD progress that had been made. In particular, she took issue with our using the completion of 75 percent of COD requirements as an indication of progress. Although the Chief Operating Officer did not disagree with the accuracy of this figure, she stated that FSA’s informal analysis indicated that between 85 to 90 percent of COD functions had been implemented, which she believed was a better gauge of progress. We believe that we have accurately portrayed FSA’s progress in implementing the COD process. First, since FSA’s analysis was “informal,” and FSA’s supporting documentation had limited detail that we could not validate, we do not agree that this should be the primary basis for an analysis of COD’s progress. Second, we included both the percentage of COD’s requirements that had been implemented and FSA’s estimate in our report. Nevertheless, we modified our report to include additional data provided by FSA regarding the number of transactions processed by the COD system to further indicate progress. The Chief Operating Officer agreed that the tracking of all of the expected benefits is not in place at this time, but stated that work is under way in this area. FSA also provided updated information and supporting documentation related to the tracking of some of the expected benefits. We made changes to the report reflecting this new information, as appropriate. The Chief Operating Officer agreed that it is important and beneficial to communicate lessons learned, but stated that FSA’s informal method for communicating lessons related to school migration to the common record worked well in the first year of COD implementation. FSA also noted that it plans to include lessons learned in a planned update to its school testing guide. We modified the report to reflect this initiative, but we do not agree that FSA’s informal method or its plan to include lessons learned in its testing guide is adequate because these approaches do not provide a continuous process for actively capturing and disseminating lessons learned. As a result, some important lessons may be overlooked, and all schools may not be aware of potential problems associated with implementing the common record. FSA’s written comments, along with our responses, are reproduced in appendix II. We are sending copies of this report to the appropriate congressional committees, the Secretary of Education, the Chief Operating Officer of Education’s Office of Federal Student Aid, and the Director of the Office of Management and Budget. This report will also be available at no charge on the GAO Web site at http://www.gao.gov. If you have any questions on matters discussed in this report, please contact me at (202) 512-9286 or Linda J. Lambert, Assistant Director, at (202) 512-9556. We can also be reached by E-mail at pownerd@gao.gov and lambertl@gao.gov, respectively. Other individuals making key contributions to this report included Jason B. Bakelar and Anh Q. Le. Legacy records (MQSeries) (MQSeries) (MQSeries) Single common record file (XML) 18 legacy files (secured FTP) FTP File Transfer Protocol LOWeb Loan Origination Web NSLDS National Student Loan Data System PEPS Postsecondary Education Participants System XML Extensible Markup Language This is a temporary interface. DLOS is targeted to be retired in fiscal year 2003. MQSeries is IBM’s proprietary message format. CThis is an information technology infrastructure that enables data exchange among disparate systems. The following are GAO’s comments on the Department of Education’s Office of Federal Student Aid’s letter dated December 18, 2002. 1. We revised the draft report title to clarify that our follow-up review was focused on assessing FSA’s progress in implementing the COD process. 2. Information related to operational problems was contained in the draft report. We asked the COD Contracting Officer’s Representative to characterize these problems, and he stated that they were very serious. In addition, we confirmed the seriousness of these problems at the conclusion of our review with FSA IT and program officials. 3. We modified this report to include data on the number of transactions processed. We also modified our report to clarify that all schools participating in the Pell Grant and/or Direct Loan programs currently use the COD system. 4. We do not agree that the report implies that FSA’s use of a phased-in approach in implementing the common record increases risks. Instead, this report notes that the implementation is not yet complete. 5. We believe that we have accurately portrayed FSA’s progress in implementing the COD process. First, since FSA’s analysis was “informal,” and FSA’s supporting documentation had limited detail that we could not validate, we do not agree that the COD Development Manager’s functionality estimate should be the primary basis for an analysis of COD’s progress. Second, we included both the percentage of COD’s requirements that had been implemented and FSA’s estimate in our report. 6. We modified our report to reflect this updated information as appropriate. 7. We do not agree that FSA’s informal process for capturing and disseminating lessons learned was adequate because (1) it may lead to important lessons being overlooked and (2) all schools may not be aware of potential problems associated with implementing the common record. 8. We modified this report to reflect that FSA plans to include lessons learned in a planned update to its school testing guide. While this is a positive step, it does not replace the need for mechanisms to continuously capture and disseminate acquired knowledge as schools implement the common record.

Summary: To address system problems and other long-standing management weaknesses, in 1998, the Congress created a discrete unit within the Department of Education, the Office of Federal Student Aid (FSA). This office subsequently adopted a new approach to systems integration using middleware (a type of software that can allow an application to access data residing in different databases) and Extensible Markup Language (XML)--a flexible, nonproprietary set of standards that is intended to make it easier to identify, integrate, and process information widely dispersed among systems and organizations. FSA's first use of this approach is the Common Origination and Disbursement (COD) process for the Direct Loan, Pell Grant, and campus-based programs. GAO initiated a follow-up review to assess FSA's progress in implementing this process. FSA has made progress in implementing the COD process. Specifically, it has implemented (1) a new information technology infrastructure that uses middleware to enable data exchange among disparate systems; (2) the initial version of the basic COD system, which replaces two existing systems and is being used by schools participating in the Pell Grant and Direct Loan programs; (3) middleware into existing systems to support the COD process; and (4) a common record based on XML that schools can use to submit student financial data for the Pell Grant and Direct Loan programs. However, the implementation of the COD process is behind schedule, and its ultimate success hinges on FSA's completing critical work, including addressing serious postimplementation operational problems, and having thousands of postsecondary schools implement the common record. Further, there are important elements to managing any information technology investment that FSA has not yet completed: Determining whether expected benefits are being achieved. FSA has only some of the metrics, baseline data, and tracking processes necessary to determine whether it is achieving all expected benefits. Tracking lessons learned. FSA has relied on an ad hoc approach for gathering and disseminating lessons learned related to schools' implementation of the common record. To address this issue, FSA plans to include lessons learned as part of an update to its school testing guide. However, this does not replace the need for an ongoing mechanism to capture and disseminate lessons learned, without which schools may encounter problems that could have been avoided or mitigated.