SEC introduced its ARP program in 1989 because of capacity and other problems in the exchanges’ and clearing organizations’ information systems. The program resulted from SEC’s November 1989 policy statement that noted that many exchanges and other organizations experienced problems in their systems during the high trading volumes that occurred in October 1987 and again in October 1989. This policy statement also cited disasters, such as fires or earthquakes, that required exchanges to implement their contingency planning procedures. Since the ARP program was created, exchanges, clearing organizations, and the systems that link the stock and options markets have continued to periodically experience capacity-related problems or other disruptions. Under the ARP program, SEC called on the SROs to ensure that the information technology systems they use to conduct market operations have adequate processing capacity for current and future estimated trading volumes. In addition, SEC sought assurances that SROs were taking steps to assess the risk to their operations from internal and external threats, such as unauthorized use, computer vandalism, or computer viruses. The first ARP policy statement called on the SROs to establish capacity planning procedures to estimate current and future information system capacity needs and to periodically conduct capacity stress tests. In addition, the statement recommended that the SROs have assessments performed of their systems capacity and their vulnerability to physical threat. In a second policy statement issued in May 1991, SEC provided more specific guidelines to the SROS that identified five primary areas it expected the SROs to have reviewed, including the general controls and security relating to computer operations and facilities, telecommunications, systems development, capacity planning and testing, and contingency planning. The ARP program is administered by staff in SEC’s Office of Technology and Enforcement within the Division of Market Regulation. To determine the adequacy and completeness of the criteria SEC uses to conduct capacity and security oversight, we compared the criteria with guidance issued by other financial regulators and organizations that have developed standards for auditing information systems, including the information security manual we developed for use by federal agencies.We also used a list of criteria we developed based on the procedures recommended in a publication written by experts in the field of capacity planning for information systems and on the findings from our prior reports or testimonies that address automation issues in the securities markets. In addition, we reviewed SEC inspection work plans that the ARP staff uses to conduct on-site inspections and held discussions with SEC staff on the criteria that they use to conduct their oversight. To determine the scope and frequency of the ARP on-site inspections, we obtained from SEC a list of on-site inspections conducted between 1995 and June 2001 of 27 SROs and electronic communication networks (ECN). We reviewed a total of 11 SEC ARP inspection reports that addressed capacity planning or security-related issues, including the written reports and supporting work papers on ARP inspections of 7 SROs which included the most active exchanges as well as some of the smaller exchanges, and just the written reports for 4 other SROs. We discussed our observations of these reviews with ARP staff. To determine the scope and frequency of the independent reviews, we examined certain recent audit reports and a summary of audit reports prepared by SEC ARP staff. We also discussed the results of our assessment of these reports with SEC staff. Specifically, we reviewed copies or summaries of 37 reviews done by SRO internal audit staff for the 3 largest SROs in 2000. In addition, we examined seven independent reviews of five SROs performed by external organizations that were included in the supporting work papers of the SEC inspections we reviewed. To address how the voluntary nature of the ARP programs affects SEC oversight capabilities, we reviewed various documents prepared by the ARP staff. The documents included analyses of SRO systems, on-site inspection reports, a printout of SEC staff’s database of the status of recommendations made during inspections, and oversight work plans. We conducted this work in Washington, D.C., from November 2000 to June 2001 in accordance with generally accepted government auditing standards. To plan and conduct inspections and other oversight activities, the ARP program uses criteria from a variety of sources that address aspects of capacity planning, security, and other information system issues. The second ARP policy statement discussed five primary areas that SEC expected that SROs would address regarding their information systems. Using these areas, the ARP staff work with SROs to develop a checklist as an initial guide for use by SEC staff in conducting their on-site inspections. This checklist was also provided to the SROs in 1991 for use as part of the independent reviews of their systems. The ARP program staff told us that they regularly update the inspection checklist by consulting professional standards and guidance relating to information systems established by other regulatory, audit, or industry bodies. Figure 1 shows how SEC staff and the SROs use the various sets of guidance. In our review of SEC on-site inspection work papers, we observed instances in which ARP staff used the steps from this checklist to plan certain segments of work they would perform at individual SROs. SEC officials said they also expect the areas examined during inspections of SRO systems to be based on the ARP policy statements as well as on industry standards for conducting systems audits and the reviewers’ professional judgment. In the work papers we reviewed, we found examples of individualized checklists that ARP staff had created that incorporated steps from the 1991 checklist and other sources for use in particular inspections of SROs. SEC staff said that checklists created for past inspections are also used to plan subsequent inspections. ARP staff also described performing frequent Internet searches to monitor the latest information on standards and issues from various auditing and information system organizations, such as the Information Systems Audit and Control Association. At a minimum, SEC officials said that they expect their staff to perform these searches before each inspection. In their view, these additional information sources provide up-to-date, comprehensive criteria for assessing capacity planning, security, and other relevant systems issues. ARP program staff also explained that they use their own knowledge and experience to plan the inspections and the ongoing monitoring they conduct. They said that they also added review steps to their inspections to address any current challenges facing the SROs that would affect information systems. For example, they added steps to inspections recently to address the industry’s transition to decimal pricing as well as for the Year 2000 date change. They also added steps to inspections of individual SROs when new systems are being implemented or outages occur. Because of continuous change in technology, SEC staff need to refer to up- to-date criteria and standards to conduct their oversight. However, they lack a consolidated guide for their staff. The 1991 inspection checklist that the ARP staff continuously updates to serve as criteria for their inspections does not address some developments in the markets and advances in information technology. For example, SEC’s checklist addresses some security issues but does not include steps relating to intrusion detection. The 1991 checklist also does not address the increased risk of unauthorized access faced by SROs with information systems connected to the Internet. Although SEC officials explained that the SROs do not generally operate critical systems that use the Internet, some are using it to transmit information for less important systems, and others are considering or are already developing Internet-based systems. SEC’s 1991 inspection checklist is also missing some elements relating to capacity planning. For example, the checklist did not specifically address certain issues relating to volume forecasts used in capacity planning in which some SROs have had problems. In 2000, the National Association of Securities Dealers’ (NASD) transition to decimal pricing was delayed because the system NASD planned to use for decimal trading lacked sufficient capacity. A review by an external organization later found that NASD’s volume forecasts had not adequately accounted for the increasing volatility in its trading and processing volumes. Although SEC ARP staff had identified deficiencies and made recommendations to NASD to improve its capacity planning processes, we did not find volatility of trading volumes specifically addressed in SEC’s 1991 checklist or the other work plans that SEC staff prepared for the inspections we reviewed. Because the ARP program does not have a consolidated guide for its staff, the burden of maintaining consistent quality in ARP oversight falls primarily on the most experienced ARP staff. Both SEC and other regulators frequently use comprehensive guides to ensure that the rule- compliance reviews their staff perform are consistent. For example, staff in SEC’s Office of Compliance Inspections and Examinations use examination modules that consolidate the procedures for the reviews they expect their staff to perform consistently at the broker-dealers and other entities they review. However, without a similar consolidated guide, the ARP program staff must make continual efforts to consult numerous sources to supplement the areas not contained in SEC’s 1991 ARP materials. Conducting quality reviews of the various SROs also requires ARP program staff to have broad knowledge of relevant issues and to be aware of how market developments could affect the systems at each SRO. We found that the various work plans, risk analyses, and other documents prepared by the ARP program staff were generally thorough and addressed issues adequately. However, the level of detail and extent of documentation varied across staff members. Although the quality of SEC’s oversight depends heavily on individual staff, the ARP program has experienced considerable staff turnover. SEC officials said that the ARP program staff has experienced turnover rates of almost 30 to 40 percent in some years. The officials said that finding replacements is always difficult, as the salaries SEC offers for people with information system skills are not competitive with the private sector. As of June 15, 2001, 4 of the 10 ARP program staff had 2 years or less of experience, including 2 staff who had just joined the program. SEC officials said that only experienced staff prepare updated workplans and lead on-site inspections. However, the lack of a consolidated written guide could lead to inconsistency in planning and conducting inspections, given the high turnover rate of ARP staff. We found that, for the most part, SEC’s on-site inspections addressed key capacity and security issues. However, resource limitations have prevented SEC from conducting inspections as frequently as their staff would prefer. During an on-site inspection, the ARP staff usually review SRO procedures, examine supporting documents, and hold discussions with SRO staff over the course of 4 to 5 days. During each inspection, ARP staff focus on the information system issues from the ARP guidance that are most relevant to the particular SRO. Although SEC staff do not conduct detailed steps to review all ARP issues during each inspection, most inspections begin with a presentation by the SRO, which the ARP staff told us covers all ARP issues. SEC staff also reported conducting some 1-day on-site inspections that focused on more limited issues. ARP staff then prepared a report that was later provided to the SROs’ management. Our review of the ARP on-site inspection reports and the supporting work papers addressing capacity and security issues indicated that, for the most part, these inspections addressed the key issues relating the SROs’ procedures. We reviewed reports and supporting work papers for the most recent on-site inspections done at seven SROs and four additional inspection reports prepared between 1996 and 2000. In these documents, we found examples of detailed audit work plans that were specifically designed to address the objectives of each ARP inspection. The work papers also included documents prepared by the SROs, including their formal capacity plans and trading volume and processing load projections, which SEC staff had asked to review as part of the inspections. We also found that SEC staff had collected documents the SROs had prepared on vulnerability assessments, as well as summaries of security staff meetings. In addition, we observed instances in which SEC staff documented their reviews of the security-related steps from the review checklist. The reports ARP staff prepared after conducting on-site inspections frequently contained numerous substantive recommendations to the SROs that addressed capacity planning, security, and other issues. For example, in an inspection done at one SRO, ARP staff made seven recommendations, including that the SRO increase the capacity of its systems, improve the security procedures for two major systems, and increase the frequency of disaster recovery testing. Although SEC officials told us that the ARP program has no formal goal for the frequency of inspections, ARP staff said that they would prefer conducting on-site inspections every 12 to 18 months. However, limited staff and the need to monitor industrywide information technology initiatives have prevented them from conducting examinations this frequently. According to the information SEC provided us, SEC staff conducted 41 on-site inspections of exchange or clearing organization SROs from 1995 through June 2000. During this 6-year period, ARP staff inspected most SROs once every 2 to 3 years and addressed capacity and security issues in most of these inspections. However, at least eight of these inspections lasted only 1 day. Furthermore, over this 6-year period the total number of days that ARP staff were actually on each SRO’s premises was very limited. According to the data SEC provided us, we calculated that the number of days that ARP staff were on site averaged a total of 7 days at each SRO during this 6-year period, with ARP staff being on site at the least visited SRO for a total of only 4 days and at the most visited SRO for a total of 19 days. ARP program officials explained that because of their small staff they conduct only seven or eight inspections per year. Although the ARP program had a staff of 10 as of June 15, 2001, it has had as few as 4 during some years because of generally high turnover. The staff also explained that they had spent a considerable amount of time addressing major industrywide initiatives, some of which spanned several years. These initiatives included preparations for the Year 2000 date change and the transition to trading using decimal instead of fractional prices. SEC officials told us that they take other steps to ensure that the SROs are adequately addressing information system issues. SEC staff meet annually with the SRO officials responsible for information systems. During these day-long annual report meetings, the SRO staff provide presentations on prior and upcoming changes to their systems and on activities relating to market events that could affect system capacities, such as decimal trading and other initiatives. SEC staff told us that these meetings allow them to question the SROs and obtain copies of relevant materials. When an SRO is subject to an on-site inspection, the officials explained that the first day is usually a presentation of the SRO staff’s annual report. Although SEC originally envisioned that SRO systems would also be reviewed under the ARP program by independent external organizations, SRO internal auditors now perform the majority of these reviews. The reviews now address the key areas of ARP cyclically based upon an annual risk analysis, but we were unable to determine whether all the issues are being addressed with sufficient frequency. In addition, SEC has requested reviews by external organizations when internal audits have been insufficient or when deficiencies existed in SRO systems and procedures. In the 1989 policy statement announcing the ARP program, SEC called for annual independent reviews of SROs that would cover capacity planning, security, and other areas. SEC staff told us that at that time, SEC proposed that external organizations perform these independent reviews. However, ARP staff said that the SROs later raised concerns about the costs of implementing such reviews and the potential overlap with the SROs’ own internal audit processes. ARP staff told us that they had also identified a need to modify the independent review guidance to ensure that the reviews were of sufficient depth. As a result, SEC issued the second ARP policy statement in 1991. In addition to expanding the areas that should be reviewed at the SROs, this statement also clarified that SROs could use their internal auditors to perform the independent reviews. However, the statement noted that, if internal auditors were to be used, they should adhere to the standards set by various groups, such as the Institute of Internal Auditors and the Information Systems Audit and Control Association. In addition, SEC asked that an external organization periodically assess the SRO internal auditors’ independence, competency, and work performance. Since this change, the majority of the independent reviews are now done by the SROs’ internal audit processes, rather than by external organizations. SEC and the SROs have also agreed to a change in the type and frequency of the independent reviews. In December 1993, SEC and the SROs agreed that SRO staff would plan the independent reviews addressing the key areas identified in the ARP guidance, using a risk-based approach. Using this approach, the SROs’ internal auditors are to determine which areas should be examined by conducting a yearly risk analysis of the SRO’s information systems. This risk analysis allows the internal audit staff to develop an audit plan that identifies the critical areas that need to be reviewed that year and less urgent issues that can be deferred until a later date. Although the SROs evaluate the risks in their systems against all of the ARP areas under this approach annually, the SEC officials explained that the SROs are not expected to review all of the key areas of the ARP guidance each year. As discussed in our Federal Information Systems Audit Control Manual (GAO/AIMD-12.19.6, January 1999), and elsewhere, such an approach is considered appropriate for reviews of this type. Although SEC staff said that they were generally satisfied with the quality and scope of the reviews the SROs’ internal auditors had performed, we could not determine from the documents SEC staff prepared whether these reviews were addressing all the areas contained in the ARP guidance with sufficient frequency. To verify the adequacy of the SROs’ efforts, ARP staff said that they also perform their own risk analyses for the SROs each year. They then are to review the SROs’ risk analyses, audit plans, and past audit results to assess whether the SROs’ independent reviews are addressing the ARP guidelines appropriately. When an SRO has not addressed an issue warranting attention, SEC requests a review of that area. The ARP program staff said they were also satisfied with the internal audits because many include testing of controls and compliance with procedures. In addition, the ARP staff told us that the SROs have increased the number of internal audit staff that review information system issues and that the quality of these audits has improved over time. When the ARP program first began, some of the SROs did not have internal auditors who could review information systems, and ARP staff said that their oversight efforts have resulted in increased internal audit staffing at the SROs. According to SEC staff, in the mid-1990s two major SROs had only one internal auditor specializing in information systems issues. As a result of SEC staff efforts, one of these SROs gradually increased the number of information systems auditors it employs to five. Nevertheless, from our review of the SROs’ internal audits conducted during 2000, we were unable to determine whether these reviews were addressing all of the important areas in the ARP policy statements with sufficient frequency. In one analysis we reviewed, ARP staff noted that the internal audit staff at one major SRO had not reviewed at least two of the five areas specified in the ARP policy statements since 1992 and did not state when reviews had last been conducted for the other three areas. ARP staff told us that in this case, auditors for the SRO’s service vendor had reviewed at least one of the areas and information had been provided to SEC about the other area periodically. At another SRO, the SEC staff’s inspection report noted that the internal auditors had not conducted an independent review of the SRO’s capacity planning process in 8 years. ARP staff told us that they had performed reviews of this area at least twice during this period. With the pace of technological change and developments in the markets, it is unclear whether this level of attention to SRO capacity planning is sufficiently frequent or appropriate. ARP staff told us that when SRO internal auditors do not address all the issues addressed in the ARP policy statements, the ARP staff take steps to see that they do. If their analysis indicates that internal audits have not reviewed particular issues, ARP staff said that they would consider the areas not addressed as high risk for the SROs and they would try to include them in their next on-site inspection. In some cases, ARP staff request that the SROs obtain independent reviews by an external organization when internal audits have not sufficiently addressed systems issues or because of recurring systems problems at some SROs. We found that SEC staff had recommended in at least five recent on-site inspection reports that the SROs contract with external organizations to perform reviews of SROs’ capacity planning processes. For example, ARP staff requested an external review of NASD’s overall capacity planning process before that market announced that the system it intended to use to transmit price quotations did not have sufficient capacity to allow it to implement decimal trading by the date SEC had set for the securities markets. In a July 2000 inspection report, the ARP program staff requested that another SRO obtain a review of all aspects of its capacity planning process because that SRO’s trading volume had grown dramatically and its internal auditors had not recently addressed this process. And in 1997, after the two systems that transmit information between the stock and options markets experienced numerous delays or queues in their transmissions, ARP staff requested an external review be done of the organization that operates these systems. In March 2000, an official whose exchange relies on price data transmitted by the intermarket system for stocks told us that systems problems had caused considerable financial losses to members until its capacity was upgraded. In addition, the options exchanges and the Options Price Reporting Authority, which administers the intermarket system for options, are under an SEC order that requires them to limit the data they transmit across this system because their systems capacity is insufficient. From a review of internal audits done at three SROs during 2000, we found that the internal audits varied in both scope and depth. We reviewed 29 internal audit reports conducted at two SROs during 2000 and a summary prepared by SEC staff of eight audits done in 2000 at another SRO that uses an external organization to conduct its internal audits. In some cases, the audits appeared to address an important ARP area thoroughly and contained substantive recommendations, including one report that identified numerous deficiencies in an SRO’s contingency planning procedures. Some of the reports also indicated that the internal auditors had taken steps to test relevant controls over systems. In general, most of the internal audit reports for the three SROs that we reviewed were limited in scope, covered only one SRO system, or made minor recommendations, such as asking that one SRO obtain the most recent version of a capacity planning software program or recommending that the staff at one SRO use only one entrance to its data center. Most of the reports we reviewed addressed security or other information system issues—such as change management processes—rather than capacity planning issues. Our review of seven reports addressing capacity and security issues that external organizations had prepared for five SROs showed that these reports generally had identified substantial deficiencies. For the most part, the SROs had obtained these reviews in response to requests by ARP staff. In one review, the external organization identified seven problems relating to an SRO’s capacity planning procedures, including finding that the SRO had not collected all the data needed for its capacity planning process, identified the applications that were generating increases in processing demand, or used a standardized forecasting approach for all systems. In addition, external audit reports recommended that SROs create formal capacity planning processes and security procedures for systems that currently lack them. Because the ARP program was not established under SEC’s rulemaking authority, it lacks specific rules that SEC can use to sanction SROs for not complying. Although SEC staff reported that the SROs generally comply with the ARP program, we found that in some cases SROs had not implemented ARP staff recommendations and had not always created the requested notices and reports sought under ARP. When establishing the ARP program, SEC left open the possibility of making the program mandatory but did not establish criteria to assess the level of cooperation under the voluntary program. The policy statements issued when SEC began the ARP program established voluntary guidelines for the SROs to follow regarding the capacity and security of their information systems. These guidelines called for the SROs to have independent reviews performed on their systems and to make various reports and notices to SEC. However, the program was not established under SEC’s rule making process. SEC officials explained that the view of the staff at the time was that any specific standards relating to information systems included in such a rule could become obsolete in a short period of time. SEC staff would then be required to seek amendments to the rule, which would also likely take considerable time and effort to complete. In their view, voluntary guidelines afford SEC staff greater flexibility. However, by issuing only voluntary guidelines, SEC staff have no specific rules to require SROs to implement key ARP recommendations or create the reports or notices called for in the policy statements and cannot sanction SROs under the ARP program for failing to do so. SEC officials said that they believed they could bring an official action against SROs whose failure to follow ARP was serious enough to represent a violation of the general requirement that exchanges maintain the ability to operate. They said, however, that SEC rarely uses such authority. ARP staff acknowledged that SROs have not addressed several significant capacity and security recommendations or concerns raised in ARP inspections. For example, we previously reported that in 1996, ARP staff recommended that NASD establish capacity alternatives to meet unexpected system demand. However, NASD has continued to experience capacity-related problems with several of its systems, disrupting the markets. For example, insufficient capacity in NASD’s price- quotation system delayed the start of decimal trading by all securities markets for 3 months and prevented NASD from fully trading in decimals for an additional 7 months. As a result, investor benefits from the reduced spreads that have resulted from decimal trading on the Nasdaq market were delayed by an additional 10 months. In addition, NASD has experienced capacity-related delays in a system that transmits orders to buy or sell shares in response to displayed price quotations. Officials from a major ECN told us in 2000 that they have experienced losses of up to $1.5 million a day because they are obligated to honor orders that arrive late through this system for shares that have already been sold to their own customers. Honoring these delayed orders can produce losses because the ECN sometimes has to execute new orders at disadvantageous prices if the price of the security has changed since the original transaction. Finally, NASD experienced trading disruptions on June 28, 2001 because the number of market participants given access to one of its systems exceeded the number of market participants that system had been programmed to handle. NASD officials said that the system was set up to handle about 90 users at once; however, by that date the number of users exceeded this figure by about 30 percent, and the system software had not been modified to account for this growth. Other important ARP recommendations and concerns that were not being implemented or addressed dealt with SROs’ security procedures, including their contingency plans for addressing physical threats or damage. In 2000, ARP staff recommended that one SRO develop and publish security policy and procedures and enforce them through a central authority, in accordance with basic industry standards. The SRO disagreed with the ARP recommendations, preferring to leave its security procedures decentralized. Another ARP staff recommendation that one SRO develop a recovery plan for trading facilities used for two of its most actively traded securities has been outstanding since at least 1995. Although this SRO has discussed various alternatives during this period for continuing operations in the event that its trading floor becomes unavailable, as of July 2001, its staff had still not implemented an alternative approach. In addition, although ARP program staff considered the lack of backup facilities to be a major deficiency, ARP program staff have recommended in other cases that SROs perform studies rather than take actions to resolve the deficiencies. In at least three cases, ARP staff recommended that SROs study the feasibility of establishing such facilities to avoid potentially lengthy shutdowns should their trading locations became unusable. One SRO disagreed with the recommendation, citing the costliness of maintaining such facilities, and the other SROs performed or are performing the studies. However, none has taken steps that fully address the ARP staff concerns that major physical damage to the trading floors could render these SROs unable to operate for an extended period. Although the ARP program calls for the SROs to create certain reports to SEC when outages or other disruptions occur that affects their systems, these reports were not always being made. As stated in the second ARP policy statement, the SROs are to report immediately to SEC systems outages that are expected to last longer than 30 minutes and report shorter outages after systems have been repaired. In addition, the second ARP statement recommended that SROs provide SEC with notices of significant system modifications. According to ARP staff, approximately 100 system outages were reported in fiscal year 2000, and for more than half of these, SEC officials said that they asked the SROs to provide analyses or other documentation of the event. SEC staff said that most SROs provide notices of outages or system modifications, but that some important outages or changes have not been reported. According to the findings from an SEC on-site inspection, one SRO lacked procedures for ensuring that notices of system modifications would be created and provided to SEC. In response, this SRO agreed to implement appropriate procedures. Another ARP inspection found that one SRO had failed to report at least six system outages during 2000. If SROs were required by SEC rule to provide SEC with notifications of significant changes to their automated systems, then the failure to have procedures in place for ensuring that notices of systems modifications are provided to SEC would likely demonstrate a weakness in the SRO’s internal controls. If the deficiency was severe enough, SEC could initiate an enforcement proceeding. In some cases, SEC staff became aware of anticipated SRO system changes from press or trade publications. For example, ARP staff learned of the proposed 1998 sale of one SRO’s options trading operations to another in a newspaper report. Although some of these instances involved proposed system changes that had not been finalized by the SROs, not knowing the most current configuration for the SROs systems could make planning inspections and other oversight activities more difficult for SEC staff. SEC stated in its initial ARP release that it would consider making the ARP program mandatory if SROs did not cooperate fully. However, SEC has yet to develop formal criteria and perform an assessment of SRO cooperation. In 1998, SEC’s Office of Inspector General reported that SEC had not indicated how it would assess compliance with the ARP program.Because of the increased importance of information technology to the functioning of the securities markets, the Inspector General’s report recommended that the agency consider making the ARP program mandatory. In response to this recommendation, ARP program staff said that they had considered the issue and determined that ARP should remain voluntary. SEC staff said that a substantial lack of cooperation with ARP would be inconsistent with an SRO’s general obligations, but they were satisfied with the extent to which SROs cooperate. The use of information technology is pervasive in the securities industry, and the quality of the SROs’ systems is vital to the functioning of the markets. Based on our review, the ARP program provides SEC staff with some assurance that SROs are addressing capacity planning, security, and other information system issues. In addition, the ARP staff performed comprehensive and in-depth inspections of SRO systems, and were actively involved in the industry’s recent completion of efforts to ready systems for the Year 2000 date change and the transition to decimal trading. Various aspects of the ARP program highlight areas in which SEC’s oversight could be strengthened to better assure that the SROs manage their critical information systems sufficiently to prevent major disruptions in the markets. Although SEC staff consulted an extensive array of standards and guidance to ensure that their oversight addresses relevant issues, the lack of a consolidated inspection guide for their staff means that the consistency and quality of SEC’s oversight is heavily dependent on the efforts of the individual ARP staff. A consolidated inspection guide could take the existing five ARP areas and provide additional topics that the SEC staff find are most relevant given the current state of technology in the markets. Rather than duplicating external guidance that SEC staff already use, a consolidated inspection guide could enumerate these other sources and incorporate, by reference, the specific areas that the SEC staff have found relevant to their work. Having a consolidated inspection guide for its staff would better ensure that SEC’s ARP program oversight is conducted thoroughly and consistently across its staff. This is particularly important because the program has high turnover that results in significant portions of its staff having little or no experience. SEC’s ability to oversee information system issues is also hampered by the limited resources available to the ARP program, which constrains its staffs’ ability to inspect the SROs more frequently. SEC now relies largely on the SROs’ own internal auditors to review systems in detail instead of more routinely using external organizations as an independent check on the activities of the SROs, as was originally envisioned under the ARP program. In cases in which the internal audits had not sufficiently addressed issues or when SROs had deficiencies in their information system procedures, SEC staff have called for SROs to obtain external reviews of their systems. When combined with the reliance on internal audits, the ARP program’s voluntary nature raised concerns that SEC’s oversight efforts are not as effective as they could be. SRO cooperation in implementing significant SEC recommendations has been uneven. The SROs’ unwillingness to make recommended improvements may have adversely affected the markets, for example, when capacity problems at one market delayed full implementation of decimal trading for all securities markets. Because some SROs have not addressed ARP staff concerns over the lack of backup trading facilities, securities trading in the United States could be severely limited if a terrorist attack or a natural disaster damaged one of these exchange’s trading floor. When SROs are not implementing significant recommendations or taking steps to remedy identified capacity and security weaknesses, SEC’s Chairman and Commissioners could focus additional SRO attention on the need to take actions to improve their systems. SEC’s ARP policy statements left open the possibility of having a rule- based program if compliance was not adequate. Developing formal criteria and performing an assessment of SROs’ compliance with the ARP program would allow SEC to determine whether a rule-based program would be warranted. Such an assessment also could weigh the advantages and disadvantages of the current voluntary program and whether it provides SEC with sufficient authority to optimally ensure that SROs’ systems are sound. Criteria and an assessment could allow SEC to determine whether failure to implement recommendations risked material disruption in the markets. Making the ARP program mandatory could give SEC the authority it needs to better assure that SROs take cost-effective steps to improve their systems and procedures and reduce the risk of systems- related problems disrupting the markets. On the other hand, if the program were to be made mandatory, SEC would need to build adequate flexibility into the governing rule to deal with technological change. Because of the importance of the proper functioning of the SROs’ information systems, we recommend that the Acting Chairman, SEC, take the following actions: ensure that the ARP program develops a consolidated inspection guide for the ARP staff that is updated on a periodic basis, ensure that significant ARP program recommendations and concerns that have not been addressed by the SROs are brought to the attention of the Chairman and the Commissioners, and develop formal criteria for assessing the SROs’ cooperation with the ARP program and perform an assessment to determine whether the voluntary status of the ARP program is appropriate. We obtained comments on a draft of this report from SEC, which are presented in appendix I. In its letter, SEC commented that the draft report was based on an inaccurate view of the ARP program, and that it did not reflect the development of the program since SEC issued its two ARP policy statements in 1989 and 1991. SEC provided an extensive discussion of the ARP program’s evolution over time. In response, we have made language changes where appropriate and believe that our report fairly presents the evolution of the ARP program over time. However, although the ARP program has achieved some important goals, we think that it could be more efficient and effective if our recommendations were adopted. SEC generally disagreed with our recommendations, noting that activities they already perform satisfy the intent of the recommendations. Specifically, SEC did not see a need to develop a consolidated inspection guide because it would quickly become outdated and the ARP staff’s approach to developing work plans for individual inspection results in oversight that addresses key capacity and security issues. The ARP staff’s approach has, to date, generally resulted in oversight that addresses key issues. However, given the high staff turnover and the relative inexperience of many staff, we are recommending that ARP develop a guide that will assure continued consistency. Moreover, we believe that such standard guides are a good business practice and a sound internal control. The type of guide that we recommend would also require minimal effort to update because it would largely incorporate by reference standards and criteria developed by other organizations, which would likely be updated by those organizations regularly. With respect to our recommendation that SEC develop a process to bring significant unimplemented ARP recommendations and outstanding concerns to the attention of the Chairman and the Commissioners, SEC commented it had a process that satisfied the recommendation. In its letter, SEC noted that it already reviews the status of all ARP recommendations. SEC also stated that where an SRO’s response to ARP recommendations is unsatisfactory, SEC has a procedure to bring the matter to the attention of the Division Director and, if necessary, to the Chairman and Commissioners. SEC commented that, based on discussions with us, the staff was enhancing its process for reviewing the status of ARP recommendations and updating the recommendations database. We note, however, that according to SEC officials, no unimplemented ARP recommendations or concerns have been escalated beyond the Division Director level. We believe that some significant unimplemented ARP recommendations and concerns regarding capacity and security weaknesses at the exchanges and clearing organizations warrant attention at the highest levels of the Commission. Involvement at this level would increase the likelihood that SROs would take meaningful action in response to such recommendations and concerns. Therefore, we reaffirm our recommendation. SEC also disagreed with our recommendation that it develop formal criteria for assessing SRO compliance with the ARP program. SEC commented that the risk assessment process the ARP program staff conducts annually for each SRO represents their assessment of the SRO’s compliance with the ARP policy statements and that when SROs do not implement ARP recommendations or remedy concerns, they call for additional inspections and reviews. Although we agree that the ARP staff’s efforts have resulted in some improvements in the SROs’ information systems, we remain concerned that some recommendations that SROs have not fully addressed pose a greater risk of further market disruptions. Moreover, seeking to address noncompliance with the ARP program by performing additional inspections would likely result in ARP staff identifying many of the same discrepancies over time. For example, ARP staff found capacity-related problems over several years at NASD and have had long-standing concerns about contingency planning alternatives at some SROs. SEC’s risk assessment process, although allowing it to adequately plan its oversight, does not constitute or supplant the type of assessment of overall program compliance that we recommend. Instead, by developing formal criteria and assessing the overall level of compliance with the ARP program, SEC would have a sound basis for evaluating the nature of the program. Even if no change in its status were made after such an assessment, periodically reapplying the criteria would allow SEC to assess the pattern of compliance by SROs over time to ensure that the program’s status is not hampering the effectiveness of SEC’s oversight of the SRO information systems that are critical for continued market functioning. SEC also commented that neither GAO nor SEC itself has any basis to believe that the voluntary nature of the program is problematic. However, we did identify various instances in which SROs were not addressing recommendations or taking actions in response to ARP staff concerns or were not making the reports that SEC has requested. Furthermore, we are not recommending that SEC make the ARP program mandatory, but instead have recommended that SEC develop formal criteria to assess whether the program is working as it is currently structured. SEC also provided technical comments that we incorporated as appropriate, including refining our presentation of the extent to which ARP program recommendations have not been implemented. In addition, we revised the language of the report and our recommendation to clarify that the SEC Chairman and Commissioners should be advised when significant recommendations to SROs are not implemented or SRO actions do not address ARP staff concerns. As agreed with you, unless you publicly release its contents earlier, we plan no further distribution of this letter until 30 days from its issuance date. At that time, we will send copies to the Chairman and Ranking Minority Member, Senate Committee on Banking, Housing, and Urban Affairs; the Chairman and Ranking Minority Member, House Committee on Financial Services; the Chairman, House Committee on Energy and Commerce; and the Acting Chairman, SEC. We will also make copies available to others upon request. If you have any further questions, please call me at (202) 512-8678 or Cody J. Goebel, Assistant Director, at (202) 512-7329. The following are GAO’s comments on the Securities and Exchange Commission’s letter dated July 18, 2001. 1. SEC’s letter states that our report overlooks the important distinction between a program that is tasked with overseeing information technology, in which no single set of standards exists, and other programs based on assessing compliance with rules that lend themselves to bright-line tests. However, we believe that our report acknowledges the evolving nature of information systems and the lack of one source for standards, but also offers suggestions to improve SEC’s oversight of this area. For this reason, we recommended that SEC create a consolidated guide for its staff of the most up-to-date and authoritative sources for criteria for planning their oversight activities. We also believe that rules can be drafted to allow sufficient flexibility for information technology advances. Furthermore, many examination programs assess compliance using professional judgment against criteria even when bright lines do not exist. 2. SEC’s letter states that our report assumes that its 1991 checklist is the principal tool used by SEC staff to conduct inspections. However, our report describes the process SEC staff uses to plan inspections, including drawing on external criteria and using work plans and checklists from more recent inspections. However, we did observe instances in which the staff continued to use the 1991 checklist, but had to supplement the areas that it does not address with all the other sources. In addition to those named above, Ronald W. Beers, Emily R. Chalmers, Heather T. Dignan, William Lew, and Jean-Paul Reveyoso made key contributions to this report. Securities Pricing: Trading Volumes and NASD System Limitations Led to Decimal-Trading Delay, GAO/GGD/AIMD-00-319, Sep. 20, 2000. Securities Pricing: Progress and Challenges in Coverting to Decimals, GAO/T-GGD-00-96, Mar. 1, 2000. Securities Pricing: Actions Needed for Conversion to Decimals, GAO/T- GGD-98-121, May 8, 1998. Financial Markets: Stronger System Controls and Oversight Needed to Prevent NASD Computer Outages, GAO/AIMD-95-22, Dec. 21, 1994. Financial Markets: Computer Security Controls at Five Stock Exchanges Need Strengthening, GAO/IMTEC-91-56, Aug. 28, 1991. Financial Markets: Active Oversight of Market Automation by SEC and CFTC Needed, GAO/IMTEC-91-21, May 10, 1991. Stock Market Automation: Exchanges Have Increased Systems’ Capacities Since the 1987 Market Crash, GAO/IMTEC-91-37, May 10, 1991.

Summary: Capacity problems and other disruptions at the securities and options exchanges have caused processing delays within the U.S. securities markets in recent years. These exchanges and clearing organizations have also been concerned about unwarranted access by hackers and other unauthorized users. To address these issues, the securities and Exchange Commission (SEC) created its automation review policy program in 1989. The program calls for the exchanges and clearing organizations that act as self-regulatory organizations to voluntarily follow SEC guidance and submit to oversight of their information systems. The program includes two key policy statements that provide voluntary guidelines to these organizations, periodic on-site inspections by SEC staff, and independent reviews of systems by internal auditors or external organizations. In addition, self-regulatory organizations are expected to provide SEC with reports of system outages and notices of system modifications. This report reviews SEC's effectiveness in its oversight roles. GAO found that the program reasonably ensures that self-regulatory organizations address capacity, security, and other information systems issues. However, SEC could improve its program oversight by consolidating criteria used by program staff into a comprehensive guide. Overall, SEC's inspections addressed the key areas of program guidance and often contained substantive recommendations designed to improve the organizations' procedures.