SEC introduced its ARP program in 1989 because of capacity and other problems in the exchanges’ and clearing organizations’ information systems. The program resulted from SEC’s November 1989 policy statement that noted that many exchanges and other organizations experienced problems in their systems during the high trading volumes that occurred in October 1987 and again in October 1989. This policy statement also cited disasters, such as fires or earthquakes, that required exchanges to implement their contingency planning procedures. Since the ARP program was created, exchanges, clearing organizations, and the systems that link the stock and options markets have continued to periodically experience capacity-related problems or other disruptions. Under the ARP program, SEC called on the SROs to ensure that the information technology systems they use to conduct market operations have adequate processing capacity for current and future estimated trading volumes. In addition, SEC sought assurances that SROs were taking steps to assess the risk to their operations from internal and external threats, such as unauthorized use, computer vandalism, or computer viruses. The first ARP policy statement called on the SROs to establish capacity planning procedures to estimate current and future information system capacity needs and to periodically conduct capacity stress tests. In addition, the statement recommended that the SROs have assessments performed of their systems capacity and their vulnerability to physical threat. In a second policy statement issued in May 1991, SEC provided more specific guidelines to the SROS that identified five primary areas it expected the SROs to have reviewed, including the general controls and security relating to computer operations and facilities, telecommunications, systems development, capacity planning and testing, and contingency planning. The ARP program is administered by staff in SEC’s Office of Technology and Enforcement within the Division of Market Regulation. To determine the adequacy and completeness of the criteria SEC uses to conduct capacity and security oversight, we compared the criteria with guidance issued by other financial regulators and organizations that have developed standards for auditing information systems, including the information security manual we developed for use by federal agencies.We also used a list of criteria we developed based on the procedures recommended in a publication written by experts in the field of capacity planning for information systems and on the findings from our prior reports or testimonies that address automation issues in the securities markets. In addition, we reviewed SEC inspection work plans that the ARP staff uses to conduct on-site inspections and held discussions with SEC staff on the criteria that they use to conduct their oversight. To determine the scope and frequency of the ARP on-site inspections, we obtained from SEC a list of on-site inspections conducted between 1995 and June 2001 of 27 SROs and electronic communication networks (ECN). We reviewed a total of 11 SEC ARP inspection reports that addressed capacity planning or security-related issues, including the written reports and supporting work papers on ARP inspections of 7 SROs which included the most active exchanges as well as some of the smaller exchanges, and just the written reports for 4 other SROs. We discussed our observations of these reviews with ARP staff. To determine the scope and frequency of the independent reviews, we examined certain recent audit reports and a summary of audit reports prepared by SEC ARP staff. We also discussed the results of our assessment of these reports with SEC staff. Specifically, we reviewed copies or summaries of 37 reviews done by SRO internal audit staff for the 3 largest SROs in 2000. In addition, we examined seven independent reviews of five SROs performed by external organizations that were included in the supporting work papers of the SEC inspections we reviewed. To address how the voluntary nature of the ARP programs affects SEC oversight capabilities, we reviewed various documents prepared by the ARP staff. The documents included analyses of SRO systems, on-site inspection reports, a printout of SEC staff’s database of the status of recommendations made during inspections, and oversight work plans. We conducted this work in Washington, D.C., from November 2000 to June 2001 in accordance with generally accepted government auditing standards. To plan and conduct inspections and other oversight activities, the ARP program uses criteria from a variety of sources that address aspects of capacity planning, security, and other information system issues. The second ARP policy statement discussed five primary areas that SEC expected that SROs would address regarding their information systems. Using these areas, the ARP staff work with SROs to develop a checklist as an initial guide for use by SEC staff in conducting their on-site inspections. This checklist was also provided to the SROs in 1991 for use as part of the independent reviews of their systems. The ARP program staff told us that they regularly update the inspection checklist by consulting professional standards and guidance relating to information systems established by other regulatory, audit, or industry bodies. Figure 1 shows how SEC staff and the SROs use the various sets of guidance. In our review of SEC on-site inspection work papers, we observed instances in which ARP staff used the steps from this checklist to plan certain segments of work they would perform at individual SROs. SEC officials said they also expect the areas examined during inspections of SRO systems to be based on the ARP policy statements as well as on industry standards for conducting systems audits and the reviewers’ professional judgment. In the work papers we reviewed, we found examples of individualized checklists that ARP staff had created that incorporated steps from the 1991 checklist and other sources for use in particular inspections of SROs. SEC staff said that checklists created for past inspections are also used to plan subsequent inspections. ARP staff also described performing frequent Internet searches to monitor the latest information on standards and issues from various auditing and information system organizations, such as the Information Systems Audit and Control Association. At a minimum, SEC officials said that they expect their staff to perform these searches before each inspection. In their view, these additional information sources provide up-to-date, comprehensive criteria for assessing capacity planning, security, and other relevant systems issues. ARP program staff also explained that they use their own knowledge and experience to plan the inspections and the ongoing monitoring they conduct. They said that they also added review steps to their inspections to address any current challenges facing the SROs that would affect information systems. For example, they added steps to inspections recently to address the industry’s transition to decimal pricing as well as for the Year 2000 date change. They also added steps to inspections of individual SROs when new systems are being implemented or outages occur. Because of continuous change in technology, SEC staff need to refer to up- to-date criteria and standards to conduct their oversight. However, they lack a consolidated guide for their staff. The 1991 inspection checklist that the ARP staff continuously updates to serve as criteria for their inspections does not address some developments in the markets and advances in information technology. For example, SEC’s checklist addresses some security issues but does not include steps relating to intrusion detection. The 1991 checklist also does not address the increased risk of unauthorized access faced by SROs with information systems connected to the Internet. Although SEC officials explained that the SROs do not generally operate critical systems that use the Internet, some are using it to transmit information for less important systems, and others are considering or are already developing Internet-based systems. SEC’s 1991 inspection checklist is also missing some elements relating to capacity planning. For example, the checklist did not specifically address certain issues relating to volume forecasts used in capacity planning in which some SROs have had problems. In 2000, the National Association of Securities Dealers’ (NASD) transition to decimal pricing was delayed because the system NASD planned to use for decimal trading lacked sufficient capacity. A review by an external organization later found that NASD’s volume forecasts had not adequately accounted for the increasing volatility in its trading and processing volumes. Although SEC ARP staff had identified deficiencies and made recommendations to NASD to improve its capacity planning processes, we did not find volatility of trading volumes specifically addressed in SEC’s 1991 checklist or the other work plans that SEC staff prepared for the inspections we reviewed. Because the ARP program does not have a consolidated guide for its staff, the burden of maintaining consistent quality in ARP oversight falls primarily on the most experienced ARP staff. Both SEC and other regulators frequently use comprehensive guides to ensure that the rule- compliance reviews their staff perform are consistent. For example, staff in SEC’s Office of Compliance Inspections and Examinations use examination modules that consolidate the procedures for the reviews they expect their staff to perform consistently at the broker-dealers and other entities they review. However, without a similar consolidated guide, the ARP program staff must make continual efforts to consult numerous sources to supplement the areas not contained in SEC’s 1991 ARP materials. Conducting quality reviews of the various SROs also requires ARP program staff to have broad knowledge of relevant issues and to be aware of how market developments could affect the systems at each SRO. We found that the various work plans, risk analyses, and other documents prepared by the ARP program staff were generally thorough and addressed issues adequately. However, the level of detail and extent of documentation varied across staff members. Although the quality of SEC’s oversight depends heavily on individual staff, the ARP program has experienced considerable staff turnover. SEC officials said that the ARP program staff has experienced turnover rates of almost 30 to 40 percent in some years. The officials said that finding replacements is always difficult, as the salaries SEC offers for people with information system skills are not competitive with the private sector. As of June 15, 2001, 4 of the 10 ARP program staff had 2 years or less of experience, including 2 staff who had just joined the program. SEC officials said that only experienced staff prepare updated workplans and lead on-site inspections. However, the lack of a consolidated written guide could lead to inconsistency in planning and conducting inspections, given the high turnover rate of ARP staff. We found that, for the most part, SEC’s on-site inspections addressed key capacity and security issues. However, resource limitations have prevented SEC from conducting inspections as frequently as their staff would prefer.