Established in 1939, NASD regulates (1) over-the-counter securities trading (that is, trading that does not occur on the floor of a stock exchange) and (2) all brokers and dealers conducting securities business with the public. NASD owns and operates NASDAQ, a computerized communication system that provides quotation information on and facilitates trade executions for 5,700 securities. Implemented in 1971, NASDAQ links a nationwide network of about 500 brokerage firms, called market makers. These firms maintain inventories of securities which they buy from or sell to investors. During 1993, 66.5 billion shares of stock—totaling $1.35 trillion—were traded in this market. These volumes represent 43.6 percent of the total shares traded on U.S. stock markets, or about 32.6 percent of the total dollar value traded. NASD’s headquarters is located in Washington, D.C. The Association’s automated quotation and trading systems are located in and operated from its primary data processing facility in Trumbull, Connecticut. Its backup systems are located at NASD’s data processing facility in Rockville, Maryland, which also houses automated administrative systems such as payroll, personnel, and market surveillance. The U.S. securities markets are primarily governed by self-regulatory organizations, such as NASD, which, in turn, are overseen by SEC. While self-regulatory organizations are responsible for maintaining smooth and dependable operations with their automated systems, SEC is responsible for overseeing overall market operations, including systems used to support such operations. To determine the nature and causes of the outages and to better learn how NASD develops, tests, and operates systems, we interviewed NASD senior officials, including the Executive Vice President and Chief Technology Officer, the Senior Vice President for Production Services, the Vice President for Computer Operations, the Director for Quality Assurance, and the Director for Performance Measurement. In addition, these officials provided us with a minute-by-minute chronology of events as they occurred on July 14 and 15 and August 1. We also obtained and reviewed NASD’s policies and procedures for quality assurance and stress testing. Finally, we visited the primary data processing facility in Trumbull, Connecticut, to observe the systems and the controls used to safeguard them. We determined the impact of the recent NASD system outages on three categories of market participants—market makers, derivatives markets, and individual investors. For the market makers, we used a structured questionnaire to collect information from the top 12 market makers—Merrill Lynch; Smith Barney Shearson; Herzog, Heine, Geduld; Mayer & Schweitzer; Troster Singer Corporation; Goldman, Sachs & Company; Lehman Brothers; Morgan Stanley & Company; Bear, Stearns & Company; The First Boston Corporation; PaineWebber; and Sherwood Securities Corporation. Together, these 12 represent over 50 percent of NASD’s total trading volume. Our questionnaire included inquiries on how the outages impacted the market makers’ ability to obtain information and execute trades, as well as questions on the impact of the outages on confidence in NASD systems and the market, and on future participation in this stock market. We met with six of the market makers and mailed the questionnaire to the others. For the derivatives markets, we interviewed NASD officials including the Chief Operating Officer, Chief Technology Officer, and Chief Economist, as well as senior officials from the Chicago Board Options Exchange and the Chicago Mercantile Exchange. Finally, to assess the impact of the outages on individual investors, we interviewed market makers, senior NASD officials, and officials from the National Association of Investors Corporation and the American Association of Individual Investors—representing about 440,000 members combined. In assessing the adequacy of NASD’s plans to respond to contingencies and disasters, we conducted a walk-through of NASD’s backup facility in Rockville, Maryland. We also interviewed those NASD officials responsible for preparing, maintaining, and testing the Association’s contingency and disaster recovery plan. In addition, we reviewed NASD’s contingency and disaster recovery plan and processes, including examinations of the 1993-94 test schedules and results. To determine how well NASD oversees its automated market systems and facilities, we examined the role of NASD’s Internal Review office and discussed the work it has done in the past and is planning to undertake in the future. We interviewed the Director of Internal Review and reviewed position descriptions for the auditors who review NASD’s systems and computer facilities. We also obtained and examined Internal Review’s audit plan detailing the scope of work to be performed through April 1995. Finally, we interviewed systems managers at both the primary and backup computing facilities to determine the extent of their involvement with Internal Review. To determine how SEC generally oversees markets’ preparedness for contingencies and disasters, we interviewed senior officials in SEC’s Division of Market Regulation and obtained a chronology of events and supporting documentation regarding the Commission’s role and response to the NASD outages experienced during July 14 and 15 and August 1. In addition, we reviewed SEC’s automation review policy, the Commission’s report of its most recent inspection at NASD which occurred in 1992, and the audit report of the most recent review of automated NASD systems conducted by an independent public accountant in 1992. We conducted our review from August through October 1994, in accordance with generally accepted government auditing standards. We discussed the contents of this report with senior officials from NASD and SEC’s Division of Market Regulation and incorporated their comments where appropriate. The system outages experienced by NASD in July and August were due to malfunctioning software and hardware. Specifically, on July 14, new communications software being implemented as part of NASD’s efforts to upgrade its system did not operate as intended and caused the system to fail. When NASD staff restarted the system, the communications software experienced additional problems. Consequently, NASD shut down the system and reconfigured it to use the old communications software. NASD operated its system this way for the remainder of the trading day with only minor problems. In total, the outages caused the system to be down for about 14 minutes. According to NASD systems officials, they corrected one problem with the new communications software that evening. These officials also told us that they disabled a function of the new communications software that was causing a second problem that could not be fixed immediately, and reconfigured the system to use the old software for this function. On July 15, before the normal market opening (9:30 a.m. EST), the system’s response time slowed to unacceptable levels while processing routine tasks, prompting NASD to delay opening the market. At about 11:00 a.m., NASD diagnosed the problem as a faulty hardware component (used to manage disk access and storage devices), took it off-line, and opened the market at 11:55 a.m., approximately 2-1/2 hours late. According to NASD systems officials, it took them about 1-1/2 hours after normal opening time to locate this problem because their focus was on the new software, while the problem was actually caused by an intermittent hardware failure. After the market opened, NASD then opened a market function for exchange-listed securities—the Consolidated Quotation Service. Because this function had been closed for the morning, the transaction rate surged. Software controls in the communications software that were designed to manage (limit) the number of transactions the system would accept did not fully protect the system from this surge and transaction backlogs began to build, resulting in the system’s response time increasing. NASD responded by turning off selected automated services to reduce the processing workload. The system functioned with only minor problems for the rest of the trading day. The outage on August 1, which lasted 34 minutes, was caused by a faulty circuit board in NASD’s backup electrical system. The backup electrical system, which consists of commercial-grade batteries and generators owned and operated by NASD, was activated because the power from the local utility company dipped to an unacceptable level. As designed, the backup battery system operated until the facility could be switched over to the backup generators; however, during the switchover, the circuit board responsible for monitoring the conversion malfunctioned. This resulted in a total loss of power to the data center. At this point, NASD switched operations to the backup data processing facility in Maryland and continued operations for the rest of the trading day. To address this problem, NASD (1) replaced the circuit board, (2) had the contractor who supplied the backup electrical system determine why the board malfunctioned, (3) is considering purchasing a second backup electrical system of batteries, generators, and circuit board to supplement its existing backup electrical system, and (4) hired a contractor to assess other single points of failure in the backup electrical system. The malfunctions that caused the July outages might have been avoided had NASD more thoroughly tested its software. Testing systems to assess their ability to operate as intended and process unusually large workloads—commonly referred to as quality assurance and stress testing, respectively—helps identify and correct system weaknesses before they cause data processing disruptions in a live operating environment. NASD performs quality assurance and stress tests on its systems. For instance, quality assurance personnel test software to determine whether it meets established business requirements. However, NASD’s quality assurance testing was limited in scope. Specifically, quality assurance did not (1) test all requirements and (2) verify that the system would not operate in inappropriate ways. For example, one requirement of NASD’s communications software was to limit the total number of transactions the system could accept into its processing queue; however, quality assurance did not test this software function.