To describe the history and nature of the NAOMS project, we researched, reviewed, and analyzed related material posted on several NASA Web sites and provided to us directly by NASA and its contractor for NAOMS. We reviewed relevant documents on the House of Representatives’ Committee on Science and Technology Web site. We examined relevant documents produced by the Battelle Memorial Institute (Battelle), National Academies, and others as well as information produced for the National Research Council. In addition, we reviewed a number of relevant reports, articles, correspondence, and fact sheets on the NAOMS project and air safety. Many of the publicly available materials we reviewed are named in the bibliography at the end of this report. To analyze the NAOMS air carrier pilot survey’s planning, design, and implementation (including pretest, interview, and data collection methods); interviewer training; development of survey questions, including which safety events to include in the survey; and sampling, we interviewed officials from NASA, the Federal Aviation Administration (FAA), and the National Transportation Safety Board (NTSB) and NAOMS project staff. We also reviewed relevant documents. We discussed the survey with NAOMS team members to obtain their recollections of the work, particularly regarding limitations, gaps, and inconsistencies in the documentation. GAO internal experts in survey research reviewed the Office of Management and Budget’s (OMB) Standards and Guidelines for Statistical Surveys and derived a number of survey research principles relevant to assessing the NAOMS survey. We compared the NAOMS survey’s design and implementation with these principles. Although OMB’s standards as they are used today were not final until 2006, the vast majority of OMB’s guidelines represent long-established, generally accepted professional survey practices that preceded the 2006 standards by several decades. We also examined the potential risk for survey error— that is, “errors inherent in the methodology which inhibit the researchers from obtaining their goals in using surveys” or “deviations of obtained survey results from those that are true reflections of the population.” Survey error could result from issues related to sampling (including noncoverage of the target population and problems with the sampling frame), measurement error, data processing errors, and nonresponse. We asked three external experts to review and assess the NAOMS air carrier pilot survey’s design and implementation as well as considerations for analysis of collected data. These external reviews and assessments were conducted independently of our own review activities. We selected the experts for their overall knowledge and experience in survey research methodology and, specifically, for their expertise in measurement (particularly the aspects of memory and recall), survey administration and management, and sampling and estimation. The experts included Robert F. Belli, Professor, Department of Psychology, University of Nebraska, Lincoln, Nebraska; Chester Bowie, Senior Vice President and Director, Economics, Labor, and Population Studies, National Opinion Research Center, Bethesda, Maryland; and Steve Heeringa, Senior Research Scientist at the Survey Research Center and Director of the Statistical Design Group at the Institute for Social Research, University of Michigan, Ann Arbor, Michigan. To determine what steps or other considerations might improve the quality and usefulness of a survey like NAOMS if one were to be implemented in the future, we identified and described methodological deviations that we found from GAO’s guidance and OMB’s standards. We also obtained the views of internal and external experts on how limitations caused by such deviations might be overcome. We assessed the potential or known effects of design or implementation limitations we identified. We focused our review on the most extensively developed part of the NAOMS effort, the air carrier pilot survey. We discuss the general aviation study as it relates to the air carrier survey and overall project evolution, but we do not focus on its development or implementation. We attempted to identify any problems that might have prevented the NAOMS survey data from producing meaningful results, and that might not materially affect the survey results but could result from accepting the reasonable risk and trade-offs inherent in any survey research project. We note that limitations may not necessarily be weaknesses. We conducted our work from March 2008 to March 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The NAOMS project was conceived and designed in 1997 to provide broad, long-term measures on trends and to measure the effect of new technologies and policies on aviation safety. Following the 1996 formation of the White House Commission on Aviation Safety and Security, and the commission’s 1997 report to the President committing the government and industry to “establish a national goal to reduce the aviation fatal accident rate by a factor of five within ten years and conduct safety research to support that goal,” NASA worked with FAA and NTSB to set up the Aviation Safety Investment Strategy Team within NASA. This team organized workshops, examined options, and recommended a strategy for improving aviation safety and security. One of its recommendations led to NASA’s Aviation System Monitoring and Modeling (ASMM) project, a program to identify existing accident precursors in the aviation system and to forecast and identify potential safety issues to guide the development of safety technology. ASMM, within NASA’s Aviation Safety and Security Program, was to provide systemwide analytic tools for identifying and correcting the predisposing conditions of accidents and to provide methodologies, computational tools, and infrastructure to help experts make the best possible decisions. ASMM was expected to accomplish this by, among other things: intramural monitoring, providing air carriers and air traffic control facilities with tools for monitoring their own performance and safety within their own organizations, and extramural monitoring, providing a comprehensive, systemwide, statistically sound survey mechanism for monitoring the performance and safety of the overall National Air Transportation System by seeking the perspectives of flight crews, air traffic controllers, cabin crews, mechanics, and other frontline operators (NAOMS was developed as the primary mechanism for collecting this information). Agencies, airlines, and other private organizations had realized that quantitative and anecdotal information they had been collecting could not be used to calculate statistically reliable risk levels. The project team identified eight major aviation safety data sources that were available when NAOMS was created. For example, flight operational quality assurance data could have helped in deriving statistically reliable estimates from digital measurements of flight parameters, but these data do not cover all airlines or include information on human cognition or affect. Another dataset was from the Aviation Safety Reporting System (ASRS), which for 30 years had been successfully collecting information from pilots, controllers, mechanics, and other operating personnel about human behavior that resulted in unsafe occurrences or hazardous situations. However, because ASRS reports are submitted voluntarily, the resulting data cannot be used to generate reliable rate estimates. Under ASRS, pilots describe events briefly by mail or on NASA’s ASRS Web site. NASA reviews each report and enters detailed information about the events into an anonymous database that it maintains. According to the ASRS Director, the system is subject to volatility in reporting, as in 2006, when the data witnessed a spike in reports of wrong runway use following a fatal accident in Kentucky, where pilots turned onto a taxiway that was too short for their aircraft to attain lift-off speed. Also, ASRS is not statistically generalizable. Although it does not constrain the types of events that can be reported, ASRS reporting is voluntary and unlikely to cover the universe of safety events, and it cannot be used to calculate trends. To complement this system and other safety databases, the NAOMS project was to interview a statistical sample of professionals participating in the air transportation system, including pilots, about their experiences. Data from the interviews were to enable statistically reliable measurements of rates and rate trends for a wide array of types of safety events, such as the professionals experiencing fire in the cargo or passenger compartment or encountering severe turbulence in clear air, collisions with birds, airframe icing, and total engine failure. As the project evolved, the NAOMS researchers decided to deemphasize NAOMS’s potential to calculate rates in isolation, instead highlighting the project’s primary capability to identify trends worthy of investigation, thereby complementing other data sources. The premise of the NAOMS project was that aviation personnel were the best source of information on day-to- day, safety-related events. In measuring the occurrence of safety incidents that might increase the risk of an accident, rather than accidents themselves, the project would serve a monitoring role rather than an investigative role. Instead of directly informing policy interventions, NASA expected that trends seen in the NAOMS data would point aviation safety experts toward what to examine in other data systems. However, to date, the accuracy of rate and trend estimates based on NAOMS data has not been established. NASA appointed two researchers with aviation safety experience to lead a project team in developing surveys for NAOMS as a part of ASMM. The researchers contracted with Battelle to administer the project. Battelle, in turn, subcontracted with experts in survey methodology and aviation safety to help with questionnaire construction and project execution. “1) plausibility and understandability of NAOMS statistics (e.g., reasonable and reliable representation of the relative frequencies with which unwanted events occur), “2) stability and interpretability of NAOMS statistical trends, “3) sensitivity to industry concerns about data misuse, and “4) timely and appropriate disclosures of NAOMS findings.” A primary objective of NAOMS was to demonstrate that surveys of personnel from all aspects of the aviation community could be cost- effectively implemented to help develop a full and reliable view of the NAS. NASA also sought to find a permanent “home” for the surveys, having planned to develop “scientific methodologies to maximize the useful information and minimize the cost, but not . . .