In September and October 2001, at least seven envelopes containing significant quantities of B. anthracis spores were mailed through the U.S. postal system to two senators at their congressional offices in the District of Columbia and to media organizations in New York City and Boca Raton, Florida. According to the FBI, the evidence supports the conclusion that the mail attacks occurred on two separate occasions. The two letters of the first attack were postmarked September 18, 2001, and sent to NBC News and the New York Post, both in New York City. Three weeks later, two letters postmarked October 9, 2001, were mailed to two senators—Thomas Daschle and Patrick Leahy—at their Washington, D.C., offices. Other letters were sent to ABC, CBS, and American Media, Inc. Hard evidence of the attacks surfaced on October 3, 2001, when Robert Stevens, an American Media Inc. employee who worked in Boca Raton, Florida, was diagnosed as having contracted inhalational anthrax, from which he later died. However, because a contaminated envelope or package was not recovered in Florida, the agencies could not initially establish how the B. anthracis spores were delivered. According to the Postal Service, the combination of the Florida incident and the opening of the letter to Senator Tom Daschle on October 15 established the link to the U.S. mail system. At least 22 victims contracted anthrax as a result of the mailings. Eleven individuals developed inhalational anthrax, and another 11 developed cutaneous infections. Five of the inhalational anthrax victims died from their infections. The attack highlighted the need for enhanced capabilities for full forensic exploitation and interpretation of microbial evidence from acts of bioterrorism. Ideally, forensic evidence obtained in an investigation is sufficient to support conclusions about the culpability of a group, an individual, or the source of material used in such an act. Forensic evidence is used to support conclusions by classifying evidence into one of several categories that distinguish possible sources from one another. While classification does not unequivocally demonstrate a connection with an individual or a single source, it can be used to reduce the number of possible sources and thus can provide important leads in an investigation. The development and application of microbial forensics was essential to the FBI’s scientific investigation, which relied heavily on genetics and comparative genomics to classify the spore materials used in the attack, reduce the number of possible sources and suspects, and provide investigative leads. In fact, according to the NAS, this investigation accelerated the development of the then nascent field of microbial forensics. The FBI’s investigation, assisted by government, university, and commercial laboratories, was an effort to develop the physical, chemical, genetic, and forensic profiles of the anthrax spores in the letters and envelopes used in the attacks. The investigation employed myriad traditional and novel investigative and scientific methods. The scientific methods involved efforts to develop the physical, chemical, genetic, and forensic profiles of the anthrax spores and letters and envelopes used in the attacks so as to identify the source of the spores. The FBI faced many difficult and complex scientific challenges over the course of this investigation, according to the NAS. New microbial forensic methods were developed and implemented over several years, and some of them provided valuable evidence and significant leads in the case. For example, according to the FBI, new methods to determine the source of the growth media for the mailed spores were inconclusive while the use of the genetic mutations provided an investigative lead. By October 2001, CDC had identified the microorganism used in the attack as Bacillus anthracis (B. anthracis). This was a key step in the classification of the microorganism used in the attack letters and was one of the first scientific findings that allowed the FBI to begin to reduce the number of possible sources of the spores. B. anthracis is a gram positive, rod-shaped bacterium that causes the disease anthrax. It is a member of the larger genus Bacillus that includes other commonly found species, such as B. cereus, B. subtilis, and B. thuringiensis. B. anthracis, a species of Bacillus that can be found on all continents except Antarctica, typically shows little genetic variation among isolates. However, during the investigation scientific methods were being developed that allowed scientists to find some genetic differences among natural isolates of B. anthracis. Applying these methods allowed the FBI to refine the classification of the spores used in the attack and further reduce the number of possible sources of the spores. Even in the most homogeneous species, some differences are usual in genome sequences among populations. Although few in number, these differences are sufficient to characterize subgroups, or “strains.” in the attack letters. In fact, this scientific evidence allowed the FBI to focus its investigation on the limited number of laboratories that had had access to the Ames strain before the attacks (see figure 1). While classifying the spore material as the Ames strain was instrumental in reducing the number of possible sources, it was not sufficient by itself to definitively identify the source of the material used in the attack as a single laboratory, flask, or person. The FBI then sought to identify additional characteristics of the spores used in the attack that could further discriminate between possible sources of the Ames strain. Scientists from the Department of Defense (DOD) tested samples of the spore materials found in the letters and identified several morphological variants (or morphs).the spores from the letters over an extended period observed that a small percentage of the colonies differed in texture, color, and growth patterns A laboratory technician who had grown (cultured) from those typical for the Ames strain of B. anthracis, referred to in figure 2 as the “wild type.” In an effort to identify the source of the letters, investigators and FBI scientists began to evaluate whether they could first identify and characterize these morphs genetically and then determine whether any of them were present in the repository of Ames samples. This involved genome sequencing to identify whether specific deoxyribonucleic acid (DNA) sequences underlay the morphs. Eventually, as shown in figure 2, the morphs were associated with several types of genetic mutations: duplications, single nucleotide polymorphisms (SNP), and deletions, referred to as INDELS. Afterward, over several years, outside contractors’ laboratories developed and validated several genetic tests to analyze the B. anthracis samples for the presence of certain genetic mutations. Specifically, the testing revealed the presence or absence in a sample of a specific DNA sequence (that is, the genetic mutation) associated with a given morph. The FBI contractors generally referred to the tests they developed as A1, A3, D, and E. Commonwealth Biotechnologies (CBI) developed the two A tests (A1 and A3), which targeted two different DNA sequences; and the Institute for Genomic Research (TIGR) developed the E test, targeting another DNA sequence. However, unlike the others, the Illinois Institute of Technology Research Institute (IITRI) and Midwest Research Institute (MRI) both developed a test targeting the same DNA sequence. For clarity, we refer to the IITRI-developed test as D-1 and the MRI- developed tests as D-2. Genetic test A1: detects the presence of a specific duplicated DNA sequence associated with morph A; Genetic test A3: detects a different duplicated DNA sequence from that targeted by A1 but also associated with morph A; Genetic test D-1: detects the presence of a specific deleted DNA sequence associated with morph D; Genetic test D-2: detects the presence of the same deleted DNA sequence associated with morph D as that targeted by the D-1 test; and Genetic test E: detects a deleted DNA sequence associated with morph E. In 2002, the FBI began collecting samples from laboratories in possession of the Ames strain to compare them with the material used in the attack. A grand jury issued subpoenas to 16 domestic laboratories and the FBI requested submissions from 3 foreign laboratories that investigators had determined possessed the Ames strain. The subpoenas required each laboratory to identify and submit two representative samples from each distinct stock of the Ames strain it held. The subpoena included instructions to the laboratories on how to identify, select, and submit samples to the FBI. Laboratories were required to ship sample submissions to DOD scientists at USAMRIID for preparation and entry into the FBI repository of Ames samples. In addition to the samples submitted in response to the subpoena, searches were conducted at three domestic laboratories to ensure that samples were taken from each stock of Ames strain in those facilities. The FBI assembled a repository of 1,070 Ames strain samples, of which From 2004 through 2007, each of the 1,059 viable 1,059 were viable. repository submissions was compared to the evidentiary material using the five genetic tests (see figure 3). The results of the genetic testing indicated that only 8 of the 1,059 FBI repository Ames samples tested positive for the presence of the four genetic mutations originally found in the anthrax letter evidence. Using submission records, investigators concluded that these 8 samples were derived from a single source—a flask identified as RMR-1029. According to the FBI, this information constituted a groundbreaking lead in the development in the investigation. It allowed the investigators to reduce drastically the number of possible suspects, because only very few individuals had ever had access to this specific flask. Armed with this new information obtained from the scientific evidence, the task force focused its investigation on researchers who had had access to the laboratory at USAMRIID where RMR-1029 was stored. In 2008, the FBI sought to conduct statistical analyses in order to determine (1) the probative value of genetic markers found in a sample, and (2) possible inferences regarding the relationships of similar samples. A contractor submitted the Final Statistical Analyses Report in October 2008. In February 2010, the FBI closed the case concluding that a scientist at USAMRIID had perpetrated the attack alone. Neither the case nor the totality of the evidence, including the scientific evidence that provided the FBI with valuable leads, was brought to trial in a court of law. The alleged perpetrator of the attack died on July 29, 2008, from an overdose of over- the-counter medication. At the start of the investigation no standards or guidelines existed for verifying and validating microbial forensic methods, including the polymerase chain reaction (PCR)-based tests that were eventually used to identify the genetic mutations in the repository samples. The first contractor, Commonwealth Biotechnologies (CBI), an established forensics laboratory, had begun developing the A1 and A3 genetic tests in 2002. A CBI official stated that it had relied upon the National Institute of Standards (NIST) guidance on the validation of methods for detecting human DNA, and also on the DNA Advisory Board standard for forensics. For the remaining three contractors, the Illinois Institute of Technology Research Institute (IITRI), Midwest Research Institute (MRI), and The Institute for Genomic Research (TIGR), the FBI provided “Quality Assurance Guidelines for Laboratories Performing Microbial Forensic Work”—guidelines that the FBI’s Scientific Working Group on Microbial Genetics and Forensics (SWGMGF) developed and published in October 2003. These guidelines defined validation as a process by which a test procedure is evaluated to determine its efficacy and reliability for analysis.confidence in the ability of the test (the measurement tool) to accurately identify the properties of interest in samples that are to be analyzed. Verifying and validating a test method provides a level of Verification, confirms by objective evidence, from laboratory experiments, that the given test meets the user’s specific requirements, such as criteria for accuracy. If the verification testing were not to produce consistent results, then the scientist or the laboratory would have to return to the optimization phase to further refine the method and materials and then revise the test’s standard operating procedure (SOP) accordingly. Verification of the acceptance criteria must include repeated testing to account for measurement uncertainty, and confidence in performance statistics should be reported. Depending on the intended use of a test, sensitivity, specificity, limit of detection (LOD), reproducibility, bias, and precision may all be measures of performance (performance statistics) that should be evaluated. The type of test (qualitative, quantitative, or semi-quantitative) may also determine which of these performance parameters is to be evaluated. The testing protocol and materials, including quantities that optimize test performance are recorded in a test’s SOP. Validation confirms by examination, from laboratory experiments, and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled. Successful validation offers some assurance that a given genetic test is sufficiently robust to provide reproducible results, regardless of the practitioner, agency, contractor, or laboratory applying it to a sample. Validation is frequently used to connote confidence, but it may also be thought of as defining the limitations of a method. Studies are conducted that enable the estimation of the limits of the procedures and the measurements of the test. To the extent possible, the validation of a method should mimic “real world” conditions. The limits of the method must be known, demonstrated and documented. In essence, validation measures the uncertainty in the test output. In 2007 the FBI convened a team of scientists to review selected scientific methods used in the case. Referred to as the AMX Red Team, it was asked to assess whether the science used was sound and to consider what additional tests might be performed to benefit the investigation. The team, finding no shortfalls or deficiencies in the basic methodologies it reviewed, concluded that the “genetic signatures correlating with specific morphs were valid tools for eliminating those repository samples not closely related to the spores used in the attack.” However, the team also stated that the extent of research and development of the genetic tests at the date of its review was insufficient to determine whether the presence or absence of one or several of the morphs in a sample was associated with the evidence, was merely characteristic of normal culture practices, or possibly was affected by the genetic tests’ sensitivity of detection. The team recommended additional studies to characterize the genetic markers as a function of growth conditions, including the influence of growth time, growth media, and temperature. It also recommended additional evaluation of the sensitivity of detection of each genetic test to ensure a reliable interpretation of analyses. In 2008, the FBI asked the National Research Council (NRC) of the National Academy of Sciences (NAS) to review the scientific approaches it had used to support its conclusions. In 2011, the NAS issued its report, concluding that “it is not possible to reach a definitive conclusion about the origins of the B. anthracis in the mailing based on the available scientific evidence alone.” Additionally, the report included the following findings related to the development and validation of the genetic tests: “Specific molecular assays were developed for some of the B. anthracis Ames genotypes (those designated A1, A3, D, and E) found in the letters. These assays provided a useful approach for assessing possible relationships among the populations of B. anthracis spores in the letters and samples subsequently collected for the FBIR. . . . However, more could have been done to determine the performance characteristics of these genetic tests. In addition, the assays did not measure the relative abundance of the variant morphotype mutations, which might have been valuable and could be important in future investigations. . . .” “The development and validation of the variant morphotype mutation assays took a long time and slowed the investigation. The committee recognizes that the genomic science used to analyze the forensic markers identified in the colony morphotypes was a large-scale endeavor and required the application of emerging science and technology. Although the committee lauds and supports the effort dedicated to the development of well- validated assays and procedures, looking toward the future, these processes need to be more efficient.” Additionally, the NAS report included the following findings related to the statistical approach taken to quantify the significance of finding the genetic markers in a small number of repository samples: “The results of the genetic analyses of the repository samples were consistent with the finding that the spores in the attack letters were derived from RMR-1029, but the analyses did not definitively demonstrate such a relationship.” . . . . . “Some of the mutations identified in the spores of the attack letters and detected in RMR-1029 might have arisen by parallel evolution rather than by derivation from RMR-1029. This possible explanation of genetic similarity between spores in the letters and in RMR-1029 was not rigorously explored during the course of the investigation, further complicating the interpretation of the apparent association between the B. anthracis genotypes discovered in the attack letters and those found in RMR-1029.” We found that the genetic tests used to screen the FBI’s repository of B. anthracis samples demonstrated through the verification and validation testing that they generally met the FBI’s minimum validation requirements. However, the FBI’s validation procedure did not require and the tests did not demonstrate a level of statistical confidence for interpreting the validation results. Also, tests conducted after validation— although not required by the quality assurance guidelines provided to the contractors—yielded valuable information on the performance characteristics of the genetic tests. Therefore, by not having a comprehensive validation approach, or framework, that sets out consistent steps for achieving minimum performance standards, and includes an assessment and measurement of the uncertainty in the test performance (see table 1 for the phases of a validation framework), the FBI cannot have statistical confidence in its validation test results. Knowledge of uncertainty is essential for subsequent statistical analysis that can provide quantitative measures of confidence in conclusions drawn from tests applied to forensic samples. According to DHS, it now validates methods and tests used to support FBI investigations and has an established ISO-accredited program. In our review of scientific literature and agency and industry standards along with guidelines regarding the verification and validation of methods for analyzing both microbial and human DNA, we found that terminology and the extent of verification and validation differed across industries. However, we identified three distinct phases in genetic test development: (1) optimization, (2) verification testing, and (3) validation testing. While the literature and various validation standards and guidelines that we reviewed identify the specific types of tasks for each phase, we found that a clear boundary does not always exist between the first two phases. That is, optimization and verification are sometimes treated as a single continuous process. Further, we found that verification and validation are sometimes used interchangeably to describe the same process. Thus, the process could combine either optimization and verification or optimization and validation. Nevertheless, what is important is that the approach, or framework, to test development generally includes these phases and the associated key tasks, as shown in table 1. The FBI set limited performance requirements for the genetic tests and relied on the contractors’ expertise to determine the processes they would use to develop (i.e., optimize and verify) their tests (see table 1 for types of performance parameters to be evaluated). According to the FBI, it provided minimal direction to the four contractors on how they were to develop their genetic tests in order to allow creative development. It stated that, with a few exceptions, it left the development mostly to the contractors who were experienced in developing tests. However, we found that the contractor’s approaches differed in their (1) use of verification and validation guidelines, (2) steps in conducting optimization and verification testing, and (3) interpretation criteria for results generated by the genetic tests. The FBI required that the genetic tests detect the target mutations in an overwhelming background of bacteria consisting of predominantly wild type B. anthracis Ames, which had been found in the evidentiary material (that is, the letters). Further, the FBI specified that sensitivity was to be demonstrated by the LOD—that is, the lowest concentration level that can be reliably detected for a qualitative and quantitative test. The FBI did not require a specific calculation or value for the LOD. According to the FBI, it was looking for the presence or absence of the morphs (genetic mutations) in the repository samples and the LOD was an important factor. Three contractors developed qualitative tests; the fourth developed a semi-quantitative test. In this regard, the FBI wanted to know the lowest concentration at which the genetic tests could detect the presence of a specific genetic mutation in a sample. Specificity was to be demonstrated by the detection of the target in a sample containing an overwhelming background of predominantly B. anthracis Ames. We found that the contractors evaluated other performance parameters at their discretion (see appendix II). We also found that standards for the verification and validation of microbial forensics methods did not exist at the start of the investigation and were only limited after it had begun. At that time, more was known about verifying and validating human DNA testing methods for forensics than about microbial forensics methods, as reflected in the revised quality assurance guidelines. In addition, we found that the contractors’ disparate experience and the FBI’s minimal instruction to them contributed to the differences in their expectations and approaches. Most of the contractors had worked for other federal agencies whose processes differed and thus their approaches to optimizing and verifying their genetic tests differed. While most of the contractors had developed methods for the federal government, one contractor said that each of its federal sponsors had its own processes for validation and that it followed a particular agency’s processes when working with it. The contractor also stated that its own internal quality assurance guidelines were more stringent than the SWGMGF guidelines for validation. One contractor was a forensics laboratory that was familiar with analyzing human DNA samples and using associated quality assurance standards, including the DNA Advisory Board standards. Another contractor was engaged in genomic research. Finally, the FBI stated that it was more confident after the two A tests were developed; it had required the contractor for the two A tests to subject material from each polymerase chain reaction (PCR) well to genetic sequence analysis, regardless of the result (positive or negative). Further, it stated that after the first four genetic tests were developed, it had been unsure as to whether it wanted to proceed with the last one— the E test. We found that the contractors generally conducted the tasks we identified in table 1 under the first two phases—optimization and verification—to develop the genetic tests and determine their performance, although one did not conduct a verification test. Specifically, CBI conducted an “internal qualification” study that is included in the SWGMGF guidelines. CBI’s qualification study involved multiple experiments using internally blinded samples following an SOP to determine whether (1) the A1 and A3 genetic tests could correctly identify the targeted genetic mutations and (2) the staff involved could be considered qualified to perform the genetic tests. The first appeared to be equivalent to verification testing and the second to proficiency testing. Similarly, both MRI and ITRI conducted internal verification testing and followed it with a qualification test of laboratory personnel. While the distinction between, interpretation of, and expectations for the verification and any qualification testing were not always explicit in the documentation we were provided, we found that they were both intended to precede the validation testing. However, TIGR—the last contractor to develop its genetic test—did not conduct the equivalent of either a verification or a qualification study. The FBI indicated that it believed that the contractors had conducted verification testing but acknowledged that it was possible that one had not been conducted for the last genetic test that was developed. Thus, the verification testing was not consistent for all the tests—with one relying solely on the validation testing to determine whether it met the FBI’s requirements and also was fit for use on the repository samples. We also found that there was no clear rationale for the lack of complementary interpretation criteria for the results generated by the two genetic tests that targeted the D mutation, which proved problematic during the repository screening, after verification and validation had been completed. Each contractor independently developed interpretation criteria for positive, negative and inconclusive results through laboratory experimentation, which when defined became part of its SOP. Initially, for the A1 and A3 tests interpretation criteria were for a positive or negative result only. For the A1 and A3 tests, validation results were reported as the number of correct positive and negative results for the FBI-provided samples, and excluded blind samples for which, at this stage of the investigation, it was not yet known whether they contained the targeted genetic mutations.were reported as the number of correct positive and negative results, detection limit, false positive rate, and inconclusive rate. For the D-1 and D-2 tests and the one E test, results Criteria for an inconclusive result included several types of occurrences that varied by the particular genetic test. The FBI stated that it reviewed the interpretation criteria for each genetic test. However, after the repository screening, disparate interpretation criteria for the D-1 and D-2 genetic tests determined the samples that had usable test results. Ultimately, contradictory interpretations of the D-1 and D-2 test results were a reason for eliminating the results of the D-1 genetic test’s screening of the repository samples; thus, the D-1 results were not part of This issue did the final statistical analyses, according to the NAS report.not surface during the validation of the genetic tests. We found that (1) the genetic tests used to screen the FBI’s repository of B. anthracis samples met the FBI’s validation requirements, (2) the validation tests were not required to and did not demonstrate a level of statistical confidence for interpreting the validation test results, and (3) some information on the sensitivity and specificity of the genetic tests was not characterized until after validation (postvalidation testing). As a result, the performance characteristics of the genetic tests were not fully understood when they were applied to the repository samples and more could have been done to strengthen the quality of the data and ultimately the validation results. The validation test results showed that the genetic tests met the FBI’s requirements in that they were able to detect the targeted genetic mutation when it was present at a low level (that is, at less than 1 percent, in a validation sample containing predominantly B.anthracis Ames), although no measurement of statistical confidence in these results was provided. As shown in table 2, the A3 genetic test had the lowest LOD, at 0.001 percent, while the others ranged from 0.005 percent to 0.01 percent. The validation results led the FBI to determine that there were no false positives for any of the genetic tests, and that the inconclusive rates were 0.12 percent and 0.02 percent for the D-1 and D-2 tests, respectively. During validation, inconclusive rates could not all be computed for all the genetic tests. Since the FBI’s requirements stated that the LOD was to be used as a measure of sensitivity, it was an important measure of the performance of a given genetic test. LOD provides interpretations of results generated by a genetic test with the known limitations of such data, but it is a difficult quantity to estimate reliably. Our review of existing and current guidelines for validation suggests that using an appropriate estimate of LOD does provide a reliable measurement of sensitivity, but LOD estimates, like any performance statistic, should be reported with some measure of confidence. For example, the Environmental Protection Agency defines the method detection limit as the “minimum concentration of substance that can be measured and reported with 99-percent confidence that the analyte concentration is greater than zero.” If test sensitivity is an important performance criterion, then both verification and validation procedures for a genetic test should report LOD, along with a measure of confidence. However, the LODs for the genetic tests the four contractors performed neither required a confidence measure nor determined one by using statistical measurements of confidence. Validation testing should to the extent possible simulate the conditions of the intended use of a given genetic test, using known case samples (see tasks under the third phase in the validation framework in table 1). Calculating uncertainties of measurement is also an important task. All steps in validation testing, such as sample collection, sample preparation, transport, storage and analysis can introduce stochasticity and increase uncertainty in the test results. Most such stochasticities (and others) will also affect the testing of repository samples. These additional uncertainties can be measured and understood using repeated (replicate) experiments including all relevant steps (from collection to analysis) of samples with known concentration levels. By designing a validation study with a sufficient number of replicate samples, the FBI could have quantified the level of statistical confidence in the sensitivity and specificity of the tests. While the SWGMGF guidelines did not require them, tests as part of validation that examined stochastic (random) effects of the process would have made it possible to draw more rigorous conclusions, with measures of confidence, regarding the test results for the repository samples. addition, we found that additional information on the sensitivity and specificity of the PCR-based genetic tests was characterized during postvalidation testing that the FBI’s expert advisers recommended. Our analysis of these post-validation test results suggests that the negative rates of the genetic tests were high for samples that could be expected to contain the genetic mutations when using the sample collection and processing methods as required for the repository samples and that there were stochastic (random) effects in the repository screening process. The SWGDAM guidelines do suggest such tests for PCR-based methods, which is important when samples contain low concentrations of the target to be detected. Sampling fluctuations can occur in PCR-based tests. According to the 2004 SWGDAM guidelines, for PCR-based assays, validation studies must address stochastic effects and sensitivity levels. sequence, growth conditions would vary slightly. However, according to the FBI, the purpose of the additional testing was to determine whether stochastic sampling error had been introduced into the repository preparation process as instructed in the subpoena. Therefore, the postvalidation tests were to determine whether the procedures by which the repository samples were processed could affect the accuracy of the interpretation of the data. The postvalidation tests were conducted in August and September 2007 under conditions that closely mimicked the intended use for each of the genetic tests. According to the FBI, the screening of the repository samples with the genetic tests was about three-fourths complete when this testing took place. Specifically, in the postvalidation tests, the contractors applied their genetic tests to replicate samples derived directly from some of the evidentiary material—including flask RMR-1029. The results revealed that the genetic tests did not always detect the genetic mutations in samples that had been derived directly from the evidence and thus were expected to contain all four mutations—a best-case scenario. Our evaluation of measures of the sensitivity and specificity of the genetic tests revealed differences between the validation and postvalidation test results. Regarding sensitivity, under the assumption that undiluted samples from flask RMR-1029 are positive for all four genetic mutations (supported by the preponderance of genetic and non-genetic data), we can estimate the negative rate as the frequency of negative results in replicate tests of undiluted samples from RMR-1029. Validation testing showed that for those results expected to be positive, no negative results were observed at or above the LOD for any of the genetic tests. However, in the postvalidation testing, the negative rates were generally high. As shown in table 3, the negative rates for the postvalidation tests ranged from 0 percent to 43 percent for the undiluted samples from flask RMR-1029. (Appendix III breaks down the results of the replicate testing for each genetic test.) The NAS report stated that the FBI did not address false negative results and inconclusive results, and it was concerned about the restriction of the statistical analyses to the repository samples that had no inconclusive or variant results.results of only D-2 were used in the FBI’s analysis of the repository screening—that is, the analysis was restricted to the 947 samples that contained no inconclusive or variant results, which resulted in the exclusion of 112 samples from the analysis. Thus, the knowledge about sensitivity and specificity obtained by the replicate testing, as well as ensuring that these two genetic tests’ interpretation criteria were complementary, would have been more useful if it had been completed in the validation process. Of the two genetic tests that targeted the D mutation, the Regarding measures of specificity, the effect of the repeated analysis of the undiluted nonpositive samples during the postvalidation testing showed evidence of a nonzero false positive rate for the D-2 genetic test. As shown in table 4, the 3.3 percent false positive rate for the D-2 genetic test demonstrates the likelihood of a random effect in the postvalidation tests that was not apparent from the validation results. Although not a requirement at the time, repeated testing—such as that conducted postvalidation—would have provided additional information on the performance of the genetic tests. We recognize that neither the FBI nor the SWGMGF guidelines required contractors to conduct replicate tests of case samples to identify the stochastic (or random) effects of the genetic tests when they were used under realistic test conditions to further evaluate the genetic tests’ sensitivity—an important step in validating PCR-based genetic tests. In contrast, the SWGDAM guidelines suggested using experiments to determine the sensitivity of real-time PCR-based tests as a part of validation. Importantly, while the LOD is a critical performance indicator for a genetic test, LOD calculations do not account for the data that PCR-based tests sometimes generate but that The FBI also stated that during the development of the are not typical.genetic tests it was concerned that stochastic effects might be a problem, stating, for example, that it had discussed its concerns with the contractors about evidence growth steps and the possible stochastic effects, that is, in the context of the growth rates of the wild type cells (B. anthracis Ames) versus the morph cells in culturing, among other things. The postvalidation tests were able to estimate valuable performance statistics of the genetic tests and under more realistic testing conditions than the original validation tests. More extensive validation testing could have reduced uncertainties in the testing procedure. For example, the sensitivity of a given genetic test relies on the sampling procedures, the rarity of the targeted genetic mutation in a sample, and other factors that vary by genetic test. Incorporating these types of tests into the validation would have resulted in more information on the uncertainties inherent in the use of the genetic tests and would have been a way to simulate the conditions of their intended use. Future validation efforts would be strengthened by including experiments designed to identify and eliminate likely uncertainties in test performance. The differences we have highlighted regarding the contractors’ approaches to verification and validation indicate that the use of a comprehensive validation framework could help ensure greater consistency. Such a framework would need to specify the defined level of statistical confidence to be calculated for the interpretation of validation results before they are applied to evidentiary samples. Minimally, the statistical confidence achievable in each test should be estimated during validation. The development of such a framework could be facilitated by DHS’s National Bioforensics Analysis Center (NBFAC), which validates tests used to support FBI bioforensic investigations. According to DHS, NBFAC will take steps to ensure that the results it generates will meet Daubert standards for “appropriate validation” and third party review and will thus meet admissibility requirements for evidence in federal court proceedings. NBFAC—an ISO 17025 accredited forensic laboratory—is experienced in working with multiple outside laboratories to verify and validate their methods. It has an established ISO 17025 accredited process. The combination of limited communication among the contractors, varied timing in the validation efforts, uncertainties the FBI faced as the investigation unfolded, and increasing knowledge about the repository samples made it clear, with hindsight, that the contractors’ verification and validation approaches were likely to differ. Thus, in the future, standardizing the approach to verification and validation testing—by the means of a validation framework—would be more efficient, especially in clearly communicating expectations to multiple contractors. In contrast to 2001, DHS’s NBFAC validates assays (or tests) that can be used to support FBI bioforensic attribution investigations. Generally, the NBFAC validation process involves the evaluation of methods transferred from others, such as DOD and academic laboratories, and sometimes the development of a new method. For forensic tests, NBFAC and the FBI are provided with a “validation package” for each test that encompasses data on testing previously conducted during the development stage or before the transfer to the laboratory.have to provide information on the performance parameters (e.g., accuracy, LOD, precision) that they have previously verified. Next, NBFAC conducts its own test, evaluation, and validation of the transferred method. When evidence stemming from the use of validated methods is needed as evidence in court, it must be defensible by meeting evidentiary standards. According to DHS, developers Questions may be raised in court about the standards used for the validation of such methods. Results generated by forensic methods, including microbial forensics, must meet a high standard. According to NBFAC, to ensure that results generated by a validated test will meet Daubert standards for “appropriate validation,” the deliverables from the Bioforensics R&D Program include SOPs for the methodologies and technical and peer-reviewed published reports. Also, quality project performance plans are required of researchers, who must define method performance parameters to provide a baseline for verification and further validation if required by law enforcement. accreditation of tests requires the demonstration of previously described method parameters in NBFAC laboratories with trained staff followed by a third-party review of the supporting data, procedures, equipment, and staff training that supports ISO 17025 accreditation. In this context, a method that has been validated elsewhere and that is transferred would be evaluated to ensure that the NBFAC successfully used the method as intended in the NBFAC laboratory. Performance parameters include accuracy, precision, specificity, selectivity, LOD, limit of quantitation, linearity, ruggedness, and robustness. We identified six characteristics of a statistical framework that would strengthen the significance of microbial forensic evidence. When we compared the FBI’s statistical approach to these six characteristics, we found that three could be improved to strengthen the significance of its evidence for future investigations. That is, the FBI (1) could do more to understand the methods and conditions that give rise to the chosen genetic markers, (2) institute more rigorous controls over sample identification and collection, and (3) include measures of uncertainty when interpreting the results. We found that the FBI has taken some steps to include such expertise in future investigations by building formal forensic statistical expertise both internally and externally. Although not always possible, an important goal of a microbial forensic investigation is to generate meaningful comparative analyses of evidentiary samples and suspect samples to establish their relatedness or to exclude suspect samples from an investigation. Statistically meaningful comparative analyses can allow the use of statistical inferences relating to the process to produce the sample, the provenance of a sample, or the relatedness of samples. The significance of such statistical inference relies on the analyst’s ability to quantify both the confidence in test results and the frequency with which results match. Confidence, in this context, refers to the level of reliability and accuracy investigators assign to the test results obtained from the measurement tools used to identify the properties of interest in the samples. The frequency of the sample properties’ presence, or generation in a relevant population of possible sources, is a measure of how common or rare the properties are and provides context to the probative value of the evidence. According to a 2009 NRC report, a statistical framework is needed to quantify the probative value of forensic evidence in terms of the frequency of that evidence in a population. Formulating an appropriate statistical framework that is adequate for all microbial forensic investigations is not feasible because the diversity of many potential pathogens is unknown or, at best, difficult to describe. For this reason, frameworks must be adapted to the specific circumstances of each case. As shown in table 5, our review of scientific literature in forensic science, statistics, epidemiology, and population genetics identified the six general characteristics that a framework needs for statistically meaningful comparative analyses of the attack material to repository samples for the specific set of circumstances of the FBI’s investigation. First, a definition of what constitutes a matching type should be clearly established. A genetic signature, or a set of genetic markers, can be chosen to establish a genetic type (or genotype) that is used to differentiate the samples. The genetic signature should be sufficient to identify the target of interest at the resolution needed for an investigation. In this case, the target of interest was the B. anthracis Ames strain, capable of producing spores with a set of specific genetic markers linked to morphs observed after a prolonged period of growth. The requisite resolution was the ability to differentiate among the individual stocks (or collections of organisms) of B. anthracis. Determining that two or more samples have a matching type must take into account the source of the organisms (for example, nature or the laboratory), the stability of genetic markers, storage conditions, and conditions giving rise to the markers. Specific growth or environmental conditions may selectively advantage or disadvantage mutations and affect the stability of genetic markers. Therefore, if the significance of a matching genetic signature is to be understood, the genetic markers should be well characterized, and the conditions giving rise to the presence of markers in a sample should be understood. Second, once the genetic signature has been established and a match has been clearly defined, it is then necessary to identify and define the population of relevant sources that may have the genetic signature in order to understand how common or rare the genetic signature is. This relevant source population is critical in identifying the probative value of any match or nonmatch between samples. In a criminal investigation, a relevant source population may be considered the population of suspects, and it should be defined as specifically as possible to identify the smallest population related to the evidentiary material. The definition of the relevant source population should be based on the population related to characteristics of the evidence and not on characteristics of a suspected source. The relevant source population in this case is all stocks of B. anthracis that could have been used to grow the material used in the attack letters. In defining the source population, the structure of the relevant source population of bacteria should be understood. When a population is divided into subgroups that do not mix freely, that population is said to have structure. In this case, the relevant reference population of stocks of B. anthracis was highly structured among the laboratories included in the investigation. The lack of independence between stocks in a structured population affects inferences about the evidentiary material and its most and least likely sources. Third, in order to quantify or estimate how common or rare a genetic signature is in the relevant source population, a database that accurately represents the relevant source population’s genetics should be created. The extent to which the database reflects the population will affect the accuracy of the match probability. The size and quality of the data in the database will affect the power of match probability, determining the potential probative power of the signature for distinguishing one source from another. A large and comprehensive database is the theoretical goal but in most cases may not be possible. However, in this case, the FBI determined that it was possible to identify all sources of the B. anthracis Ames strain, and it set out to create a comprehensive database. For completeness the genetic information in the database should have included samples from all sources of the B. anthracis Ames strain. In such cases, the database should be complete—excluding sources results in underrepresentation—and should avoid duplication (although replication can be beneficial)—unknowingly including sources more than once results in overrepresentation. Methods used to select samples from each stock should be adequate to ensure representation of the organisms within each stock. In an ideal situation, the database of genetic information should be constructed to the same quality standards as the actual evidentiary analysis. These quality standards should apply to the selection of samples from stocks to the results of the genetic tests. Fourth, the limitations of the measurement tools used to generate the genetic information in the database should be identified. When quantitative inference is attempted, care must be taken not to overemphasize data; the limits of the methods used to generate the data should be considered. The power and limitations of microbial forensics methods need to be understood through validation. Validation frequently connotes confidence, but it may be thought of as defining the limitations of the method. This does not mean that a method must be 100 percent accurate to be useful. Studies should allow the estimation of the limits of the measurements. The limits of the methods must be demonstrated and documented for all steps in the process, including sample collection, preservation, extraction, analytical characterization, and data interpretation. Fifth, the choice of statistical methods should be appropriate for the data and should properly account for the mode of inheritance of the genetic markers and any structure in the populations. An important aspect of computing association statistics and probability estimates is properly accounting for the mode of inheritance. Methods appropriate for computing probability estimates and statistical tests of significance differ by the mode of inheritance of the genetic markers. In organisms that reproduce asexually, such as B. anthracis, genetic diversity is driven by mutation processes, not by random mixing. Computing match probabilities using methods that assume independence and random mixing within populations is not appropriate because the genetic variation in such organisms is highly correlated. In organisms that reproduce asexually, the frequency of a particular genetic type in the population must be determined by direct observation. The frequency of the evidentiary genotype in a relevant source population can be based on counting the number of times the genotype is observed in a reference database. The strength of this approach is affected greatly by the genetic database and whether it has sufficiently sampled relevant populations. Sixth and finally, the interpretation of results should include quantifications of uncertainty. It is crucial to clarify the type of question the analysis is addressing when evaluating the accuracy of a forensic analysis. Although some techniques may be too imprecise to permit the accurate identification of a specific individual, they may still provide useful and accurate information about questions and classification. The interpretation of results will be stronger with the proper use of statistical and probabilistic analyses, but the strengths and weakness of any result should be communicated. Results should indicate the uncertainty in the measurements, and studies must be conducted that enable the estimation of those values. We believe that the six general characteristics described above make up a comprehensive statistical framework that could have allowed the FBI to quantify significance and probative value of the scientific evidence collected in a statistically meaningful way and could have strengthened the evidence it collected. However, we found that at the outset of its investigation, the FBI did not have a comprehensive framework that would allow for statistically meaningful comparative analyses between samples from the attack letters and samples in the FBI repository of B. anthracis Ames strain. Specifically, we found that the FBI’s approach to three of the six characteristics could be improved to strengthen the significance of evidence in future investigations. Although the specific genetic mutations used as genetic markers to determine a match or exclusion were adequately characterized, the FBI did not conduct studies to understand the methods and environmental conditions that gave rise to the mutations. The FBI convened a team of scientists in 2007 to review the scientific methods. Finding no shortfalls or deficiencies in the basic methodologies they reviewed, they determined that the usefulness of the genetic markers was sufficient. The team also stated that the extent of research and development of the genetic tests at the date of their review was insufficient to determine whether the presence or absence of one or several of the genetic markers was associated with the evidence, was merely characteristic of normal culture practices, or possibly was affected by the sensitivity of detections of the genetic tests. The team recommended additional studies to characterize the genetic markers as a function of growth conditions, including the influence of growth time, growth media, and temperature. In response to questions from the NAS panel about this recommendation, the FBI stated that it considered such studies academic and did not conduct the recommended research. Consequently, experimental data are missing that would have shown the frequency with which particular genetic mutations occur under growth conditions that could affect their retention or loss. In its report, NAS opined that some of the morphs used as genetic markers might have arisen independently from RMR-1029. According to the report and experts we spoke with, the genetic markers might have had a selective advantage under growth conditions used for large-scale production of spores, such as in a fermenter or in a batch culture. If so, the presence of the genetic markers would be a function of the growth conditions rather than direct derivation from parent material, such as RMR-1029. This is problematic for the quantification of the rarity of the results because it is not possible to calculate the probability of two independent cultures having the same genetic markers if either was subjected to growth conditions that provide selective advantage or disadvantage. Without the experimental data, the usefulness of the genetic markers as an identifying signature to determine a match or exclusion was not fully understood. For example, it is not known whether the genetic markers could have arisen independently. To identify repository samples that received a direct or indirect transfer from the laboratory that possessed RMR-1029 after it was created in 1997, we examined the FBI’s documentation of historic transfer records of B. anthracis Ames strain between laboratories from 1981 through 2001. We supplemented this with information from laboratory officials and researchers we interviewed. Then, we compared the frequency of positive genetic markers in these groups of samples to the 119 samples that we verified were independent of transfers from the laboratory that possessed RMR-1029. Our analysis of repository data found no evidence of independent evolution in three of the four genetic markers (A1, A3, and E). However, we found that repository samples with no direct or indirect relationship to RMR-1029 tested positive for the D genetic marker at rates similar to those of the samples that were submitted from laboratories with direct transfers from the laboratory that possessed RMR-1029. table 6, the D genetic marker was detected in about 6.6 percent of the repository samples submitted from laboratories with direct transfers from the laboratory that possessed RMR-1029 compared to 6.7 percent of the samples that were independent of the laboratory that possessed RMR- 1029. Additionally, the NAS report found that in repository samples associated with experiments conducted before the 2001 attacks, the D genetic marker was the only marker detected and it occurred in about 1 percent (3 of 296) of those samples. This provides additional evidence that the D genetic marker may have arisen independently of RMR-1029. Additional studies recommended to the FBI that it did not conduct could have provided the experimental data needed to fully understand the probative value of this genetic marker. Because the FBI adequately identified the relevant source population as all stocks of B. anthracis Ames strain, it significantly reduced the number of possible sources. The NAS report found that the dominant organism in the letters was correctly and efficiently identified as the Ames strain of B. anthracis. The science performed on behalf of the FBI for identifying the Bacillus species and B. anthracis strain was appropriate, was properly executed, and reflected the contemporary state of the art. The correct identification of the specific strain of B. anthracis allowed the FBI to adequately define the relevant source population as stocks of the Ames strain in laboratories that had the Ames strain in their inventories before the attacks. This significantly reduced the number of possible sources. We found that the FBI’s effort to create a comprehensive repository containing samples from all known stocks of the Ames strain of B. anthracis was appropriate for assessing the rarity of the genetic markers in the relevant source population. Its adequacy, however, was affected by the incompleteness and inaccuracy in the repository. The NAS report found that the repository was not optimal for a variety of reasons. It stated, for example, that the instructions in the subpoena issued to laboratories for preparing samples were not precise enough to ensure that they would follow a consistent procedure for producing samples that would be most suitable for later comparisons. Our analysis of FBI documents shows that FBI searches at three specific laboratories identified hundreds of additional relevant stocks that laboratories did not submit to the repository in response to the subpoena. Specifically, we found that the FBI collected about 29 percent of the 1,059 repository samples through these searches. The proportions of samples thus obtained were 34 percent, 96 percent, and 22 percent in these laboratories (see table 7). We were unable to determine how two of the three laboratories identified and selected samples from relevant stocks in response to the subpoena, but we found that individuals at one laboratory differed in interpreting the subpoena’s instructions. Laboratory officials acknowledged differences in interpreting the instructions on how to identify distinct Ames strains of B. anthracis. Identifying the specific stocks to submit in response to the subpoena at that laboratory was left up to the principal investigator because, at that time, no one else actually working with the stocks would have understood what was in them. FBI officials acknowledged that the interpretation of the instructions to determine what strains to submit to the repository varied across laboratories, stating that the subpoena was not as precise as it needed to be. However, they emphasized that every laboratory that submitted samples to the repository was investigated thoroughly and that, when the FBI conducted searches at the three laboratories, those investigations eliminated many laboratories from being suspects. Furthermore, FBI officials told us that the decision to conduct searches at these three laboratories was an investigative decision, not a scientific one. The NAS report also raised concerns that the decision to remove samples with inconclusive or variant results contributed to the lack of completeness of the repository data. The report stated that a major concern was the restriction of its statistical analyses to the 947 samples that contained no inconclusive or variant results. Notably, the report showed that 4 of the 112 samples that were disregarded for having a single inconclusive or variant result scored positive for the three remaining genetic tests. In addition, our analysis of FBI documents shows that FBI searches contributed to inaccuracies in the repository by collecting samples from stocks that had already been submitted to the repository. We identified 14 duplicate samples from a search conducted at one laboratory in April 2004. FBI officials stated that they were not concerned about duplicate samples in the repository because duplicate samples may have served other important investigative purposes such as verifying if two samples were related or answering other important questions related to investigative information. They also stated that additional information collected about the samples would allow them to reconcile duplicates. However, our analysis of the FBI repository data indicates that known duplicates were not removed from the repository before the statistical analysis. As a result of these examples of incompleteness and inaccuracies in the repository, a statistically meaningful extrapolation of the statistics and frequencies derived from the repository to the relevant source population was not possible. By instituting more rigorous controls over sample identification and collection for future investigations, the FBI can improve the completeness and accuracy of a repository. The results from statistical analyses conducted in 2008 did not adequately account for the mode of inheritance of the genetic markers, and they added little probative value to the investigation. Many of the methods used for the 2008 statistical analyses inappropriately relied on the assumption of independence among the repository samples. For example, the NAS report stated that because the repository samples were not independent, the proportion of samples testing positive for all four genetic markers was not a meaningful estimate of the probability of occurrence. The FBI did not use the results of the statistical analyses and did not quantify the confidence it had in, or the probative value of, the repository results in its conclusions included in its final investigative summary. An FBI official stated that the statistical analyses were viewed from an academic standpoint and were not part of the investigation. That official also stated that the results of the statistical analyses did not contradict the conclusions of the investigation. In its final investigative summary, the FBI concluded that only 8 of more than 1,000 samples tested positive for all four genetic markers, but it did not provide any measure of the confidence it had in this conclusion. We found that the genetic tests show variability in the results on samples selected from the same stock. As we previously indicated in our assessment of the validation of the genetic tests, the additional postvalidation tests conducted in 2007 demonstrated variability in the results of the genetic tests when they were applied to samples under conditions intended to mimic their use on repository samples. Additionally, the two genetic tests for the D marker did not always give the same result for the same sample. An analysis included in the FBI contractor’s Statistical Analysis Report identified 24 repository samples for which the two genetic tests yielded opposite results from the same sample. The NAS report stated that this lack of agreement between the two genetic tests for the D mutation illustrated the differing sensitivities and specificities of the tests. This lack of agreement was also evident in the eight samples that tested positive for all four genetic markers. As shown in figure 4, our analysis of the repository data demonstrated that one of these eight samples also tested negative using the other genetic test for the D marker. Further, our analysis of duplicate samples in the repository showed differences in the results of genetic tests on samples selected from the same stock. As shown in figure 5, only 3 of the 14 duplicate samples we identified showed the same results across the five genetic tests. For example, FBI repository sample number 049-004 tested positive for all five genetic tests while a duplicate sample selected from the same stock (066-044) tested positive for only four of the five genetic tests. In another example, FBI repository sample number 049-016 tested positive for all five genetic tests while the duplicate sample (047-002) tested negative for all five genetic tests. FBI officials stated that these results may have differed for a number of reasons, including uncertainty from the sampling process (sampling error) and uncertainty from the genetic test itself (stochastic error). Each step in the process the FBI used to collect, prepare, and test repository samples could have added uncertainty to the results of the genetic test. As noted previously, before its searches, the FBI relied on laboratory officials to identify and select subsamples of distinct Ames strains for submission to the repository. The NAS report stated that the subpoena’s instructions to laboratories for preparing samples were not precise enough to ensure consistent procedures for producing samples that would be most suitable for later comparisons. For example, the subpoena instructed laboratories to select a representative sample from each stock but did not provide guidance on how many cells or colonies to select. Although steps were taken in the genetic tests to standardize the number of cells being tested, the number of initial cells or colonies selected from each stock would have affected the probability of selecting material capable of producing the genetic markers. This is particularly important because the mutations chosen as genetic markers were infrequent in the evidentiary material. For example, we interviewed the scientist who submitted the duplicate samples we identified above as having opposite results (all five negative versus all five positive). He told us that, in the presence of an FBI investigator, he had not followed the subpoena instructions when he selected the sample (047-002) that tested negative for all five genetic markers. In addition to the selection methods we have discussed in this report, the methods used to prepare and test the repository samples could have introduced uncertainty to the results of the genetic tests. The NAS report stated that replication could have been used in the design of the FBI repository to provide measures of the uncertainty of the genetic tests. Although laboratories were required to submit to the repository two samples from each stock, only one of those samples was tested for the genetic signature. Without replication, the FBI was unable to assess uncertainty in the results of the genetic tests in the context of testing actual repository samples. Because the FBI did not include measures of uncertainty when presenting the results of the genetic testing, questions have been raised about samples that tested positive for three or fewer genetic markers. For example, NAS stated that the FBI did not address false negative results and raised concern regarding the restriction of the statistical analyses to the repository samples that contained no inconclusive or variant results. NAS further highlighted 21 samples that contained an inconclusive or variant result and tested positive for 1, 2, or 3 genetic markers. To illustrate the potential effect this uncertainty could have had on the interpretation of the results, we conducted an analysis using the estimates of false negative rates obtained from the additional replicate testing, combined with a sensitivity analysis accounting for the decision to restrict the statistical analyses to the 947 samples that contained no inconclusive or variant results. We computed a range of probabilities, given the observed results of the genetic testing, that each repository sample was selected from a stock that could have produced all four genetic markers. We found an additional 16 repository samples with probabilities that exceeded a 1 percent chance of being selected from a stock that contained all four genetic markers. We determined that 15 of these 16 additional samples were selected from stocks held at the same two laboratories that were the source of one or more of the 8 samples that tested positive for all four genetic markers. The remaining sample identified in our analysis was a sample that we had determined was independent from RMR-1029 and tested positive for the D marker. In addition, this sample was inconclusive for both the A1 and A3 markers and negative for the E marker. We computed a 0 to 19 percent range of probabilities for this sample, the maximum occurring when the model made the assumption that both inconclusive results for A1 and A3 markers were positive. Additionally, the results of the genetic tests for this sample further highlight the importance of including measures of uncertainty. According to the transfer inventory records we reviewed and the laboratory official we interviewed, this sample was selected from a stock that was one of four copies of the same material. As shown in figure 6, the repository samples selected from the remaining three copies tested negative for all five genetic markers. This demonstrates that the genetic tests could have yielded different results for samples selected from the same material and, as the NAS stated, replication could have been used to provide measures of the uncertainty induced by these varying results. The FBI has taken steps to include statistical expertise in future investigations. The NAS report stated that the FBI appeared not to have sought formal statistical expertise early in this investigation and that similar investigations would benefit from including statistical expertise in their design and implementation. It noted that because many inferences depend on the design and analysis of complex data, the FBI should consult with expert statisticians throughout experimental design and planning, sample collection, sample analysis, and data interpretation. Further, the 2009 NRC report on strengthening forensic science in the United States highlighted the importance of statistical and quantitative proficiency for improving forensic science methods. An FBI official told us that since the 2009 NRC report, the FBI has been building formal forensic statistical expertise both internally and externally. For example, he said that the FBI laboratory division had created an internal statistical working group to examine the FBI’s statistical needs in its forensic methods. The group included a professor of statistics visiting for 6 months to examine the statistical questions related to patterns, such as fingerprints, and also other science, such as chemistry and explosives. Additionally, the FBI has established a working relationship with members of the American Statistical Association’s Ad-Hoc Advisory Committee on Forensic Science in order to discuss its statistical capacity. The FBI has also worked with other agencies to identify areas of statistical research needed for future investigations. After the 2001 attack, the FBI did not conduct a lessons learned study but considers the NAS report to be one. The NAS report identified some scientific gaps related to the development of genetic tests and statistical analyses. In addition, we identified a key scientific gap that is related to the verification and validation of the genetic tests and the statistical analyses—that is, the significance of using genetic mutations in B. anthracis as genetic markers for analyzing evidentiary samples. DHS has funded some research on this gap but this research is not yet complete, and it is not yet known whether it will fully address the gap. The FBI has not conducted a formal lessons-learned study of the scientific and technical methods it used in the investigation and thus has not specifically identified any scientific gaps in research related to the validation of genetic tests and statistical approaches. An FBI official stated that such a study was not needed because the 2001 incident was unique and the case is closed. This FBI official also told us that he considered the NAS report to be the lessons-learned study because it had identified several scientific gaps. For example, the NAS report indicated that the investigation lacked 1. a method for interpreting the genetic similarity between the attack spores in the letters and those in RMR 1029; and 2. an experimental design that included statistical input in the early stages of the investigation. Nevertheless, the FBI does not necessarily agree with the scientific gaps that NAS highlighted in that report. However, the FBI stated in 2010 that the active dynamics of the microbial genome for any given species need to be understood—for example, the location on the genome of “hot spots” for mutation and diversity and whether there is a high rate of genetic mobility and change in any given species. Further, in September 2014, according to an FBI official, technology has changed since the investigation, and in the future genome sequencing will be used to analyze evidence samples. In addition to the gaps identified in the NAS report, we identified a key scientific gap that has not been fully addressed. This gap is related to the significance of using genetic mutations as genetic markers for analyzing evidentiary samples to determine their origins. Recognized by NAS, this issue is associated with the gaps it identified. With respect to verification and validation, the genetic tests targeted specific DNA sequences of certain genetic mutations in their screening of the repository samples. The FBI used the results of the analysis of the repository screening by those tests to narrow the source of the attack spores. However, during the investigation, it was not known how stable genetic mutations were in a microbial genome or how significant they were as genetic markers. We found that conditions causing the rise of the genetic mutations in the evidence were not known before or after validation or during the subsequent statistical analysis of the results of the repository screening. During the investigation, it was not known what conditions would have promoted or inhibited the presence of the genetic mutations at detectable levels. Such knowledge would have indicated whether they were associated with the evidence itself or with the culture practices normally used in a laboratory. Although FBI expert advisers recommended experiments, none were conducted at that time to attempt to obtain this information. Such experiments could have helped in understanding the evolution of these particular genetic mutations. DHS has recognized the need for a methodology to determine how a material has been grown and produced and for obtaining information on the biology of agents, including their mutation rates and genome “hotspots” for mutation, so that their “relatedness” can be measured. In this context, an expert who reviewed this report stated that computational methods are also needed to reconstruct (or assemble) genome sequencing data so that the relationship between markers that are not independent, as is common in asexually reproducing bacterial genomes, can be inferred. As a result, DHS has funded research that is intended to provide a better understanding of how morphological variants, or mutations, could emerge and evolve in bacterial genomes. Some of the technologies involved in DHS’s research, such as whole genome sequencing, are still evolving. DHS-funded research includes studies of the population genetics of bacterial agents, including B. anthracis, at Northern Arizona University This research involves studies of diversity that include mutations (NAU).among these agents. DHS’s NBFAC is also studying genome sequencing methods. The purpose of these studies is to develop the capability to perform a metagenomic analysis of an entire sample using a hybrid- assembly. According to DHS, the field of “metagenomics,” is broad but unified by its focus on a community of genomes rather than individual isolates. Such research is a step in the right direction, since the FBI has indicated that it is likely to use genome sequencing methods in future investigations to analyze evidence. However, since this research is ongoing it is not clear when it will close the gap or whether it can do so alone. Although we identified several aspects of the FBI’s scientific methods we reviewed that could be improved in a future investigation, we recognize that in 2001, the FBI was faced with an unprecedented case. Determining the source of the spores in the envelopes was complicated by many factors, including the uncertain provenance of samples in the FBI repository, an unknown mutation rate for B. anthracis under laboratory growth conditions, and the performance of the genetic tests under “real- world” conditions. The genetic tests were generally verified, validated and demonstrated through the validation testing that they met the FBI’s acceptance criteria, but the lack of a comprehensive approach—that is, a validation framework—allowed for differences in the contractors’ approaches. Further, the results of the postvalidation testing raise questions about whether additional information could have been obtained during verification and validation and, thus, whether the validation testing could have been more rigorous. The use of a standardized approach to verification and validation from the beginning could have more definitively established the performance of all the genetic tests. It could have helped in communicating expectations clearly, ensuring confidence in results generated by any genetic tests developed. DHS could be instrumental in developing a validation framework and future efforts using a framework could help achieve minimum performance standards during verification and validation, particularly under multiple contracts. Also, incorporating statistical analyses in the framework would allow the calculation of statistical confidence for interpreting the validation testing results. The FBI’s statistical approach to its study design and plan, sample collection and analysis, and interpretation of data and scientific evidence lacked several important characteristics that could have strengthened the significance of that evidence. Although the complexity and novelty of the scientific methods at the time of the FBI’s investigation made it challenging for the FBI to adequately address all these problems, the agency could have improved its approach by including formal statistical expertise early in the investigation and establishing a statistical framework that could identify and account for many of the problems. In future investigations, statistical expertise early in the investigation will help identify the importance and role of fully understanding the (1) evolution of the genetic markers, (2) sources of dependence between samples, and (3) uncertainty in the measurement tools used to identify a genetic signature. This expertise could influence an investigation’s methods and strengthen the significance of scientific evidence. A key scientific gap—how stable genetic mutations are in a microbial genome and thus their suitability as genetic markers—remains an issue. Lack of this knowledge has implications for both the development of genetic tests, or other investigative approaches and technologies, and the analysis of the results they generate. For example, how likely it is that the same genetic mutations will arise independently in separate cultures is currently unknown, and so is whether different culture conditions can change the ratio of the mutations significantly enough to provide a negative rather than a positive result. DHS-funded research into the evolutionary behavior of variants in the genome of B. anthracis and other microbial agents and the use of genome sequencing is a step in the right direction because the FBI is planning to use sequencing in future investigations to analyze all the material in evidence samples. However, in determining the significance of using mutations as genetic markers, an understanding is still needed about the stability of genetic mutations. DHS’s ongoing research is likely to take several years and some of the technologies it entails, such as whole genome sequencing, are still evolving. Therefore, it is not clear when and whether this research alone will address this gap. To ensure that a structured approach guides the validation of the FBI’s future microbial forensic tests, we recommend that the Director of the Federal Bureau of Investigation work with the Secretary of Homeland Security to develop a verification and validation framework. The framework should be applied at the outset of an investigation involving an intentional release of B. anthracis, or any other microbial pathogen. It should (1) incorporate specific statistical analyses allowing the calculation of statistical confidence for interpreting the results and specifying the need for any additional testing to fully explore uncertainties relative to the type of genetic test being validated and (2) applied and adapted to a specific scenario and employs multiple contractors. In addition, we recommend that the Director of the FBI establish a general statistical framework that would require input from statistical experts throughout design and planning, sample collection, sample processing, sample analysis, and data interpretation that can applied and adapted to address a specific scenario involving an intentional release of B. anthracis or any other microbial pathogen. We provided a draft of this report to the FBI and DHS for review and comment. The FBI provided written comments, which are reprinted in appendix IV. In its comments, the FBI agreed with our recommendations and stated that it had taken significant steps toward addressing them. In addition, the FBI provided technical comments that we have addressed in the body of our report as appropriate. DHS stated that it had no comments on the draft report. With respect to the first recommendation, the FBI stated that “NBFAC programs have developed analytical capabilities in microbial forensics for numerous biological agents” in “support of investigations of the use or suspected use of biological weapons.” It stated that “these assays are validated and accredited under international standards (ISO17025) . . . .” According to the FBI, these capabilities, and those still being developed, “address part 2” of our recommendation “…applied and adapted to a specific scenario…” in as much as they represent capabilities addressing numerous biological agents and toxins. Further, the FBI stated that the NBFAC is pursuing the most current techniques of microbial genetic analyses and that some of these may soon be accredited. The FBI added that it actively participates in the National Strategy for Countering Biological Threats, under which the agency has helped in “Establishing a National level research and development strategy and investment plan for advancing the field of microbial forensics.” Further, it stated that it is helping to maintain “the National Biological Forensics Analysis Center (NBFAC) as the Nation’s lead Federal facility for forensic analysis of biological material in support of law enforcement investigations,” which advances the field of microbial forensics through scientific workshops sponsored by the FBI. According to the FBI, such workshops have included work on interpreting microbial genetic data acquired by next generation sequencing platforms. The FBI stated that this work has included “statistical analyses of the confidence in base calling” using these platforms and “bioinformatic software.” We recognize the importance of the FBI’s active participation in microbial forensic research and scientific workshops that address key issues related to the performance of emerging microbial forensic tests. We also recognize that establishing the error rates of genome sequencing platforms, which the FBI stated it may use in future investigations, would be an important step in verification and validation. Further, as we state in this report, developing a framework for verification and validation when employing multiple contractors in the same investigation could help standardize the process with minimum performance standards. Thus, we believe that the FBI’s continued work with DHS could help ensure the development of such a framework and improve its approaches to future investigations. A written plan could assist in the development of the framework. With respect to the second recommendation, the FBI stated that scientists from the FBI and NBFAC participate in the Food and Drug Administration’s related efforts, the “Global Microbial Identifier” symposiums, “whose activities include statistical analyses for interpreting microbial genetic data in investigations of food-borne illness.” We recognize the importance of the FBI’s continued participation in research on the statistical interpretation of microbial genetic data. The evidence we present in this report suggests that if statistical expertise had been included early in the FBI’s investigation, it could have improved the significance of the collected microbial forensic evidence. By establishing a general statistical framework, the FBI will be able to provide some assurance that input from statistical experts will be included in future investigations so that they will benefit from statistical expertise. Developing such a framework could also be facilitated by a written plan. We believe that the actions that the FBI states it has taken are a step in the right direction toward addressing our two recommendations. We are sending copies of the report to the FBI and DHS, appropriate congressional committees, and other interested parties. The report is also available at no charge on the GAO website at www.gao.gov. If you or your staff have any questions about this report, please contact Timothy M. Persons, Ph.D. at (202) 512-6412 or personst@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs appear on the last page of this report. Key contributors to the report are listed in appendix V. The scope of our work was limited to a review of the scientific methods employed to validate the genetic tests used to screen the FBI’s repository of Ames B. anthracis samples, the procedures used to identify and collect samples of Ames B. anthracis in the creation of the FBI’s repository, and the statistical analyses and interpretation of the results of the genetic tests. We did not address any other scientific methods or any of the traditional investigative techniques used to support the FBI’s conclusions in this case, and we take no position on the FBI’s conclusions when it closed its investigation in 2010. Our objective for this performance audit was to answer the following questions: 1. To what extent were the genetic assays used to screen the FBI repository of Ames samples scientifically verified and validated? 2. What are the characteristics of an adequate statistical approach for analyzing the repository samples and to what extent was the statistical approach used adequate? If not adequate, how could this approach be improved for future efforts? 3. What remaining scientific concerns and uncertainties, if any, regarding the validation of genetic assays and statistical approaches will need to be addressed in future analyses? What additional research, if any, would be helpful in resolving such scientific uncertainties in any future investigation? To determine the extent to which the genetic tests were verified and validated, we collected and reviewed data regarding (1) the FBI’s requirements for validation, (2) documentation from the FBI’s contractors on their verification and validation testing, and (3) documentation from the FBI on the contractors’ efforts to develop their genetic tests as well as results from the validation testing. We also reviewed related scientific literature and agency and industry standards and guidelines regarding the verification and validation of analytical methods, including real-time PCR- based tests for detecting B. anthracis, among others. We developed criteria for assessing the extent of the validation. We used references from agency standards, reports, and guidelines for validation and from scientific literature to identify the essential phases in an approach, or framework, for developing genetic tests. We compared what the FBI and its contractors had done to verify and validate the genetic tests against these phases and tasks. Specifically, we reviewed the FBI’s and its contractors’ laboratory documentation to determine for each genetic test (1) the steps each took to verify the genetic tests’ performance and conduct the FBI-administered validation, (2) whether the validation test results met the FBI’s acceptance criteria and minimum requirements, and (3) whether the FBI’s postvalidation testing of the genetic tests on the flask RMR-1029 provided further insights into the sensitivity and specificity of the genetic tests beyond those obtained by the validation. We also determined whether the processes the contractors’ laboratories followed for verifying and validating their genetic tests were consistent. Finally, we reviewed the NAS report’s observations on the performance of the genetic tests in screening the FBI’s repository samples. We interviewed officials and scientists at the FBI contractors, the FBI, and elsewhere on how the genetic tests had been verified and validated, standards or guidelines had been applied, and the FBI’s rationale for its requirements and acceptance of the five genetic tests as validated. We also compared the validation test results with the results of the additional testing that was conducted after validation to determine if any additional information was provided on the performance characteristics of the genetic tests. We did not independently verify whether the contractors followed their quality assurance guidelines in developing, verifying, and validating their genetic tests, but we assumed that they did so from the documentation provided. To determine the extent to which the statistical approach used for analyzing the repository samples was adequate, we used three approaches. First, we collected and analyzed documentation from the FBI, the three domestic laboratories searched by the FBI, and the contractor who did the statistical analyses. We reviewed contract records and conducted interviews with the FBI and laboratory officials. We conducted a literature review to collect relevant references from forensic science, statistics, epidemiology, and population genetics. Informed by the relevant literature, we identified and developed the set of characteristics that would be a statistical approach adequate to achieve the stated purposes of the FBI’s statistical analyses. We submitted the set of desirable characteristics described in this report to our experts and a subcommittee of the American Statistical Association’s (ASA) Ad Hoc Advisory Committee on Forensic Science for their review and comment. To obtain information about how samples were selected from stocks and submitted to the repository, we reviewed the FBI subpoena protocols, conducted semi-structured interviews with officials, and collected relevant laboratory documentation from the three laboratories that the FBI searched. Second, to obtain information about samples collected through the three follow-up searches, we interviewed FBI officials and reviewed the agency’s documentation, conducted semi-structured interviews with officials from the three laboratories that the FBI searched, and reviewed relevant laboratory documentation. To identify duplicate samples in the repository, we compared the documentation of samples obtained through the searches to samples submitted through the subpoena process. Third, to demonstrate the impact of the sensitivity of the genetic tests and data trimming assumptions made in the statistical analyses, we analyzed the FBI repository data and estimated false negative rates for each genetic test under repository conditions, using the post-validation results from replicate testing of RMR-1029 and evidentiary material. We conducted sensitivity analyses to examine the impact of data trimming assumptions made in the FBI’s statistical analysis by varying the assumptions made to remove all inconclusive, no-growth, and variant results from the analysis. We computed conditional probabilities that a repository sample was selected from a stock containing all four morphs, given the observed combinations of genetic test results. We combined the probability analysis with the data trimming sensitivity analysis to compute a range of conditional probabilities for each repository sample. We identified the samples that had a maximum conditional probability of greater than 1 percent (nontrivial). To assess the reliability of the FBI repository data, we summarized the data and compared the results to the contractor’s final report on the statistical analysis and to published reports by the FBI and the National Academies to ensure external validity of the data. From the results of this testing, we found the data to be sufficiently reliable for the purposes of our review. To determine any remaining scientific concerns and uncertainties regarding the validation of the genetic tests and statistical approaches that would need to be addressed in future analyses, we reviewed relevant federal agencies’ and their contractors’ documents, published literature, and industry documentation on the validation of polymerase chain- reaction based tests, such as those for detecting rare variants, and related scientific concerns and uncertainties that could affect a future investigation. We reviewed the contractors’ final reports on the statistical analysis, reviewed contract documents, and interviewed FBI officials to identify where improvements to the approach could be made. In addition, we reviewed the Centers for Disease Control and Prevention’s (CDC), the Animal and Plant Health Inspection Service’s (APHIS), and the Department of Defense’s (DOD) select agent requirements for storing, handling, shipping, and maintaining inventory controls. We interviewed agency officials to determine if gaps exist in documenting important information about the provenance of B. anthracis stocks. Further, to identify scientific concerns arising during the FBI’s investigation of the validation of the genetic tests and statistical approaches, we reviewed pertinent documentation on scientific issues or problems the FBI and NAS had identified and their effect on the FBI’s ability to validate the genetic tests or develop appropriate statistical approaches. Assisted by experts, we determined which gaps were significant and their potential effect on a future investigation with a similar scenario. We also interviewed officials and scientists at the contractors, the FBI, DHS, the National Bioforensic Analysis Center (NBFAC), DOD (at Dugway and USAMRIID), the Department of Energy’s (DOE) Lawrence Livermore National Laboratory (LLNL), the Joint Genome Institute (JGI), EurekaGenomics, and the Executive Office of the President’s Office of Science and Technology Policy (OSTP), regarding scientific challenges to genetic test validation, statistical analyses of the repository data, scientific gaps related to the FBI’s investigation, and any federal research being conducted, or planned, to fill those gaps. To determine additional research that would be helpful in resolving such scientific uncertainties in any future investigation, we reviewed documentation on research DHS is conducting to address any scientific gaps we found related to the validation of the genetic assays and issues related to the statistical analyses of the results of the repository screening. We reviewed the identified gaps and DHS’s research and determined the progress that had been made to close them. Further, following on interviews with scientists and agency officials, and input by our experts, we determined whether any additional research is needed. We asked scientists with expertise in public health and microbial forensic investigations to review and comment on a draft of our report. They included Jim Bristow, M.D., Deputy Director for Scientific Programs, DOE Joint Genome Institute; Karin S. Dorman, Associate Professor, Departments of Statistics and Genetics, Development, and Cell Biology, Iowa State University; George V. Ludwig, Ph.D., Deputy Principal Assistant for Research and Technology, U.S. Army Medical Research and Materiel Command; Jack Melling Ph.D., Director (retired), U.K. Centre for Applied Microbiology and Research, Porton Down, U.K.; Jeff Mohr, Ph.D., Chief (retired), Life Sciences Division, U.S. Army, Dugway Proving Grounds; and Stephen Velsko, Ph.D., Senior Scientist and Associate Program Leader, Lawrence Livermore National Laboratory. Finally, we asked a subcommittee of the American Statistical Association’s (ASA) Ad-hoc Advisory Committee on Forensic Science for its review and comment on the statistical aspects of a draft of our report. The subcommittee provided us with detailed comments that expressed general agreement with the statistical aspects of the draft, suggested changes to terminology related to the frequency with which microbial properties are present in a population, and suggested appropriate caveats and limitations to analyses we conducted. We incorporated these comments as appropriate throughout the report. We conducted this performance audit from January 2013 to November 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Performance Parameters Evaluated by Genetic Test n.a. n.a. n.a. n.a. n.a. Legend:  = evaluated; ο = not evaluated; n.a. = not applicable for qualitative tests. To illustrate the potential effect of the sensitivity of the genetic tests and data trimming assumptions made in the statistical analyses, we analyzed the FBI repository data and estimated false negative rates for each assay under repository conditions using the results from postvalidation replicate testing of RMR-1029 and evidentiary material. We conducted a sensitivity analysis to examine the effect of data trimming assumptions in the FBI’s statistical analysis by varying the assumptions to remove all inconclusive, no-growth, and variant results. We computed the conditional probabilities that a repository sample was selected from a stock containing all four genetic markers, given the observed combinations of results. We combined the probability analysis with the data trimming sensitivity analysis to compute a range of conditional probabilities for each repository sample. We then identified the samples that had a maximum conditional probability of greater than 1 percent (nontrivial). To build a model to compute this probability, we defined the sample space of possible outcomes. There are 16 combinations for a binary measure of the presence (+) or absence (-) of each of the four genetic markers. Therefore we defined the possible outcomes for the four genetic markers (A1, A3, D, and E) as S1 through S16, as shown in figure 7. The observed assay results have the 16 possible outcomes listed in figure 7. Since the goal of this analysis was to compute the probability that a repository sample had been selected from a stock that contained all four genetic markers, given the observed test result, we are interested in the probability of S, given the observed test result for a repository sample, 𝑃(𝑆1 |𝑜𝑏𝑠). Using Bayes’ theorem, this can be written as a , where posterior probability, (𝑆1|𝑜𝑏𝑠)= 𝑃(𝑜𝑏𝑠|𝑆1)𝑃(𝑆1) 𝑃(𝑜𝑏𝑠|𝑆𝑖)𝑃(𝑆𝑖) 𝑃(𝑜𝑏𝑠|𝑆𝑖) 𝑃(𝑜𝑏𝑠|𝑆1)=𝑃𝐴1(𝑜𝑏𝑠𝐴1|+)∗𝑃𝐴3(𝑜𝑏𝑠𝐴3|+)𝑃𝐷(𝑜𝑏𝑠𝐷|+)∗𝑃𝐸(𝑜𝑏𝑠𝐸|+) We used statistics derived from the results of postvalidation replicate testing conducted on RMR-1029 and letter material to estimate false negative rates. Figure 8 shows the breakdown of the results of the replicate testing. The sensitivity analysis examined the effect of two data trimming decisions made in the FBI’s statistical analysis of the repository samples─the choice of D assay results and the treatment of inconclusive results. The D marker was typed by two assay procedures (D-1 and D-2), only one of which (D-2) the FBI used in its analysis. The Statistical Analysis Report was restricted to the analysis of 947 samples that contained no inconclusive or variant results and, therefore, excluded 112 samples. To explore the potential effect of the inconclusive exclusion on the probabilities of observing all four morphs, we explored three possible outcomes for inconclusive results. We treated all inconclusive results as first positive and then negative, and then we excluded the inconclusive results from the analysis. The sensitivity analysis examined the six different combinations of outcomes, the two D assay possibilities, and the three potential outcomes of the inconclusive data. The computation included all 1,059 repository samples and varied the assumptions made around data trimming from most to least conservative. The results for each set of estimated false negative rates show that 7 of the 16 possible outcomes of the genetic testing had a range of probabilities that included values exceeding a 1 percent chance of being selected from a stock that contained all four genetic markers (table 8). Further, when we computed the probabilities for the repository samples, we found that only a small subset of the 1,059 repository samples had a range of probabilities that included values that exceeded a 1 percent chance of being selected from a stock that contained all four genetic markers. Specifically, we identified 24 repository samples, including the 8 that tested positive for all four genetic markers, which had a nontrivial chance of being selected from a stock that contained all four genetic markers. By using estimates of false negative rates from the results of the postvalidation replicate tests on RMR-1029 and the letter material, we have made an assumption that the genetic variants in all samples in the FBI repository were at least as concentrated as in RMR-1029 or the letter material. Additionally, since these replicate samples were selected in a controlled environment, false negative rates may have been underestimated because they are not affected by variation in test results caused by the sampling procedures used to submit samples to the repository. These assumptions contribute to a conservative estimate of the probability of a source matching all four genetic markers. Timothy M. Persons (Chief Scientist), (202) 512-6412 or personst@gao.gov. In addition to the contact named above, Sushil Sharma, Assistant Director, Pille Anvelt, James Ashley, Hazel Bailey, Amy Bowser, Mae Liles, Jan Montgomery, Penny Pickett, and Elaine Vaurio also made key contributions to this report. Anthrax: DHS Faces Challenges in Validating Methods for Sample Collection and Analysis GAO-12-488 (Washington D.C.: July 31, 2012). Federal Agencies Have Taken Some Steps to Validate Sampling Methods and to Develop a Next-Generation Anthrax Vaccine, GAO-06-756T (Washington D.C.: May 9, 2006). Anthrax Detection: Agencies Need to Validate Sampling Activities in Order to Increase Confidence in Negative Results, GAO-05-251 (Washington D.C.: March 31, 2005). U.S. Postal Service: Better Guidance Is Needed to Ensure an Appropriate Response to Anthrax Contamination, GAO-04-239 (Washington D.C.: September 9, 2004). U.S. Postal Service: Issues Associated with Anthrax Testing at the Wallingford Facility, GAO-03-787T (Washington D.C.: May 19, 2003). U.S. Postal Service: Better Guidance Is Needed to Improve Communication Should Anthrax Contamination Occur in the Future, GAO-03-316 (Washington D.C.: April 7, 2003).

Summary: In 2001, the FBI investigated an intentional release of B. anthracis , a bacterium that causes anthrax, which was identified as the Ames strain. Subsequently, FBI contractors developed and validated several genetic tests to analyze B. anthracis samples for the presence of certain genetic mutations. The FBI had previously collected and maintained these samples in a repository. GAO was asked to review the FBI's genetic test development process and statistical analyses. This report addresses (1) the extent to which these genetic tests were scientifically verified and validated; (2) the characteristics of an adequate statistical approach for analyzing samples, whether the approach used was adequate, and how it could be improved for future efforts; and (3) whether any remaining scientific concerns regarding the validation of genetic tests and statistical approaches need to be addressed for future analyses. GAO reviewed agency and contractor documentation, conducted literature reviews, and conducted statistical analyses of the repository data. GAO's review focused solely on two aspects of the FBI's scientific evidence: the validation of the genetic tests and the statistical approach for the analyses of the results. GAO did not review and is not taking a position on the conclusions the FBI reached when it closed its investigation in 2010. After the 2001 Anthrax attacks, the genetic tests that were conducted by the Federal Bureau of Investigation's (FBI) four contractors were generally scientifically verified and validated, and met the FBI's criteria. However, GAO found that the FBI lacked a comprehensive approach—or framework—that could have ensured standardization of the testing process. As a result, each of the contractors developed their tests differently, and one contractor did not conduct verification testing, a key step in determining whether a test will meet a user's requirements, such as for sensitivity or accuracy. Also, GAO found that the contractors did not develop the level of statistical confidence for interpreting the testing results for the validation tests they performed. Responses to future incidents could be improved by using a standardized framework for achieving minimum performance standards during verification and validation, and by incorporating statistical analyses when interpreting validation testing results. GAO identified six characteristics of a statistical framework that can be applied for analyzing scientific evidence. When GAO compared the approach the FBI used to this framework, it found that that the FBI's approach could have been improved in three of six areas. First, the FBI's research did not provide a full understanding of the methods and conditions that give rise to genetic mutations used to differentiate between samples of B. anthracis . Second, the FBI did not institute rigorous controls over the sampling procedures it used to build the repository of B. anthracis samples. Third, the FBI did not include measures of uncertainty to strengthen the interpretation of the scientific evidence. GAO found that since 2001 the FBI has taken some steps to build formal forensic statistical expertise. The FBI's approach to future incidents could benefit from including such expertise early in an investigation. The lack of an understanding of how bacteria change (mutate) in their natural environment and in a laboratory is a key scientific gap that remains and could affect testing conducted in future incidents. Specifically, the significance of using such mutations as genetic markers for analyzing evidentiary samples to determine their origins is not clear. This gap affects both the development of genetic tests targeting such mutations and statistical analyses of the results of their use on evidentiary samples. The Department of Homeland Security is currently funding some research on genetic changes in bacteria and genome sequencing methods, among others. Such research is a step in the right direction since the FBI is planning to use genome sequencing methods in future investigations. However, because this research may not be complete for several more years, the extent to which it will close this gap is not known. GAO recommends that the FBI develop a framework for validation and statistical approaches for future investigations. The FBI agreed with our recommendations.