The 2018 Ebola outbreak in Equateur Province, Democratic Republic of the Congo (DRC), was 28 brought under control following 54 cases between 5 th April and 2 nd June (1). Another 29 unconnected outbreak was declared in DRC on 1 st August 2018, and that outbreak is still in 30
progress. It is about to become the second largest in history, with 421 probable and confirmed 31 cases as of 26 th November 2018 (2). Increasingly, decision-makers use forecasts generated using 32 mathematical models to guide control measures when outbreaks are ongoing (3,4). However, less 33 attention has been directed towards using mathematical modelling to inform decision-making at 34 the ends of outbreaks (5). 35
Determining when an outbreak of any infectious disease is over is important for decision-makers, 37 as they need to choose when to relax control measures, scale-back the deployment of personnel 38 and resources, adjust communication messages to the public, and re-establish confidence in 39 commercial sectors such as agriculture and tourism. However, the difficulty of end-of-outbreak 40 decision-making was illustrated during the 2013-16 Ebola epidemic, when the World Health 41 Organization (WHO) declared Liberia disease-free four times only to have new cases detected 42 after the first three declarations (Fig 1A) . This raises an important question: how confident can 43 public health decision-makers be when declaring an outbreak over? 44
The proportion of cases identified by public health authorities through passive or active case 46 finding, otherwise called the sensitivity of a surveillance system (6,7), is a critical parameter that 3 underlies how confident a decision-maker can be when declaring the end to an outbreak (8). The 48 surveillance sensitivity is the ratio of the number of infectious cases detected to the total number 49 of cases (including both cases that are detected and those that go unnoticed), and should not be 50 confused with the sensitivity of a diagnostic test (i.e. the probability that the diagnostic test 51 correctly identifies an infected host). Intuitively, there will be a lower confidence that an 52 outbreak is over if the surveillance system has low sensitivity. However, decision-makers do not 53 typically make quantitative assessments about the confidence in their end-of-outbreak decisions. 54
A retrospective modelling study of the MERS-CoV outbreak in South Korea in 2015 concluded 55 that, with no quantitative end-of-outbreak assessment, decision-makers took longer than 56 epidemiologically necessary to declare the outbreak over (9), although we note that the 57 sensitivity of the surveillance system was assumed to be 100% in that study. 58
To illustrate how the confidence that an outbreak is over can be estimated, and that the 60 confidence level can be increased by improving surveillance, we consider the situation of 61 declaring the end of an Ebola outbreak. The WHO considers an Ebola outbreak to be over once 62 42 days have passed since the last possible exposure to a confirmed case without any new cases 63 being detected (10), with this rule most often deployed at the scale of a single country. The 64 incubation period (the time between an individual becoming infected and displaying 65 recognisable symptoms) for Ebola has been estimated to be in the range of 2-21 days (11), and so 66 the period of 42 days is based on two maximal incubation periods. For a disease that is 67 transmitted directly from person-to-person, the passing of two incubation periods is 68 epidemiologically relevant because additional between-person transmission is then unlikely. We 69 use mathematical modelling to show that, if the surveillance sensitivity is 100%, then it is likely 70 4 that an Ebola outbreak is over after 42 days without symptomatic cases. However, we also 71 demonstrate that, if the surveillance sensitivity is lower, there is an increased chance of 72 undetected infected cases remaining after 42 days. This leads to a lower confidence that the 73 outbreak is over after this time period. 74 75 Although we focus on Ebola virus disease here, we note that the question of whether or not an 76 infectious disease outbreak is over is not only important for diseases of humans, but also those of 77 animals and plants. Declaring an outbreak over allows disease management interventions to be 78 lifted, including restrictions on travel (12) and plant trade quarantine (13). The idea that 79 improved surveillance may lead to increased confidence in an end-of-outbreak declaration is 80 related to well-established theory regarding conducting surveys to ascertain the absence of a 81 pathogen (see e.g. (14-19) ). In that context, the more hosts are tested and found to be disease-82 free, the higher the confidence that the entire population is disease-free. This can in turn be used 83 to generate sample-size requirements to establish freedom from disease to pre-specified 84 confidence levels. While initial studies in this area -motivated by the desire to limit pathogen 85 transmission via the animal trade -assumed that the level of disease in the host population was 86 static, more recent elaborations have included incorporation of dynamic models describing 87 parasite/pathogen transmission in the host population (see e.g. (20,21)). Statistical disease 88 freedom studies have not only been applied to animal disease epidemics, but the theory has also 89 been used in the context of epidemics in populations of plants (22, 23) and humans (24). 90 91 In this paper, rather than considering surveys of the host population at the apparent end of an 92 outbreak, we show how the confidence in end-of-outbreak assessments can be estimated using 93 5 epidemiological models once the surveillance system sensitivity has been approximated. Our 94 approach, which can be used when the outbreak in question is still ongoing, provides decision-95 makers with a practical way to gauge whether current surveillance efforts will be sufficient to 96 declare the outbreak over with conviction, or whether an intensification of surveillance is 97 necessary instead. 98 99
We extended an epidemiological model commonly used for Ebola (the SEIR model, see e.g. 101 (25,26)) to include imperfect surveillance ( Fig 1B) . In the resulting model (the SEICR model) 102
individuals were classified according to whether they were (S)usceptible, (E)xposed, (I)nfectious 103 and reporting disease, (C)ryptically infectious (i.e. infectious but not reporting disease), or 104 (R)emoved. The parameters of the model and the baseline values used in our analyses to 105 illustrate the model behaviour are given in Table 1 , although we also tested the robustness of our 106 results to these values ( Fig S1) . We ran stochastic simulations of the model, thereby including 107 randomness in whether or not each outbreak was over when the number of symptomatic 108 individuals (I) reached zero (for additional details, see the Supplementary Material). 109
The surveillance sensitivity was implemented in the model via the proportion, d, of infectious 111 individuals that reported disease (I) as opposed to remained cryptically infectious (C). When an 112 individual left the exposed class, they either transitioned into the I class (with probability d) or 113 into the C class (with probability 1 -d). The parameter d represents a proportion/probability and 114 therefore lies between zero and one, whereas the surveillance sensitivity is reported as a 115 percentage. As an example, the value d = 0.1 corresponds to a surveillance sensitivity of 10%. 116 6 For simplicity, we assumed that whether or not an infectious individual reported disease did not 117 alter their infectiousness or duration of infection, although this simplification could be relaxed 118 straightforwardly. As described above, the cryptically infectious class represents individuals that 119 are infectious but do not report disease -this could include asymptomatic carriers that are 120 infectious (27) or symptomatic individuals not reporting for reasons including a lack of access to 121 healthcare (8). 122 123 By continuing to run simulations after the number of symptomatic infectious individuals (I) 124
reached zero, the confidence that an outbreak will be over, defined as the probability that no 125 undetected infected hosts (E or C) remained in the population, was estimated at different time 126
periods beyond the removal of the last detected case. 127 128
We inferred the expected number of undetected infected cases once the number of symptomatic 130 cases reached zero (Fig 1C) , considering only outbreaks that successfully invaded the host 131 population (outbreaks in which more than 20 individuals were ever infected). We estimated the 132 confidence that the outbreak is over for different time periods beyond the removal of the last 133 detected case (Fig 1D) . For additional results with different model parameters, see the 134
Since a period of 42 days has been estimated as twice the maximal incubation period for Ebola, it 137 is unsurprising that, when the sensitivity of the surveillance system was perfect so that 100% of 138 infectious cases were detected accurately, the model suggested a high confidence (more than 139 7 97%) that an Ebola outbreak is over after 42 days without symptomatic cases. Additional new 140 cases could only occur if existing infected individuals remained pre-symptomatic, and this was 141 very unlikely after this time period. However, when we assumed that the surveillance sensitivity 142 was only 40%, an estimate made for Ebola surveillance in Liberia (28), the probability that Ebola 143 cases remained in the population was 16%, leading to only an 84% confidence that the outbreak 144 was finished (red line in Fig 1D) . With such a low surveillance sensitivity, a period of 62 days 145 with no cases would need to elapse to be 95% confident that an outbreak is over (light green line 146 in Fig 1D) , or 88 days to be 99% confident (dark green line in Fig 1D) . decision-makers to be confident that Ebola outbreaks are over after 42 days. We therefore 156 considered the end-of-outbreak confidence for varying levels of the surveillance sensitivity (Fig  157   1E ). To be at least 95% confident that an Ebola outbreak is over after 42 days, surveillance 158 needed to be at least 79% sensitive (light green line in Fig 1E) . For lower surveillance levels, 159 there is a significant chance (> 5%) of residual infectious cases remaining in the population, and 160 these might generate outbreak flare-ups. 161 162 8
We have proposed an approach for decision-makers to estimate their confidence that an Ebola 164 outbreak is over after 42 days (two maximal incubation periods) have passed with no new cases. 165
In scenarios with a low surveillance sensitivity, decision-makers may either choose to wait 166 longer than two incubation periods before declaring the end of an outbreak, take measures to 167 increase the surveillance sensitivity, or adopt both of these approaches. Communicating that an 168 outbreak is over following two incubation periods is epidemiologically coherent when the 169 surveillance level is high, and so decision-makers may prefer to focus efforts on achieving a high 170 surveillance sensitivity rather than adjusting the guideline period before declaring the end of an 171 outbreak. However, in contexts that prevent strengthening of disease surveillance, for example if 172 there is poor security due to armed conflict or other factors, extending the period with no cases 173 before declaring an outbreak over may be the more pragmatic option. 174 175 Sensitivity measurements are sometimes carried out for evaluation of surveillance systems (31). 176
Analysis of the percentage of cases being recorded can be conducted using serological surveys 177 (32) or by comparing multiple data sources (33). When an outbreak is ongoing, however, 178 measuring the surveillance sensitivity might not be the first priority. For assessing the confidence 179 in a potential end-of-outbreak declaration, it is most important to measure the surveillance 180 sensitivity towards the apparent end of the outbreak, and so resources can be directed to this task 181 after the acute outbreak period has passed. In scenarios in which the surveillance sensitivity is 182 insufficient for declaring an outbreak over with confidence, remedial actions can be taken such 183 as strengthening case finding for example via contact tracing (34), closer working with 9 community leaderships to establish a case finding and reporting network (35), and/or providing 185 incentives for successful case reporting (36), among other approaches. 186
In this paper, we sought to use a simple approach to demonstrate how the confidence in end-of-188 outbreak declarations could be assessed, and to show that rigorous surveillance is extremely 189 important. While accurate case reporting will minimise the chance of incorrect declarations that 190
Ebola outbreaks are over in future, we note that surveillance during the outbreak alone is not 191 always sufficient. In the 2013-16 Ebola epidemic in West Africa, additional cases occurred after 192 regions were declared disease-free due to factors including persistently infected sources (37) Extending our approach to other disease outbreaks might require elaborations to the underlying 209 model. To illustrate the principle that the surveillance sensitivity affects the confidence in end-210 of-outbreak declarations, we modelled surveillance as simply as possible -by assuming that a 211 proportion of infectious hosts report disease, but that reporting did not impact on the underlying 212 transmission process. In practice, individuals that report disease are more likely to be subject to 213 interventions that reduce infectiousness or shorten their infectious period, such as isolation or 214 treatment. This could straightforwardly be built into the framework that we have presented. We 215 also use a single parameter to denote the surveillance sensitivity, whereas in practice a 216 surveillance program is likely to encompass many aspects, including both passive and active case 217 finding strategies, that could be built explicitly into an epidemiological model. One of the 218 benefits of our approach is that, in contrast to methods relying on surveys to prove disease 219 absence, our analysis can be conducted in advance of the apparent end of the outbreak to see 220 whether or not surveillance needs to be intensified. However, it might be possible to combine our 221 approach with surveys to establish the end of an outbreak, and to make use of statistical methods 222 for estimating the number of hosts to survey so that the probability of the population being 223 disease-free exceeds a pre-specified threshold (14-24). 