dataset to determine which crowdsourced samples realistically originate from COVID-19 patients.
In addition to COVID-19 diagnoses, our expert labels and metadata provide a wealth of insights beyond those of existing public cough datasets. These datasets either do not provide labels or contain a small number of samples. For example, the Google Audio Set 12 contains 871 cough sounds, but it does not specify the diagnoses or pathologies of the coughs. Conversely, the IIIT-CSSD 13 labels coughs as wet vs dry and short-term vs long-term ailments, but it only includes 30 unique subjects. The COUGHVID dataset contributes over 2,000 expert-labeled coughs, all of which provide a diagnosis, severity level, and whether or not audible health anomalies are present, such as dyspnea, wheezing, and nasal congestion. Using these expert labels along with subject metadata, our dataset can be used to train models that detect a variety of subjects' information based on their cough sounds. Overall, our dataset contains samples from a wide array of subject ages, genders, COVID-19 statuses, pre-existing respiratory conditions, and geographic locations, which potentially enable AI algorithms to successfully perform generalization.
Finally, we assert the validity of our data by ensuring that samples labeled as COVID-19 originate from countries where the virus was prevalent at the time of recording, and that the experts exhibit a reasonable degree of agreement on the cough diagnoses. The first step to building robust AI algorithms for the detection of COVID-19 from cough sounds is having a reliable, high-quality dataset, and the COUGHVID dataset effectively meets this pressing global need.
All of the recordings were collected between April 1st, 2020 and September 10th, 2020 through a Web application deployed on a private server located at the facilities of the École Polytechnique Fédérale de Lausanne (EPFL), Switzerland. The application was designed with a simple workflow and following the principle "one recording, one click", according to which if someone simply wants to send a cough recording, they should have to click on no more than one item.
The main Web interface has just one "Record" button that starts audio recording from the microphone for up to 10 seconds. Once the audio recording is completed, a small questionnaire is shown to get some metadata about the age, gender, and current condition of the user, but even if the questionnaire is not filled, the audio is sent to the server. The variables captured in the questionnaire are described in Table 1 . Also, the user is asked for permission to provide their geolocation information, which is not mandatory. Finally, since coughing is a potentially dangerous activity in the scope of a global pandemic, we provide easy-to-follow safe coughing instructions, such as coughing into the elbow and holding the phone at arm's length, that can be accessed from the main screen.
A common pitfall of crowdsourced data is that it frequently contains samples unrelated to the desired content of the database. In order to allow users of the COUGHVID database to quickly exclude non-cough sounds from their analyses, we developed a classifier to determine the degree of certainty to which a given recording constitutes a cough sound. These output probabilities of the classifier were subsequently included in the metadata of each record under the cough_detected entry.
To train the classifier, we first hand-selected a set of 121 cough sounds and 94 non-cough sounds including speaking, laughing, silence, and miscellaneous background noises. These recordings were preprocessed by lowpass filtering ( f cuto f f = 6 kHz) and downsampling to 12 kHz. Next, 68 audio features commonly used for cough classification were extracted from each recording. 40 of the features are those described by Pramono et al. 6 , and the implementation of our 19 Energy Envelope Peak Detection features is detailed by Chatrzarrin et al. 14 . We also included a signal length feature, as well as the power spectral density (PSD) of the signal in 8 hand-selected frequency bands. These bands were chosen by analyzing the PSDs of cough vs non-cough signals and selecting the frequency ranges with the highest variation between the two classes. Finally, we trained an eXtreme Gradient Boosting (XGB) 15 classifier using 78% of the available data. The hyperparameters of the XGB model were tuned using Tree-structured Parzen Estimators (TPE) 16 with a precision objective using the training data for 10-fold, 20%-test shuffle-split cross-validation.
The ROC curve of the cough classifier is displayed in Figure 1 , which users of the COUGHVID database can consult to set a cough detection threshold that suits their specifications. As this figure shows, only 10.4% of recordings with a cough_detected value less than 0.8 actually contain cough sounds. Therefore, they should be used only for robustness assessment, and not as valid cough examples.
To enhance the quality of the dataset with clinically validated information, we were assisted by three expert pulmonologists. Each of them revised 1000 recordings, selecting one of the predefined options to each of the following 10 items:
1. Quality: Good; Ok; Poor; No cough present.
2. Type of cough: Wet (productive); Dry; Can't tell. 10. Impression: the cough is probably...: Pseudocough / Healthy cough (from a healthy person); Mild (from a sick person); Severe (from a sick person); Can't tell.
As a labeling support tool, we used an online spreadsheet using Google Sheets c . Thus, the experts could play the recordings directly inside the browser and select their answers in a convenient way. Once a recording had been labeled, the background color of the full row turned to green for easier navigation. The time required for each expert for labeling the 1000 recordings was around 10 hours, without significant differences among the three experts.
In addition to the personal spreadsheets, the experts were provided with the following general instructions:
• For binary variables (Columns E-J) the box should be ticked for any reasonable suspicion of the sounds being heard.
• In Column K ("Impression: I think this patient has..."), you should check what you consider the most likely diagnosis, knowing that the records were collected between 01/04/2020 and 18/05/2020.
• Each row is automatically marked as "completed" after providing an answer to column K. However, please try to give an answer to every column.
Also, the following criteria for assessing quality was indicated to the experts:
• Good: Cough present with minimal background noise.
• Ok: Cough present with background noise.
• Poor: Cough present with significant background noise.
The selection of the recordings to be labeled was done through stratified random sampling and after pre-filtering using the automatic cough classifier described above, requiring a minimum probability of 0.8 of containing cough sounds. The stratification was based on the self-reported status variable, as follows:
• 25% of the recordings with COVID value.
• 35% of the recordings with symptomatic value.
• 25% of the recordings with healthy value.
• 15% of the recordings with no reported status.
Finally, we ensured that 15% of the recordings were labeled by all three reviewers, so that we could assess the level of agreement among them.
Each cough recording consists of two files with the same name but different extensions. One of the files contains the audio data, as it was directly received at the COUGHVID servers, and it can be in the WEBM 17 or OGG 18 formats, respectively with the .webm and .ogg extensions. In all cases, the audio codec is Opus 19 and the sampling frequency 48 kHz. The second file contains the metadata encoded as plain text in the JSON format 20 , and has the .json extension. The file name is a random string generated according to the UUID V4 protocol 21 .
In the metadata we may distinguish three types of variables, related to: 1) context information (timestamp and the probability that the recording actually contains cough sounds), 2) self-reported information provided by the user, and 3) the labels provided by expert medical annotators about the clinical assessment of the cough recording. The only processing performed on the metadata was the reduction in the precision of the geolocation coordinates to just one decimal digit to ensure privacy protection. A full description of all the metadata variables is provided in Tables 1 and 2.
As an illustrative example, let us consider the recording with UUID 4e47612c-6c09-4580-a9b6-2eb6bf2ab40c. We can see the audio properties using a tool such as ffprobe (included in the ffmpeg software package 22 4e47612c-6c09-4580-a9b6-2eb6bf2ab40c.json { "datetime": "2020-04-10T10:30:31.576207+00:00", "cough_detected": "0.9466", "age": "50", "gender": "male", "respiratory_condition": "True", "fever_muscle_pain": "False", "status": "COVID-19", "expert_labels_1": { "quality": "ok", "cough_type": "dry", "dyspnea": "False", "wheezing": "False", "stridor": "False", "choking": "False", "congestion": "False", "nothing": "True", "diagnosis": "COVID-19", "severity": "mild" } } For convenience, a file compiling all of the available metadata is also provided. This file is named metadata_compiled.csv, and is a CSV file with 40 columns and one row per record. The first column corresponds to the UUID of each recording, and may be used as an index, while the rest of the columns correspond to the variables described in Tables 1 and 2. The expert annotation variables have been expanded, and are named quality_1, cough_type_1 ... diagnosis_3, severity_3.
Demographic representativeness An important requirement for large datasets is that they must represent a wide range of subject demographics. Demographic statistics were collected across all recordings that provided metadata and that were classified as coughs with a probability above 0.8 by the XGB cough detection model. There were slightly more male subjects than female (65.5% and 33.8%, respectively), and the majority of subjects did not have pre-existing respiratory conditions (81.9%). The percentages of healthy, COVID-19 symptomatic, and COVID-19 positive subjects were 77%, 15.5%, and 7.5%, respectively. The average age of the recordings was 34.4 years with a standard deviation of 12.8 years. This shows that a wide variety of ages, genders, and health statuses are captured within our dataset.
In order to assess the plausibility that samples labeled as COVID-19 truly originated from people who tested positive for the disease, we analyzed the geographic locations of the samples for which this data was provided. We then evaluated the COVID-19 statistics of the countries at the time each recording was sent to determine if these countries had high infection rates at the time of recording. Of the 10,743 samples that the XGB model classified as cough sounds, 6,485 of them provided GPS information, 676 of which reported COVID-19 symptoms and 372 claim to have been diagnosed with COVID-19.
Studies have shown that coughing persists in a significant proportion of COVID-19 cases 14-21 days after receiving a positive PCR test 23 . Therefore, we combined the World Health Organization's statistics on daily new COVID-19 cases 1 with the United Nations 2019 population database 24 to determine the rate of new infections in the country from which each recording originated in the 14 days prior to it being uploaded to our web server. This analysis revealed that 94.4% of our recordings labeled as COVID-19 came from countries with more than 20 newly-confirmed COVID-19 cases per 1 million people. Similarly, 91.3% of recordings labeled as symptomatic originated from these countries. Figure 2 shows a map of the world with countries color coded according to the cumulative COVID-19 positive tests in April and May 2020 per 1 million population. We also show the COVID-19 and symptomatic recordings providing GPS information that were collected within this time period. This figure shows that most recordings were sent from countries with moderate-to-high COVID-19 infection rates at the time.
In order to determine the extent to which the three expert pulmonologists agreed on their cough sound labeling, Fleiss' Kappa scores 25 , K Fleiss , were computed for each question among the 150 common recordings. The results are displayed in Table 3 . This analysis revealed moderate agreement on audible nasal congestion, fair agreement on the type of cough, as well as slight agreement on the cough severity, nothing specific, audible wheezing, and audible dyspnea.
There was a poor agreement among experts on the cough diagnosis (K Fleiss = 0.0031), which is reflective of the fact that COVID-19 symptomology includes symptoms of both upper respiratory tract infections (e.g., rhinorrhea, sore throat, etc.) and lower respiratory tract infections (e.g., pneumonia, ground-glass opacities, etc.) 26 . There was a slight agreement between Experts 1 and 2 on the diagnosis (K Fleiss = 0.0774), whereas Expert 3 did not label any of the common recordings as COVID-19. Out of the 86 coughs that at least one rater labeled as COVID-19, 22 had a majority consensus. When the majority agreed that a cough was COVID-19, Expert 3 diagnosed it as an upper respiratory infection 59.1% of the time, a lower respiratory tract infection 27.3% of the time, and nothing 13.6% of the time. A confusion matrix between Experts 1 and 3 is displayed in Figure  3 , which shows a higher degree of agreement between Experts 1 and 2 than either with Expert 3.
All of the coughs labeled as COVID-19 among the three experts were subsequently pooled together and analyzed for trends in the attributes of the cough recordings. In the case of overlaps, the diagnoses of Expert 1 were taken into account, producing 632 total COVID-19-labeled cough records. The vast majority of coughs do not exhibit audible dyspnea (93.0%), wheezing (90.5%), stridor (98.7%), choking (99.1%), or nasal congestion (99.2%). Additionally, 87.3% of COVID-19-labeled coughs are annotated as dry, which is consistent with literature stating that a dry cough is a common COVID-19 symptom 5, 26 . Finally, 86.2% of these coughs are labeled as mild. These commonalities among COVID-19 labeled coughs reflect the consistency of the database.
In order to ensure the reproducibility of the experimental results that use the COUGHVID crowdsourcing dataset, a private test set has been kept out from publishing. Recordings in this private set have been randomly selected from those having at least labels from one expert.
The evaluation of models on the private test set will be open to the entire scientific community, but to ensure a fair use, the performance measurements will be obtained by an independent evaluator. Any researcher that demonstrates promising results on the public dataset using cross-validation may apply for an independent evaluation on the private test set according to the protocol described in the data repository. This protocol shall be regularly updated in line with the available technology to ensure that it is as convenient as possible for all parties.
Since one of the aims of this project is to go beyond the study of COVID-19, every variable except datetime and cough_detected may be considered the target of a prediction model. This opens the possibility to study several different problems within the same dataset, from just cough identification and sound quality assessment to the detection of different conditions, or even the estimation of age or gender. Conversely, a prediction model may require as input not just the sound recording, but also other metadata variables to provide the necessary context, such as the age or gender of the subject. The restrictions on the sets of variables that can be used as input or as a result of the algorithms will be kept to a minimum.
The aforementioned XGB classifier used to remove non-cough recordings and feature extraction source code are available on our public c4science repository. The patient self-reports that has been diagnosed with COVID-19 (COVID), that has symptoms but no diagnosis (symptomatic), or that is healthy (healthy).
No JSON dictionary with the manual labels from expert 1, 2 or 3
The expert annotation variables are described in Table 2 . Impression of the expert about the severity of the cough. It can be a pseudocough from a healthy patient, a mild or severe cough from a sick patient, or unknown if the expert can't tell. 