For decades, contact tracing has been known as an effective method for controlling the spread of infectious diseases. In the traditional contact-tracing model, trained personnel use inperson or telephone interviews to identify and list those who have had meaningful exposure to diagnosed individuals during a disease's likely transmission period. The risk levels of those contacts can then be determined based on factors such as the physical distances between them and the infected people, the duration of such exposure, their mobility trajectories over time, whether masks were worn, and whether the contact occurred indoors or outdoors.
With the escalating scale of the COVID-19 pandemic, the vast numbers of new cases identified via testing each day exceed the capacity of traditional contact tracing, the failures of which have been extensively reported [1] . Some local health services have even abandoned their former practice of directly communicating with the close contacts of a case. Thus, governments are turning to digital contact tracing that utilizes mobile providers or mobile applications to help identify potential contacts by means of GPS data, cell-tower connection history, QR-code scanning, and so forth.
However, large-scale collection of citizens' digital footprints by governments has sparked concerns about mass surveillance. Accordingly, to enhance the privacy of digital contact tracing, a number of decentralized approaches have been developed, capable of detecting close proximity between phones (and thus, presumably, those phones' owners) without needing to report each phone's location to a centralized server. Many of them use Bluetooth received signal strength indication (RSSI) as a proxy for distance. The core of these apps is the periodic broadcasting via Bluetooth of anonymized, frequently changing tokens (e.g., broadcasting every 100-270 ms and changing every 15 minutes [2] , [3] ). These tokens contain no information about the phone or its owner, and each phone stores the sent as well as received tokens. Thus, a user, Alice, who has tested positive can voluntarily publish all the tokens her phone sent during her potential transmission period (e.g., the past 14 days), such that another user, Bob, can download the tokens published by infected people and determine for himself whether he has encountered any of them, and for how long, by matching them to his own phone's received tokens. Bob can then voluntarily seek help from health authorities if his risk of exposure is considered high. Many apps of this kind have built-in risk-estimation models, and may recommend different measures (e.g., self-monitoring, commencing a 14day quarantine, or seeking medical attention) based on the risk levels they calculate.
As countries lift their lockdowns and reopen facilities, there is a rush to deploy these Bluetooth-based decentralized contact-tracing apps. As of October 2020, some 20 countries and six U.S. states have adopted them. The EU is currently planning a Europe-wide coronavirus-tracing network, based on new infrastructure that will enable data-sharing between national contact-tracing apps [4] . In addition, Apple and Google are planning to jointly release a new system called Exposure Notification Express, aimed at local health authorities that have not built their own apps [5] .
Nevertheless, few studies have gauged the effectiveness of these apps in practice, with existing evaluations of Bluetoothbased contact-tracing techniques relying on either simulation with mathematical models, or experiments involving either mobile devices only, or a very limited number of participants. Accordingly, the present work empirically investigates the effectiveness of Bluetooth-based contact tracing, in two phases. In Phase 1, we will examine whether Bluetooth RSSI can reliably predict distance in a controlled experimental setting. Then, in Phase 2, based on estimated distances between pairs of participants' phones over time, we compare detected proximity and contact events in a semi-controlled event: a real-world academic gathering, the ground truth of which will be carefully recorded. We recruited a total of 80 participants to use one Bluetooth-based contact-tracing app, which we modified from Covid-Watch-TCN, in our controlled (30 participants) and semi-controlled (50 participants) settings. Our modifications allowed us to collect ground-truth data and the phones' usage logs. After both experimental phases were completed, we also conducted a followup survey with participants to enrich the data we had already obtained. This paper will address the following questions:
• RQ1. Can the app reliably estimate the distance between its users based on Bluetooth RSSI under different crowd parameters, e.g., standing still vs. walking, with or without physical barriers, varying interpersonal distances, and the presence of jamming? • RQ2. How accurately does the app detect proximity and contact events in realistic crowd environments, as compared with the ground truth of such events? • RQ3. What are users' perceptions and experiences of using these apps?
Our experimental results confirmed that Bluetooth RSSI was unreliable for detecting proximity, and revealed that such inaccuracy worsened in crowded environments. This implies that this technique may be least useful when it is most needed, and fragile when confronted by low-cost jamming. Particularly, the app failed to capture the majority of the proximity events: Only 16 of 67 (24%) proximity events were detected by the app when setting a 2-meter distance threshold; and only 19 out of 67 (28%) were detected even when no distance threshold was set. In terms of user experience and perceptions, technical problems such as high energy consumption and phone overheating caused by the app were found to negatively influence users' willingness to adopt it. On the bright side, the app captured 63% of contacts lasting one hour in a room containing 50 participants and more than 150 other people. Divided by the operating systems, 80% of the Android devices were able to be discovered by both nearby Android and iOS devices in about an hour. This implies that this technique may still be useful for detecting coarse-grained contact events, for example, contacts within 20m that last for at least an hour. The sampled phone users said they were more willing to use the similar apps 1) when in crowded environments and 2) while contact tracers from health departments were also using them.
We summarize several representative privacy-preserving contact-tracing methods and previous studies that review them.
A. Privacy-preserving contact tracing for COVID-19
There are two major approaches to privacy-preserving contact tracing. The first adopts a decentralized design intended to minimize the amount of data that needs to be sent to a centralized server. The other applies cryptographic algorithms to protect sensitive user data.
a) Decentralized design: The former approach is exemplified by Safe Path [6] , DP3T [7] , and Covid-Watch [3] . Safe Path logs a user's movement routes in his/her mobile device and only exports that data to health authorities if that user is diagnosed with the virus. When this happens, the exported dataset is first redacted to ensure privacy, and then is broadcast to other users to allow them to self-determine their likelihood of having been exposed. Rather than capturing and storing route information, DP3T and Covid-Watch continuously broadcasts ephemeral Bluetooth identities, which are published to other relevant users in the event of a positive diagnosis, again to aid their individual risk calculations. Many of them have been deployed in the wild, but their effectiveness is yet to be investigated, partially due to the lack of empirical studies.
b) Cryptographic algorithms: The second, cryptographybased approach has been adopted by Private Set Intersection (PSI) [8] and TraceSecure [9] . PSI enables two parties to compute the intersection of their data in a privacy-preserving way, with only the common data values being revealed. The data used for computation contains only hashed location points, and location privacy can therefore be guaranteed. TraceSecure, on the other hand, incorporates a public-key based security protocol for message exchange and storage. An optional homomorphic encryption scheme can be used to further enhance data protection. While the cryptographic-based approach guarantees better privacy than the decentralized one, it is less deployable on consumer devices and existing infrastructure. The present study therefore focuses only on decentralized contact tracing, as being more feasible to deploy on the massive scale required for this purpose. c) Implementation: Among the decentralized contacttracing apps, a number of them use Bluetooth RSSI as a proxy for close proximity between devices, and therefore, between the owners of those devices. Early implementations such as the DP3T project and Covid-Watch had to confront Bluetooth interoperability issues between the two major mobile platforms, iOS and Android. In April 2020, the two major smartphone OS providers, Apple and Google, announced that they had formed a coalition to release APIs that would help contact-tracing apps work across iOS and Android devices, with the first such APIs appearing the following month. Android version 6.0 and iOS 13.7 and higher have supported the fundamental functions of Bluetooth contact tracing, including broadcasting and listening to tokens, at the OS level. To prevent abuse of these APIs, the companies restricted each country to one official app. An increasing number of national projects have adopted the GAEN APIs, including Switzerland's SwissCOVID, Italy's Immuni, and Germany's Corona-Warn-App. Since our use of it in our experiments, Covid-Watch has also now switched to using the GAEN APIs. We plan to use our proposed methodology to evaluate GAEN-API-based apps once the GAEN APIs are enabled in Taiwan. d) Privacy concerns: Tracing contacts by accessing devices' relative locations (e.g., using Bluetooth signal reception) is considered more protective of user privacy than capturing absolute locations (e.g., via GPS). Several prior studies have investigated privacy threats such as replay attacks and deanonymization attacks by state-level or resourceful adversaries [10] , [11] , or have proposed advanced cryptographic solutions to enhance privacy [12] . While detailed privacy concerns are beyond the scope of our empirical study, we feel it should be noted that privacy enhancement beyond a certain level is likely to degrade the detection accuracy and other aspects of the performance of contact-tracing apps. Our methodology and protocols used in this study will be useful in assessing whether the negative performance impacts of future privacy-enhancement efforts outweigh their benefits.
Broadcasting of anonymized tokens to nearby devices is fundamental to the implementation of privacy-preserving contact tracing. In contemporary smartphones, the most widely deployed techniques that support such broadcasting are Bluetooth and Wi-Fi. Of the two, however, developers prefer to use Bluetooth because it was originally designed to function within ad hoc networks, and because it has also been widely used for distance measurement and indoor positioning [13] , [14] . By reading an RSSI reported by a receiver, an application can estimate the distance between the receiving and sending devices; and indoor positions can be calculated based on three or more RSSIs from fixed-location broadcasters or beacons. a) RSSI-based distance estimation: The major drawback to RSSI-based distance estimation is variation in its measurement results, which can be caused by various environmental factors such as interference, emission power, and receiver sensitivity, all of which introduce noise. Studies [15] , [16] have shown that distances derived from RSSIs without calibration can be quite diverse, even in a controlled-experiment scenario with no human participants and with the same settings on all devices. Some have proposed to augment Bluetooth with other sensors to improve accuracy [14] . Nevertheless, we felt that Bluetooth-based distance estimation has the potential to provide helpful information to pandemic investigators, and tested its performance for this purpose with human subjects in controlled and uncontrolled settings, as explained in Section III.
b) Empirical evaluation of contact tracing: Although many Bluetooth-based contact-tracing apps have been deployed in the field, their effectiveness remains an open question due to the lack of ground-truth information. Prior to our present effort to help fill that gap, an empirical study [17] was conducted in April 2020 among a group of 48 soldiers in Germany. They were divided into five scenarios with different moving patterns, with at most 10 people in any one scenario. A follow-up report documented the experimental protocol and provided some preliminary analysis, but drew no clear conclusions and made no recommendations.
Leith and Farrell conducted a series of studies [18] to empirically measure RSSI between mobile phones indoors and outdoors, as well as on a bus and a tram, and considered factors that could affect such signal strength, including distance, phone orientation, and absorption and/or reflection by surroundings such as building walls or even human bodies. Their follow-up measurement study recruited five participants on a commuter bus to investigate the relationship between Bluetooth attenuation and distance in an environment prone to signal reflection. They made several recommendations for increasing the accuracy of Bluetooth-based contact tracing, such as leaving phones on tables instead of keeping them in bags or pockets. Our study considers scenarios involving much larger groups of participants as well as jamming devices, which allows us to simulate crowded scenarios and to observe issues that might only occur only or mostly in large groups of people, e.g., rapid battery depletion, interference, and interoperability problems between different phone models. In addition, we also collected users' feedback after using a Bluetooth-based contact-tracing app, particularly their perceptions and concerns regarding privacy and usability.
Since our focus is on Bluetooth-related issues, the following studies were also important to our thinking, despite being beyond the scope of our own research. Existing exposurenotification apps often feature fixed thresholds for identifying contact events and calculating exposure risks. For example, if a user has been in close proximity with a confirmed patient for a sufficiently long time (e.g., 15 minutes), that user will be warned of potential exposure by the app. Wilson et al. [19] proposed a calibrated measure of infection risk based on empirical measurements, and devised a risk-scoring system that aims to provide better quarantine recommendations. Some scholars have evaluated the effectiveness of contact tracing via mathematical modeling and simulation, and compared it against other countermeasures such as social distancing or lockdowns [20] , [21] . Our findings can provide more realistic parameters for contact tracing that can assist the refinement of such models and simulations.
Despite the security and privacy issues involved in the adoption of digital contact tracing, as of mid-October 2020, a considerable number of national and local governments have either already introduced such measures in the fight against COVID-19, or are planning to do so.
On the flipside, a relatively small number of governments have launched new human-based tracing services or announced improvements to existing ones, and remain on the fence regarding the adoption of digital contact tracing, repeatedly citing concerns about uptake rates and false positives/negatives. The UK, for instance, is about to launch a coronavirus app that enables users to report symptoms and book tests, but does not allow contact tracing [22] .
Several large-scale questionnaire surveys have been conducted to capture phone users' general perceptions toward contact-tracing mobile apps. A multi-country survey of Europe and North America has shown high user acceptance of downloading such apps (74.8%) [23] . However, the results of another survey, conducted in the U.S., suggest that support for the policy of encouraging use of these apps is relatively weak (42%), as compared to traditional measures; but also that the implementation of decentralized data storage helps increase acceptance [24] . A team in Jordan, meanwhile, reported that 71.6% of their respondents accepted the use of contacttracing technology, but only 37.8% actually used it [25] . It is noteworthy, therefore, that all of the respondents to our poststudy survey had actually experienced using such an app, and were thus able to provide meaningful, app-specific answers about their usage experience and privacy perceptions.
Our research design comprises four elements, as shown in Table I . These are: 1) modification to the Covid-Watch-TCN mobile application; 2) a controlled experiment with 30 participants; 3) semi-controlled experiment with 50 participants in the wild; and 4) a followup user survey administered to the participants from both experiments.
To evaluate the effectiveness of Bluetooth-based decentralized contact tracing, our experiments collected information that would help us reconstruct the ground truth required for conducting comparative analysis: e.g., the sender of each token, and the distance between each pair of participants. Some of this information was collected by the mobile app and reported to a backend server; some was pre-assigned based on our protocol; and some was derived from direct observation. This subsection describes how we customized and configured an open-source app, and the following two subsections present our protocol and observational approaches, respectively.
We modified the source code of Covid-Watch-TCN Exposure Notification App [3] , whose Android and iOS versions are both open-source and can be found on GitHub. We modified the version based on the TCN Coalition's implementation. Since our use of it in our experiments, Covid-Watch has also now switched to using the GAEN APIs, which nevertheless are not enabled in Taiwan. To minimize interference with the app's main functionality, we did not modify its token generation algorithm, where tokens are sent every 100 ms and changed every 15 minutes, but only locally logged data, and sent the logs back to our server at the end of each task. We manually inspected the app's logic to identify the Android or iOS system API calls that created or sent tokens, and inserted our logging code before such calls.
On starting up, our modified app prompted each participant to enter the unique ID that was assigned to him/her at the beginning of the experiment, and to log device information including the running operating system and phone model.
While running, the app logged all the sent and received tokens, along with RSSI values, timestamps, and phone battery status. Specifically, when a device transmitted a token, the app logged the sent token with the unique ID of the device, the current battery status, and a timestamp. When a token was received from another participant's device, on the other hand, the app logged that received token with measured RSSI and calculated distance; the unique ID of the device; and a timestamp. At the end of each task, the participants were asked to click on a "Submit" button to upload the logged information to our backend server. How the logged information was processed for further analysis will be explained in Section IV.
We assigned each participant a position and movement pattern (see Figure 2 ), such that the actual and Bluetooth-RSSI-estimated distances between each pair of participants can be calculated and compared at multiple time-points. a) Research site and participants: The controlled experiment, conducted in July 2020, comprised two scenarios: indoor and outdoor. The indoor scenario utilized an empty classroom 1, 411f t 2 (131m 2 ) in size, and the outdoor scenario, a covered patio measuring 5, 413f t 2 (503m 2 ). Both of them are 9.94f t (3.03m) in height.
We recruited 30 college students from our institutions. All the participants brought their personal mobile devices. Among these 30 devices, 14 were Android and the remainder, iOS. In both scenarios, we physically labeled each participant with a unique ID and marked the floor with tape. Before starting the experiment, we provided instructions to all participants explaining the research purpose and the overall experiment flow, and collected informed consent from all of them. Figure 1 provides an overview of the experiment settings and process.
b) Protocol: The indoor scenario was broken down into five sessions, and the outdoor one into three, as shown in Table II . Both scenarios included three sessions-i.e., sessions 1, 3, and 4 in the indoor scenario, and sessions 6, 7, and 8 in the outdoor scenario-that required the participants to hold their devices in hands and i) stand still, ii) equidistantly walk in a given area, or iii) gradually move closer to each other.
The other two sessions in the indoor scenario required the participants to stand still in a jammed environment characterized by continuous sending of useless data by Bluetooth beacons (Session 2), and near a wall that divided the participants into two groups (Session 5). To set up the jammed environment, we placed six RaspberryPi 3 model b devices in the same room. Each of these devices emitted a unique, useless data via Bluetooth every 20 ms, i.e., five times faster than a normal device. This emulates two cases: 1) the existence of a malicious user jamming the wireless channels by sending tokens at a higher frequency, and 2) a very crowded place containing an additional 30 (= 6 * 5) phones.
In all, the eight sessions comprised 28 one-minute tasks. The multiple tasks within a session were set up to test how Bluetooth signal propagation varied across devices 1) running different operating systems (i.e., Android, iOS, and mixed) and 2) held at different distances from one another (i.e., 0.5d, 1d, 1.5d, where d = 1.5meters).
• Operating systems. In sessions 1 and 6, where the participants stood still in both the indoor and outdoor scenarios, they were first grouped by their devices' operating systems. After completing the tasks (i.e., tasks 1-6 and The experiment was performed during a conference in an auditorium. Participants carried their devices when they were listening to a speech, having a group discussion, taking a tea break, etc.
•RSSI values •Calculated distances •Ground-truth records RQ3
User survey 24 respondents A follow-up survey was developed to reveal the participants' 1) technical problems encountered during both experiments, 2) willingness to use contact-tracing apps, and 3) privacy concerns. 18-23), the participants were taken out of these operatingsystem-based groups, and the experiment continued, any further groupings being randomized. • Distances. To simulate a real-world setting in which people may or may not maintain social distancing, we asked the participants in sessions 1, 2, 3, and 6 to keep a distance of 0.5d, 1.0d, or 1.5d (d=1.5 m) from each other. This yielded data that subsequently allowed us to compare the estimated distances generated by the app against the ground-truth distances, and thus to evaluate the accuracy of the app's proximity-detection techniques. Due to time limitations, sessions 4, 5, 7, and 8 were conducted with the participants attempting to maintain a single fixed distance of 0.5d, 1.0d, 1.0d, and 0.5d, respectively. At the beginning of each task, the participants were prompted to turn on the app, as well as their devices' Bluetooth and GPS. We required all users to turn on GPS during the experiment for consistency, because for Android version 6.0 and above, location services (e.g., GPS) need to be enabled when performing Bluetooth scanning [26] . During each task, a series of slides would show the participants their assigned positions and/or movement paths (Fig. 2) . Once a task ended, the participants were asked to manually upload the logged data to our backend server using the app.
Participants were allowed to move freely within a large auditorium, and proximity events and contact events were reconstructed based on video footage and on-site direct observation.
a) Research sites and participants: We additionally conducted a semi-controlled experiment at a cybersecurity summer school, which was held from July 27 through August 2, 2020 in a campus auditorium measuring roughly 4, 259f t 2 (396m 2 ) with a seating capacity of 250.
Out of the 216 attendees, we recruited 50 participants: 24 were using Android devices and the other 26, iOS ones. The participants could be told apart from the other attendees by their differently colored lanyards. b) Protocol: On the first day, we asked participants to turn on our research app, Bluetooth, and GPS for at least 90 minutes, starting around 6:30pm; and on the second day, this was increased to 150 minutes, starting around 5pm. At the points when the participants were asked to turn on the app, they might be sitting still and listening to a speech, divided into groups and taking part in discussions, or having a tea break outside the auditorium.
On the first day, one conference staff member was secretly assigned to be "the source of the virus". This individual turned on the app on his device, randomly passed by the participants, and recorded these actions with a GoPro camera so that after the experiment we could reconstruct his close contacts. In addition, to enhance the analyzability of the collected signals, four researchers observed and manually documented the ground truth of proximity events in the auditorium, including the IDs of the participants involved and the times when they occurred.
After the experiments, we administered a questionnaire. Its three sections covered 1) technical problems the participants had encountered during the experiments, 2) their attitudes toward the use of the contact-tracing app, and 3) their attitudes toward personal privacy. a) Section 1: Technical Problems: In this section, the respondents could choose to agree with any or all of the following seven statements: Phone overheating, Seriously increased energy consumption, App crash, Unstable receiving token, Phone performance negatively affected, Couldn't log in, and Other. We also asked a yes/no question regarding whether the respondents had encountered upload failure during the experiments. b) Section 2: Willingness to Use the Contact-tracing App: Section 2 aimed to capture how different technical factors and situations influenced the participants' willingness to use the contact-tracing app. Its questions were divided into two groups.
In the first, each of the seven answer options from Section 1 regarding technical problems was repeated, along with the question, Will this technical problem affect your willingness to use the app in the next six months? The respondent was then asked to select how much each problem s/he had selected would affect such willingness, on a three-point Likert-scale ranging from -2=gradually decrease to 0=not influenced, though an answer of N/A could also be given in place of a scaled response.
The second group of items in Section 2 asked the respondents to select how various non-technical conditions would affect their willingness to use the app over the following six months. These conditions were Regulation by law or my school; Social influence from my family or colleagues; Planning a trip domestically or abroad; Entering a crowd of more than 100 people; and Current use of such apps by epidemic investigators. These were rated on a five-point Likertscale ranging from -2=gradually decrease to +2=gradually increase, plus an N/A option. c) Section 3: Privacy Considerations: In the final section of the questionnaire, the respondents were first asked about when and why they usually turned on their devices' Bluetooth and GPS functions. Then, they rated the statement My data are secure and my privacy is protected while using the app on a five-point Likert scale ranging from -2=strongly disagree to +2=strongly agree, again with an N/A option. If a person's response revealed a negative attitude toward the app's privacy and security, i.e., was lower than 0, s/he would further be asked to select from among the following list of five datasecurity and privacy concerns: Data being tampered with; The app developer or associates may take advantage of security weaknesses; The app developer or associates may use my data for other purposes; and My identity past contacts, or past locations may be recognized.
We sent out the questionnaire to all 80 participants, but in fact this represented only 78 individuals, as two had participated in both experiments. Of these 78, 24 completed the questionnaire: a response rate of 30.8%.
This study was reviewed and approved in July 2020 by National Taiwan University's Research Ethics Office (equivalent to an Institutional Review Board in North America), and meets all criteria for minimal-risk research (#202006HS001).
We will open the participant instructions shortly after publication, so that other research teams can reuse our protocols or reproduce our research.
Based on the participants' devices' logs, we reconstructed a directed multigraph, on which a vertex represents a participant, and an edge from A to B represents a token sent by A and received by B. Each edge is labeled with a unique tuple (token, RSSI, timestamp) representing the corresponding token's RSSI value and timestamp. Because tokens are sent every 100ms and changed every 15 minutes, the same token may be seen multiple times and have different RSSI values and timestamps.
Tokens missing either sender or receiver information were removed. Missing sender information could have been caused by technical glitches (e.g., device malfunctioning, phone overheating and network congestion), while missing receiver information could have been caused by any of the same factors, or simply by no device having received them. Tokens with non-negative RSSI values or unrecognized sender/receiver IDs were also removed. In all, around 524,000 tokens, representing 86% of the total received, were removed.
Because the participants all used their own devices, it was not possible for us to determine the root causes of all technical glitches; nor can we be certain that they were not specific to our experiments. However, app-store reviews and news reports reveal that many similar apps have struggled to resolve similar glitches in real-world settings. Thus, it seems relatively unlikely that our settings and/or app modifications caused them.
During the controlled experiment, five of the Android smartphones encountered technical issues, and thus data collected from them were corrupted and discarded. The remaining 25 valid smartphones included nine Android and 16 iOS devices, and over the whole course of the experiment transmitted 700 unique tokens and received around 85,000 unique tuples of (token, RSSI, timestamp), all of which were included in the analysis described below.
Estimated distances between pairs of devices were calculated directly by the Covid-Watch-TCN app based on Bluetooth RSSI data.
The relation between measured RSSI and estimated distance, d, can be expressed as
where n is the environment factor, and A is the reference signal strength at 1m.
In both Android and iOS Covid-Watch-TCN apps, n is set to 2. The A value is determined based on the sender's transmission power level, encoded in Bluetooth tokens. When receiving a token, the app extracts the transmission power level, and determines the value of A according to what range that level falls within. Then, the app estimates based on the measured RSSI and A, using the equation shown above.
1) The Influence of Operating Systems: Figure 3 represents the transmission and reception statistics for all the devices used during the controlled experiment, classified by operating system. It shows that there was a significant difference between Android and iOS devices' token-transmission capabilities. Most of the Android devices were able to transmit tokens to both Android and iOS devices directly, while all but one of the iOS devices relied on nearby Android devices to broadcast tokens to others. However, two out of nine Android devices According to the Covid-Watch-TCN app's specifications, iOS versions 13.4 and older do not support discoverability between third-party iOS apps in the suspended or backgroundrunning state if the devices' screens are locked [3] . Therefore, an iOS device running version 13.4 or older should rely on an Android device as a relay to broadcast Bluetooth packets when running the app in the background. On the other hand, iOS devices running the app in the foreground exchange Bluetooth packets with one another directly.
Due to the iOS Bluetooth platform's reliance on relays from other devices, its RSSI and estimated-distance information cannot represent actual values. Therefore, we chose to focus only on directly transmitted tokens, i.e., Android-to-Android or Android-to-iOS, in our further analysis.
2) The Influence of Distance: Figure 4 illustrates the relationships of the estimated and true distances between each sender-receiver pair of Android devices in Session 1. In that session, the participants stood still in an indoor environment, and within each task were 0.5d, 1d, or 1.5d apart. It can be seen from the figure that the standard deviation of the estimated distance increased as the true distance increased, suggesting that Bluetooth signals attenuate during transmission and be- Fig. 5 : Estimated distance vs. ground truth, Session 1, phones with both systems running the app, both systems receiving come more easily influenced by radio noise. The correlation coefficient between true distance and estimated distance was 0.68. Figure 5 also indicates that the app tended to underestimate the distance between devices when the receiver was an Android one, potentially leading to high numbers of false-positive results. When the receiver was an iOS device, in contrast, the app tended to overestimate the distance, potentially leading to high numbers of false negatives.
3) The Influence of Background Radio Noise or Jamming: Next, we tested if and how background radio noise or jamming affected token reception and RSSI.
Our results indicated that Bluetooth was susceptible to packet drop due to jamming. The devices received fewer tokens when there were higher levels of background radio noise due to jamming. Looking at Fig. 6 , the number of received tokens was reduced in the presence of jamming. Additionally, the app had a lower accuracy when the participants were in a jammed environment. By applying a Wilcoxon Rank Test with α = 0.05, we confirm that the estimation errors across these two types of settings were drawn from two non-distinguishable distributions, implying that jamming did affect the app's ability to estimate distance.
Even the app itself became a source of noise when a large number of app users were gathered in the same place. Tasks 7-9 can be seen as more "noisy" conditions than Tasks 1-3, i.e., with 16 iOS devices placed between each pair of Android devices. As shown in Figure 5 , as compared to the previous three tasks' results, estimates of distance in "noisy" environments became more inaccurate in general, and even at distances of less than 2m. For these tasks, the correlation coefficient between true distance and estimated distance was just 0.26.
Additionally, in Tasks 1-3, the app recorded transmission events for 137 out of 170 device pairs, a rate of 81%. However, when more participants joined the experiment in Tasks 7-9, the number of recorded pairs dropped by to 111, or 65%; i.e., one-third of the senders were no longer able to successfully transmit tokens to receivers, due to the "noise" caused by iOS devices in the immediate vicinity.
On average, the phone battery dropped by 11.3% per hour in the uncontrolled experiment. As a reference, the per-hour drop was 14.3% in the semicontrolled experiment. We also observed a bigger battery drop in larger crowds: The per-hour drop for small and large groups were 10.4% and 29.6%, respectively.
Data collected in the semi-controlled experiment were also analyzed to evaluate the effectiveness of the Covid-Watch-TCN app in a spacious indoor environment. After data preprocessing, 41 valid devices were retained, including 24 Android and 17 iOS devices. Collectively, over the two days of the second experiment, they transmitted a total of 39,000 tokens and received 1.8 million.
A proximity event was deemed to have occurred if 1) two devices were detected by the app as having exchanged tokens at below a particular estimated-distance threshold, and 2) the time at which this exchange was recorded as occurring by the app was within 15 minutes before or after the time at which the same event was recorded by the researchers observing the conference and/or the GoPro videos.
A contact event between two devices was defined as a continuous proximity event lasting for a particular period, for example, 15 minutes. We further defined a strict contact event as one meeting the additional condition that every minute during the exposure period included at least one proximity event.
1) Proximity Detection and Contact Tracing: There were 67 proximity events documented by the four researchers during the experiment and extracted from the GoPro videos. If we set the distance threshold as 2m, commonly recommended as a social-distancing measure, only 16 of these proximity events were detected, implying a proximity detection rate of 24%. However, even when no distance threshold was set (i.e., no lower bound was placed on the RSSI value), the number of detected events only increased to 19, i.e., 28% of the total known to have occurred.
Under an exposure-duration rule of 15 minutes, meanwhile, the app could only detect 7.5% of the relevant contact events. Decreasing the exposure duration to 5 minutes and 1 minute resulted in only slight increases in its contact-detection rate: to 9.0% and 10.4%, respectively. And, when the strict contact rule was applied, the app failed to detect any contact events at all. The proximity and contact detection rates at various RSSI and contact-duration thresholds are shown in Figure 7 and Figure 8 , in which a measured signal with RSSI of -80 dB equates roughly to a 2m separation.
We also evaluated the proximity and contact detection rates of "the source of the virus". A total of 11 proximity events with this individual were recorded, two via direct observation and nine via screening of the GoPro videos. However, only four of these 11 proximity events were recorded by the app, despite none of them being fleeting. That is, exposure to the "virus" lasted for at least 5 minutes in each case, according to our observations. Moreover, among the 11 documented 5mincontact events involving the "virus", none were detected. 2) Exposure Duration: During the span of the semicontrolled experiment, each device sent at least one token to every other device, and received at least one, with an average of around 1,000 tokens per device being sent, and about 38,000 per device being received. Figure 9 illustrates trends in the number of unique recorded sender-receiver pairs over a 90-min period on the first day of the experiment. This number steadily increased over time and This indicates that, if one of its users remains in an indoor environment long enough, the app will be able to discover most of the nearby devices.
We further classified the recorded device pairs according to their operating systems, as shown in Figure 10 . Notice that most (> 80%) of the Android devices were able to be discovered by both nearby Android and iOS devices. As for iOS devices, 55% of them were discovered by iOS devices, with only 17% of them discovered by Android devices. Such results are consistent with our findings in the controlled experiment that limitations of the iOS Bluetooth platform could significantly influence the transmission capability of iOS devices.
Among the 24 questionnaire respondents, about half of them indicated that they had encountered at least one technical problem, including the app crashing(n=7), unstable receiving tokens (n=6), phone overheating (n=4), unexpectedly high energy consumption (n=3), login issues (n=3), and difficulty uploading their devices' token data (n=15). Nine out of the 15 participants tried re-uploading and successfully uploaded the data eventually.
Among these, the technical problem with the strongest negative influence on the respondents' intentions to use the app was high energy consumption, followed by phone overheating. We also found that, although marginally more respondents mentioned experiencing crash problems (n=13) than inefficient phone performance (n=12), the latter problem had a greater negative impact on their willingness to use the app.
The external conditions that the respondents selected most often as likely to affect their willingness to use the contacttracing app were entering crowds of 100 people or more (n=19); current use of the app by epidemic investigators (n=19); regulations (n=16); and domestic-trip planning (n=16). The top two of these conditions were also the most positively influential on the respondents' willingness to use the contacttracing app. Some of the participants even rated the influence of regulations on their willingness to use the app as negative.
Turning now to privacy issues, half our respondents stated that their habits regarding Bluetooth and GPS functions would not change in the wake of our experiments, whereas half said that they would. However, since our questionnaire did not ask about how/why Bluetooth and GPS usage impacted the respondents' privacy concerns, we cannot make any conclusions about this split in attitudes.
Surprisingly, only a small minority of our respondents expressed a belief that, due to using the focal contact-tracing app, their data (n=4) or privacy (n=5) might be unsafe, with the others either deeming them to be safe, or expressing no opinion on this matter. Among the minority, the top two datasecurity concerns cited were that the app developer might take advantage of security weaknesses (n=4), and that the app developer did not build in sufficient protections (n=3). Their top privacy concerns included their past routes being recognized (n=4), developers' associations using the data for other purposes (n=4), and the developer itself using the data for other purposes (n=3).
Our empirical findings have four important implications for Bluetooth-based contact tracing, each of which is discussed in turn below.
First and foremost, RSSI does not produce reliable estimates of physical distance. Our comparison between RSSI estimates and measured distances showed that the former often spanned -11.3 to 11.7 db, resulting in errors of 0.27 to 3.85 times of the ground truth. Unreliable distance estimates lead to inaccurate proximity or contact detection.
While increasing the number of samples might reduce this variance, we also observed system bias caused by contextual factors such as phone models and crowd size, in addition to those investigated in previous work, including wall geometry, phone orientation, and whether users were indoors or outdoors. These system biases would be hard to eliminate in the absence of extensive, detailed prior knowledge of the context in which contact tracing would need to occur.
Although the app was unreliable in estimating the distance between app users, information about whether they are in the same indoor location or not could still be useful to the broader contact-tracing process.
Another interesting observation was that variance increased with the density of the crowd. That is, given the same number of participants in the same room, variance was lower when the participants were farther away from each other. This could be due to the status of human bodies as obstacles, as well as to wireless channels becoming congested when all devices in the room are transmitting signals simultaneously.
In addition, the six Raspberry Pi devices that emitted tokens at a high rate in one session of our experiment, which we added to investigate possible jamming effects, had similar an impact to more crowded conditions. That is, all else being held equal, variance was higher in the more "noisy" environment that resulted from the inclusion of these extra devices.
Reducing the token broadcast frequency (e.g., from 100ms to 1s) in dense areas may alleviate packet loss due to interference but its effect on the proximity and contact detection remains to be investigated.
In all our experimental tasks, the participants' phone batteries drained quickly regardless of brand. This excessive consumption means that our research app would not be usable in real-world scenarios, even if people were willing to try. Some also complained that their phone overheated while running the app.
The reason for this extreme power consumption may be attributable to how Bluetooth is handled in our app. The early version of Covid-Watch-along with many other apps implemented before the release of the Google-Apple Exposure Notification (GAEN) API-did not have native access to Bluetooth, and had to use some hacks to bypass low-level restrictions. For example, iOS versions below 13.4 can only send tokens when either the sender or the receiver is running in the foreground. These hacks likely consume unnecessary resources, including energy. Although we were unable to test GAEN API-based apps, we anticipate that they will have better power efficiency.
Some participants also experienced app crashes or hangs, and thus could not broadcast or submit tokens. Although this was likely caused by our rapid development cycle and lack of testing on a variety of phone models and OSs, it is worth noting that similar issues have been reported by users of other contact-tracing apps.
To be effective, apps need to be interoperable and produce consistent results regardless of what OSs, phone models, app configurations and implementations are involved. Our experiments used the existing Android and iOS versions of the same app, and about half of our participants used Android, and the rest used iOS. To emulate realistic scenarios, we did not place any limits on the phone models involved, apart from a requirement that all support Bluetooth.
We found asymmetric results across phone models and versions. The differences we observed among phone models might be due to differences in Bluetooth chips, transceiver modules, and signal-processing methods, among other factors. This complicates interoperability by implying that each receiving phone may need to know the model and version of each sending phone if the app's detection accuracy is to be improved.
We observed that iOS devices tended to overestimate distances, while Android ones tended to underestimate them. The overestimation of distance by the iOS devices may have been caused by the calibration of the default reference RSSI value (i.e., A) at 1m across both versions of the Covid-Watch-TCN app. For the Android version of the app, there were only three possibilities for the value of A; and the iOS version had the same possible values as the Android version, except that its default value was greater by 10, i.e., Android is -67db and iOS -57db. This coarse ranges of transmission power levels could lead to inaccuracy in distance estimates.
Google and Apple's exposure-notification system [2] recommends that Android devices be calibrated to a typical iPhone according to their model designations. However, even with improved calibration to compensate biases due to inter-device differences, inaccuracies caused by environmental factors may be difficult to eliminate in the absence of prior knowledge of the context.
Our survey result may indicate that people's willingness to use contact-tracing apps is rooted in self-protection concerns and/or a public-spirited desire to aid epidemic investigation work, but that the enforced use of such apps might nevertheless provoke opposition.
However, it should be borne in mind that most of our participants in the second experiment were students with information-engineering backgrounds and an interest in cybersecurity, who may have been less likely to worry about datasecurity issues than an equivalent-sized sample of the general public.
Our study has the following limitations and thus may not be generalized to other settings. Additional studies are needed to address these limitations and to further improve Bluetoothbased contact tracing.
• We did not have access to the GAEN APIs developed by Google and Apple, and our evaluation was limited to a specific implementation (the TCN Coalition version of Covid-Watch). • We used just one RSSI-to-distance function in our experiment, so apps using different distance-estimation algorithms might produce different results. • Our logging code may introduce additional overheads.
• Our experiments were of short duration, and were not representative of the full range of real situations. • Falsely identified non-contacts (false positives) were not analyzed in our semi-controlled setting.
Digital contact tracing has strong potential to ease the burdens of traditional contact tracing by narrowing down the set of cases requiring further investigation. To achieve this potential, however, it is important to understand the its limits in real-world settings. In this study, we evaluated a type of digital contact tracing that preserves user privacy through Bluetooth technology and decentralized design. Based on our findings, we make the following recommendations, including some for future research.
When using existing/unmodified Bluetooth-based contacttracing apps, users should be aware of the inaccuracy of their estimates of distance and contact duration, and avoid making important decisions based solely on such estimates. Also, users should maintain social distancing even when these apps are on, not merely as an extra layer of protection, but also to enhance the accuracy of their proximity-detection function.
When it is possible to change their detection logic, we recommend using Bluetooth-based contact tracing for proximity detection that is coarse-grained (e.g., within 20m for an hour) rather than fine-grained (e.g., within 1m for 15 minutes, as a means of identifying social-distancing violations). Coarsegrained proximity detection would tend to have similar outcomes to digital approaches based on QR code scanning or cell tower history, insofar as it would capture the fact that a particular set of people were co-present with an infected person (e.g., attended the same convention or ate in the same restaurant), but preserve privacy.
When it is possible to modify their apps to collect additional information, designers should consider incorporating mechanisms to reduce false/missed detection and resource consumption. This could be achieved by taking additional samples or collecting additional contextual information to calibrate distance estimates and broadcast frequencies. An app might be able to collect some contextual information by itself, e.g., whether the phone is in an area dense with other phones, or whether it has been placed in a bag. Further studies are needed to evaluate the influence of such dynamic adjustment on detection rates, interoperability, and privacy. In addition, it might be helpful for apps to encode certain sender information in addition to maximum transmission strength level in its broadcasts, though such information should be narrowed down to a small number of common options to limit potential privacy abuse (e.g., tracking a particular device based on its unique maximum transmission strength). App designers may also wish to consider augmenting other sensing technologies such as ultrasound and Wi-Fi [14] , [27] . As well as facilitating the development of better distance-estimation functions, added contextual information should help those seeking to estimate infection probabilities.
On a broader picture, many more questions need to be answered before we can conclude: Can Bluetooth-based contact tracing help manual contact tracing? These include but are not limited to investigating other potential attacks and privacypreserving countermeasures, improving app adoption, handling panics and fatigues due to false positives, and estimating exposure risks based on collected contact information and epidemiological infectiousness models.