  Contact tracing has been globally adopted in the fight to control the
infection rate of COVID-19. Thanks to digital technologies, such as smartphones
and wearable devices, contacts of COVID-19 patients can be easily traced and
informed about their potential exposure to the virus. To this aim, several
interesting mobile applications have been developed. However, there are
ever-growing concerns over the working mechanism and performance of these
applications. The literature already provides some interesting exploratory
studies on the community's response to the applications by analyzing
information from different sources, such as news and users' reviews of the
applications. However, to the best of our knowledge, there is no existing
solution that automatically analyzes users' reviews and extracts the evoked
sentiments. In this work, we propose a pipeline starting from manual annotation
via a crowd-sourcing study and concluding on the development and training of AI
models for automatic sentiment analysis of users' reviews. In total, we employ
eight different methods achieving up to an average F1-Scores 94.8% indicating
the feasibility of automatic sentiment analysis of users' reviews on the
COVID-19 contact tracing applications. We also highlight the key advantages,
drawbacks, and users' concerns over the applications. Moreover, we also collect
and annotate a large-scale dataset composed of 34,534 reviews manually
annotated from the contract tracing applications of 46 distinct countries. The
presented analysis and the dataset are expected to provide a baseline/benchmark
for future research in the domain.
