The outbreak of the SARS-CoV-2 virus in late 2019 and subsequent evolution of the COVID-19 disease has affected the world on an enormous scale. While hospitals are at the forefront of trying to mitigate the life-threatening consequences of the disease, practically all societal levels are dealing directly or indirectly with an unprecedented situation. Most countries -are at the time of writing this paper -in various stages of a lockdown. Schools and universities are closed or operate online-only, and merely essential shops are kept open.
At the same time, lockdown measures such as social distancing (e.g., keeping a distance of at least 1.5 meters from one another and only socializing with two people at most) might have a direct impact on people's mental health. With an uncertain outlook on the development of the COVID-19 situation and its preventative measures, it is of vital importance to understand how governments, NGOs, and social organizations can help those who are most affected by the situation. That implies, at the first stage, understanding the emotions, worries, and concerns that people have and possible coping strategies they use. Since a majority of online communication is recorded in the form of text data, measuring the emotions and worries around COVID-19 will be a central part of understanding and addressing the impacts of the COVID-19 situation on people. This is where computational linguistics can play a crucial role.
In this paper, we present and make publicly available a high quality, ground truth text dataset of emotions and worries around COVID-19. We report initial findings on linguistic correlates of emotions, topic models, and prediction experiments.
Tasks like emotion detection (Seyeditabari et al., 2018) and sentiment analysis (Liu, 2015) typically rely on labeled data in one of two forms. Either a corpus is annotated on a document-level (i.e., judging a document as positive, neutral, or negative) or individual n-grams are judged on their polarity (i.e., assigning a score to each n-gram on a word list). These annotations are done (semi) automatically (e.g., exploiting hashtags such as #happy) (Mohammad and Kiritchenko, 2015) or manually through third persons (Mohammad and Turney, 2010) . While these approaches are common practice and have accelerated the progress that was made in the field, they are limited in that they propagate a pseudo ground truth. This is problematic because, as we argue, the core aim of emotion detection is to make an inference about the authors emotional or cognitive state. The text as the product of an emotional state functions as a proxy for the latter. For example, rather than wanting to know whether a Tweet is written in a pessimistic tone, we are interested in learning whether the author of the text actually felt pessimistic.
The limitation inherent to third-person annotation, then, is that they might not be adequate measurements of the emotional state of interest. The solution, albeit a costly one, lies in ground truth datasets. Whereas real ground truth would requirein its strictest sense -a random assignment of people to experimental conditions (e.g., a group that is given a positive product experience vs. the opposite), variations that rely on self-reported emotions can also mitigate the problem. Datasets that rely on self-reports are the International Survey on Emotion Antecedents and Reactions (ISEAR) 1 , which asked participants to recall from memory situations that evoked a set of emotions. The COVID-19 situation is unique and calls for datasets that capture peoples affective responses to it while it is happening.
Several datasets mapping how the public responds to the pandemic have been made available. For example, tweets relating to the Coronavirus have been collected since March 11, 2020, yielding about 4.4 million tweets a day (Banda et al., 2020) . Tweets were collected through the Twitter stream API, using keywords such as 'coronavirus' and 'COVID-19'. Another Twitter dataset of Coronavirus tweets has been collected since January 22, 2020, in several languages, including English, Spanish, and Indonesian (Chen et al., 2020) . Further efforts include the ongoing Pandemic Project 2 which has people write about the effect of the coronavirus outbreak on their everyday lives.
This paper reports initial findings for the Real World Worry Dataset (RWWD) that captured the emotions and worries of UK residents at a point in time where the impact of the COVID-19 situation affected the lives of all individuals in the UK. The data were collected on the 6th and 7th
of April 2020, a time at which the UK was under lockdown (news, 2020), and death tolls were increasing. On April 6, 5,373 people in the UK had died of the virus, and 51,608 tested positive (Walker , now) . On the day before data collection, the Queen addressed the nation via a television broadcast (Guardian, 2020). Furthermore, it was also announced that Prime Minister Boris Johnson was admitted to intensive care in a hospital for COVID-19 symptoms (Lyons, 2020) . The RWWD is a ground truth dataset that used a direct survey method and obtained written accounts of people alongside data of their emotions and worries. As such, the dataset does not rely on third-person annotation but can resort to direct self-reported emotions. We present two versions of RWWD, each consisting of 2,500 English texts representing the participants' genuine worries about the Corona situation in the UK: the Long RWWD consists of texts that were open-ended in length and asked the participants to express their feelings as they wish. The Short RWWD asked the same people also to express their feelings in Tweet-sized texts. The latter was chosen to facilitate the use of this dataset for Twitter data research.
The dataset is publicly available. 3 .
We collected the data of n = 2500 participants (94.46% native English speakers) via the crowdsourcing platform Prolific 4 . Every participant provided consent in line with the local IRB. The sample requirements were that the participants were resident in the UK and a Twitter user. In the task, all participants were asked to indicate how they felt about the current COVID-19 situation using 9-point scales (1 = not at all, 5 = moderately, 9 = very much). Specifically, each participant rated how worried they were about the Corona/COVID-19 situation and how much anger, anxiety, desire, disgust, fear, happiness, relaxation, and sadness (Harmon-Jones et al., 2016) they felt about their situation at this moment. They also had to choose which of the eight emotions (except worry) best represented their feeling at this moment. All participants were then asked to write two texts. First, we instructed them to "write in a few sentences how you feel about the Corona situation at this very moment. This text should express your feelings at this moment" (min. 500 characters). The second part asked them to express their feelings in Tweet form (max. 240 characters). Finally, the participants indicated how well they felt they could express their feelings (in general/in the long text/in the Tweet-length text) and how often they used Twitter and whether English was their native language. The overall corpus size of the dataset was 2500 long texts (320,372 tokens) and 2500 short texts (69,171 tokens).
Below are two excerpts from the dataset:
Long text: I am 6 months pregnant, so I feel worried about the impact that getting the virus would have on me and the baby. My husband also has asthma so that is a concern too. I am worried about the impact that the lockdown will have on my ability to access the healthcare I will need when having the baby, and also about the exposure to the virus [...] There is just so much uncertainty about the future and what the coming weeks and months will hold for me and the people I care about.
Tweet-sized text: Proud of our NHS and keyworkers who are working on the frontline at the moment. I'm optimistic about the future, IF EVERYONE FOLLOWS THE RULES. We need to unite as a country, by social distancing and stay in.
We excluded nine participants who padded the long text with punctuation or excessive letter repetitions. The dominant feelings of participants were anxiety/worry, sadness, and fear (see Table  1 ). For all emotions, the participants' self-rating ranged across the whole spectrum (from "not at all" to "very much"). The participants' self-reported ability to express their feelings in the long text (M = 7.12, SD = 1.78) was larger than that for short texts (M = 5.91, SD = 2.12), Bayes factor > 1e + 96.
We correlated the self-reported emotions to matching categories of the LIWC2015 lexicon (Pen- , 2015) . The overall matching rate was high (92.36% and 90.11% for short and long texts, respectively). Across all correlations, we see that the extent to which the linguistic variables explain variance in the emotion ratings (indicated by the R 2 ) is larger in long texts than in Tweetsized short texts (see Table 2 ). There are significant positive correlations for all affective LIWC variables with their corresponding self-reported emotions (i.e., higher LIWC scores accompanied higher emotion scores, and vice versa). These correlations imply that the linguistic variables explain up to 10% and 3% of the variance in the emotion ratings for long and short texts, respectively. The LIWC also contains categories intended to capture people's concerns, which we correlated to the self-reported worry. Positive (negative) correlations would suggest that the higher (lower) the worry score of the participants, the larger their score on the respective LIWC category. We found no correlation between the categories "work", "money" and "death" suggesting that worry was not associated with these categories. Significant positive correlations emerged for long texts for "family" and "friend": the more people were worried, the more they spoke about family and -to a lesser degree -friends.
We constructed topic models for both the long and short texts separately using the stm package in R (Roberts et al., 2014a) . The text data were lowercased, punctuation, stopwords and numbers were removed, and all words were stemmed. For the long texts, we chose a topic model with 20 topics as determined by semantic coherence and exclusivity values for the model (Mimno et al., 2011; Roberts et al., 2014b,a) . Table 3 shows the five most prevalent topics with ten associated frequent terms for each topic (see online supplement for all 20 topics). The most prevalent topic seems to relate to following the rules related to the lockdown. In contrast, the second most prevalent topic appears to relate to worries about employment and the economy. For the Tweet-sized texts, we selected a model with 15 topics. The most common topic bears a resemblance to the government slogan "Stay at home, protect the NHS, save lives." The second most prevalent topic seems to relate to calls for others to adhere to social distancing rules.
We used linear regression models to predict the reported emotional values (i.e., anxiety, fear, sadness, worry) based on text properties. Specifically, we applied regularised ridge regression models using TFIDF and part-of-speech (POS) features extracted from long and short texts separately. TFIDF features were computed based on the 1000 most frequent words in the vocabularies of each corpus; POS features were extracted using a predefined scheme of 53 POS tags in spaCy 5 .
We process the resulting feature representations using principal component analysis and assess the performances using the mean absolute error (MAE) and the coefficient of determination R 2 . Each experiment is conducted using five-fold crossvalidation, and the arithmetic means of all five folds are reported as the final performance results. Table 4 shows the performance results in both long and short texts. We observe MAEs ranging between 1.26 (worry with TFIDF) and 1.88 (sadness with POS) for the long texts, and between 1.37 (worry with POS) and 1.91 (sadness with POS) for the short texts. We furthermore observe that the models perform best in predicting the worry scores 5 https://spacy.io for both long and short texts. The models explain up to 16% of the variance for the emotional response variables on the long texts, but only up to 1% on Tweet-sized texts.
This paper introduced the RWWD as a ground truth dataset as a resource to measure emotional responses to the Corona pandemic. We reported initial findings on the linguistic correlates of emotional states, used topic modeling to understand what people in the UK are concerned about, and ran prediction experiments to infer emotional states from text using machine learning. These analyses provided several core findings. (1) Some emotional states correlated with word lists made to measure these constructs, (2) longer texts were more useful to identify patterns in language that relate to emotions than shorter texts, (3) Tweet-sized texts served as a means to call for solidarity during lockdown measures while longer texts revealed peoples worries, and (4) preliminary regression experiments indicate that we can infer from the texts the emotional responses with an absolute error of 1.26 on a 9-point scale (14%).
Affective reactions to the Coronavirus were obtained through self-reported scores. When we used psycholinguistic word lists that measure these emotions, we found weak, positive correlations. The lexicon-approach was best at measuring anger, anxiety, and worry and did so better for longer texts than for Tweet-sized texts. In behavioral and cognitive research, small effects (here: a maximum of 10.63% of explained variance) are the rule rather than the exception (Gelman, 2017; Yarkoni and Westfall, 2017) . It is essential, however, to interpret them as such. If 10% of the variance in the anxiety score is explained through a linguistic measurement, 90% are not. An explanation for the imperfect correlations -aside from random measurement error -might lie in the inadequate expression of someone's felt emotion in the form of written text. The latter is partly corroborated by even smaller effects for shorter texts, which may have been too short to allow for the expression of one's emotion. 
Long texts 9.52 people, take, think, rule, stay, serious, follow, virus, mani, will 8.35
will, worri, job, long, also, economy, concern, impact, famili, situat 7.59 feel, time, situat, relax, quit, moment, sad, thing, like, also 6.87 feel, will, anxious, know, also, famili, worri, friend, like, sad 5.69 work, home, worri, famili, friend, abl, time, miss, school, children 10.70 stay, home, safe, live, pleas, insid, save, protect, nhs, everyone 8.27 people, need, rule, dont, stop, selfish, social, die, distance, spread 7.96 get, can, just, back, wish, normal, listen, lockdown, follow, sooner 7.34 famili, anxious, worri, scare, friend, see, want, miss, concern, covid 6.81 feel, situat, current, anxious, frustrat, help, also, away, may, extrem 
Prevalent topics in our corpus showed that people worry about their jobs and the economy, as well as their friends and family -the latter of which is also corroborated by the LIWC analysis. For example, people discussed the potential impact of the situation on their family, as well as their children missing school. Participants also discussed the lockdown and social distancing measures. In the Tweet-sized texts, in particular, people encouraged others to stay at home and adhere to lockdown rules in order to slow the spread, save lives and/or protect the NHS. Thus, people used the shorter texts as a means to call for solidarity, while longer texts offered insights into their actual worries.
While there are various ways to select the ideal number of topics, we have relied on assessing semantic coherence of topics and exclusivity of topic words. Since there does not seem to be a consensus as to the best practice for selecting topic numbers, we encourage others to examine other approaches or models with varying numbers of topics.
Prediction experiments reveal that ridge regression models can be used to approximate emotional responses to COVID-19 based on encodings of the textual features extracted from the participants' statements. Similar to the correlational and topic modeling findings, there is a stark difference between the long and short texts: the regression models are more accurate and explain more variance for longer than for shorter texts. Additional experiments are required to investigate further the expressiveness of the collected textual statements for the prediction of emotional values.
The current analysis leaves several research questions untouched. First, to mitigate the limitations of lexicon-approaches, future work on inferring emotions and worries around COVID-19 could expand on the prediction approach (e.g., using binary classification and different feature sets and models). Carefully validated models could help to provide the basis for large scale, real-time measurements of emotional responses. Of particular importance is a solution to the problem hinted at in the current paper: the shorter, Tweet-sized texts contained much less information, had a different function, and were less suitable for predictive modeling. With much of today's stream of text data coming in the form of (very) short messages, it is important to understand the limitations of using that data and worthwhile examining how we can better use that information.
Second, with a lot of research attention paid to readily available Twitter data, we hope that future studies also focus on non-Twitter data to capture emotional responses of those who are underrepresented (or non-represented) on social media but are at heightened risk.
Third, future research may focus on manually annotating topics to precisely map out what people worry about with regards to COVID-19. Several raters could assess frequent terms for each topic, then assign a label. Then through discussion or majority votes, final topic labels can be assigned to obtain a model of COVID-19 real-world worries.
This paper introduced the first ground truth dataset of textual emotional responses to COVID-19. Our findings highlight the potential of inferring concerns and worries from text data but also show some of the pitfalls, in particular, when using concise texts as data. We encourage the research community to use the dataset so we can better understand the impact of the pandemic on people's lives.