Eight million people have been confirmed positive of COVID-19 across 110 countries as of mid-June 2020, and the death toll has reached close to 435,000 [1] . The widespread utilization of social media, such as Twitter, accelerates the process of exchanging information and expressing opinions about public events and health crises [2] [3] [4] [5] . COVID-19 is one of the trending topics on Twitter since January 2020 and has continued to be discussed to date. Since quarantine measures have been implemented across most countries (e.g., the Shelter-in-Place order in the United States), people have been increasingly relying on different social media platforms to receive news and express opinions. Twitter data are valuable in revealing public discussions and sentiments to interesting topics, and real-time news updates in global pandemics, such as H1N1 and Ebola [6] [7] [8] [9] . Chew and Eysenbach's study shows that Twitter can be used for real-time "infodemiology" studies, a source of opinions for health authorities to respond to public concerns [6] . In the current COVID-19 pandemic, many government officials worldwide are using Twitter, as one of the main communication channels, to regularly share policy updates and news related to COVID-19 to the general public [10] .
Since the COVID-19 outbreak, a growing number of studies have collected Twitter data to understand the public responses to, and discussions around COVID-19 [11] [12] [13] [14] [15] [16] . For instance, Abd-Alrazaq and colleagues adopt topic modeling and sentiment analysis to understand main discussion themes and sentiments around COVID-19, using Tweets collected between February 2 and March 15, 2020 [11] . Budhwani and Sun compare Tweets discussions before and after March 16, 2020, when President Trump tweeted the "Chinese virus," and find a significant increase use of "Chinese virus" in people's Tweets across many states in the United States [14] . Mackey and colleagues analyze about 3,465 tweets collected between March 2 and 20, 2020, using a topic model to explore users' self-reported experiences with COVID-19 and related symptoms [16] . Ahmed and colleagues conduct social network analysis and content analysis of the collected Tweets between March 27 and April 4, 2020 , to understand what may have driven the misinformation that linked 5G towers in the United Kingdom to the COVID-19 pandemic [12] . As conversations on Twitter continue to take place and evolve, it is worth continuing to use tweets as a source of data to track and understand what the salient topics discussed on Twitter in response to the COVID-19 pandemic and track their changes in different time periods are.
To expand the literature on public reactions to the COVID-19, the present study aims to examine the public discourses and emotions about the COVID-19 pandemic by analyzing more than 4 million Tweets collected between March 7 and April 21, 2020.
We used a purposive sampling approach to collect COVID-19 related Tweets published between March 7 and April 21, 2020. Our Twitter data mining approach followed the pipeline displayed in Figure 1 . Data preparation included three steps (1) sampling, (2) data collection, and (3) preprocessing the raw data. The data analysis stage included unsupervised machine learning, sentiment analysis, and qualitative method. The unit of analysis was each message-level tweet.
Unsupervised learning is one approach in machine learning, and used to examine data for patterns, and derives a probabilistic clustering based on the text data. We chose unsupervised learning because it is commonly used when existing studies have little observations or insights of the unstructured text data [17] . Since a qualitative approach has challenges analyzing large-scale Twitter data, unsupervised learning allows us to conduct exploratory analyses of large text data in social science research. In the present study, we first employed an unsupervised machine learning approach to identify salient latent topics. Using the topics, we used a qualitative approach to develop themes further, as a qualitative approach allows a deeper dive into the data, such as through manual coding and inductively developing themes based on the latent topics generated by machine learning algorithms.
We used a list of COVID-19 related hashtags as search terms to fetch Tweets, such as "#coronavirus," "#2019nCoV," "#COVID19," "#coronaoutbreak," and "#quarantine," (Appendix Figure 1 , we used Python to clean the raw data. The process was as follows [18] , (1) We removed the hashtag symbol, @users, and URLs from the Tweets in the dataset.
(2) We removed non-English characters (non-ASCII characters) because the present study focuses on Tweets in English.
(3) We removed special characters, punctuations, and stop-words listed in [19] from the dataset as they do not contribute to the semantic meanings of the messages.
Latent Dirichlet Allocation (LDA) [20] is one of the widely used unsupervised machine learning approaches allowing researchers to analyze unstructured text data (e.g., Twitter messages). Based on the data itself, the algorithm produces frequently mentioned pairs of words, the pairs of words co-occur together, and the latent topics and their distributions over topics in the document [21] .
Existing studies have indicated the feasibility of using LDA in identifying the patterns and themes of the Tweets texts related to COVID-19 [11 22] .
To triangulate and contextualize findings from the LDA model, we employed a qualitative approach to develop themes further. Specifically, using Braun and Clarke's [23] six steps of thematic analysis: (1) getting familiar with the keyword data, (2) generating initial codes, (3) searching for themes, (4) reviewing potential themes, (5) defining themes, and (6) reporting. Since the thematic approach relies on human interpretation, a process that can be significantly influenced by the personal understanding of the topics and a variety of bias, we had two team members conduct the first five steps independently. Then, the two members reviewed all identified themes and resolved disagreements together. Finally, we finalized themes corresponding to each one of the 13 topics.
We used sentiment analysis, a natural language processing (NLP) approach, to classify the main sentiments of a given twitter message, such as fear and joy [24] . In the study, we used the NRC Emotion Lexicon, which consisted of eight primary emotions: anger, anticipation, fear, surprise, sadness, joy, disgust, and trust [25] . We followed the four steps to calculate the emotion index for each twitter message, including (1) removing articles, pronouns (e.g., "and," "the," or "to"), (2) applying a stemmer by removing the predefined list of prefixes and suffixes (e.g., "running" after stemming becomes "run") [26] , and (3) calculating the emotion index. We only kept one emotion with the maximum matching counts if one sentence has multiple emotions; and (4) calculating the scores for each eight-emotion type. We discussed the four steps in detail in the previous study [18] .
A total of four million (n=4,196,020) Tweets consisted of our final dataset after pre-processing all raw data. We identified the most popular tweeted bigrams (pairs of words) related to COVID-19.
Bigrams captured "two concessive words regardless of the grammar structure and semantic meaning and may not be self-explanatory" [27] , including "covid 19," "stay home," "social distancing," "new cases," "don't know," "confirmed cases," "home order," "New York," "tested positive," "death toll," and "stay safe." Popular unigrams included virus, lockdown, quarantine, people, new, home, like, stay, don't, and cases. We presented the most popular unigrams and bigrams related to COVID-19 in Table 1 and visualized them using the word clouds in figure 3 and figure 4. 
Our approach, Latent Dirichlet Allocation (LDA), produced frequently co-occurred pairs of words related to COVID-19 and organized these co-occurring words into different topics. LDA allowed researchers to manually define the number of topics (e.g., ten topics, twenty topics) that we would like to generate. Consistent with the previous studies, we used the coherence model -gensim [28] to calculate the most appropriate number of topics based on the specific data itself. For this dataset, the number of topics (n=13) returned by LDA had the highest coherence score as well as the smallest topic number. For example, the numbers of topics (n=19 or n=20) had higher coherence scores than the number of topics (n=13), but they represented larger topic numbers, shown in Table 2 . For example, Topic 3 had the highest distribution (8.87%) among all 13 common latent topics. The bigrams were associated with Topic 3, including "tested positive," "coronavirus outbreak," "New York," "shelter place," and "mental health." These pairs of words frequently co-occurred together, and therefore the LDA model assigned them to the same topic. 
We presented the results of the sentiment analysis for each of the thirteen latent topics in Table 3 .
We also ran a one-tailed z test to examine if each of the eight emotions is statistically significantly different across topics. The p-value smaller than .01 was set as a threshold for significance. For example, statistical significance for anticipation in Topic 2 indicated that it was very likely (p < .001) that the anticipation emotion is more prevalently expressed in Topic 2 (21.7%) than all other topics. 13.20%*** 20.90%*** 3.30%*** 15.10% 4.20% 3.30%*** 2.20%*** 13.30%*** 5
12.40%*** 23.80%*** 2.60%*** 14.30% 4.30% 3.50%*** 2.10% 13.40%*** 6
13.10%*** 22.50%*** 2.40% 13.40% 4.60%*** 3.50%*** 3.00%*** 12.80%*** 7
12.50%*** 21.90%*** 2.50%*** 17.00%*** 3.70% 3.20%*** 3.30%*** 13.10%*** 8
13.80%*** 20.70%*** 2.40% 16.50%*** 3.80% 3.10%*** 2.40%*** 12.10%*** 9
12.50%*** 20.70%*** 2.80%*** 15.50% 7.90%*** 3.40%*** 2.40%*** 12.30%*** 10 14.60%*** 17.40% 3.00%*** 18.80%*** 3.30% 3.30%*** 1.90% 11.30% 11 11.80% 20.60%*** 2.50%*** 15.50%*** 6.00%*** 3.70%*** 2.70%*** 11.90%*** 12
12.50%*** 21.40%*** 2.80%*** 17.90%*** 4.20% 3.30%*** 2.60%*** 14.20%*** 13 13 .30%*** 20.80%*** 2.60%*** 14.80% 4.30% 4.20%*** 3.10%*** 11.50%*** Notes: The sum of the percentage under each topic is not equal to 100%. The rests are either neutral or other emotions. p-value from Z-test, *** p< .001 
The qualitative content analysis approach allowed users to categorize these topics into different distinct themes. Two team members discussed these bigrams and generated Tweets samples in each topic and then categorized the identified 13 topics into 5 different themes. To protect the privacy and anonymity of the Twitter users, we did not present any user-related information, such as users' twitter handles or other identifying information; therefore, sample Tweets shown in Table   4 were excerpts drawn from original Tweets. "Worldwide it is now 182,726.." "New Zealand Prime Minster Jacinda Ardern says the government will partially relax its lockdown in a week, as a decline in …"
We organized thirteen topics into five themes, including "public health measures to slow the spread of COVID-19 (e.g., facemasks, test kits, vaccine)," "social stigma associated with COVID-19 (e.g.,
Chinese virus, Wuhan virus)," "coronavirus news cases and deaths (e.g., new cases, deaths),"
"COVID-19 in the United States (e.g., New York, protests, task force)," and "coronavirus cases in the rest of the world (e.g., UK, global issue)." For example, theme "public health measures to slow the spread of COVID-19" included relevant topics of "facemasks," "quarantine," "test kits,"
"lockdown," "safety," "a need for the vaccine," and "U.S.'s shelter-in-place." In addition, "home quarantine," "self-quarantine" were two of the most commonly co-occurred words under topic quarantine.
In this study, we address the problem of public discussions and emotions using COVID-19 related messages on Twitter. Twitter users discuss five main themes related to COVID-19 Between March 7 to April 21, 2020. Topic modeling of the Tweets is effective in providing insights about coronavirus topics and concerns on Twitter. The results show several essential points. First, the public is using a variety of terms referring to COVID-19, including "virus," "COVID 19," "coronavirus," "corona virus." In addition, coronavirus has been referred to as the "China virus" that can create stigma and harm efforts to address the COVID-19 outbreak [14] . Second, discussions about the pandemic in New York are salient, and its associated public sentiments are anger. Third, public discussions about the Chinese Communist Party (PPC) and the spread of the virus emerge as a new topic, which is not identified in a previous study [18] , suggesting the connection between the COVID-19 and politics is increasing to be circulating on Twitter as the situation evolves. Fourth, public sentiment on the spread of coronavirus is anticipation for the potential measures that can be taken and followed by a mixed feeling of trust, anger, and fear.
Results suggest that the public is not surprised by the rapid spread of growth. Fifth, the public reveals a significant feeling of fear when they discuss the coronavirus crisis and deaths. Lastly, results show that trust for the authorities no longer remain as a prominent emotion when Twitter users discuss COVID-19, which is different than an early study [18] showing that people hold trust for public health authorities based on a sample of Tweets posted from January 20 to March 7, 2020.
Our findings are consistent with previous studies using social media data to assess the public health response and sentiments for COVID-19, and suggest that public attention has been focusing on the following topics since January 2020, including
(1) the confirmed cases and death rates [11, 18, 30] ;
(2) preventive measures [11, 18, 30] ;
(3) health authorities and government policies [10, 18] ;
(4) an outbreak in New York [18] ;
(5) COVID-19 stigma by referencing the coronavirus as the "Chinese virus" [11, 14] .
(6) negative psychological reactions (e.g., fear) or mental health consequences [11, 31, 32] Compared with the study examining public discussions and concerns for COVID-19 using Twitter data from January 20 to March 7, 2020, we find that several salient topics are no longer popular, 
First, future research could further explore public trust and confidence in existing measures and policies, which is essential. Compared to prior work, our study shows that Twitter users reveal a feeling of joy when talking about "herd immunity." We also find sentiments of fear and anticipation related to topics of quarantine and shelter-in-place. In addition, future studies could evaluate how government officials (e.g., President Trump) and international organizations (e.g., WHO) deliver and convey messages to the public, and its impact on the public opinions and sentiments. Third, future studies could examine the spread of anti-Chinese/Asian sentiments social media and how people use social media platforms to resist and challenge COVID-19 stigma.
Fourth, our findings do not show that misinformation during the COVID-19 pandemic is a prominent theme. An existing study shows that 25% (n=153) of sampled Tweets contained misinformation [35] . They also find that the term "COVID-19" has lower rates of misinformation than that associated with the terms of "#2019_ncov" and "Corona." Future research is suggested in the investigation of misinformation and how it expands on social media. Finally, the study finds that trust is no longer prominent when people tweet about confirmed cases and deaths. Instead, fear has replaced trust to be the dominant emotion. Future research is suggested to examine the changes in trust over time.
The study shows that Twitter data and machine learning approaches can be leveraged for infodemiology study by studying the evolving public discussions and sentiments during the COVID-19 pandemic. Our findings facilitate an understanding of public discussions and concerns about COVID-19 pandemic among Twitter users between March 7 and April 21, 2020. Several topics were consistently dominant on Twitter, such as "the confirmed cases and death rates," "preventive measures," "health authorities and government policies," "COVID-19 stigma," and "negative psychological reactions (e.g., fear)." As the situation evolves rapidly, new salient topics emerge accordingly. Anticipation for the potential measures that can be taken to stop the spread of COVID-19 is still significantly prevalent across all topics. However, Twitter users reveal fear when tweeting about COVID-19 new cases or death rather than trust [18] . Real-time monitoring and assessment of the Twitter discussion and concerns can be promising for public health emergency responses and planning. Hearing and reacting to real concerns from the public can enhance trust between the healthcare systems and the public as well as prepare for a future public health emergency.