Synchronizing Approach in Designing Annotation Guidelines for Multilingual Datasets: A COVID -19 Case Study Using E nglish and J apanese Tweets (2024)

Abstract: The difference in culture between the U.S. and Japan is a popular subject for Western vs. Eastern cultural comparison for researchers. One particular challenge is to obtain and annotate multilingual datasets. In this study, we utilized COVID-19 tweets from the two countries as a case study, focusing particularly on discussions concerning masks. The annotation task was designed to gain insights into societal attitudes toward the mask policies implemented in both countries. The aim of this study is to provide a practical approach for the annotation task by thoroughly documenting how we aligned the multilingual annotation guidelines to obtain a comparable dataset. We proceeded to document the effective practices during our annotation process to synchronize our multilingual guidelines. Furthermore, we discussed difficulties caused by differences in expression style and culture, and potential strategies that helped improve our agreement scores and reduce discrepancies between the annotation results in both languages. These findings offer an alternative method for synchronizing multilingual annotation guidelines and achieving feasible agreement scores for cross-cultural annotation tasks. This study resulted in a multilingual guideline in English and Japanese to annotate topics related to public discourses about COVID-19 masks in the U.S. and Japan.

## introduction the close political bond and distinct cultural viewpoints have resulted in the u.s.-japan comparison being a frequently studied topic among researchers interested in cultural contrasts and variations in western and eastern societal conventions. this extends to how they behave in daily life and how they responded to major world events, such as the pandemic, making it a great target for a multilingual annotation case study. in the early stages of covid-19, many health personnel and epidemiologists advocated the importance of mask in curbing the infection @xcite @xcite . however, the u.s. and japan displayed contrasting attitudes regarding policy implementation to control the spread of the disease and adherence to mask mandates, as demonstrated by their respective governments and citizens @xcite @xcite . this was also observed by @xcite , who summarized the general use of masks from several countries, including the u.s. and japan, and suggested further measure to improve mask effectiveness. while the general population recognized the importance of mask in the u.s., whether they actually wore them largely related to their demographics and their beliefs about societal value @xcite . on the other hand, wearing masks was common in japan even before covid-19, as observed in the majority of respondents in @xcite who had no issue with wearing masks. further information about how the two countries reacted to covid-19 mask-wearing mandate can also be observed from social media. a study by @xcite found that mask mandates and mask-wearing, as obtained from geo-tagged twitter (now x) images, had a strong association in the u.s. tweets also provided insights on public opinion about mask-wearing and their reasons for not wearing any @xcite . another study identified mask as one of the most frequently used words in tweets from korea and japan @xcite , and it also appeared in the list of top concerns expressed through social media during the pandemic in japan @xcite . undoubtedly, the vast number of social media posts is often considered a valuable resource for understanding, analyzing, and even informing policymakers about societal perceptions or attitudes toward certain events. however, a significant challenge for cross-cultural comparison studies is obtaining a comparable data from two different languages with different cultural backgrounds. to get such kind of data, we applied a synchronizing approach in annotating our data. given the differing responses to mask mandates as a measure against covid-19 in the u.s. and japan, we are interested in exploring the variations in the public responses. for instance, we are interested in whether individuals actually wear masks despite their stated stance against masking. however, this presents a complex challenge in the annotation process, hence the decision to split the question into a simpler form. annotation plays a crucial role in many natural language processing (nlp) research, as the annotation results will provide a foundation for future model training work. we adopted a synchronized approach in which we borrowed linguistic nuances and annotation insights from both languages to design and refine the guidelines to ensure the guidelines achieve a desirable level of agreement, accuracy, and effectiveness in capturing issues related to covid-19 masking we aim to assess in both cultures. the multilingual guidelines can be accessed as supplementary material of this paper foot_0 . ## related work utilizing human annotated data in training machine learning classifiers was a practice often employed in twitter studies @xcite . previous covid-19 studies have also employed such a method, as observed in @xcite who used annotated tweets for covid-19 tracking to identify potential tweets reporting covid-19 cases in the u.s. research involving multiple languages, especially in tweets, is often done in comparative studies where two or more target populations are using different languages. for example, @xcite compared stance detection in two languages (catalan and spanish), designing their guidelines based on the approach introduced by @xcite , with annotations performed by two annota-tors who are skilled in both languages. another study by jahan (2020) completed a task of offensive language comparison of tweets in five different languages using the english translation of the tweets. translating all the tweets into english seems feasible to a degree and can benefit from a powerful english-based language model. similarly, @xcite studied covid-19 vaccination attitudes using translated tweets from four western european countries, resulting in a dataset with potential use for covid-19 analysis. however, using translation tools might result in the inaccurate conveyance of the sentiment @xcite . analyzing the tweet in its native language ensures that the annotators get to comprehend the original meaning, as well as the cultural nuances of the texts, which could easily get lost through translation. considering the positive and negative aspects of involving translation in dealing with multi language dataset, we decided to focus on annotating the tweets in their native language, utilizing our synchronized approach on guidelines creation and annotation to ensure the comparability of our datasets. ## conclusion creating a guideline covering more than one language can be challenging. our approach consists of simultaneous annotation to synchronize the guideline in order to achieve a reliable and comparable dataset. the final guideline designed in this study shows adequate results for both languages, even if the two sets of tweet sources come from different cultural backgrounds. this study resulted in multilingual annotation guidelines in english and japanese for classifying tweets' relevancy, stance, and mask-wearing status. the final version of the guideline can be utilized to obtain more annotated sample for the future work on the comparison between mask opinion in the two countries.