Building Large-Scale E nglish and K orean Datasets for Aspect-Level Sentiment Analysis in Automotive Domain (2020)

Abstract: We release large-scale datasets of users’ comments in two languages, English and Korean, for aspect-level sentiment analysis in automotive domain. The datasets consist of 58,000+ commentaspect pairs, which are the largest compared to existing datasets. In addition, this work covers new language (i.e., Korean) along with English for aspect-level sentiment analysis. We build the datasets from automotive domain to enable users (e.g., marketers in automotive companies) to analyze the voice of customers on automobiles. We also provide baseline performances for future work by evaluating recent models on the released datasets.

## introduction aspect-level sentiment analysis (alsa) has been actively studied to understand authors' opinion on aspects from texts. for example, in a given text, "although the space is smaller than most, it is the best service you will find in even the largest restaurants", the author's sentiment to space and service is negative and positive, respectively. since devising deep learning models for alsa recently received substantial attention @xcite , building large-scale datasets in different languages has been an essential line of research @xcite . however, due to the high cost of human annotation, the size and language of datasets are still limited. specifically, only one of public datasets contains more than 20,000 instances, and existing datasets cover only three languages (i.e., english, spanish, and arabic). to this end, we release large-scale datasets of users' comments with two languages such as english and korean in automotive domain. the total size of these datasets is 58,603, which is the largest compared to existing datasets for alsa. in addition, the datasets include new language (i.e., korean), which extends the coverage of the datasets in terms of languages. in this work, we focus on automotive domain to build the datasets to enable users (e.g., marketers in automotive companies) to analyze the voice of customers on aspects related to automobiles. to build the datasets, domain experts define the 12 largest automotive manufacturers (e.g., ford) by production volume and their popular automobiles (e.g., mustang) as aspects. given the aspects, we collect users' comments from automotive communities in the united states and south korea. to annotate aspect-level sentiments, we perform crowdsourcing by assigning at least three annotators to each comment-aspect pair. the annotated datasets consist of 28,571 and 30,032 comment-aspect pairs in english and korean, respectively. inter-annotator agreements are 0.36 (fair agreement) for english dataset and 0.54 (fair agreement) for korean dataset in terms of fleiss' kappa 1 . we perform extensive experiments with deep learning models for alsa to provide the baseline performance on the released datasets for future work. the datasets are publicly available at our website. ## related datasets researchers in alsa have built labeled datasets as supervised learning has been a major approach to tackle alsa. annotated sentiments for aspects related to organizations and persons. @xcite annotated tweets that include aspects related to celebrities and products. semantic evaluation (semeval) in 2014 @xcite built two english datasets from laptop and restaurant domains. semeval in 2017 @xcite built the largest dataset consisting of english tweets and a non-english dataset consisting of arabic tweets with popular events as aspects such as named entities (e.g., iphone) and geopolitical entities (e.g., palestine). @xcite built a dataset to include multiple aspects in each text (i.e., tweet), and they define aspects related to uk election (e.g., greens and labour). similar to @xcite , @xcite included multiple aspects in each text, and the aspects are related to restaurants (e.g., food and service). we note that a recent work built a dataset for alsa in korean @xcite , but we omit it from our comparison as the dataset is not publicly available. in this work, we release large-scale datasets consisting of users' comments in english and korean from automotive domain. the total size of our datasets is 58,000+, which is the largest compared to the other datasets. the datasets also include new non-english language (i.e., korean) in addition to spanish and arabic. we believe that the released datasets further relieve the lack of large-scale and non-english datasets for alsa. to cover a wide range of automotive domain, experts from hyundai, a korean automotive company, defined the 12 largest automotive manufacturers (e.g., ford) by production volume and their popular automotive models (e.g., mustang) as aspects in both english and korean. the predefined list contains 12 automotive manufacturers and 341 automotive models. ## conclusion we release large-scale datasets consisting of 58,000+ comments in english and korean from automotive domain. the total size of the datasets is currently the largest, and the datasets include new non-english (i.e., korean) language for alsa. for future work, we also provide the baseline performances on the released datasets using deep learning models for alsa. we also thank the domain experts from hyundai, jung-mi park and kye-yoon kim, for helpful opinions, and young-woo kim for developing korean annotation website.