Text Normalization for J apanese Sentiment Analysis (2025)

Abstract: We manually normalize noisy Japanese expressions on social networking services (SNS) to improve the performance of sentiment polarity classification.Despite advances in pre-trained language models, informal expressions found in social media still plague natural language processing.In this study, we analyzed 6,000 posts from a sentiment analysis corpus for Japanese SNS text, and constructed a text normalization taxonomy consisting of 33 types of editing operations.Text normalization according to our taxonomy significantly improved the performance of BERT-based sentiment analysis in Japanese.Detailed analysis reveals that most types of editing operations each contribute to improve the performance of sentiment analysis.

## introduction for research and development of sentiment analysis models, datasets with sentiment labels for text on social networking services (sns) are available @xcite @xcite . in japanese, sentiment analysis datasets for sns posts such as wrime foot_0 @xcite are available. text from social media often contains informal japanese expressions such as misspellings and internet slang. these noisy texts may degrade the performance of natural language processing, including sentiment analysis. in this study, to improve the performance of sentiment analysis in japanese, various noisy expressions in sns texts were manually normalized. we performed text normalization on 6,000 posts from the wrime dataset, and organized the editing operations contained therein into 6 major categories and 33 subcategories. then, our detailed analysis based on this japanese text normalization taxonomy revealed which type of normalization contributes to improved performance of sentiment analysis in japanese. experimental results showed that our text normalization improved the performance of sentiment analysis in japanese. furthermore, our detailed analysis reveals that most types of normalization contribute to improved performance in sentiment analysis. among them, there were notable improvements due to the normalization of casual/formal sentence endings, missing symbols, abbreviations, and inconsistencies in hiragana, katakana, and kanji. 2 in contrast, since the normalization of pronunciation variations worsened the performance of sentiment analysis, the pronunciation variations may express the writer's emotions. we plan to release 1 our 6,000 normalized post pairs with our japanese text normalization taxonomy. ## related work noisy expressions found in social media deteriorate the performance of various natural language processing such as word segmentation and sentiment analysis. to address this issue, text normalization has been studied. text normalization corpora have been developed for various languages, including english @xcite @xcite , german @xcite , spanish @xcite , turkish @xcite , danish @xcite , italian (van der goot et al., 2020), thai @xcite , and vietnamese @xcite , to facilitate the development of data-driven approaches for text normalization. for text normalization in japanese, approaches to sequence labeling @xcite and sequence-tosequence generation @xcite have been proposed. however, these previous studies are based on small parallel corpora of about 1,000 sentence pairs @xcite @xcite , automatically generated corpus @xcite , and non-public corpora @xcite . therefore, a larger-scale parallel corpus that is freely available for japanese text normalization is desired. ## experiment our experiments evaluate the performance of sentiment polarity classification on sentences with individual or all normalizations, and assess the effectiveness of preprocessing with text normalization. ## conclusion in this study, we worked on text normalization as a preprocessing to improve the performance of sentiment analysis for japanese sns texts. we defined a japanese text normalization taxonomy consisting of 33 types of editing operations and manually normalized 6,000 posts. experimental results showed that both automatic and manual text normalization consistently improved the performance of sentiment analysis. in manual text normalization, most types of normalization improved the performance of sentiment analysis, respectively. our detailed analysis reveals that pronunciation variations should not be edited, and are a useful linguistic phenomenon for sentiment analysis.