User-Generated Text Corpus for Evaluating J apanese Morphological Analysis and Lexical Normalization (2021)

Abstract: Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.

## introduction japanese morphological analysis (ma) is a fundamental and important task that involves word segmentation, part-of-speech (pos) tagging and lemmatization because the japanese language has no explicit word delimiters. although ma methods for well-formed text @xcite have been actively developed taking advantage of the existing annotated corpora of newswire domains, they perform poorly on usergenerated text (ugt), such as social media posts and blogs. additionally, because of the frequent occurrence of informal words, lexical normalization (ln), which identifies standard word forms, is another important task in ugt. several studies have been devoted to both tasks in japanese ugt @xcite @xcite to achieve the robust performance for noisy text. previous researchers have evaluated their own systems using in-house data created by individual researchers, and thus it is difficult to compare the performance of different systems and discuss what issues remain in these two tasks. therefore, publicly available data is necessary for a fair evaluation of ma and ln performance on japanese ugt. in this paper, we present the blog and q&a site normalization corpus (bqnc), foot_0 which is a public japanese ugt corpus annotated with morphological and normalization information. we have constructed the corpus under the following policies: for the first requirement, we extracted and used the raw sentences in the blog and q&a site registers compiled by (the non-core data of) the balanced corpus of contemporary written japanese (bccwj) @xcite , in which the original sentences are preserved. 2 for the second requirement, we followed the short unit word (suw) criterion of the national institute for japanese language and linguistics (ninjal), which is used in various ninjal's corpora, including manually annotated sentences in the bccwj. for the third requirement, we organized linguistic phenomena frequently observed in the two registers as word categories, and annotated each word with a category. we expect that this will contribute to future research to develop systems that manage ugt-specific problems. the bqnc comprises sentence ids and annotation information, including word boundaries, pos, lemmas, standard forms of non-standard word tokens, and word categories. we will release the annotation information that enables bccwj applicants to replicate the full bqnc data from the original bccwj data. 3 using the bqnc, we evaluated two existing category example reading translation standard forms type of vocabulary (general words) neologisms/slang コピペ copipe copy and paste proper names ドラクエ dorakue dragon quest onomatopoeia キラキラ kirakira glitter interjections おお ō oops dialect words ほんま homma truly foreign words ｅａｓｙ easy emoticons/aa (＾-＾) type of variant form (standard forms) character type variants カワイイ kawaī cute かわいい,可愛い alternative representations 大きぃ ōk ī big 大きい sound change variants おいしーい oishīi tasty おいしい,美味しい typographical errors つたい tsutai tough つらい,辛い ## overview of word categories based on our observations and the existing studies @xcite , we organized word tokens that may often cause segmentation errors into two major types with several categories, as shown in from another perspective, any word can have multiple variant forms. because the japanese writ-4 "onomatopoeia" typically refers to both the phonomime and phenomime in japanese linguistics literature, similar to ideophones. we follow this convention in this paper. 5 we observed a few examples of other vocabulary types, such as japanese archaic words and special sentence-final particles in our corpus, but we treated them as general words. ## corpus construction process the bqnc was constructed using the following steps. the annotation process was performed by the first author. token translation standard form id イイ good 38988:良い 歌 song です (polite copula) ねェ (emphasis marker) 28754:ね ヨカッ good 38988:良い_ヨカッ タ (past tense marker) 21642:た ## evaluation we present the statistics of the bqnc in in the following subsections, we evaluate the existing methods for ma and ln on the bqnc and discuss correctly or incorrectly analyzed results. ## conclusion we presented a publicly available japanese ugt corpus annotated with morphological and normalization information. our corpus enables the performance comparison of existing and future systems and identifies the main remaining issues of ma and ln of ugt. experiments on our corpus demonstrated the limited performance of the existing systems for non-general words and non-standard forms mainly caused by two types of difficult examples: complicated variant forms and unknown words of non-general vocabulary types. in the future, we plan to ( 1 ) expand the corpus by further annotating of 5-10 times more sentences for a more precise evaluation and (2) develop a joint ma and ln method with high coverage.