A Text Editing Approach to Joint J apanese Word Segmentation, POS Tagging, and Lexical Normalization (2021)

Abstract: Lexical normalization, in addition to word segmentation and part-of-speech tagging, is a fundamental task for Japanese user-generated text processing. In this paper, we propose a text editing model to solve the three task jointly and methods of pseudo-labeled data generation to overcome the problem of data deficiency. Our experiments showed that the proposed model achieved better normalization performance when trained on more diverse pseudo-labeled data.

## introduction user-generated text (ugt), such as social media and blog posts, is a valuable source of knowledge and opinions from diverse users. a notable characteristic of ugt is that it contains non-canonical sentences, and this degrades the performance of natural language processing (nlp) systems trained on canonical sentences. to reduce the gap between the performance on general text and on ugt, lexical normalization techniques-which convert nonstandard word forms to standard forms-have been explored, particularly for english @xcite . in addition, japanese ugt requires a further step: to identify nonstandard words in unsegmented sentences written without word delimiters. for this reason, the problem of japanese lexical normalization has been solved by predicting word boundaries, part-of-speech (pos) tags, and normalized word forms simultaneously @xcite . similarly to previous work, we tackle the joint task comprising japanese word segmentation, pos tagging, and lexical normalization (spn). a critical problem in lexical normalization is the lack of labeled data. manual annotation of normalized forms is a time-consuming task; therefore, the size of the available annotated corpora is quite small @xcite . a prospective solution to this problem is the use of pseudo-labeled data. in this paper, we propose methods of generating pseudo-labeled data using (auto-) segmented sentences and standard and nonstandard word variant pairs. to generate high quality labels, we acquire reliable variant pairs based on lexical knowledge, namely, a dictionary with lemma definition and hand-crafted rules. for efficient learning from a limited amount of data, we adopt a text editing approach. our neural tagging model predicts edit operations to normalize input characters, while predicting segmentation and pos tags at the same time. the editing process is similar to that proposed in previous work on english lexical normalization @xcite , but we design a specific tag set for the japanese spn task, which requires the management of a large number of character types. our extensive experiments on the spn task demonstrated that our model achieved better normalization performance when the model used more additional features, it was trained on more types of pseudo-labeled data, and it was trained on training instances with more diverse context. ## task definition as shown in a system is required to predict the word boundaries of an input sentence and the pos tag of each word, detect nonstandard words, and generate one of the standard forms of each nonstandard word. ## pseudo-labeled data generation to overcome the lack of training data for the normalization task, we construct a set of standard and nonstandard word variant pairs v and then generate different types of pseudo-labeled data by two approaches: distant supervision on formal target-side (ds tgt ) and informal source-side text (ds src ). ds tgt generate a sentence where the original tokens are retained but nonstandard tokens among them are annotated with pseudo standard tokens; specifically, given a segmented sentence, a token matching with a nonstandard form v nst in v is annotated with sedit and cconv tags to convert to its standard form, while other tokens are annotated with keep tags. on the other hand, ds src generate a sentence where one or more of the original tokens are replaced by pseudo nonstandard forms; given a standard and nonstandard variant pair (v st , v nst ) ∈ v, ds src extracts a segmented sentence containing a token with the same lemma as that of the pair, replaces the token by v nst , and generates sedit and cconv tags to convert v nst to lemma (語彙素) 〈大きい〉 ōkī 'big' word form (語形) オオキイ ōkī オッキイ okkī orthographic form (書字形) 大きい おおきい おっきい surface form (出現形) 大きい [term] 大きく [cont] … おおきい [term] おおきく [cont] … おっきい [term] おっきく [cont] … figure 1: hierarchical lemma definition in unidic. termination forms (term) and continuative forms (cont) are illustrated as examples of surface forms. as the main difference between two approaches, ds tgt does not change original sentences but ds src does. although we can use actual sentences with ds tgt , we can easily obtain any number of synthetic sentences containing nonstandard words of interest with ds tgt . however, both approaches require reliable variant pairs to generate useful data. for this purpose, we use two strategies for variant pair acquisition: dictionary-based and rule-based. ## conclusion this paper presents our text editing model and methods of pseudo-labeled data generation for the joint segmentation, pos tagging, and normalization task. the experiments demonstrated that the proposed model was successfully trained on generated pseudo-labeled data, but more exhaustive detection and accurate normalization of nonstandard words have the potential to improve the model. future work includes (1) explicit consideration of nonstandard word spans for accurate segmentation and normalization, (2) the use of a large-scale pretrained lm for better normalization coverage, and (3) evaluation on broader ugt domains.