KWJA : A Unified J apanese Analyzer Based on Foundation Models (2023)

Abstract: We present KWJA, a high-performance unified Japanese text analyzer based on foundation models.KWJA supports a wide range of tasks, including typo correction, word segmentation, word normalization, morphological analysis, named entity recognition, linguistic feature tagging, dependency parsing, PAS analysis, bridging reference resolution, coreference resolution, and discourse relation analysis, making it the most versatile among existing Japanese text analyzers.KWJA solves these tasks in a multi-task manner but still achieves competitive or better performance compared to existing analyzers specialized for each task.KWJA is publicly available under the MIT license at https://github.com/ku-nlp/kwja .

## introduction end-to-end neural network-based models have become mainstream for many nlp tasks including machine translation @xcite @xcite and dialogue response generation @xcite . however, end-to-end models are not always the best means of developing an nlp application because exploratory tasks, such as information analysis and causal analysis, inherently require manual trial-and-error processes. we believe that for such tasks, text analysis still plays an important role. text analysis, including morphological analysis, dependency parsing, predicate-argument structure (pas) analysis, and discourse relation analysis, saw shifts in model architectures. recent studies demonstrate that foundation models @xcite drastically improve the performance in dependency parsing @xcite , pas analysis @xcite , and discourse relation analysis @xcite . moreover, improvements on foundation models tend to have a greater impact on performance than incremental improvements tailored to individual tasks @xcite . in this study, we design and build a unified japanese text analyzer, kwja, foot_0 , foot_1 in view of the fact that recent high-performance text analysis models are all based on foundation models. kwja supports a wide variety of text analysis tasks: typo correction, word segmentation, word normalization, morphological analysis, named entity recognition, linguistic feature tagging, dependency parsing, pas analysis, bridging reference resolution, coreference resolution, and discourse relation analysis (figure 1 , our emphasis is on usability in addition to performance. kwja provides a single command to perform a variety of text analyses, collapsing the painstaking steps previously needed to obtain the same result, namely, installing and combining multiple text analyzers, one for each task. the design policy of kwja is to minimize the amount of code and hand-written rules by maximally exploiting the power of foundation models. this is a drastic departure from the traditional japanese analysis suite, including the morphological analyzers juman @xcite 3 and juman++ @xcite 4 and the dependency parser knp @xcite , 5 which rely heavily on manually constructed dictionaries, rules, and features. such lexical knowledge is context insensitive and suffers from limited coverage. motivated by the observation that foundation models learn massive knowledge through pre-training on large raw corpora, we narrow our efforts to supervised learning from relatively small annotated corpora. this approach enables us to support a new task just by constructing an annotated a base phrase is a unit consisting of a single independent word and its ancillary words @xcite . we group the components into three blocks (separated by horizontal lines) for multi-task learning. kwja reorganizes text analysis tasks into a dozen of components, as shown in each analysis component utilizes a fine-tuned transformer. as transformer consumes a considerable amount of computational resources, we resort to multi-task learning to reduce the model parameters. we group the components into three modules: the typo module, character module, and word module. within each module, analysis components share most of their model parameters and run concurrently. consequently, kwja executes all the components in three steps (figure 1 ). although kwja is extremely slow for users who only need word segmentation, it is the most practical choice for advanced text analysis, for which, after all, only transformer achieves state-ofthe-art performance. kwja is publicly available under the mit license at https://github.com/ ku-nlp/kwja . ## related work traditionally, japanese text analysis tasks have been tackled individually, with separate analyzers for each task. juman++ @xcite and mecab @xcite are examples of morphological analyzers, while knp @xcite is a dependency parser. juman++ and mecab segment japanese text into words and assign lexical information to each word using manually constructed dictionaries. knp assigns dependency relations between phrases using linguistic features obtained from juman++ and external dictionaries. in addition to dependency parsing, knp handles named entity recognition, linguistic feature tagging, and pas analysis. kwja, however, supports an even broader range of tasks. the universal dependencies (ud) project @xcite standardizes the annotation of dependency structures across languages. while their focus is on dependency relations, the ud guidelines define word units, pos tags, and other linguistic features. the tasks supported by major ud-based analyzers, ud-pipe @xcite , spacy, foot_5 and stanza @xcite , are sentence segmentation, word segmentation, pos tagging, lemmatization, and dependency parsing. 7 in other words, higher-level tasks such as pas analysis and discourse relation analysis are out of the scope of these analyzers. a major advantage of ud-based analyzers is that they can handle multiple languages. this is done at the expense of ignoring language-specific features @xcite . for the purpose of pioneering task design, it is reasonable to focus on a single language. ginza foot_7 is also a ud-based analyzer but specializes in japanese. ginza supports morphological analysis, dependency parsing, and named entity recognition. ginza v5 improved the dependency parsing performance by utilizing the foundation model electra. kachako @xcite and jigg @xcite have been proposed as frameworks for combining existing analysis tools to form a pipeline. these works aim to improve the usability of existing analysis tools. in contrast, our goal is to design and build a unified analyzer itself. ## resources this section presents the model and data resources used when training the modules in kwja. ## architecture each analysis component of kwja uses a transformer-based @xcite foundation model. we add two layers of feed-forward neural networks for each task and fine-tune the whole model. kwja formulates the text analysis tasks as a sequence labeling task, word selection task, or word relation classification task. a sequence labeling task assigns a label to each character or word in a text. figure 2 shows an example of solving named entity recognition as a sequence labeling task. in a word selection task, a word is selected from a text for each given word in the text. figure 3 shows an example of solving dependency parsing as a word selection task. a word relation classification task assigns a label to each word pair in a text. to reduce computational time and space, the model parameters of the analyzer are shared as much as possible through multi-task learning. the tasks in each block separated by the horizontal lines in while morphological analysis and dependency parsing use a sentence as the smallest unit of analysis, pas analysis and discourse relation analysis b-org o o b-date i-date o o b-mon i-mon i-mon o o foundation model (deberta) [cls] toyota announced on sep ##. 31 to invest 730 billion yen in evs [sep] foundation model (deberta) [cls] i visited kyoto ##, a city in japan ##, and saw … [sep] [root] softmax 𝑃(𝑤 ! |𝑤 " ) 𝑠(𝑤 ! , 𝑤 " ) 𝑤 " 2.6 0.8 8.3 5.5 2.3 1.5 ffn ffn ffn ffn ffn ffn ffn ffn ffn ffn … 7.4 1.2 0.9 0.00 0.00 0.67 0.04 0.00 0.00 … 0.28 0.00 0.00 require document-level processing. in this study, we apply document-level processing for all tasks to perform multi-task learning. with this formulation, obtaining sentence boundaries becomes less important. thus, we only use simple rules based on regular expressions for sentence segmentation. ## analysis components kwja consists of eleven analysis components belonging to three modules. in this section, we describe the details of each component, including its definition, formulation, and evaluation. ## experiments and discussion in this section, we investigate the performance of each analysis component through fine-tuning foundation models. ## conclusion in this study, we designed and built a unified japanese text analyzer, kwja, on top of foundation models. kwja supports typo correction, word segmentation, word normalization, morphological analysis, named entity recognition, linguistic feature tagging, dependency parsing, pas analysis, bridging reference resolution, coreference resolution, and discourse relation analysis in a unified framework. users can quickly obtain analysis results by inputting a text and specifying the desired level of analysis. one of the advantages of kwja is its simplified design, thanks to the use of foundation models. various analysis tasks, previously solved separately, are now performed only with three modules. for further simplification, we plan to solve all the analysis tasks with a character-level foundation model.