Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages (2025)

Abstract: Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text, yet the absence of comprehensive evaluations leaves it unclear which methods excel across multiple perspectives. Focusing on unsegmented languages, we make three key contributions: (1) creating a large-scale, multi-domain Japanese normalization dataset, (2) developing normalization methods based on state-of-the-art pre-trained models, and (3) conducting experiments across multiple evaluation perspectives. Our experiments show that both encoder-only and decoder-only approaches achieve promising results in both accuracy and efficiency.

## introduction user-generated text (ugt) is invaluable textual content produced by users' activities on web platforms such as review sites and social media. ugt's informality-its frequent use of colloquial expressions-poses substantial challenges to accurate analysis in natural language processing applications. consequently, researchers have explored lexical normalization (ln), the task of converting non-standard word forms into standard ones. ln has been actively studied particularly for space-delimited languages such as european languages @xcite . in this study, we investigate ln for unsegmented languages in writing, focusing primarily on japanese. major issues in existing ln research are summarized as a lack of comprehensive evaluation, leaving unclear which methods excel under different evaluation aspects. specifically, (i) comparative evaluations of recent model architectures are absent, and (ii) multi-perspective analysesexamining required training data size, inference cost, and domain-specific accuracy across diverse domains-have not been conducted. these issues are common to ln research but are particularly severe for underexplored unsegmented languages. this study addresses these gaps through three key contributions: (1) dataset construction, (2) method development based on cutting-edge pre-trained models, and (3) comprehensive experiments. first, we introduce the japanese multi-domain lexical normalization dataset (jmln), a large collection of 21,402 sentences drawn from a variety of ugt sources. jmln's size exceeds existing japanese ln datasets @xcite , and its domain diversity surpasses that of any current ln datasets. these properties enable both the development of methods suitable for japanese and multi-perspective evaluation. second, we develop ln methods based on three modern transformer @xcite architectures-encoder-only, encoder-decoder, and decoder-only-including a novel encoder-based infilling approach, as well as variants of generative approaches. while these boundary-aware methods are tailored for unsegmented languages, they remain broadly applicable. third, we evaluate these methods on jmln and an existing thai dataset. multi-perspective experiments on jmln yield in-depth insights into methods' characteristics and trade-offs, while experiments on the thai dataset further validate crosslingual applicability and generalizability. our evaluation reveals three main findings. first, compact encoder-only models deliver the highest inference throughput, and decoder-only models excel in normalization recall-while both yield high normalization precision. second, training models on 4k-8k sentences yields reasonable precision of around 0.7, and cutting-edge decoder-only models deliver superior recall even with fewer instances. third, domains rich with unknown informal words exhibit low performance, especially the typo-correction domain. to date, research on ugt normalization can be broadly divided into three categories based on the dominant methodologies of each period: rulebased and statistical methods @xcite @xcite , pre-transformer neural methods @xcite @xcite , and transformer-based methods @xcite @xcite . to achieve robust word segmentation (ws) for ugt, previous studies have often addressed ln jointly with ws, as exemplified by research on japanese @xcite @xcite @xcite , chinese @xcite , and thai @xcite . some recent studies have adopted transformer masked language models (mlms) with a text-editing approach @xcite and a two-step approach-first detecting informal tokens and then predicting formal tokens @xcite . decoder-only transformer models have recently made remarkable advances and have been applied to text normalization for tts @xcite and other sequence transduction tasks @xcite . however, these models remain underexplored in ugt normalization, resulting in a lack of comparative evaluation of state-of-the-art models for this task. regarding evaluation domains, many studies have focused on building short message and social media datasets for european @xcite @xcite @xcite and asian languages @xcite @xcite , which has led to extensive model development and evaluation in these domains. some studies have focused on other domains such as blog and q&a site @xcite , and maintenance short text @xcite . however, cross-domain evaluation research covering three or more domains remains scarce. ## japanese dataset construction for our primary target language, japanese, existing datasets @xcite @xcite have limited domain diversity and size-covering one or two domains with approximately 1,000 to 6,000 sentences/posts. in this study, we have constructed jmln, as a large-scale dataset sourced from a variety of ugt. ## task definition following previous studies @xcite , we define ln as a task of boundary-aware span extraction and conversion, in which a system not only generates a normalized text but also identifies the original spans of each informal words (or phrases). an example inputoutput pair for our task is shown in figure 1 . this task can be carried out independently of the tokenization unit and applied to any ln dataset, as long as informal-to-formal alignments are annotated. 5 in contrast to the text-to-text conversion task @xcite , the boundary-aware task allows fine-grained evaluation at the normalizationspan level and provides better interpretability of system outputs. this is because span-level evaluation metrics directly assess the validity of each normalization instance, leading to more robust evaluation across sentences with varying densities of non-standard tokens. further discussion on the task definition is provided in appendix b formally, an ln system takes as input a source sentence, namely, a sequence of n character (or subword) tokens @xmath0:@xmath1, . . . , x n-1 ], and is required to predict the set of non-standard word spans and their standard forms @xmath2, e, s)}. here, (b, e) (0 ≤ @xmath3:e with length eb in the source sentence, and s indicates its standard form. foot_5 each standard form s is a string with @xmath4, where @xmath5. when @xmath6, a zero-length span indicates some token(s) should be inserted into the position b. ## methods we present boundary-aware ln methods based on three transformer architectures: encoder-only, encoder-decoder, and decoder-only. our encoderbased infilling approach is a novel method for unsegmented languages, and comparing multiple approaches across different architectures offers a valuable, novel evaluation. ## experiments we set the following experimental questions (eqs): to address these eqs, we evaluated various ln methods using our japanese dataset and an existing thai dataset. specifically, experiments in §7.1-7.3 (and §7.6) correspond to eq1, §7.4 to eq2, and §7.5 and §7.7 to eq3. we use precision, recall, and f 0.5 score at the normalization-span level for individual nonstandard words as evaluation metrics. while normalized results with low recall do not harm downstream task performance compared to original texts, low-precision outputs introduce spurious tokens that can degrade downstream accuracy. we therefore adopt the f 0.5 score, which emphasizes precision over recall, similarly to other text correction ## conclusion this paper presented our multi-domain japanese ln dataset, ln methods based on three transformer architectures for unsegmented languages, and multi-perspective experiments and analysis. the answers to the three evaluation questions ( §6) are summarized as follows. (1) compact encoder-only models achieved high precision and offered the best inference throughput, while cuttingedge decoder-only models delivered high precision, notably high recall, and reasonable throughput on a high-spec gpu. (2) normalization accuracy consistently increased with training data size, yet even 4k-8k training sentences yielded reasonable precision around 0.7. sarashina-series models, in particular, achieved superior recall with fewer training sentences. (3) domains with higher rates of unknown non-standard tokens correlated with decreased performance across models. typo correction emerged as the most challenging category, reflecting the difficulty posed by diverse typo patterns. in future work, we will evaluate the impact of ln on downstream tasks and explore the development of a general-purpose decoder model with robust normalization capabilities.