Word Segmentation by Separation Inference for E ast A sian Languages (2022)

Abstract: Chinese Word Segmentation (CWS) intends to divide a raw sentence into words through sequence labeling. Thinking in reverse, CWS can also be viewed as a process of grouping a sequence of characters into a sequence of words. In such a way, CWS is reformed as a separation inference task in every adjacent character pair. Since every character is either connected or not connected to the others, the tagging schema is simplified as two tags “Connection” (C) or “NoConnection” (NC). Therefore, bigram is specially tailored for “C-NC” to model the separation state of every two consecutive characters. Our Separation Inference (SpIn) framework is evaluated on five public datasets, is demonstrated to work for machine learning and deep learning models, and outperforms state-of-the-art performance for CWS in all experiments. Performance boosts on Japanese Word Segmentation (JWS) and Korean Word Segmentation (KWS) further prove the framework is universal and effective for East Asian Languages.

## introduction in natural language processing (nlp), word segmentation is the commencement of part-of-speech (pos) tagging, semantic role labeling (srl), and other similar studies. particularly for chinese, japanese and korean languages, the absence of explicit boundaries between characters makes the word segmentation (ws) task indispensable in nlp tasks. dominant word segmentation methods considered ws as a sequence tagging task @xcite . various tagging schemas such as "bmes" (begin, middle, end, single), "bies" (begin, inside, end, single), "sep-app" (separate, append), "bi" (begin, inside), and "start-nonstart" were employed to tackle the sequence labeling task. these tagging schemas are all characterbased and summarized as four-tags ("bmes", "bies") and two-tags ("sep-app", "bi" "start-nonstart"). despite diverse tagging schemas, they all carry implicit position information. for four-tags tagging schemas, the implicit information restricts the transition between tags. take "bmes" as an example; tag "b" can not be followed by "b" or "s". these two tagging schemas heavily rely on the precise prediction of the relative position of each character in one segment. however, the exact position information is not essential for the ws task. any unreasonable inner prediction representing the character's relative position results in incorrect segmentation, although the correct boundary prediction. there is no limitation of tag-to-tag transition for the two-tags schema, but according to common sense, the first character of a sentence must be predicted as "sep", "b" or "start". the implicit constraint of position for the first tag of the sentence still exists. it is necessary to ensure the prediction accuracy of the first tag during the inference. therefore, crf is required to revise unreasonable tag-to-tag transitions and learn the implicit restriction including the first tag of a sentence. the crf has alleviated the unreasonable tag prediction to some degree, but the simultaneous learning of transition and emission matrix still results in the tag inference being intractable. current works attempt to complicate the network @xcite and introduce more information @xcite such as rich context, linguistic and extra knowledge to tackle the abovementioned problem. however, the intrinsic problem, which is the implicit restriction of the position in the existing tagging schemas, is not well solved. in this paper, we propose "connection(c)-no-connection(nc)", which targets on characterto-character connections, to deal with the ws task directly. "c-nc" is independent of the previous state, and there is no dependency between states. moreover, there is no restriction for the first state as it is located between the first and the secondary characters. it can be either "c" or "nc". "c" or "nc" is a binary classification task. therefore, crf is not required and can then be substituted with a classification network. the tag-transition and implicit restriction burdens can be substantially alleviated through such "c-nc" states. because "c-nc" describes the connection state between two adjacent characters, we employ bigram features to cooperate with the "c-nc". compared with existing tagging schemas, which are character-based and the bigram features are considered as extra information, the bigram features in spin are the basic processing unit. therefore, a brand-new separation inference (spin) framework is proposed and constructed on the bigram features and the classification layer. sliding one-after-one along all the bigrams, words are yielded by allocating "c" and "nc" tags in the interval of characters. spin significantly reduces the inference complexity (inference layer crf is degraded as the softmax network); dispels extra context information (merely bigram feature is in consideration); and gains competitive performance of cws on the machine learning in contrast with the deep learning models. besides its effectiveness on chinese word segmentation, our extensive experiments also verify the universality by attaining state-of-the-art (sota) performance in japanese and korean word segmentation benchmark tests. our contributions are summarized as follows: ## related work researchers have explored the cws task from various directions since the 1990s @xcite . widely applied methodologies considered it as the sequence tagging task based on various label schemas. cws was first treated as a sequence tagging task in @xcite . the maximum entropy @xcite model and the crf @xcite were the most adopted sequence tagger. there are two main problems in the ws task: the ambiguities and the out-of-vocabulary (oov) words. researchers tried to leverage extra context features such as the bigram @xcite @xcite @xcite and the word features @xcite @xcite to tackle word ambiguities and improve the model's generalization capability. moreover, language-specific knowledge such as dictionaries was employed @xcite for better cws. extra punctuation marks from large manually segmented corpus were introduced to the learning model and proved effective for solving the unknown words @xcite . meanwhile, the external knowledge was explored through the semi-supervised models for better segmentation @xcite @xcite . along with the development of pre-trained models like bert @xcite , elmo @xcite , and gpt @xcite , striking improvements on cws are observed by replacing the feature extraction layer with these powerful pretrained models. except for the investigation of the effect of features, various tagging schemas were also discussed. widely applied tagging schema in cws contains "bmes" @xcite @xcite , "bies" @xcite , "sep-app" @xcite @xcite , "bi" @xcite , and "start-nonstart" @xcite . there is either the limitation of tagto-tag transitions or the implicit constraint for the first tag for these tagging schemas. these inherent problems were not well solved. hence, we propose the spin framework constructed on the "c-nc" tagging schema and its specially tailored bigram features. spin eliminates the implicit restriction of existing tagging schemas and boosts the performance of the ws task in east asian languages. ## proposed method we propose adopting the bigram feature to adapt to the "c-nc" tagging schema to model the connection of adjacent characters. distinguished from character-based models leveraging bigram feature as extra information, merely bigram is employed and set up as input unit. adaptation of spin involves machine and deep learning models. figure 1 and figure 5 summarize the spin framework architecture adapted to the machine and deep learning models separately. before exploring the structure of spin, we firstly elaborate definition of the proposed "c-nc" and distinction with the traditional two-tags tagging schema that indicates whether the current character is the boundary or not. in the later part of this section, we present the detailed structure of spin, including how to apply the spin framework to the machine learning and deep learning models. for machine learning, we explain how to build features based on the bigram through applying feature templates. meanwhile, we present how to build the bigram features based on the feature extractor layer for the deep learning model. in the last subsection, we illustrate the inference layer. ## experiments evaluation is first conducted on the cws to prove the sota performance of spin. contrast experiments involve both machine learning and deep learning models for further demonstrating the robustness of spin. an ablation study is conducted to investigate the effect of each component. ## adaptation to asian languages