Syllable-based Neural T hai Word Segmentation (2020)

Abstract: Word segmentation is a challenging pre-processing step for Thai Natural Language Processing due to the lack of explicit word boundaries. The previous systems rely on powerful neural network architecture alone and ignore linguistic substructures of Thai words. We utilize the linguistic observation that Thai strings can be segmented into syllables, which should narrow down the search space for the word boundaries and provide helpful features. Here, we propose a neural Thai Word Segmenter that uses syllable embeddings to capture linguistic constraints and uses dilated CNN filters to capture the environment of each character. Within this goal, we develop the first ML-based Thai orthographical syllable segmenter, which yields syllable embeddings to be used as features by the word segmenter. Our word segmentation system outperforms the previous state-of-the-art system in both speed and accuracy on both in-domain and out-domain datasets.

## introduction word segmentation presents a fundamental challenge for thai language processing. many of the down stream natural language processing tasks require that texts be broken into a sequence of words before applying any models. for example, bagofword models or rnn models usually require that the text is represented as a sequence of words. notable examples of writing systems without clear word boundaries are japanese, chinese, and khmer, which often require an automatic word segmentation. however, the thai script differs from chinese and japanese in that a thai character represents just a consonant or a vowel, but a chinese character represents a whole syllable. the thai script uses 44 consonant symbols, 15 vowel symbols, and 4 tonal symbols. a word is composed of one or more syllables, and each syllable is formed by a set of intricate orthographical rules for valid sequences of thai alphabet symbols. hence, it is not straightforward to extend the techniques from chinese or japanese to thai word segmentation problem. thai word segmentation is complicated by linguistic ambiguity and outofvocabulary cases. a word can be formed by juxtaposing two "words" e.g. เห็ นชอบ (/hěn tc h ô:p/, approve) = เห็ น (/hěn/, see) + ชอบ (/tc h ô:p/, like). this kind of word formation can be detected with a simple dictionary lookup, but harder cases that require context abound in the language. for example, กอดอกไม้ can be segmented into กอ|ดอก|ไม้ (/ko: dò:k máj/, flower bush) or กอด|อก|ไม้ (/kò:t pók máj/, hugging a wooden chest), but the latter is rather nonsensical and statistically unlikely. the local context is needed to select the right segmenta tion; therefore, dictionaries only provide partial solutions to this problem. further, a constant stream of new words and loanwords complicates the task of word segmentation because they never appear in the dictionaries. in recent years, a few opensourced thai word segmenters have been introduced and used widely in the industry. notable examples of opensourced thai word segmenters include pythainlp @xcite , sertis @xcite , and deepcut @xcite , which claim good performance according to their own respective benchmarks. the accuracies of these word segmen tation systems are not benchmarked on the same datasets for a rigorous comparison within and across domains. in this paper, we reimplement some of these baselines for comparison. more importantly, these techniques formulate the word segmentation problem as a charactersequence tagging problem and ignore the fact that characters form a syllable as an intermediate step to word formation. our systems take advantage of the fact that every word can be parsed into orthographical syllables. a sequence of thai characters can be thought of as a sequence of orthographical syllables, each of which consists of at least two consonant characters or a pair of a consonant and a vowel. we hypothesize that syllable segmentation should simplify the downstream task of word segmentation. we explore the use of syllables in two ways. we use syllables as features (embeddings) for the word segmentation model, and we also try posing the word segmentation as a syllabletagging problem, as opposed to charactertagging problem. we present a neural word segmentation model that utilizes character and syllable embeddings as the representation and achieve stateoftheart result. we conclude that underlying linguistic structures (such as syllables) can serve as a suitable structure for neural architecture, which yields better performance as a result. our contributions can be summarized as follows: ## problem statement: thai word and syllable segmentation given a string of thai text, a thai word segmenter identifies the word boundaries within the string. this problem is formulated as a classification problem or sequencetagging problem where we try to identify a character that starts a new word in the string. a thai word is defined as the smallest lexical unit that still conveys the meaning @xcite . most ambiguous and debatable cases revolve around the degree of compositionality of nouns and verbs. the word การบ้ าน (/ka:n.bâ:n /, homework) is one word, although การ (/ka:n/, nominalization morpheme) and บ้ าน (/bâ:n/, house) are also words in other context. the meaning of การบ้ าน is not compositional from the two potential morphemes and therefore should be treated as one word. as a more uncertain example, it is arguable that the meaning of กอดอก (/kò:t pòk/, to cross arms) is composed of กอด (/kò:t/, to hug) and อก (/pòk/, chest) and therefore should not be treated as one word. our dataset follows the guidelines that favor the segmentation takes the degree of compositionality into account and not grammatical function changes such as nominalization or verbalization. syllable segmentation task is defined similarly, but we try to find syllable boundaries instead of word boundaries. it is noteworthy that word boundaries are always subset of syllable boundaries because each syllable belongs to exactly one word. we use this fact as a basis for our model architecture. an orthographical syllable is defined as a substring in a word that can be pronounced as one or one and a half phonological syllable. for example, the word จิ นตนา can be segmented into two orthographical syllables จิ น (/tcin/), which is also phonologically one syllable, and ตนา (/ta.nǎ:/), which is phonologically two syllables. the mismatch between orthographical and phonological syllables is very common, and we believe orthographical syllable is more tractable computationally because each syllable is guaranteed to be at least two characters long and each syllable can also be a word itself in some cases. ## related works thai word segmentation share some similarities with chinese and japanese word segmentation, but the linguistic differences require different types of models, but some insights from their techniques inspire our approaches. both chinese and japanese word segmentation problems are formulated as character sequence tagging problem. bilstm has been found to be very effective for chinese @xcite and japanese @xcite . interestingly, lengthsensitive bies tagset proves to be a better tagset that the standard bi tagset @xcite . the bies tagset cannot be readily applied to thai word segmentation since thai words are at least two characters long. in this study, we automatically group characters into syllables first, so that such lengthsensitive tagset can be applied. previous approaches to thai word segmentation can be categorized into two categories: dictionary based and learningbased. @xcite proposes the first dictionarybased method using a greedy algorithm to decide when a word should be formed. dictionarybased methods inevitably suffer from unseen words, and hence are harder to generalize to other domains. @xcite uses maximal matching, to handle such unseen word cases. past machinelearning word segmenters utilize subword features. @xcite pro pose thai character clusters (tccs). tccs differ from the notion of syllables used in our work in that tccs are based loosely on the orthographical syllables and designed with information retrieval applica tions in mind. using tccs, @xcite develop a decision tree classifier to determine whether a word should be formed from tccs, based on a predefined metric. @xcite presents a twostage word segmentation that incorporates handcrafted syllable features with dy namic programming to form the most reasonable segmentation, while @xcite use a hid den markov model to form words that are then verified with a dictionary. modern approaches to this problem are used by practitioners, but it is unclear which approach is the most accurate or fastest. @xcite present deepcut, a 1dimension cnn for thai word segmentation. the alleged stateoftheart system @xcite and @xcite use a birnn to segment words with multiple possible segmentation candidates. these results are not systematically benchmarked or published, but we use these architectures as our baselines. we propose two thai word segmentation sys tems based on bilstm and cnn architectures. the bilstmbased system follows the stan dard formulation of sequencetagging bilstm model. for the cnnbased system, unlike deepcut that uses multiple convolutions on features, we employ iterated dilated convolutions proposed by @xcite . iterated dilated convo lutions is a hierarchical dilated convolutional lay ers that the dilation number increases doubly at each layer. it allows the model to consume suffi cient context with fewer parameters, and @xcite demonstrate its advantages in pre diction performance on entity recognition tasks and computational efficiency. we use a stack of three levels with width three and dilation num bers 1, 2, and 4; in total, it covers the context of seven characters on each side. after the convolution layers or the bilstm layers, we have two fullyconnected layers, combined extract features to the probability of tags. figure 1 depicts the convolution architecture in details. our systems use syllables as additional features and structures to the model. both systems use char acter embeddings and syllable embeddings as features. for a given character, we first find the syllable that contains the character and then look up the syllable embedding. syllable embeddings should provide higherlevel information than a character type or a character embedding can afford. as an additional ex periment, we reformulate the task as a syllablesequence tagging problem instead of a charactersequence tagging one. in this formulation, we use syllable embeddings only for both bilstm and cnnbased models. we use new tagsets to take advantage of the fact that words with different lengths are distributed differently. the standard tagset uses binary tags: b for the beginning of a new word and i for the inside of a word. we explore a lengthsensitive tagset: bishort, bimid, and bilong for words that have 12 syllables, 34 syllables, and 5+ syllables, which we call scheme a. we also experiment with another lengthsensitive tagset: bik tag where @xmath0, 2, 3, 4+} is the length of the word in terms of number of syllables, which we call scheme b. all of the proposed models require an automatic syllable segmenter as a preprocessing system for the word segmenters. thai syllables are mostly bound by rules, but ambiguity and typos are natural occur rences in real data. to the best of our knowledge, there is no previous work in thai syllable segmentation, and syllables have never been used as features for mlbased word segmentation systems. we propose a conditional random fields (crf) syllable segmenter primarily to be used as a preprocessing step for the word segmenter. we use templatebased ngram features to capture the local context of the sequence. evaluating the quality of word segmenters is typ ically done on a characterlevel (cl) basis. stan dard measured metrics are precision, recall, and f 1 of startingword characters. however, in tuitively, when a token is tokenized wrongly, it would consequentially affect the tokenization of following tokens. thus, measuring only the characterlevel metrics would overestimate the tokenization performance of syllable or word to kenizers. therefore, in this work, we also mea sure the chunklevel metrics-the syllable level (sl) or world level (wl). sl and wl f 1 are cal culated based on the number of correct syllables and words respectively rather than characters. figure 2 illustrates the characterlevel and chunklevel measures. ## speed benchmark we perform speed benchmark of two existing methods, namely pythainlp and deepcut, and our word segmenters. we conduct the benchmark on two aws cloud instances: t2.medium and p2.xlarge; the former is a cpu instance, while the latter is a gpu instance. we use unix's time to measure the wall clock of segmentation and aws's official deep learning image foot_2 . for each method or configuration, we perform five runs in which the first two runs are burnin, and we average segmentation time from the last three. we refer to appendix a for our benchmark code. for p2.xlarge(@xmath3, we did not benchmark deepcut in this setting because its api does not allow batch size configuration. we use bilstm(ch)bi's time as the reference instead. from the ta ble, we see that both bilstmcrf(sy)bi and idcnncrf(sy)schemea are on par in terms of speed. nevertheless, we consider here the results on t2.medium the most important one because word segmentation is one of the first step in nlp, and it should be efficient without a special hardware. ## conclusion in this work, we propose to use syllable knowledge for thai word segmentation. we perform a system atic comparison between existing thai word segmenters and our different model configurations based on lstms and cnns. our results show that incorporating syllable can improve word segmentation performance significantly both indomain and out domain evaluations. we also analyze results from hy perparameter search, and the analysis shows that using syllable features require less computational budget for the search to find word segmenters with decent performance. we also explain two characterlevel word segmenters using integrated gradients, showing that the lstmbased word segmenter relies only a faction of neighbor character when predicting tags of the sequence, suggesting that using cnns models are more efficient than lstms. our speed benchmark shows that our best and fastest syllable model idcnncrf(sy)schemea is 8.9× and 3.5× faster than deepcut, the current state of the art, on cpu and gpu instances respectively. in future, we would like to investigate 1) the endtoend learning paradigm, combining syllable and word segmentation into one model; 2) the effect of pretrained syllable embeddings; and 3) the effect of word segmentation on downstream tasks.