Character-based T hai Word Segmentation with Multiple Attentions (2021)

Abstract: Character-based word-segmentation models have been extensively applied to agglutinative languages, including Thai, due to their high performance. These models estimate word boundaries from a character sequence. However, a character unit in sequences has no essential meaning, compared with word, subword, and character cluster units. We propose a Thai word-segmentation model that uses various types of information, including words, subwords, and character clusters, from a character sequence. Our model applies multiple attentions to refine segmentation inferences by estimating the significant relationships among characters and various unit types. The experimental results indicate that our model can outperform other state-of-the-art Thai word-segmentation models.

## introduction thai running text has neither essential word delimiters nor sentence periods. however, spaces are arbitrarily allowed to separate words, phrases, clauses, and sentences. these characteristics make word segmentation in thai more difficult than in other languages, such as english, german, and finnish, which have spaces and periods to identify word and sentence boundaries. thai word segmentation can be categorized as a sequential-labeling task that assigns a word-boundary label to each character in a fine-grained tagging scheme such as bies (beginning, inside, end, and singleton) @xcite , as shown in figure 1 . neural-network models have been applied and perform well on character-based thai word segmentation. @xcite applied bidirectional recurrent neural networks with gated recurrent units, while @xcite accurate word inferring motivated by deepcut (kittinaradorn et al., 2019). despite the fact that neural-network models with linguistic knowledge can perform almost perfectly, adapting models with pre-trained neural networks, such as word vectors and language models, is still useful @xcite . @xcite proposed a transfer-learning approach for thai word segmentation by using a pre-trained character language model for character-based word segmentation. although it exhibited state-of-the-art performance regarding thai, it merely uses characters and does not use other information such as word and subword @xcite . additional linguistic knowledge, such as thai character clusters (ccs) @xcite and subword units, has also been successfully used for word segmentation and related tasks @xcite @xcite @xcite . an attention mechanism @xcite has been successfully applied to various downstream tasks, particularly a sequencelabeling task @xcite . we propose a character-based thai wordsegmentation model with multiple attentions that jointly uses corresponding words and ccs. our model is based on the bidirectional long short-term memory with conditional random field (bilstm-crf) architecture, which is the baseline model for this study, because it has been successfully applied in sequence-labeling tasks for thai and other languages @xcite @xcite @xcite . our contributions are as follows: ## proposed model incorporating candidate words with an attention mechanism into the character-based bilstm-crf architecture could yield superb segmentation performance @xcite ). an attention mechanism has the advantage of being flexible for use with additional linguistic knowledge such as ccs and subword units. thus, we use the concept of cc with an attention mechanism in characterbased word segmentation, as shown in figure 3 , by extending the bilstm-crf architecture with word attention from @xcite . our model estimates cc-integrated character vectors z are incorporated on top of wordintegrated character vectors g, which are almost identical in architecture. we discuss the major components of our model, i.e., the character-embedding layer, word-and cc-embedding layers, bilstm layers for character representation, attention integrations with the bilstm layers for integrated representations, and crf layer. ## conclusion we proposed a character-based thai wordsegmentation model that uses multiple attentions on various types of linguistic knowledge, i.e., words, subwords, and ccs. the best version of our model achieved thai state-of-the-art performance by using the word attention along with cc attention in the bilstm-crf architecture. further analysis also indicates that using cc can be more beneficial than using subword units in word-segmentation tasks.