Optimizing Word Segmentation for Downstream Task (2020)

Abstract: In traditional NLP, we tokenize a given sentence as a preprocessing, and thus the tokenization is unrelated to a target downstream task. To address this issue, we propose a novel method to explore a tokenization which is appropriate for the downstream task. Our proposed method, optimizing tokenization (OpTok), is trained to assign a high probability to such appropriate tokenization based on the downstream task loss. OpTok can be used for any downstream task which uses a vector representation of a sentence such as text classification. Experimental results demonstrate that OpTok improves the performance of sentiment analysis and textual entailment. In addition, we introduce OpTok into BERT, the state-of-the-art contextualized embeddings and report a positive effect.

## introduction tokenization is a fundamental problem in natural language processing (nlp). we must split a given sequence into a sequence of words for languages that do not contain obvious boundaries, such as chinese and japanese. in addition, it is also better to explore appropriate segmentations for languages containing obvious boundaries indicated by whitespaces, such as english @xcite @xcite @xcite . in traditional nlp, we tokenize a given sentence as a preprocessing. thus, as shown in figure 1 (a), we apply an existing tokenizer to the given sentence, and then input the tokenized sentence into a model for a target downstream task. in the conventional approach, we obtain the most plausible tokenized sentence based on the tokenizer; however, some studies have varied the tokenization using a sampling during the training to enable the downstream model to adapt to various tokenizations @xcite ; provilkov we directly optimize the tokenizer to improve the performance of the model for a downstream task using the loss of the target task. @xcite . although such a strategy makes the downstream model robust, little attention has been paid to optimizing the tokenizers for a downstream task. thus, if we acquire an appropriate tokenization to a downstream task, we might improve the task performance. by contrast, some studies have used multiple tokenized sentences to prevent the damage depending on the tokenization @xcite @xcite . their methods compute various tokenizations for a given sentence, and then encode the tokenizations using an architecture based on the lstm @xcite . although their methods prevent the error propagation from the tokenizer, they are intractable when handling all possible tokenizations owing to the computational costs required. this paper describes an exploration into an appropriate tokenization to the downstream tasks. we propose a novel method to optimize a tokenizer based on the downstream task, as shown in figure figure 2 : outline of the proposed method for calculating a sentence vector h s with the 3-best tokenizations during the training phase. at the inference, we use the 1-best tokenization as well as general neural architectures. the arrows along the continuous line indicate the differentiable paths for backpropagation. we can use various architectures as the encoder, which converts a sequence of tokens into a single vector. downstream model is the architecture for the downstream tasks, i.e., mlp for text classification. we design the proposed method to be used for any downstream task that uses a vector representation of a sentence. we conduct experiments on text classification in three languages, and show the effectiveness of the proposed method. moreover, we indicate that we can also introduce our proposed method into the pre-trained architecture. we combine the proposed method with the state-of-the-art contextualized embeddings, bert @xcite , and improve its performance. ## code: https://github.com/tathi/optok to search s satisfying argmax s (q(z, f (s ))), we train optok based on the score of the downstream task (q(z, f (s ))). thus, we optimize both optok and the downstream model simultaneously in contrast to a traditional pipeline approach, which tokenizes a given sentence as a preprocessing. op-tok generates multiple tokenized sentences as candidates, and we train optok to assign a high probability to a better tokenization based on the score of the downstream task. during the inference step, we make optok output only the most plausible tokenized sentence to reduce the computational costs. figure 2 shows an overview of optok with the downstream model during training. optok constructs n tokenized sentences and converts them into vector representations with a neural encoder. then, optok combines the probabilities of each tokenization with the vector representations. we compute the sum of the vector representations weighted by the probabilities, and then input it into the downstream model. thus, optok becomes to assign a high probability to the tokenization which improves the performance of the downstream task. we therefore can obtain s satisfying argmax s (q(z, f (s ))) through the training. we describe the details of each module in this section. ## experiments the goal of this study is to improve the performance of downstream tasks by optimizing the tokenization. therefore, we evaluate optok on various text classification tasks to validate its effect. ## related work numerous studies have aimed to improve the nlp tasks from the perspective of tokenization. some studies have proposed an approach to prevent segmentation errors by encoding multiple tokenizations jointly. recent studies investigated lattice lstm, which expands lstm to allow multiple segmentations to be taken as a lattice @xcite @xcite . @xcite followed them to utilize a transformer. subword regularization is a famous solution to this problem @xcite . the authors demonstrated that training models with various tokenizations contribute to an improved performance for machine translations. @xcite followed this approach by dropping tokens during the bpe process to vary tokenization, and @xcite by updating the language model during training. optimization of the tokenization has attracted attention mainly in the field of machine translation. some studies have attempted to optimize the tokenization using simple criteria for a machine translation @xcite @xcite . recent studies also tackled this issue for generation tasks. @xcite developed incremental bpe, which automatically defines the number of bpe's merge operation for neural machine translation. @xcite proposed a neural architecture to find a better subword sequence from both the source and target corpora by enhancing the study in @xcite . our work differs from the existing research in that the proposed method is appliable to various neural encoders, and we can optimize the tokenization directly using only backpropagation from the training loss of the downstream tasks without any hand-crafted criteria. ## conclusion in this paper, we propose optok which explores an appropriate tokenization to the target downstream task. we combine optok with the downstream model and train them simultaneously. the experimental results indicate that optok improves the performance of several downstream tasks through better tokenization. moreover, optok also has a positive effect on pre-trained contextualized word embeddings such as bert.