A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning (2022)

Abstract: Subword tokenization is a commonly used input pre-processing step in most recent NLP models. However, it limits the models’ ability to leverage end-to-end task learning. Its frequency-based vocabulary creation compromises tokenization in low-resource languages, leading models to produce suboptimal representations. Additionally, the dependency on a fixed vocabulary limits the subword models’ adaptability across languages and domains. In this work, we propose a vocabulary-free neural tokenizer by distilling segmentation information from heuristic-based subword tokenization. We pre-train our character-based tokenizer by processing unique words from multilingual corpus, thereby extensively increasing word diversity across languages. Unlike the predefined and fixed vocabularies in subword methods, our tokenizer allows end-to-end task learning, resulting in optimal task-specific tokenization. The experimental results show that replacing the subword tokenizer with our neural tokenizer consistently improves performance on multilingual (NLI) and code-switching (sentiment analysis) tasks, with larger gains in low-resource languages. Additionally, our neural tokenizer exhibits a robust performance on downstream tasks when adversarial noise is present (typos and misspelling), further increasing the initial improvements over statistical subword tokenizers.

## introduction subword tokenization methods, such as bpe @xcite , word-piece @xcite , and unigram @xcite , rely on a predefined vocabulary to tokenize text. this vocabulary is built based on frequencies of word fragments. as a result, rare words are highly fragmented into many subpieces, whereas the integrity of the most frequent words is substantially preserved @xcite . this vocabulary bias is magnified in multilingual settings, where low-resource languages are heavily discriminated in favor of high-resource ones @xcite @xcite (see to address the aforementioned issues, we design a vocabulary-free neural tokenizer, which we train in two phases. first, in the pre-training phase, we train our neural tokenizer by distilling the segmentation information from a subword tokenizer. in the multilingual setting, our neural tokenizer learns from the language-specific subword tokenizers so that it is not biased towards high-resource languages. after the pre-training phase, the neural tokenizer segments the character sequence without requiring a predefined vocabulary. in the second phase, we employ an endto-end learning approach, which allows our neural tokenizer to adapt the tokenization behavior to the downstream task. such an end-to-end approach is not feasible for models with subword tokenizers due to the predefined vocabulary and their strong ties to the models' embedding layer. additionally, unlike the subword tokenizers, our neural tokenizer does not require a vocabulary, and its versatile alphabet reduces the bias towards high-resource languages (i.e., there is not unbalanced word coverage favoring specific languages). we compare the impact of our approach with respect to the subword tokenizers in downstream monolingual, multilingual, and code-switching tasks. for multilingual nli, the results show that our neural tokenizer generally improves the model performance, with substantially larger gains for low-resource languages (+11 absolute points of accuracy for thai, +8 for arabic, and +4 for swahili). for code-switched spanish-english sentiment analysis, our neural tokenizer also outperforms the baseline tokenizers, demonstrating better language generalization capabilities. we inspect the robustness of our neural tokenizer in the presence of noisy text through adversarial manipulation (e.g., typos and spelling variations), and we find that the tokenization result is much more resilient to generate junk tokens (i.e., excessive fragmentation of subword pieces) than the subword tokenizers. finally, we provide extensive experimental analyses that consistently suggest to adopt our approach for more robust and versatile representations of text. ## method we propose a learnable tokenizer that is trained to convert sequence of characters into meaningful subword-level tokens. consider the multilingual alphabet ψ (i.e., a closed set of letters) and the character sequence @xmath0, . . . , c n ] that represents a word of length n and c @xmath1. we aim at learning the corresponding iob foot_0 sequence of tags @xmath2, . . . , t n ] that groups characters into the desired tokenization: here ℓ denotes the language of the word. the model f θ can be any neural architecture that allows a oneto-one mapping from the input to the output. 2 we condition the model on ℓ in the multilingual setting, while the monolingual variant does not require it. a trained neural tokenizer, f θ , is capable of providing tokenization as a stand-alone tool, which can be compared directly to the standard subword tokenizers (e.g., controlling by the task-specific model in a downstream setting). additionally, a trained neural tokenizer can expose the internal representations of a segmentation so that it enables end-to-end task learning by optimizing the tokenization towards the task particularities. we describe both scenarios in more detail in the following subsections. ## experimental results and discussion we have evaluated the impact of our neural and subword tokenizers on multilingual and monolingual natural language inference (nli) tasks and on a sentiment analysis task with code-switched language. we also evaluated the impact of tokenizers in the presence of noisy data (typos and misspelling). ## conclusion we propose a neural tokenizer to segment text without a vocabulary, which allows end-to-end task learning. the experimental evaluations on multilingual nli task suggest that our neural tokenizer reduces the model size and improves the task's performance for low-resources languages, such as arabic, swahili, and thai. moreover, the neural tokenizer outperforms subword tokenizers on the nli task with noisy text (typos and misspelling). the qualitative analysis also suggests that our neural tokenizer improves the tokenizations of the subword tokenizers, which is used to train our neural tokenizer. additionally, the neural tokenizer shows comparable performance on sentiment analysis task with code-switched languages. the experimental results suggest that our neural tokenizer can distill the segmentations knowledge from multiple subword tokenizers to improve the tokenization. this finding opens future research avenues to design a learnable tokenizer for improving the state-of-the-art subword tokenization and the downstream task's performance.