HFT : High Frequency Tokens for Low-Resource NMT (2022)

Abstract: Tokenization has been shown to impact the quality of downstream tasks, such as Neural Machine Translation (NMT), which is susceptible to out-of-vocabulary words and low frequency training data. Current state-of-the-art algorithms have been helpful in addressing the issues of out-of-vocabulary words, bigger vocabulary sizes and token frequency by implementing subword segmentation. We argue, however, that there is still room for improvement, in particular regarding low-frequency tokens in the training data. In this paper, we present “High Frequency Tokenizer”, or HFT, a new language-independent subword segmentation algorithm that addresses this issue. We also propose a new metric to measure the frequency coverage of a tokenizer’s vocabulary, based on a frequency rank weighted average of the frequency values of its items. We experiment with a diverse set of language corpora, vocabulary sizes, and writing systems and report improvements on both frequency statistics and on the average length of the output. We also observe a positive impact on downstream NMT.

## high frequency tokenizer hft uses the advantage of pretokenization, where sentences are split into tokens on the borders of alphanumeric and non-alphanumeric characters. the current prototype uses the regular expression \b of the unix sed 1 command . both the beginning and the end of each token is explicitly annotated. hft subwords are learnt from these tokens, they never cross the token boundaries, each token from the pretokenization is handled independently from other tokens. it speeds up both vocabulary learning and actual subword tokenization. we also use case normalization for characters with both uppercase and lowercase. a single uppercase letter is changed to a @xmath0. a sequence of uppercase letters is changed to lowercase with a @xmath1. figure 2 gives the special characters hft uses in pretokenization and tokenization. the learning algorithm starts from a vocabulary containing all characters from the training text as possible subwords. the vocabulary contains the number of occurrences of the given subword (character). then it gradually increase the vocabulary in the following steps: the best subword tokenization (in step 1) searches in all possible subword segmentation sequences the one with the lowest number of tokens and (for same number of tokens) the highest minimum frequency. ## evaluation we train and compare hft with the sentence piece @xcite implementation of bpe @xcite and unigram @xcite on two of the metrics presented by gowda and may (2020). we use portions of different bilingual and monolingual datasets: the english section of the english-marathi and the irish part of the english-irish from the loresmt 2021 shared task foot_0 ; a sample of the hindi half of the english-hindi iitb parallel corpus @xcite and of the lithuanian portion of the lithuanian-english of europarl @xcite used in the wmt19 news translation shared task foot_1 . as for monolingual corpora, we used different translations of the bible foot_2 @xcite in a diverse range of languages and writing systems retrieved from opus @xcite . figure 3 gives an overview of the size of the datasets. ## limitations and future work as we mentioned in section 2.2, we opted to include in the final vocabulary each character seen in the training data at least once to avoid out-ofvocabulary tokens. this, however, has the sideeffect of saturating and inflating the vocabulary with low-frequency tokens. at lower vocabulary sizes, these entries make up a bigger percentage of the overall vocabulary, becoming less and less relevant as the size increases. this explains why our method becomes effective over a threshold in the size of the vocabulary, which varies depending on the size and, more importantly, on the amount of unique characters in the dataset. the presence of these character may be due to the inherent complexity of the writing system. this is the case of japanese, which uses two sets of syllabic characters, hiragana and katakana (collectively referred to as kana), and a huge amount of chinese-derived ideograms, called kanji. the other source for unique characters in the data is noise, in the form of non-standard orthography, special characters, non-linguistic text, and so on. if present in the dataset, these sections can quickly saturate our vocabulary. this issue makes the method somewhat susceptible to noise, and must be addressed in future work. another aspect to investigate further is the relationship between µ and different writing systems. from our evaluation, we have seen that hft improves the performance on this metric for most of the data and writing systems we included in our evaluation. however, for japanese and burmese, µ is higher than other methods. it is worthwhile to investigate this matter in the future. while the preliminary evaluation of hft's impact on downstream nmt seems to show promising results, we acknowledge that the testing sample is not vast. moreover, using an unoptimized transformer does not completely reflect real-world applications, where the nmt system would be carefully tuned to the specific dataset. we plan to undertake a more comprehensive evaluation on downstream translation in the future, by enlarging the testing sample and employing hft in settings closer to real applications. lastly, we report that hft has longer training time than other algorithms in the current implementation, which are however still in the range of minutes for the bigger datasets we used. we plan to work on this shortcoming in the next implementation of the tokenizer. ## related work before @xcite , mt coped with the problem of out-of-vocabulary words by backing off to a dictionary with sub-optimal assumptions regarding morphological identities and transliterations. bpe addressed this issue by adapting a compression algorithm to the task of word segmentation. the methods initializes the symbol vocabulary with the character vocabulary, plus a special end-of-word symbol. then it iteratively counts all symbols pairs and replaces every occurrence of the most frequent pair ('a', 'b') with the new symbol 'ab'. these character n-grams are then merged together in a similar fashion. they do not consider pairs that cross word boundaries. following these steps, bpe allows for open-vocabulary nmt, which better handles out-of-vocabulary and rare words, by representing them as a sequence of subword units. the unigram method by @xcite is based on a unigram language model. this makes the assumption that each subword occurs independently, thus formulating the probability of a subword sequence @xmath2, ..., x m ) as the product of the subword occurrence probabilities p(x i ). to find the vocabulary set and their probabilities, they employ an iterative algorithm which starts by creating a seed vocabulary of unique characters and most frequent substrings, without considering those that cross word boundaries. then the following steps are repeated until the vocabulary reaches the desired size: i. fixing the set of vocabulary, optimize p(x) with the expectation maximization (em) algorithm; ii. compute the loss i , for each subword x i , as the amount the likelihood l is reduced when removing x i from the vocabulary; iii. sort the symbols by loss, and keeping the top n% of subwords, while always keeping single characters to avoid outof-vocabulary. thus unigram can output multiple segmentations and their probabilities, making it more flexible than bpe. in sentence piece @xcite both these segmentation algorithms are implemented in a way that removes the need for preprocessing steps, such as pretokenization, and trains subword models directly from the raw sentences. this allows for the creation for a purely end-to-end and language-independent system. ## conclusions in this paper we present high frequency tokenizer, or hft, a new language-independent subword tokenization algorithm to improve on the frequency coverage of tokens in the vocabulary of nmt systems. we demonstrate its performance on a diverse dataset of languages and writing systems, and show that our approach can be beneficial to downstream nmt. however, some issues still remain to be investigated, such as the frequency coverage for smaller vocabularies and the mean output length for some figure 6 : example from the mr-en translation systems. the first line gives the reference translation, the second gives the translation from a bpe-based system, while the last gives the translation from an hft-based system. the named entity naxals is preserved by hft. the hft scripts are available on github, 8 together with the evaluation's data and results. 9