Implementing a neural machine translation engine for mobile devices: the Lingvanex use case (2018)

Abstract: In this paper, we present the challenge entailed by implementing a mobile version of a neural machine translation system, where the goal is to maximise translation quality while minimising model size. We explain the whole process of implementing the translation engine on an English–Spanish example and we describe all the difficulties found and the solutions implemented. The main techniques used in this work are data selection by means of Infrequent n-gram Recovery, appending a special word at the end of each sentence, and generating additional samples without the final punctuation marks. The last two techniques were devised with the purpose of achieving a translation model that generates sentences without the final full stop, or other punctuation marks. Also, in this work, the Infrequent n-gram Recovery was used for the first time to create a new corpus, and not enlarge the in-domain dataset. Finally, we get a small size model with quality good enough to serve for daily use.

## introduction lingvanex 1 is a trademark for linguistic products made by nordicwise llc company. the main focus of the company are translator and dictionary applications that work without internet connection on mobile and desktop platforms. c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 https://lingvanex.com/en/ in collaboration with sciling 2 , an agency specialised in providing end-to-end machine learning solutions, a small-sized translation model from english to spanish was implemented. when implementing a mobile translator, it is crucial to understand its purpose. in our case, the purpose was to be able to generate translations on a daily usage scenario, without requiring a internet connection. this is the typical use case in a travel scenario, where travellers often do not have an internet connection, either because they do not want to assume the cost of a roaming connection, because they do not want to purchase a local sim card, or even because there is no good connection in the places they are travelling to, such as some countries of africa. in this scenario, the main purpose of the translation engine is to be able to translate correctly short sentences, composed of common words in a traveller domain, but where other words belonging to e.g. a parliamentary or a medical domain are less frequent. in addition, the model requires to be contained in terms of size, since large models would perform poorly in a mobile device. in this work, we focus on reducing model size mainly through data selection techniques, until a size of 150mb per model. however, there are other techniques which bring promising results as compressing the nmt model via pruning @xcite . along this article we determine what is the main influence to model size. for that, we conducted experiments comparing model size with total vocabulary size and word embedding size. also, we compare the model size with different layer number on encoder and decoder 2 https://sciling.com/ side, and the size of recurrent layer. next step is to select data for training the engines through sentence length filtering and leveraging a ds technique. during the implementation of our translation engines we found several problems in the translations generated. we describe each of the problems and we propose appropriate solutions. after implementing these solutions, we evaluate the quality of our final model on a test set, and compare the results with google's and microsoft's mobile translators. ## data description the data used to train the translation model was obtained from the opus foot_0 corpus. in total, there were 76m parallel sentences. we also leveraged the tatoeba corpus for ds described in section 4. tatoeba is a free collaborative online database of example sentences geared towards foreign language learners. the development set was also built from the tatoeba corpus, by selecting 2k random sentence pairs. main figures of tatoeba corpus are shown in to determine how the previously enumerated hyper-parameters affect model size, we trained different models varying these hyper-parameters. in the first experiment, we set the number of units in the recurrent layer to 128, with a single layer on both encoder and decoder sides. we analysed the effect of considering a total combined (source and target) vocabulary size |v | was pruned to |v | = {5k, 10k, 20k, 50k, 100k}, selected according to the most frequent words in the opus corpus, with souce and target vocabulary size set to |v |/2. in addition, we also studied different embedding vector sizes |ω| = {64, 128, 256, 512}. the results obtained are shown in figure 1a . next, we analysed the effect of considering different number of hidden units and the number of layers. in this case, we fixed to |ω| = 128. we found that the number of layers, using 128 hidden units, has almost no effect on model size. in figure 1b , we only show 1 and 4 layers for 128 units. looking at figure 1 , we can conclude that the number of layers has small effect on model size comparing with number of hidden units and embedding size. figure 1 can be leveraged to decide on adequate values for these hyper-parameters, once model size has been fixed to 150mb. ## data filtering data filtering involved two main steps: first, sentences with length over 20 words were discarded. we did this under the assumption that a mobile translator is mainly designed for translating short sentences. second, we performed data selection, leveraging infrequent n-gram recovery @xcite . the intuition behind this technique is to select, from the full available bilingual data, those sentences that maximise the coverage of n-grams in a small, domain-specific dataset. the full available bilingual data is sorted by infrequency score of each sentence in order to select first the most informative. let be χ the set of n-grams that appear in the sentences to be translated and w one of them; c(w) denotes the counts of w in the source language training set; t the threshold of counts when an n-gram is considered infrequent, and n (w) the counts of w in the source sentence f to be scored. the infrequency score of f is: we applied infrequent n-gram recovery to the 60m sentences from the opus corpus as out-of-domain. intuitively, we selected sentences from the available data until all n-grams, with n up to 5, extracted from the tatoeba corpus have a maximum of 30 occurrences (if such a thing is possible with the data available). however, applying this technique on the full set of 60m sentences would have led to very long execution time. hence, we divided the corpus into 6 partitions, and the selection was performed on each one of these partitions. then, we joined the selections from all 6 partitions and conducted a second selection step on this corpus, since some n-grams could well have 6 • 30 occurrences. this led to a final selection of 740k sentences. the selected data set presented a vocabulary size of 19.4k words in source and 22.9k on target side. the total (combined) vocabulary was |v | = 42.4k. note that selection was conducted on the tokenised and lowercased corpus. ## experimental setup the system was trained using the opennmt @xcite deep learning framework based in torch. opennmt is mainly focused at developing sequence-to-sequence models covering a variety of tasks such as machine translation, summarisation, image to text, and speech recognition. byte-pair encoding (bpe) @xcite was trained on the selected training dataset, and then applied to training, development, and test data. in each experiment we trained a recurrent neural network with long short term memory (lstm) @xcite . we use global attention layer to improve translation by selectively focusing on parts of the source sentence during translation. also, we use input feeding to feed attentional vectors as inputs to the next time steps to inform the model about past alignment decisions @xcite . however, this option only had a visible effect with 4 or more layers. training was performed with 50 epochs using the adam (kingma and ba, 2014) optimiser, with learning rate of 0.0002. finally, we selected the best model according to higher bleu @xcite score on the development set, and used that model to translate the test set. given that the test set is very small, we performed a human evaluation to analyse whether the quality obtained was good enough. ## results and analysis we trained different typologies of neural networks observing the conclusions in section 3. in each of the experiments we varied the hyper-parameters described in section 3. since the total combined vocabulary was fixed to |v | = 42.4k, from figure 1 we can infer the combination of hyper-parameters with which the allowed model size will not be exceeded. the best model according to the bleu score on the development set is the model trained with 2 ## final evaluation we noticed that our model was especially accurate when using punctuation marks and capital letters, whereas google's translator introduced punctuation marks in wrong places. also, only in a few cases, google's translator, uses capital letters. we believe this is the reason why google's translator achieved such a low bleu score, as compared to the other two systems. however, google's translator features a much smaller than the other two others. also, google's and microsoft's models are bidirectional, which means that the size of our model should be doubled (2 • 121mb) to be comparable. ## conclusions in this work, we have presented our approach to developing a small size mobile neural machine translation engine, in the specific case of english-spanish. we leveraged a data selection technique to select more suitable data for real use of our translator. we have presented some adjustments to the selection algorithm the translation quality obtained. also, we proposed a solution to deal with the problem of repeated words, and another one for dealing with missed sentence translations within some segments. finally, we compared the quality of our model with google's and microsoft's mobile translator versions. we overcome significantly the bleu score of both translators, partially due to being able to translate punctuation marks and capital letters correctly. our model reached a size of 121mb, which is even much smaller than the size we considered initially as acceptable, presenting good translation quality for the specific purpose (travel domain). the translations obtained by our model are perfectly understandable and fluent, and can be used in a scenario where there is no internet connection. in addition, we are still working on improving its quality and on reducing model size even further, using other effective techniques such as weight pruning.