Robust Open-Vocabulary Translation from Visual Text Representations (2021)

Abstract: Machine translation models have discrete vocabularies and commonly use subword segmentation techniques to achieve an ‘open vocabulary.’ This approach relies on consistent and correct underlying unicode sequences, and makes models susceptible to degradation from common types of noise and variation. Motivated by the robustness of human language processing, we propose the use of visual text representations, which dispense with a finite set of text embeddings in favor of continuous vocabularies created by processing visually rendered text with sliding windows. We show that models using visual text representations approach or match performance of traditional text models on small and larger datasets. More importantly, models with visual embeddings demonstrate significant robustness to varied types of noise, achieving e.g., 25.9 BLEU on a character permuted German–English task where subword models degrade to 1.9.

## introduction machine translation models degrade quickly in the presence of noise, such as character swaps or misspellings @xcite . part of the reason for this brittleness is the reliance of mt systems on subword segmentation @xcite as the solution for the openvocabulary problem, since it can cause even minor variations in text to result in very different token sequences, needlessly fragmenting the data ( after presenting the visual text embedder (sec tion 2), we demonstrate the potential of visual rep resentations for machine translation across a range of languages, scripts, and training data sizes (sec tion 4). we then look at a variety of types of noise, and show significant improvements in model ro bustness with visual text models (section 5). 2 visual text embedder ## chasing translation parity stateoftheart translation models use subword vocabularies, which yield best performance when tuned per language pair and task @xcite . our visual text approach avoids predetermining a fixed model vocabulary. on the one hand, this allows us to represent even unanticipated characters; on the other, optimizing a finite model vocabulary per task may improve performance. our first question, therefore, is whether visual text can recover scores produced by baselines with optimized subword vocabularies. on the smaller mttt dataset, we can nearly recover the best results from the most optimal bpe segmentation without explicit input segmentation, solely from visual representations with a sliding window. ## robustness to noise we hypothesize that without a fixed vocabulary and with associations between visually similar character spans, our visual text models will be more robust to noise than textbased representa tions, where noise causes divergent subword rep resentations (see ## related work visual representations of text have previously been explored for other nlp tasks, primarily for chinese, with mixed results. @xcite used visual representations from cnns over rendered text in chinese, japanese, and korean for text classification. @xcite similarly used convolutions over characterlevel images for chi nese for downstream language modeling and word segmentation. @xcite used hybrid con volutional networks to improve several nlp tasks for chinese, using historical scripts for additional pictographic information. @xcite created dense fixedsize square text renderings in chinese and english for convolutions for down stream sentiment analysis. @xcite used visual similarity for russian romanization. in machine translation, visual information was also first used for chinese. initial work improved translation models by initializing character em beddings with linearized bitmaps of each character @xcite , and more recently, with linearized images compressed with pca, which improved model robustness @xcite incorporated visual compositionality of chinese characters for mt through wubi ascii encodings. @xcite assessed the impact of visual perturbations on current nmt systems, and augmented characterlevel text embeddings with visual information to make systems more robust to such attacks. @xcite explored the challenging task of imagetoimage translation; while their approach is exploratory and not yet competitive, and producing images introduces difficulties with evaluation, it is simi larly motivated by a desire to do away with fixed, predefined vocabularies and tokenization. previous work has explored the impact of syn thetic and natural noise on neural mt @xcite , and the use of characteraware word embeddings @xcite to increase generalizability and robustness. while research regularization and dropout techniques for bpe @xcite have im proved model robustness, discrete vocabulary sets still creates challenge in many use cases. recent work has also explored bytelevel bpe @xcite to create mod els which are not restricted to the unicode ranges seen in training, though models using bbpe may require additional training examples. ## conclusion & future work we introduced visually rendered text for continu ous openvocabulary translation. we showed that our models, trained on seven language pairs and in two data settings, approach or match the per formance of traditional text models. further, we showed that visual text models are more robust to many kinds of induced noise, including the sub stitution of visually similar characters and charac ter permutations. an important benefit of our ap proach is that it operates on raw text, doing away with the standard preprocessing routines that in clude normalization, tokenization, and subword segmentation. we believe our approach opens many avenues for future work. standard data techniques from ocr (such as varied font and font size) and train ing on noise would likely further improve robust ness. there are many possible visual architec tures, and visual pretraining has benefited vision tasks @xcite . there is nothing to preclude our approach from working on larger datasets. while effective, it is not clear that slid ing window segmentation is optimal; improving segmentation could close remaining performance gaps. since our approach does away with discrete vocabularies, visual text models could be used to transfer to new languages and scripts without re quiring transliteration or normalization, or retrain ing models from scratch. finally, it is appealing to consider this approach for additional tasks such as language identification @xcite ,