C ipher DA ug: Ciphertext based Data Augmentation for Neural Machine Translation (2022)

Abstract: We propose a novel data-augmentation technique for neural machine translation based on ROT- k ciphertexts. ROT- k is a simple letter substitution cipher that replaces a letter in the plaintext with the k th letter after it in the alphabet. We first generate multiple ROT- k ciphertexts using different values of k for the plaintext which is the source side of the parallel data. We then leverage this enciphered training data along with the original parallel data via multi-source training to improve neural machine translation. Our method, CipherDAug, uses a co-regularization-inspired training procedure, requires no external data sources other than the original training data, and uses a standard Transformer to outperform strong data augmentation techniques on several datasets by a significant margin. This technique combines easily with existing approaches to data augmentation, and yields particularly strong results in low-resource settings.

## introduction one naturally wonders if the problem of translation could conceivably be treated as a problem in cryptography. [...] frequencies of letters, letter combinations, [...] etc., @xcite are to some significant degree independent of the language used @xcite indeed, to a system which treats inputs as atomic identifiers, the alphabet behind these identifiers is irrelevant. distributional properties are of sole importance, and changes in the underlying encoding should be transparent provided these properties are preserved. in light of this, a bijective cipher such as rot-k (figure 1 ) is in effect invisible to modern nlp techniques: distributional features are invariant under such a cipher, guaranteeing that the meaning of an enciphered text is the same as the un-enciphered text, given the key. this work exploits this fact to develop a novel approach to data 1 our code is available at https://github.com/ protonish/cipherdaug-nmt plain abcdefghijklmnopqrstuvwxyz rot-1 bcdefghijklmnopqrstuvwxyza rot-2 cdefghijklmnopqrstuvwxyzab rot-3 defghijklmnopqrstuvwxyzabc src : es ist diese pyramide. rot-1(src): ft jtu ejftf qzsbnjef. rot-2(src): gu kuv fkgug rßtcokfg. tgt : it's that pyramid. ## ciphertexts for data augmentation a rot-k cipher (figure 1 ) produces a ciphertext by replacing each letter of its input (plaintext) with the kth letter after it in the alphabet. past work @xcite has explicitly used decipherment techniques @xcite to improve machine translation. we emphasize that decipherment itself is not the purpose of the present work: rather, we use ciphers simply to re-encode data while preserving its meaning. this is possible because rot-k is a 1:1 cipher where each ciphertext symbol corresponds to a unique plaintext symbol; this means it will preserve distributional features from the plaintext. this makes rot-k cryptographically weak, but suitable for use in data augmentation. concretely, given a set of n training samples @xmath0, y i )} n @xmath1, we use algorithm 1 to generate |k|n new samples; giving (|k| + 1)n samples when added to the training set. ## related work data-augmentation @xcite ) can be broadly categorized into back-translation based methods and those which perturb or change the input @xcite . back-translation @xcite is arguably the de-facto data augmentation method for nmt. besides back-translating external monolingual data @xcite , @xcite forward-translate the source @xcite and/or backward-translate the target side @xcite of the original (in-domain) parallel data. our technique produces lexically diverse samples using only the original source data, rather than relying on model predictions which may be of limited quality. @xcite showed that nmt models can be sensitive to orthographic variation, and that training with noise improves their robustness @xcite . common noising techniques include token dropping @xcite , word replacement @xcite , word-dropout (randomly zeroing out word embeddings; @xcite and adding synthetic noise by swapping random characters or replacing words with common typos @xcite . adding enciphered data is distinct from noising as the ciphertexts are generated deterministically and follow the same distribution as the underlying natural language, simply using shifted letters of the same alphabet. 15 to extend the support of the empirical data distribution, @xcite introduced raml on the target side; @xcite proposed switchout as a more general method which they applied to the source side. special cases of switchout include word-dropout and sequence-mixing @xcite , which exchanges words between similar source sentences to encourage compositional behaviour. such methods generate several different samples for each sentence because of the large vocabulary to choose replacements from; they often give poor coverage despite this. in contrast, ci-pherdaug guarantees lexically diverse examples with semantic equivalence to the source sentences without having to choose specific replacements. adversarial techniques @xcite ) perform soft perturbations of tokens or spans 15 cipherdaug can also apply to non-alphabetic scripts (e.g. mandarin, japanese) by incrementing unicode codepoints modulo the size of the block containing the script in question. @xcite ). an advantage of soft replacements over hard ones is that they take into account the context of the tokens being replaced @xcite . these methods require architectural changes to a model whereas cipherdaug does not. ciphertext-based augmentation is orthogonal to most other data-augmentation methods and can be seamlessly combined with these to jointly improve neural machine translation. ## conclusion we introduce cipherdaug, a novel technique for augmenting translation data using rot-k enciphered copies of the source corpus. this technique requires no external data, and significantly outperforms a variety of strong existing data augmentation techniques. we have shown that an agreement loss term, which minimizes divergence between representations of plain-and ciphertext inputs, is crucial to the performance of this model, and we have explained the function of this loss term with reference to co-regularization techniques from multi-view learning. we have also demonstrated other means by which enciphered data can improve model performance, such as by reducing the impact of rare words. overall, cipherdaug shows promise as a simple, out-of-the-box approach to data augmentation which improves on and combines easily with existing techniques, and which yields particularly strong results in low-resource settings.