Semantic Diversity by Phonetics: Towards Accurate and Robust Machine Translation
Abstract
Neural Machine Translation (NMT) learns from examples, and thus often lacks robustness against noise.
Previous work has shown that integrating noise into the training process is effective at improving such robustness, but this solution can be inefﬁcient due to the exponential number of string perturbations, i.e., exponential in the number of words or characters.
To robustify the translation input, we treat human phonetic interaction throughout history as a pre-compiled computational device.
This device implements a many-to-one function that converts text into phonetics.
To the best of our knowledge, we are the ﬁrst in Machine
Translation, to apply the phonetic algorithms
Soundex, NYSIIS, and MetaPhone to foreign word/character sequences.
We also apply another linguistic representation, the logogram inference, Wubi, for Chinese.
To explain why phonetic encodings improve NMT, we introduce, quantify, and empirically verify our hypothesis: “one phonetic representation usually corresponds to words that are semantically diverse.”
Driven by our hypothesis, we simulate this “natural” phonetic device and introduce an artiﬁcial method called random clustering.
We achieved signiﬁcant and consistent improvements overall language pairs and datasets we experimented with: French-English, GermanEnglish, and Chinese-English in IWSLT’17, with up to nearly 2 BLEU points over the stateof-the-art.
Moreover, our approaches are more robust than baselines when evaluated on unknown noisy or out-of-domain test sets, with up to about 5 BLEU point increase.
Upon acceptance, all software source code and experiments will be available as Open Source
Introduction
Machine translation (MT) has achieved remarkable success with milestone contributions including, but not limited to, (Koehn et al., 2007; Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017;
Vaswani et al., 2017).
State-of-the-art MT systems are trained with massive human labeled samples and have achieved high accuracy in evaluations (Bojar et al., 2018).
However, in real life, an outof-domain test set can easily make MT fail.
Moreover, noisy sets due to string distortions like typos, slang, dialect, idiolect and informal use of languages such as acronym, abbreviation, and emoji can have adverse effects on MT and lower translation quality (Belinkov and Bisk, 2018; Khayrallah and Koehn, 2018; Wang et al., 2018).
Important past works have branched in two main directions: domain adaptation (Jiang, 2008;
Carpuat et al., 2013; Freitag and Al-Onaizan, 2016;
Wang et al., 2017; van der Wees et al., 2017; Zhang et al., 2019) and noisy data augmentation (Liu et al., 2018; Karpukhin et al., 2019; Vaibhav et al., 2019).
Both leverage the datasets between training and testing but one with aspects of the domain and the other with text expression, respectively.
Though experiments have demonstrated that these methods are effective, they can be inefﬁcient for NMT training due to a large number of string perturbations and possible domains.
Importantly, they cannot be generalized to an arbitrary distorted test set if we know nothing about its distribution in advance.
We aim to improve MT robustness and accuracy in general.
We introduce a novel framework for
NMT using phonetic information “computed” by the human interaction throughout the evolution of spoken language.
We view social interaction as a computational device that generates pre-computed knowledge.
Phonology has been shown to preserve semantic meanings (Tyler et al., 1996; Beaver et al., 2007), which coincides with neurological discoveries about the correlation between phonology and semantics in the human brain (Wang et al., 2016;
Amenta et al., 2017; Paz-Alonso et al., 2018).
Table 1 shows examples of Pinyin and Soundex.
We
[Figure] [Figure]
view these encodings as many-to-one functions, where multiple words are mapped to one.
Thus, we decided to give, by using phonetic algorithms, this new form of phonetic sentence representation together with its written form as an input when training and decoding the neural networks.
It is a challenging task to purely rely on neural networks to extract all hidden features in
NMT.
Therefore, adding auxiliary information will potentially allow a simpler network structure.
As shown in Figure 1, we ﬁrst apply phonetic encoding, logogram, or random clustering to the foreign input-sentences, then use Byte-Pair-Encoding to learn a word embedding (marked as empty boxes) on each individual coding representation.
Finally, we concatenate them with the embedded original text to feed into the NMT model.
We achieved signiﬁcant and consistent improvements over the state-of-the-art on all language pairs that we experimented with.
Our approaches robustify the baseline NMT.
In particular, they lead to a higher accuracy even on an arbitrary test set whose distribution is oblivious at training time.
This is a general approach that is applicable to any language that can be compiled with phonetics, and it generically beneﬁts any NMT system, which it treats as a black-box.
Importantly, we performed a systematic empirical analysis to explain why phonetic encoding helps in NMT.
Our hypothesis of semantic diversity by phonetics is stated as “One phonetic representation usually corresponds to characters/words that are semantically diverse.”
We performed three quantitative analyses to verify this hypothesis.
Then, driven by our hypothesis, we introduced a new random clustering algorithm that casts words or characters into classes, which also improved translation accuracy and robustness.
This work contains two areas of study: phonetic encoding and random clustering, where the former inspires the latter in accordance with our stated hypothesis.
Our contributions mainly include the hypothesis of semantic diversity by phonetics and below models:
Phonetic and logogram encodings We convert the source input text using various algorithms such as Soundex, NYSIIS, and MetaPhone for western languages: English, French, and German.
For Chinese, we apply a Logogram encoding, Wubi.
Both phonetic and logogram encodings as auxiliary inputs significantly improve translation results.
Random clustering Word/character clustering signiﬁcantly improves NMT.
This empirical ﬁnding aligns with the empirical justiﬁcation of why phonetic encoding improves translation accuracy.
We conducted extensive experiments and achieved up to nearly 2 BLEU points on IWSLT’ tasks over the state-of-the-art in translation directions of English-German, German-English,
English-French, French-English, and ChineseEnglish.
We veriﬁed that our approaches are more robust on French-English experiments with about 5 BLEU point improvement on a foreign test set whose distribution is oblivious during training.
Below, we will ﬁrst introduce the phonetic and logogram encodings .
Then, we will study why using these as auxiliary inputs improves NMT and propose our hypothesis.
Consequently, we will introduce one artiﬁcial method, random clustering , as a generalization to text encoding .
Finally, we will demonstrate that all of these approaches significantly boost our NMT accuracy and robustness.
Background
NMT is an approach to MT using neural networks, which takes as an input a source sentence ( x 1 , .., x t , .., x I ) and generates its translation ( y 1 , .., y t (cid:48) , .., y I (cid:48) ) , where x t and y t (cid:48) are source and target words respectively (Bahdanau et al., 2015;
Sutskever et al., 2014; Cho et al., 2014).
NMT models with attention have three components, namely, an encoder, a decoder, and an attention mechanism.
The encoder summarizes the meaning of the input sequence by encoding it with a bidirectional recurrent neural network (RNN).
We apply the sequence-to-sequence learning architecture by
Gehring et al.
(2017), where the intermediate encoder and decoder states are calculated using convolutional neural networks (CNNs).
Phonetic Encodings
A phonetic algorithm is used to index words by their pronunciation.
Taking as the input a sequence of words, we apply the phonetic algorithm to each word and output a sequence of encodings.
3. Soundex
Soundex is the most widely known phonetic algorithm for indexing names by sound, as pronounced in English, and avoids misspelling and alternative spelling problems.
It maps homophones to the same representation so that they can be matched despite minor differences in spelling (Russel, 1918).
It clusters the letter with exceptions.
For example, the Soundex key letter codes ‘b, f, p, v’ to ‘1’, and ‘c, g, j, k, q, s, x, z’ to ‘2’, and ‘d, t’ to ‘3’.
3. NYSIIS
The New York State Identiﬁcation and Intelligence System Phonetic Code, commonly known as
NYSIIS, is a phonetic algorithm devised inas part of the New York State Identiﬁcation and
Intelligence System (Rajkovic and Jankovic, 2007).
It produces better results than Soundex because it takes special care to handle phonemes that occur in
European and Hispanic surnames.
3. MetaPhone
Metaphone (Philips, 1990) is another algorithm that improves on earlier systems such as Soundex and NYSIIS.
The Metaphone algorithm is signiﬁcantly more complicated than the others because it includes special rules for handling spelling inconsistencies and for looking at combinations of consonants in addition to some vowels.
3. Hanyu Pinyin
We also studied Hanyu Pinyin (Pinyin), the ofﬁcial romanization system for Standard Chinese in mainland China. Pinyin means ‘spelled sound’ and is
Algorithm
Input : translation units Parameter : baseline encoding
Output : mapping of units to clusters 1: perform a phonetic or logogram encoding as baseline 2: for each unique code in the baseline encoding vocabulary do 3:
Z = “how many units are mapped” 4: uniformly random sample Z units to form a new cluster 5: end for 6: return usually used for the purpose of teaching Mandarin.
One Pinyin corresponds to multiple Chinese characters.
One Chinese word is usually composed of one, two, or three Chinese characters.
3. Logogram Encoding: Chinese Wubi
The Wubizingxing (Wubi or Wubi Xing) is a Chinese character input method primarily for efﬁciently inputting Chinese text with a keyboard.
The
Wubi method is based on the structure, namely the decomposition of characters rather than their pronunciation.
Every character can be written with at most 4 keystrokes including -, | , 丿 , hook, and 丶 .
Random Clustering
Driven by our hypothesis, which will be elaborated in Section 5, we further introduce an artiﬁcial way to encode the text in order to simulate “natural” encoding, i.e. phonetics and logogram.
We call this random clustering as described in Algorithm 1.
We cluster words (or characters) uniformly at random.
The cluster size follows the distribution of how many words/characters are associated with each phonetic, here MetaPhone.
For example, in
Chinese, each Pinyin is a cluster, and, the number of clusters equals the number of unique Pinyins.
Furthermore, each cluster’s size is the same as the number of characters mapped to each Pinyin.
Hypothesis
Hypothesis:
One phonetic representation (for example, Pinyin in Chinese) usually corresponds to characters/words that are semantically diverse.
At ﬁrst, this hypothesis may seem counterintuitive.
However, it is made because, otherwise, humans would not be able to communicate effectively due to confusion.
For example, red (Pinyin: ‘hong’) and green (Pinyin: ‘lv’) in Chinese appear in similar contexts.
To reduce ambiguity in oral communication, it seems plausible to think that part of the development of phonetics is that one re-uses the same sound when context can be used to distin-
Input : points (embedded R 2 vectors) in a cluster Parameters : β : threshold; r : radius
Output : The convex hull’s vertices 1: for for each point do 2: draw a circle with r 3: if the total number of points in the circle is less than β then 4: remove this point 5: end if 6: end for 7: return the convex hull guish among multiple interpretations.
For example, “fair” (county fair) versus “fair” (equitable).
How do we set up experiments to verify this?
We test our hypothesis using geometric interpretations of semantics, precisely, word embeddings (Bengio and Heigold, 2014).
Intuitively, an embedding (Mikolov et al., 2013; Arora et al., 2016) preserves pairwise semantic distances, where two words/characters are close if they are semantically similar and far away otherwise.
For instance, see the work of Zouzias (2010); Molitor (2017) about volume preserving embeddings, which formalizes the concept of this term.
That is, if we have a set of words, for example, and all the words correspond to a Pinyin, then the points themselves may mean nothing, but the distances among the points are our focus.
Typically in geometry, three points in space are sufﬁcient to quantify a volume.
We embed each word or characters from ChineseEnglish translation data (in Section 6) intodimensions and then project this embedding into two dimensions using PCA.
Algorithm 2 describes how we compute a smooth convex hull of points.
The convex hull of a Pinyin is the convex hull of the embedding of all words or characters that are pronounced with this Pinyin.
Observations.
Figure 2 shows all embedded Chinese characters in red dots, and black dots are the
Chinese character(s) of one random Pinyin in each plot.
We can see that characters with the same pronunciation tend to have distributed meaning - that is, well-distributed over the Euclidean plane.
In Figure 3, we measure the convex hull (the smallest convex set that contains all points - implemented in Matlab) of all characters.
We exclude the outliers (blue dots) by removing all points that are encircled along with less than β other points in a ball of the radius of r .
The ﬁrst plot shows the hull enclosing characters of one random group (either cluster or Pinyin).
The second plot shows adding characters of a second random group to the ﬁrst group, and so on.
The convex hull volume (here, 2D volume) increases as more groups are added.
We can see that Soundex, Pinyin, and random grouping covers the space faster than the
K-Means clustering when increasing the number of groups, namely the convex hull volume is greater for one or two clusters or Pinyins.
Quantitative veriﬁcations.
These are carried out with three experiments.
First, Figure 4 shows the empirical CDF of the convex hull volume of characters of each Pinyin, random clustering, and
K-Means clustering, where the x-axes indicate the volume, and y-axes indicate the frequency.
Random clustering and Pinyin grouping have a larger volume than K-Means, respectively.
This means that for each group, Pinyin is slightly better distributed (more widespread) than uniformly random clustering, and both of these are better distributed than K-Means.
This is quite interesting and is probably due to the isoperimetry of the uniform random sampling for these data points.
Second, we deﬁne the concentration factor as
[Equation]
where C [Equation] =
[Equation]
p ki is the
i − th point in group k
(either cluster or Pinyin).
The smaller the value, the better distributed the points located in each cluster are over the whole space.
The concentration factor Γ is 9350 for K-Means, 3. for Pinyin, 1.476 for the random clustering in Chinese; 3543K for K-Means, 0.3674 for Soundex, and 0.0191 for random clustering.
Finally, we deﬁne the density measure as in Algorithm 3, which intuitively seems to be a more robust test.
For each point x in the smoothed convex hull of all words/characters, let D i ( x ) be the distance between x and the i -th nearest neighbor of x in the space X .
We then look at either the maximum of D i ( x ) over all x , or the average.
Choosing a larger i captures the “density” of the point-set at larger scales, which is a parameter that can be tuned to be more robust against noise.
We numerically integrate the convex hull surface by randomly sampling the points, which are a linear combination of the convex hull corner weighted uniformly at random.
The density result is shown in Table 2.
This is consistent with the CDF in Figure 4.
The
Pinyin is most well-distributed, then the random clustering, after that the K-Means.
Experiments
Adding auxiliary information:
First, we convert all the words or characters in the source sentences (in the training, development, and test set) into phonetic or other encodings.
Then, those sequences of encodings are segmented with the Byte
Pair Encoding compression algorithm (Sennrich et al., 2016b; Gage, 1994) (BPE).
We learn a new [Table] embedding on this encoded training data only, for example, MetaPhone encodings.
Afterward, the embedded vectors are either concatenated with the original sentences’ embedding (after BPE) or used alone to be fed into the encoder of the CNN neural translator.
This decomposition results in a signiﬁcant improvement over the baseline.
6. Datasets and Vocabularies
We carried out experiments for ﬁve translation directions from the IWSLT 2017 bilingual tasks: Chinese to English (ZH-EN), English to French (ENFR), French to English (FR-EN), English to German (EN-DE), and German to English (DE-EN).
Algorithm
Input: Set of points P , clusters Q , nearest neighbor i Output: Density density chosen = {} for i = 1 to 5 do cluster = get random cluster( Q ) for word in cluster do chosen = chosen ∪ wordvector ( word ) end for end for Remove outliers from P using β = 0 .
3 & r = 10 X
=
convex hull( P )
C = corner points( X ) for i = 1 to m − 1 do for i = 1 to | C | do q i = random number between 0 &end for for i = 1 to | C | do p i
= qi (cid:80)
| C | k
=1 qk C (cid:48)
i =
C i ∗ p i end for hullpt = sum ( C (cid:48) )
density =repeat dist =
KNN ( hullpt,chosen,i ) density + = dist until noChange is
[Table]
[Table]
We used the IWSLT 2017 training data (IWSLT, 2017), the development data combines test sets in 2013, 2014, and 2015, and the evaluation data is the 2017 test set.
Figure 3 and Figure 4 show vocabulary statistics on source/target tokenized text (Cettolo, 2015) before and after applying encodings (Turk and
Stephens, 2010).
We apply a BPE with 89K and 16K (Denkowski and Neubig, 2017) operations for FR and 89K for DE, and 18K operations for
ZH, then we train an individual embedding on source/target jointly for each encoding.
6. Translation Results
For each encoding scheme, we carried out two experiments, one with only the encoded sentences and another one with the source sentence concatenated with the encoded sentence .
For example, W+Soundex means the source sentence in words concatenated with all words converted into
Soundex as the input to the Neural Networks.
As the Soundex algorithm does not support French text, we do not have results for Soundex and
W+Soundex for EN-FR.
The translation results are evaluated with Bojar (2006).
Table 5 shows that encoding as an auxiliary input (concatenated with the original sentence) signiﬁcantly improves the translation quality in all language directions.
W+MetaPhone indicates adding MetaPhone to the word-based NMT baseline, which gives the best results for EN-FR and
DE-EN, with an improvement of 1.71 and 1.2 in
BLEU points, respectively.
In our experiments, random clustering consistently improves over baselines on all languages.
The non-uniform random clustering method in algorithm 1 achieves a higher
BLEU score of 37.95% than a uniform random clustering after tuning on the cluster size.
For EN-FR ( 16 k BPE operations) data in Table 5, we uniform randomly sample words for each cluster.
We get the BLEU score of 37.74%, 37.77%, 37.38%, and 37.63% when setting the number of clusters to be 20%, 40%, 60%, and 80% of the vocabulary size (63615 words), i.e. the average cluster size to be 5, 2.5, 1.6, 1.25, respectively.
However, for most languages, the best codings are phonetic ones.
Phonetic linguistic knowledge is helpful in MT, and we explained the underlying reason with our hypothesis of semantic diversity by phonetics.
Linguistic information is typically language dependent, thus different phonetic algorithms serve better for certain languages.
NYSIIS handles phonemes that occur in European and Hispanic surnames.
Thus, it performs best in French.
MetaPhone is a more advanced algorithm with spelling variations and inconsistencies, hence, it works best for English and German (both Germanic languages).
Table 7 shows the results of the ZH-EN translation system (BPE 18 k operations).
We apply
Pinyin (Yu, 2016a), Pinyin segmented into letters, and Wubi encoding (Yu, 2016b).
We achieve significant improvement over the baseline by adding auxiliary information: 0.87 BLEU points with Pinyin, 1.68 BLEU points with Pinyin in letters, and 1.
BLEU points with Wubi, respectively.
The randomly clustering on Chinese characters and on words both improve the baseline with 1.49 and 1.47 BLEU points, respectively. This is a larger improvement than that of the K-Means clustering.
Coding
FR-EN (89 k )
FR-EN (16 k )
EN-FR (89 k )
EN-FR (16 k )
DE-EN (89 k ) EN-DE (89 k )
Baseline: Words 35.
Soundex - - 27.
NYSIIS 30.
MetaPhone 29.
W+Soundex - - 35.
W+NYSIIS 35.
[Table]
6. Model Complexity
We tune the drop out parameter for three experiments: Words, W+Pinyin, and W+Pinyin letters on
ZH-EN.
The drop out is set by default to 0.2, and the beam-size to 12.
Figure 5 shows how translation accuracy changes.
Our approach consistently performs better than the baseline systems.
The peak BLEU score is achieved at drop out 0.05 for the baseline, but between 0.2 and 0.3 for our approach.
This implies that adding auxiliary inputs will reduce the model complexity, indicated by a higher drop out parameter value opt for the best translation performance.
6. Training Speed
Table 6 shows the system training time (with BPE 89 k operations and for ZH-EN 18 k ).
The total time (in thousands of seconds) is in the ﬁrst column, and the time per epoch is in the second.
Given the auxiliary information reduces the model complexity as in Section 6.3, the training becomes more efﬁcient and needs a smaller number of epochs to converge.
In most systems, the total training time with the auxiliary information is comparable to those without it, sometimes even less.
6. Robustness
To test the system robustness, we evaluated EnglishFrench systems ( 89 k ) in Table 5 that are trained on
IWSLT’17.
The tests are the out-of-domain and informal language data, the WMT’15 (Ondˇrej Bojar and Turchi, 2015) News and MTNT test sets.
MTNT’18 is a dataset in the informal domain, recently released by Michel and Neubig (2018) for the robustness task in WMT’19 (WMT, 2019), which includes MTNT’18 test data and MTNT’ test data.
Note that unlike the robustness task itself, we did not use MTNT training data when building systems.
We aim to verify how the system would behave in a new domain that was entirely unknown during the system building process - aligning with a real-life scenario.
As in Table 8, all of our approaches achieved higher accuracy, showing more robustness in this experiment.
+MetaPhone outperforms all other encoding methods for all the three out-of-domain test sets and improves over the baseline by up to 1 BLEU points.
Table 5 also has results for system trained on the
MTNT’18 data and tested on both MTNT’18 and
MTNT’19 test sets as well as the out-of-domain
WMT’15 test data.
+MetaPhone outperforms all other systems and improves over the baseline by about 5 BLEU points.
Related Work
Phonological rules or constraints have been previously applied to tasks such as word segmentation (Hayes, 1996; Johnson et al., 2015).
Phonetics involves gradient and variable phenomena, whereas phonology is characteristically categorial and far less variable.
Instead of optimizing towards phonological constraints, we directly learn from phonetic data and discover hidden phonetic features to optimize NMT performance.
Coding
FR-EN EN-FR
DE-EN EN-DE
Words (W) 2.92/ 2.84/ 2.57/88.
Soundex3.99/ 2.27/83.
NYSIIS 2.02/ 3.25/ 2.22/79.
Metaphone 2.72/ 2.95/ 1.75/83.
W+Soundex3.25/ 2.00/ 3.66/
W+NYSIIS 1.48/ 3.12/ 2.07/94.
W+Metaphone 1.12 / 3.92/ 1.98 /98.
[Table]
Coding
ZH-EN
Words (W) 1.82/79.
Pinyin 2.74/85.
Wubi 2.60/81.
W+Pinyin 2.95/
W+Wubi 3.06/
[Table] [Table]
Discriminatively learning phonetic features has demonstrated success in various Language technology applications.
Huang et al.
(2004) used phonetic information to improve the named entity recognition task.
Bengio and Heigold (2014); Zhu et al.
(2018) integrate speech information into word embedding and subword unit models, respectively.
Du and Way (2017) converted Chinese characters to subword units using Pinyin to alleviate the unknown words.
Our work aims to improve NMT overall rather than to only translate unknown Chinese words.
We are the ﬁrst to introduce several phonetic algorithms: Soundex, NYSIIS, MetaPhone; and Logogram, Wubi to improve NMT.
We also develop new algorithms driven by an empirically veriﬁed observation, which works for all languages in any NMT framework.
Leading research has investigated auxiliary information to NLP tasks, such as polysemous word embedding structures by Arora et al.
(2016), factored models by Garc´ıa-Mart´ınez et al.
; Sennrich and Haddow (2016), as well as compiling various features as in Kobus et al.
(2017) and Sennrich et al.
(2016a).
In this paper, we focus on the introduction of phonetic encoding and random clustering and demonstrate that our approaches are effective even when applied in a simple way (namely, concatenation without the help of a factor model).
Treating
NMT as a black-box can be beneﬁcial when experimenting with different NMT models such as CNN, seq2seq, and attention-based one.
Closely related, but independent to this work, is the approach of word segmentation or character based NMT (Chung et al., 2016), which focuses on the decomposition of the translation unit.
Smaller text granularity helps in unseen word forms and tokenization challenges (Ling et al., 2015), while more extended translation units reduce model complexity and input lengths (Lee et al., 2017).
Finding the optimal granularity when feeding information to an NMT is undoubtedly impressive, but stratifying the translation unit does not necessarily only take place at the next level (in the form of character or word sequences).
We take a different angle and view MT input as an information source encoded in various forms.
We study the source sentence representations other than text such as phonetic encodings, which works surprisingly well when combined with word segmentation methods.
Conclusions
We introduce phonetic and logogram encodings to convert foreign text into phonetic and logogram forms.
We deploy them into NMT systems and signiﬁcantly improve NMT translation quality and robustness.
When analyzing this improvement, we introduce and verify our hypothesis of semantic diversity by phonetics.
Driven by this hypothesis, we further introduce the random clustering which also enhance the NMT accuracy and robustness