Original: T RUE B ILINGUAL NMT
Revised: T RUE B ILINGUAL NMT

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Bilingual machine translation permits training a single model that translates monolingual sentences from one language to another.
Revised: Bilingual machine translation permits training a single model that translates monolingual sentences from one language to another.

Original: However, a model is not truly bilingual unless it can translate back and forth in both language directions it was trained on, along with translating code-switched sentences to either language.
Revised: However, a model is not truly bilingual unless it can translate back and forth in both language directions it was trained on, along with translating code-switched sentences to either language.

Original: We propose a true bilingual model trained on WMT14 English-French (En-Fr) dataset.
Revised: We propose a true bilingual model trained on WMT14 English-French (En-Fr) dataset.

Original: For better use of parallel data, we generated synthetic code-switched (CSW) data along with an alignment loss on the encoder to align representations across languages.
Revised: For better use of parallel data, we generated synthetic code-switched (CSW) data along with an alignment loss on the encoder to align representations across languages.

Original: Our model strongly outperforms bilingual baselines on CSW translation while maintaining quality for non-code switched data.
Revised: Our model strongly outperforms bilingual baselines on CSW translation while maintaining quality for non-code switched data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Machine Translation (NMT) systems can be divided into two categories: bilingual models and multilingual models.
Revised: Neural Machine Translation (NMT) systems can be divided into two categories: bilingual models and multilingual models.

Original: Bilingual models are able to translate from one language to another; while multilingual models are able to translate between multiple languages (Firat et al.
Revised: Bilingual models are able to translate from one language to another; while multilingual models are able to translate between multiple languages (Firat et al.

Original: Johnson et al.
Revised: Johnson et al.

Original: We argue that bilingual models in that sense aren’t actually bilingual since they can’t translate in the opposite direction, and can’t translate code-switched sentences either.
Revised: We argue that bilingual models in that sense aren’t actually bilingual since they can’t translate in the opposite direction, and can’t translate code-switched sentences either.

Original: Code-Switching (CSW) denotes the alternation of two languages within a single utterance (Poplack (1980); Sitaram et al.
Revised: Code-Switching (CSW) denotes the alternation of two languages within a single utterance (Poplack (1980); Sitaram et al.

Original: According to Vilhanova (2018), Africa is the most multilingual continent in the world, and this requires rethinking what we expect from our NMT models when they’re deployed.
Revised: According to Vilhanova (2018), Africa is the most multilingual continent in the world, and this requires rethinking what we expect from our NMT models when they’re deployed.

Original: In this paper, we aim to build a true bilingual NMT model.
Revised: In this paper, we aim to build a true bilingual NMT model.

Original: It is a single model that is able to translate in both directions and also translate code-switched sentences.
Revised: It is a single model that is able to translate in both directions and also translate code-switched sentences.

Original: This model was trained using only parallel data accompanied with synthetic code-switched data.
Revised: This model was trained using only parallel data accompanied with synthetic code-switched data.

Original: To make the encoder create language-agnostic representations, we propose an alignment loss function applied only on the encoder.
Revised: To make the encoder create language-agnostic representations, we propose an alignment loss function applied only on the encoder.

Original:  D ATA
Revised:  D ATA

Original: We used WMT14 English-French benchmark for training, newstest2008-2013 for validation, and newstest2014 for test.
Revised: Since English and French are among the most high-resource spoken languages in Africa, we used WMT14 English-French benchmark for training, newstest2008-2013 for validation, and newstest2014 for test.

Original: Parallel corpora for code-switched data is very scarce (Menacer et al.
Revised: Parallel corpora for code-switched data is very scarce (Menacer et al.

Original: (2019)), however, there have been works on generating synthetic code-switched data.
Revised: (2019)), however, there have been works on generating synthetic code-switched data.

Original: Similar to Song et al.
Revised: Similar to Song et al.

Original: (2019) and Xu & Yvon (2021), we created code-switched data, by first extracting word alignment using fast-align toolkit (Dyer et al.
Revised: (2019) and Xu & Yvon (2021), we created code-switched data, by first extracting word alignment using fast-align toolkit (Dyer et al.

Original: and then extracting minimal alignment units following the approach of (Crego (2005)).
Revised: and then extracting minimal alignment units following the approach of (Crego (2005)).

Original: We chose the matrix language — defined by the Matrix Language
Revised: We chose the matrix language — defined by the Matrix Language

Original: Frame (MLF) theory (Poulisse (1998)) — randomly (50%).
Revised: Frame (MLF) theory (Poulisse (1998)) — randomly (50%).

Original: Similar to MLM pre-training used by
Revised: Similar to MLM pre-training used by

Original: BERT (Devlin et al.
Revised: BERT (Devlin et al.

Original: , we randomly replaced 15% of the sentence length with its aligned segments in the embedded language.
Revised: , we randomly replaced 15% of the sentence length with its aligned segments in the embedded language.

Original: We combined the code-switched data generated with the parallel bidirectional data after prepending a target language token (Johnson et al.
Revised: We combined the code-switched data generated with the parallel bidirectional data after prepending a target language token (Johnson et al.

Original: For more details, check Appendix A.
Revised: For more details, check Appendix A.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Following Arivazhagan et al.
Revised: Following Arivazhagan et al.

Original: (2019) steps, we use parallel data, and enforce the encoder to make language-agnostic representations about the input sequences by minimizing the max-pooled cosine distance of the encoder representations of the parallel data as shown in the following equation:[Equation]
Revised: (2019) steps, we use parallel data, and enforce the encoder to make language-agnostic representations about the input sequences by minimizing the max-pooled cosine distance of the encoder representations of the parallel data as shown in the following equation:

Original: [Equation]
Revised: [Equation]

Original: Where Ω is the encoder loss, D ( en,fr ) is our data containing parallel language pairs combined with code-switched ones, x src is the source sentence which could be monolingual or code-switched while x tgt is the target which is always monolingual, Enc ( x ) is the max-pooled encoder representation of sentence x similar to Gouws et al.
Revised: Where Ω is the encoder loss, D ( en,fr ) is our data containing parallel language pairs combined with code-switched ones, x src is the source sentence which could be monolingual or code-switched while x tgt is the target which is always monolingual, Enc ( x ) is the max-pooled encoder representation of sentence x similar to Gouws et al.

Original: (2016) and Coulmance et al.
Revised: (2016) and Coulmance et al.

Original: (2016), and sim is the cosinesimilarity.
Revised: (2016), and sim is the cosinesimilarity.

Original: Unlike Arivazhagan et al.
Revised: Unlike Arivazhagan et al.

Original: (2019) where the whole model’s parameters were updated, we updated the encoder parameters only, as shown in Figure 1.
Revised: (2019) where the whole model’s parameters were updated, we updated the encoder parameters only, as shown in Figure 1.

Original: Figure 1: shows the loss functions used where CE is the cross entropy, Ω is the encoder loss.
Revised: Figure 1: shows the loss functions used where CE is the cross entropy, Ω is the encoder loss.

Original:  E XPERIMENTS AND RESULTS
Revised:  E XPERIMENTS AND RESULTS

Original: In all of our experiments, we used Transformer-Base (Vaswani et al.
Revised: In all of our experiments, we used Transformer-Base (Vaswani et al.

Original: (2017)) configuration with the
Revised: (2017)) configuration with the

Original: Fairseq (Ott et al.
Revised: Fairseq (Ott et al.

Original: (2019)) framework.
Revised: (2019)) framework.

Original: All models were trained on four Tesla T400 GPUs using
Revised: All models were trained on four Tesla T400 GPUs using

Original: WMT-14 data for training, with a shared vocabulary of 40K BPE Sennrich et al.
Revised: WMT-14 data for training, with a shared vocabulary of 40K BPE Sennrich et al.

Original: (2016) sub-words.
Revised: (2016) sub-words.

Original: The model’s hyperparameters can be found in Appendix B.
Revised: The model’s hyperparameters can be found in Appendix B.

Original: We created two baselines using the same training details as mentioned above: 1) Bidirectional :
Revised: We created two baselines using the same training details as mentioned above: 1) Bidirectional :

Original: En ↔ Fr model trained only on parallel data in both directions.
Revised: En ↔ Fr model trained only on parallel data in both directions.

Original: 2) csw :
Revised: 2) csw :

Original: En ↔ Fr model trained only on synthetic code-switched data.
Revised: En ↔ Fr model trained only on synthetic code-switched data.

Original: Our model bi+csw+cosine was trained on bidirectional and code-switched data along with the encoder criterion mentioned in eq.
Revised: Our model bi+csw+cosine was trained on bidirectional and code-switched data along with the encoder criterion mentioned in eq.

Original: To check the effect of the encoder alignment loss, we trained another model without the encoder criterion bi+csw .
Revised: To check the effect of the encoder alignment loss, we trained another model without the encoder criterion bi+csw .

Original: Tableshows the results of all models trained using the same parameters seen in Table 5.
Revised: Tableshows the results of all models trained using the same parameters seen in Table 5.

Original: to Fr to En to Frto En
Revised: to Fr to En to Fr

Original: Bidirectional 642K 39.

Original: csw 594K 8. bi+csw 420k 38.

Original: [Table]
Revised: [Table]

Original: From Table 1 we see that the Bidirectional baseline performs well for bi-directional translation.
Revised: From Table 1 we see that the Bidirectional baseline performs well for bi-directional translation.

Original: However, csw baseline performs well only on CSW translation.
Revised: However, csw baseline performs well only on CSW translation.

Original: Our combined models bi+csw and bi+csw+cosine work well across the board where bi+csw+cosine has the best performance on CSW data while achieving competitive results on bi-directional translation compared to the bidirectional baseline.
Revised: Our combined models bi+csw and bi+csw+cosine work well across the board where bi+csw+cosine has the best performance on CSW data while achieving competitive results on bi-directional translation compared to the bidirectional baseline.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced two ways to make best-use of parallel data that can improve model’s performance on both unidirectional and code-switched data: 1) A statistical way to generate codeswitched data that can be aggregated with unidirectional data for training.
Revised: In this paper, we introduced two ways to make best-use of parallel data that can improve model’s performance on both unidirectional and code-switched data: 1) A statistical way to generate codeswitched data that can be aggregated with unidirectional data for training.

Original: A loss function that trains the encoder to generate language-independent representations.
Revised: A loss function that trains the encoder to generate language-independent representations.

Original: We show that these two techniques boosted our model’s performance on both unidirectional and code-switched data.
Revised: We show that these two techniques boosted our model’s performance on both unidirectional and code-switched data.

Original: This is still  work-in-progress, and we arestill exploring new ways to improve our model even more.
Revised: This is still a work-in-progress, and we are exploring new ways to improve our model even more; and more importantly experimenting with Arabic, one of the most spoken language in Africa.

Original: A A PPENDIX A. D ATASET
Revised: A A PPENDIX A. D ATASET

Original: We used WMT-14 benchmark for training, with a shared vocabulary of 40K BPE Sennrich et al.
Revised: We used WMT-14 benchmark for training, with a shared vocabulary of 40K BPE Sennrich et al.

Original: (2016) sub-words.
Revised: (2016) sub-words.

Original: We used 4096 tokens per batch, and we removed any sentences whose length are less than 2 tokens and more than 250 tokens.
Revised: We used 4096 tokens per batch, and we removed any sentences whose length are less than 2 tokens and more than 250 tokens.

Original: Also we prepended a target-language tag to the sentences as shown in Table 2.
Revised: Also we prepended a target-language tag to the sentences as shown in Table 2.

Original: All data was tokenized and normalized using Moses SMT (Koehn et al.
Revised: All data was tokenized and normalized using Moses SMT (Koehn et al.

Original: The stats of the data after cleaning can be found in Table 3.
Revised: The stats of the data after cleaning can be found in Table 3.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] Il fait beau aujourd’hui .
Revised: the source and monolingual on the target.

Original: [Table]
Revised: [Table]

Original: -14 French-English data stats used for training, validation and test

Original: A. C ODE - SWITCHED DATA GENERATION METHOD
Revised: A. C ODE - SWITCHED DATA GENERATION METHOD

Original: The following are the steps that we followed to generate the code-switched (CSW) data; a sample can be found in Table 4.
Revised: The following are the steps that we followed to generate the code-switched (CSW) data; a sample can be found in Table 4.

Original: These steps were adapted from (Xu & Yvon (2021) which can be summarized into the following:
Revised: These steps were adapted from (Xu & Yvon (2021) which can be summarized into the following:

Original: Data preprocessing : data tokenization and normalization was done using this perl script: clean-corpus-n.perl from moses SMT (Koehn et al.
Revised: Data preprocessing : data tokenization and normalization was done using this perl script: clean-corpus-n.perl from moses SMT (Koehn et al.

Original: Alignment : Fast-Align (Dyer et al.
Revised: Alignment : Fast-Align (Dyer et al.

Original: tool with gdfa (grow-diag-final-and).
Revised: tool with gdfa (grow-diag-final-and).

Original: Random replacement : Aligned segments were replaced by considering the following: (a) Matrix Language is chosen randomly (50-50)%.
Revised: Random replacement : Aligned segments were replaced by considering the following: (a) Matrix Language is chosen randomly (50-50)%.

Original: (b) Replace around 15% of the input sequence.
Revised: (b) Replace around 15% of the input sequence.

Original: (c) Short sequences (less than 7 tokens) have just one replacement.
Revised: (c) Short sequences (less than 7 tokens) have just one replacement.

Original: (d) Positions of aligned segments are chosen uniformly.
Revised: (d) Positions of aligned segments are chosen uniformly.

Original: [Table]
Revised: [Table]

Original: English Translation

Original: French Translation
Revised: French Translation

Original: MatrixLanguage
Revised: MatrixLanguage

Original: Difficult Year pour les Pharmacists .
Revised: Difficult Year pour les Pharmacists .

Original: Difficult
Revised: Difficult

Original: Year for
Revised: Year for

Original: Pharmacists .
Revised: Pharmacists .

Original: Ann´ee difficile pour les pharmaciens .
Revised: Ann´ee difficile pour les pharmaciens .

Original: English
Revised: English

Original: Il ne believe pas que l’Ontario emboˆıtera le pas .
Revised: Il ne believe pas que l’Ontario emboˆıtera le pas .

Original: He does not believe that Ontario will follow suit .
Revised: He does not believe that Ontario will follow suit .

Original: Il ne croit pas que l’Ontario emboˆıtera le pas .
Revised: Il ne croit pas que l’Ontario emboˆıtera le pas .

Original: French
Revised: French

Original: Asked how he had developed his character , the acteur and singer Justin Timberlake avait rappel´e how he ” grandi dans
Revised: Asked how he had developed his character , the acteur and singer Justin Timberlake avait rappel´e how he ” grandi dans

Original: Tennessee , bathed in the blues and country music ” .
Revised: Tennessee , bathed in the blues and country music ” .

Original: When asked how he came up with his character, actor and singer
Revised: When asked how he came up with his character, actor and singer

Original: Justin
Revised: Justin

Original: Timberlake recalled that he ” grew up in Tennessee, surrounded by blues and country music ”.
Revised: Timberlake recalled that he ” grew up in Tennessee, surrounded by blues and country music ”.

Original: Interrog´e sur la fac¸on dont il a compos´e son personnage , l’acteur et chanteur
Revised: Interrog´e sur la fac¸on dont il a compos´e son personnage , l’acteur et chanteur

Original: Justin
Revised: Justin

Original: Timberlake avait rappel´e avoir ” grandi dans le
Revised: Timberlake avait rappel´e avoir ” grandi dans le

Original: Tennessee, baign´e par le blues et la country ” .
Revised: Tennessee, baign´e par le blues et la country ” .

Original: English
Revised: English

Original: Mes camarades cried with joie et mes parents ont conserv´e every journaux qu’ils ont trouv ´ es .
Revised: Mes camarades cried with joie et mes parents ont conserv´e every journaux qu’ils ont trouv ´ es .

Original: My classmates cried with joy , and my parents saved every newspaper they could find .
Revised: My classmates cried with joy , and my parents saved every newspaper they could find .

Original: Mes camarades de classe ont pleur ´ e de joie, et mes parents ont gard´e tous les journaux qu’ils ont pu trouver .
Revised: Mes camarades de classe ont pleur ´ e de joie, et mes parents ont gard´e tous les journaux qu’ils ont pu trouver .

Original: French
Revised: French

Original: B A PPENDIX
Revised: B A PPENDIX

Original: Table 5 holds all the hyper-parameters we used for training all models.
Revised: Table 5 holds all the hyper-parameters we used for training all models.

Original: All models were trained till convergence with patience = 10.
Revised: All models were trained till convergence with patience = 10.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]