Original: O N

Original: THE

Original: USE

Original: OF

Original: LINGUISTIC

Original: SIMILARITIES

Original: TO

Original: IMPROVE N EURAL M ACHINE T RANSLATION
Revised: On the use of linguistic similarities to improve Neural Machine Translation for African Languages

Original: FOR

Original: A FRICAN L ANGUAGES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: Abstract

Original: In recent years, there has been a resurgence in research on empirical methods for machine translation.
Revised: In recent years, there has been a resurgence in research on empirical methods for machine translation.

Original: Most of this research has been focused on high-resource,
Revised: Most of this research has been focused on high-resource,

Original: European languages.
Revised: European languages.

Original: Despite the fact that around 30% of all languages spoken worldwide are African, the latter have been heavily under investigated and this, partly due to the lack of public parallel corpora online.
Revised: Despite the fact that around 30% of all languages spoken worldwide are African, the latter have been heavily under investigated and this, partly due to the lack of public parallel corpora online.

Original: Furthermore, despite their large number (more than 2,000) and the similarities between them, there is currently no publicly available study on how to use this multilingualism (and associated similarities) to improve machine translation systems performance on African languages.
Revised: Furthermore, despite their large number (more than 2,000) and the similarities between them, there is currently no publicly available study on how to use this multilingualism (and associated similarities) to improve machine translation systems performance on African languages.

Original: So as to address these issues, we propose a new dataset  for African languages that provides parallel data for vernaculars not present in commonly used dataset like JW300.
Revised: So as to address these issues, we propose a new dataset (from a source that allows us to use and release) for African languages that provides parallel data for vernaculars not present in commonly used dataset like JW300.

Original: To exploit multilingualism, we ﬁrst use a historical approach based on migrations of population to identify similar vernaculars.
Revised: To exploit multilingualism, we first use a historical approach based on migrations of population to identify similar vernaculars.

Original: We also propose a new metric to automatically evaluate similarities between languages.
Revised: We also propose a new metric to automatically evaluate similarities between languages.

Original: This new metric does not require word level parallelism like traditional methods but only paragraph level parallelism.
Revised: This new metric does not require word level parallelism like traditional methods but only paragraph level parallelism.

Original: We then show that performing Masked Language 
Revised: We then show that performing Masked Language Modelling and

Original: Modelling and Translation Language Modeling in addition to multi-task learning on a cluster of similar languages leads to a strong boost of performance in translating individual pairs inside this cluster.
Revised: Translation Language Modeling in addition to multi-task learning on a cluster of similar languages leads to a strong boost of performance in translating individual pairs inside this cluster.

Original: In particular, we record an improvement of 29 BLEU on the pair Baﬁa-Ewondo using our approaches compared to previous work methods that did not exploit multilingualism in any way.
Revised: In particular, we record an improvement of 29 BLEU on the pair Bafia-Ewondo using our approaches compared to previous work methods that did not exploit multilingualism in any way.

Original: Finally, we release the dataset and code of this work to ensure reproducibility and accelerate research in this domain.
Revised: Finally, we release the dataset and code of this work to ensure reproducibility and accelerate research in this domain.

Original:  I NTRODUCTION
Revised:  Introduction

Original: Machine Translation (MT) of African languages is a challenging problem because of multiple reasons.
Revised: Machine Translation (MT) of African languages is a challenging problem because of multiple reasons.

Original: As pointed out by Martinus & Abbott (2019), the main ones are:
Revised: As pointed out by Martinus and Abbott (2019) the main ones are:

Original: • Morphological complexity and diversity of African languages : Africa is home to around 2144 languages out of nowadays 7111 living languages (they thus make 30.15% of all living languages) with often different alphabets (Eberhard et al.
Revised: • Morphological complexity and diversity of African languages : Africa is home to around 2144 languages out of nowadays 7111 living languages (they thus make 30.15% of all living languages) with often different alphabets (Eberhard et al., 2020).

Original: • Lack/Absence of large parallel datasets for most language pairs.
Revised: • Lack/Absence of large parallel datasets for most language pairs.

Original: • Discoverability: The existing resources for African languages are often hard to ﬁnd.
Revised: • Discoverability: The existing resources for African languages are often hard to find.

Original: • Reproducibility: Data and code of existing research are rarely shared, making it difﬁcult for other researchers to reproduce the results properly.
Revised: • Reproducibility: Data and code of existing research are rarely shared, making it difficult for other researchers to reproduce the results properly.

Original: • Lack of benchmarks: Due to the low discoverability and the lack of research in the ﬁeld, there are no publicly available benchmark or leaderboard to compare new machine translation techniques to.
Revised: • Lack of benchmarks: Due to the low discoverability and the lack of research in the field, there are no publicly available benchmark or leaderboard to compare new machine translation techniques to.

Original: However, despite the strong multilingualism of the continent, previous research focused solely on translating individual pairs of language without taking advantage of the potential similarities between all of them.
Revised: However, despite the strong multilingualism of the continent, previous research focused solely on translating individual pairs of language without taking advantage of the potential similarities between all of them.

Original: The main purpose of this work is to exploit this multilingualism, starting from a few languages spoken in Central and West Africa, to produce better machine translation systems.
Revised: The main purpose of this work is to exploit this multilingualism, starting from a few languages spoken in Central and West Africa, to produce better machine translation systems.

Original: Our contributions can be summarised as follows :
Revised: Our contributions can be summarised as follows :

Original: We provide new parallel corpora extracted from the Bible for several pairs of African vernacular  that were not available until now as well as the code used to perform this extraction.
Revised: We provide new parallel corpora extracted from the Bible for several pairs of African vernacular 3 that were not available until now as well as the code used to perform this extraction.

Original: We present a method for aggregating languages together based on their historical origins, their morphologies, their geographical and cultural distributions etc...
Revised: We present a method for aggregating languages together based on their historical origins, their morphologies, their geographical and cultural distributions etc...

Original: We also propose of a new metric to evaluate similarity between languages : this metric, based on language models, doesn’t require word level parallelism (contrary to traditional like (Levenshtein, 1965) and (Swadesh, 1952))
Revised: We also propose of a new metric to evaluate similarity between languages : this metric, based on language models, doesn’t require word level parallelism (contrary to traditional like (Levenshtein, 1965) and (Swadesh, 1952))

Original: but only paragraph level parallelism.
Revised: but only paragraph level parallelism.

Original: It also takes into account the lack of translation of words present in (Swadesh, 1952) but not in African vernaculars (like ”snow”).
Revised: It also takes into account the lack of translation of words present in (Swadesh, 1952) but not in African vernaculars (like "snow").

Original: Using the language clusters created using the previous similarities, we show that Translation Language Modelling ((Lample & Conneau, 2019)) and multi-task learning generally improve the performance on individual pairs inside these clusters.
Revised: Using the language clusters created using the previous similarities, we show that Translation Language Modelling (Lample and Conneau, 2019) and multi-task learning generally improve the performance on individual pairs inside these clusters.

Original: We make our data, code and benchmark  publicly available.
Revised: Our code, data and benchmark are publicly available at https://github.com/Tikquuss/meta_ XLM

Original: The rest of the paper is organised as follows:
Revised: The rest of the paper is organised as follows:

Original: In section 2, we discuss the motivation of this work i.e. problems faced by African communities that could be solved with machine translation systems.
Revised: In section 2, we discuss the motivation of this work i.e. problems faced by African communities that could be solved with machine translation systems.

Original: Related work is described in 3 and relevant background is outlined in 4.
Revised: Related work is described in 3 and relevant background is outlined in 4.

Original: In section 5, we describe our dataset and provide more details on the language aggregation methods.
Revised: In section 5, we describe our dataset and provide more details on the language aggregation methods.

Original: In section 6 we present the experimental protocol and the results of our methodology on machine translation of 3 African (Cameroonian) vernaculars.
Revised: In section 6 we present the experimental protocol and the results of our methodology on machine translation of 3 African (Cameroonian) vernaculars.

Original: Lastly, we summarise our work and conclude in 7.
Revised: Lastly, we summarise our work and conclude in 7.

Original:  M OTIVATION
Revised:  Motivation

Original: Africa, because of its multilingual nature, faces a lot of communication problems.
Revised: Africa, because of its multilingual nature, faces a lot of communication problems.

Original: In particular, most
Revised: In particular, most

Original: African countries have colonial languages (French, English, Portuguese, etc...) as ofﬁcial languages (with sometimes one or two African dialects).
Revised: African countries have colonial languages (French, English, Portuguese, etc...) as official languages (with sometimes one or two African dialects).

Original: The latter are those taught in schools and used in administrations and workplaces (which are often located in urban areas).
Revised: The latter are those taught in schools and used in administrations and workplaces (which are often located in urban areas).

Original: In contrast, rural and remote areas citizens mainly speak African dialects (especially the older ones and the less educated ones).
Revised: In contrast, rural and remote areas citizens mainly speak African dialects (especially the older ones and the less educated ones).

Original: This situation creates the following fairness and ethical concerns (amongst others):
Revised: This situation creates the following fairness and ethical concerns (amongst others):

Original: • Remote area population often have difﬁculties communicating during medical consultations since they mainly speak the local languages, contrary to the doctors who mostly speak French, English, etc....
Revised: • Remote area population often have difficulties communicating during medical consultations since they mainly speak the local languages, contrary to the doctors who mostly speak French, English, etc....

Original: Similar situations arise when NGOs try to intervene in rural regions.
Revised: Similar situations arise when NGOs try to intervene in rural regions.

Original: • This discrepancy between spoken languages across different regions of individual African countries make the spread of misinformation easier (especially in rural regions and especially during election periods).
Revised: • This discrepancy between spoken languages across different regions of individual African countries make the spread of misinformation easier (especially in rural regions and especially during election periods).

Original: • Young Africans (in particular those living in urban areas) struggle to learn their vernaculars.
Revised: • Young Africans (in particular those living in urban areas) struggle to learn their vernaculars.

Original: Lack of documentation and schools dealing with African languages as well as the scarcity of translators and trainers make the situation even more complicated.
Revised: Lack of documentation and schools dealing with African languages as well as the scarcity of translators and trainers make the situation even more complicated.

Original: For all of the reasons above, it is essential to set up translation systems for these languages.
Revised: For all of the reasons above, it is essential to set up translation systems for these languages.

Original:  R ELATED WORK
Revised:  Related work

Original: Several scholarships have been conducted on machine translation for African languages, Wilken et al.
Revised: Several scholarships have been conducted on machine translation for African languages, Wilken et al.

Original: (2012) performed statistical phrase-based translation for English to Setswana translation.
Revised: (2012) performed statistical phrase-based translation for English to Setswana translation.

Original: This research used linguistically-motivated pre and post processing of the corpus in order to improve the translations.
Revised: This research used linguistically-motivated pre and post processing of the corpus in order to improve the translations.

Original: The system was trained on the Autshumato dataset (Groenewald & Fourie (2009)) and also used an additional monolingual dataset.
Revised: The system was trained on the Autshumato dataset (Groenewald and Fourie, 2009) and also used an additional monolingual dataset.

Original: They reached a BLEU score of 28.8.
Revised: They reached a BLEU score of 28.8.

Original: McKellar (2014) used statistical machine translation for English to Xitsonga translation.
Revised: McKellar (2014) used statistical machine translation for English to Xitsonga translation.

Original: The models were trained on the Autshumato data (Groenewald & Fourie (2009)), as well as a large monolingual corpus.
Revised: The models were trained on the Autshumato data (Groenewald and Fourie, 2009), as well as a large monolingual corpus.

Original: A factored machine translation system was used, making use of a combination of lemmas and part of speech tags.
Revised: A factored machine translation system was used, making use of a combination of lemmas and part of speech tags.

Original: It obtained a BLEU score of 37.31.
Revised: It obtained a BLEU score of 37.31.

Original: Niekerk (2014) used unsupervised word segmentation with phrase-based statistical machine translation models.
Revised: Niekerk (2014) used unsupervised word segmentation with phrase-based statistical machine translation models.

Original: These models translate from English to Afrikaans, N. Sotho, Xitsonga and isiZulu.
Revised: These models translate from English to Afrikaans, N. Sotho, Xitsonga and isiZulu.

Original: The parallel corpora were created by crawling online sources and ofﬁcial government data and aligning these sentences using the HunAlign software package.
Revised: The parallel corpora were created by crawling online sources and official government data and aligning these sentences using the HunAlign software package.

Original: Large monolingual datasets were also used.
Revised: Large monolingual datasets were also used.

Original: They reached a BLEU score of 71 on English to Afrikaans and 37.9 on English to N.Sotho.
Revised: They reached a BLEU score of 71 on English to Afrikaans and 37.9 on English to N.Sotho.

Original: Wolff & Kotze (2014) performed word translation for English to isiZulu.
Revised: Wolff and Kotze (2014) performed word translation for English to isiZulu.

Original: The translation system was trained on a combination of Autshumato, Bible, and data obtained from the South African Constitution.
Revised: The translation system was trained on a combination of Autshumato, Bible, and data obtained from the South African Constitution.

Original: All of the isiZulu text was syllabiﬁed prior to the training of the word translation system.
Revised: All of the isiZulu text was syllabified prior to the training of the word translation system.

Original: Abbott & Martinus (2018) trained Transformer models for English to Setswana on the parallel
Revised: Abbott and Martinus (2018) trained Transformer models for English to Setswana on the parallel

Original: Autshumato dataset (Groenewald & Fourie (2009)) and obtained a BLEU score of 33.53.
Revised: Autshumato dataset (Groenewald and Fourie, 2009) and obtained a BLEU score of 33.53.

Original: Data was not cleaned nor was any additional data used.
Revised: Data was not cleaned nor was any additional data used.

Original: This is the rare study reviewed that released datasets and code.
Revised: This is the rare study reviewed that released datasets and code.

Original: Martinus & Abbott (2019) used Google Translate to perform English to Afrikaans translation and achieved a BLEU score of 41.18 To the best of our knowledge, Masakhane (2020) are the only ones to have carried out research on a wide variety of African languages.
Revised: Martinus and Abbott (2019) used Google Translate to perform English to Afrikaans translation and achieved a BLEU score of 41.18 To the best of our knowledge, Masakhane (2020) are the only ones to have carried out research on a wide variety of African languages.

Original: They used the JW300 dataset (Agi´c & Vuli´c, 2019) from opus collection 1 and trained Transformer models on it.
Revised: They used the JW300 dataset (Agi ´ c and Vuli ´ c, 2019) from opus collection 4 and trained Transformer models on it.

Original: However, they did not do any preliminary linguistic study of their languages of interest.
Revised: However, they did not do any preliminary linguistic study of their languages of interest.

Original: Lastly,
Revised: Lastly,

Original: Emezue & Dossou (2020) provided a benchmark for Fon to French translation using the same
Revised: Emezue and Dossou (2020) provided a benchmark for Fon to French translation using the same

Original: JW300 dataset (Agi´c & Vuli´c, 2019).
Revised: JW300 dataset (Agi´c and Vuli´c, 2019).

Original: They used a GRU (Chung et al. sequence to sequence model with attention (Bahdanau et al. and obtained a BLEU score of 30.55.
Revised: They used a GRU (Chung et al., 2014) sequence to sequence model with attention (Bahdanau et al., 2015) and obtained a BLEU score of 30.55.

Original: These last two studies are ones of the very few to have released datasets and code.
Revised: These last two studies are ones of the very few to have released datasets and code.

Original: Regarding the use of multilingualism in machine translation, (Johnson et al., 2016) applied multitask learning to a single encoder-decoder LSTM model to translate between English, French and German.
Revised: Regarding the use of multilingualism in machine translation, (Johnson et al., 2016) applied multi-task learning to a single encoder-decoder LSTM model to translate between English, French and German.

Original: The best gain of performance of their many-to-many model was 0.43 BLEU on (French,English) compared to a single-task model trained on that pair.
Revised: The best gain of performance of their many-to-many model was 0.43 BLEU on (French,English) compared to a single-task model trained on that pair.

Original: However, its worse decrease of performance was 2.75 BLEU on (English, German) (Ha et al., 2016) also worked on English, French and German.
Revised: However, its worse decrease of performance was 2.75 BLEU on (English, German) Ha et al. (2016) also worked on English, French and German.

Original: Their best gain of performance compared to a single task model was 2.64 BLEU on (English, German) in an under-resourced scenario.
Revised: Their best gain of performance compared to a single task model was 2.64 BLEU on (English, German) in an under-resourced scenario.

Original: They used an original approach called mix-source where the pair (German,German) was added to the translation model training data.
Revised: They used an original approach called mix-source where the pair (German, German) was added to the translation model training data.

Original: The goal was to help the model obtaining a better representation of the source side and learn the translation relationships better.
Revised: The goal was to help the model obtaining a better representation of the source side and learn the translation relationships better.

Original: They argued that including monolingual data in this way might also improve the translation of some rare word types such as named entities.
Revised: They argued that including monolingual data in this way might also improve the translation of some rare word types such as named entities.

Original:  B ACKGROUND 4. N EURAL M ACHINE T RANSLATION (NMT)
Revised:  Background 4.1 Neural Machine Translation (NMT)

Original: Machine Translation (MT) refers to fully automated software that can translate source content into target languages.
Revised: Machine Translation (MT) refers to fully automated software that can translate source content into target languages.

Original: Neural Machine Translation (NMT) is an approach to MT that uses an artiﬁcial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model.
Revised: Neural Machine Translation (NMT) is an approach to MT that uses an artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model.

Original: NMT aims to translate an input sequence from a source language to a target language.
Revised: NMT aims to translate an input sequence from a source language to a target language.

Original: An NMT model usually consists of an encoder to map an input sequence to hidden representations, and a decoder to decode hidden representations to generate a sentence in the target language.
Revised: An NMT model usually consists of an encoder to map an input sequence to hidden representations, and a decoder to decode hidden representations to generate a sentence in the target language.

Original: NMThasmade rapid progress in recent years.
Revised: NMT has made rapid progress in recent years.

Original: The attention module is ﬁrst introduced by Bahdanau et al.
Revised: The attention module is first introduced by Bahdanau et al.

Original: (2015) , which is used to better align source words and target words.
Revised: (2015), which is used to better align source words and target words.

Original: The encoder and decoder can be specialized as LSTM (Hochreiter & Schmidhuber (1997),Sutskever et al.
Revised: The encoder and decoder can be specialized as LSTM (Hochreiter and Schmidhuber, 1997; Sutskever et al., 2014;

Original: (2014),Wu et al. (2016)), CNN (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).
Revised: Wu et al., 2016), CNN (Gehring et al., 2017) and Transformer (Vaswani et al., 2017).

Original: A Transformer layer consists of three sublayers, a self-attention layer that processes sequential data taking the context of each timestep into consideration, an optional encoder-decoder attention layer that bridges the input sequence and target sequence which exists in decoder only, and a feed-forward layer for non-linear transformation.
Revised: A Transformer layer consists of three sublayers, a self-attention layer that processes sequential data taking the context of each timestep into consideration, an optional encoder-decoder attention layer that bridges the input sequence and target sequence which exists in decoder only, and a feed-forward layer for non-linear transformation.

Original: Transformer achieves the state-of-the-art results for NMT (Barrault et al., 2019).
Revised: Transformer achieves the state-of-the-art results for NMT (Barrault et al., 2019).

Original: 4. C ROSS - LINGUAL L ANGUAGE M ODEL (XLM) P RETRAINING
Revised: 4.2 Cross-lingual Language Model (XLM) Pretraining

Original: Recently, pre-training techniques, like ELMo (Peters et al., 2018), GPT/GPT-2 (Radford et al., 2018),BERT (Devlin et al., 2019), cross-lingual language model (brieﬂy, XLM) (Lample & Conneau, 2019),
Revised: Recently, pre-training techniques, like ELMo (Peters et al., 2018), GPT/GPT-2 (Radford et al., 2018),

Original:  XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) have attracted more and more attention in machine learning and natural language processing communities.
Revised: BERT (Devlin et al., 2019), cross-lingual language model (briefly, XLM) (Lample and Conneau, 2019), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) have attracted more and more attention in machine learning and natural language processing communities.

Original: The models are ﬁrst pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-aware embeddings of an input sequence (Peters et al., 2018), or initializing the parameters of the downstream model (Devlin et al., 2019) for ﬁne-tuning.
Revised: The models are first pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-aware embeddings of an input sequence (Peters et al., 2018), or initializing the parameters of the downstream model (Devlin et al., 2019) for fine-tuning.

Original: Such pre-training approaches lead to signiﬁcant improvements on natural language understanding tasks.
Revised: Such pre-training approaches lead to significant improvements on natural language understanding tasks.

Original: A Cross-lingual language model (Lample & Conneau, 2019) is used to obtain a better initialization of sentence encoders for zero-shot cross-lingual classiﬁcation, a better initialization of supervised and unsupervised neural machine translation systems, a language models for low-resource languages and unsupervised cross-lingual word embeddings.
Revised: A Cross-lingual language model (Lample and Conneau, 2019) is used to obtain a better initialization of sentence encoders for zero-shot cross-lingual classification, a better initialization of supervised and unsupervised neural machine translation systems, a language models for low-resource languages and unsupervised cross-lingual word embeddings.

Original: This is done through four objectives:
Revised: This is done through four objectives:

Original: • Shared sub-word vocabulary : XLM processes all languages with the same shared vocabulary created through Byte Pair Encoding (BPE) (Sennrich et al., 2015) • Causal Language Modeling (CLM) :
Revised: • Shared sub-word vocabulary : XLM processes all languages with the same shared vocabulary created through Byte Pair Encoding (BPE) (Sennrich et al., 2015) • Causal Language Modeling (CLM) :

Original: XLM causal language modeling (CLM) task consists of a Transformer language model trained to model the probability of a word given the previous words in a sentence P ( ω t | ω 1 , ..., ω t − 1 , θ ) .
Revised: XLM causal language modeling (CLM) task consists of a Transformer language model trained to model the probability of a word given the previous words in a sentence P ( ω t | ω 1 , ..., ω t − 1 , θ ) .

Original: • Masked Language Modeling (MLM) :
Revised: • Masked Language Modeling (MLM) :

Original: XLM consider the masked language modeling (MLM) objective of Devlin et al.
Revised: XLM consider the masked language modeling (MLM) objective of Devlin et al.

Original: (2019), also known as the Cloze task (Taylor, 1953).
Revised: (2019), also known as the Cloze task (Taylor, 1953).

Original: Following Devlin et al.
Revised: Following Devlin et al.

Original: XLM sample randomly 15% of the BPE tokens from the text streams, replace them by a [MASK] token 80% of the time, by a random token 10% of the time, and keep them unchanged 10% of the time.
Revised: XLM sample randomly 15% of the BPE tokens from the text streams, replace them by a [MASK] token 80% of the time, by a random token 10% of the time, and keep them unchanged 10% of the time.

Original: Differences between XLM approach and the MLM of Devlin et al.
Revised: Differences between XLM approach and the MLM of Devlin et al.

Original: (2019) include the use of text streams of an arbitrary number of sentences (truncated at 256 tokens) instead of pairs of sentences.
Revised: (2019) include the use of text streams of an arbitrary number of sentences (truncated at 256 tokens) instead of pairs of sentences.

Original: • Translation Language Modeling (TLM) :
Revised: • Translation Language Modeling (TLM) :

Original: Both the CLM and MLM objectives are unsupervised, only require monolingual data, and cannot be used to leverage parallel data when it is available.
Revised: Both the CLM and MLM objectives are unsupervised, only require monolingual data, and cannot be used to leverage parallel data when it is available.

Original: XLM introduce a new translation language modeling (TLM) objective for improving cross-lingual pretraining.
Revised: XLM introduce a new translation language modeling (TLM) objective for improving cross-lingual pretraining.

Original: TLM objective is an extension of MLM, where instead of considering monolingual text streams,we concatenate parallel sentences.
Revised: TLM objective is an extension of MLM, where instead of considering monolingual text streams, parallel sentences are concatenated.

Original: XLM randomly mask words in both the source and target sentences.
Revised: XLM randomly mask words in both the source and target sentences.

Original: To predict a word masked in an source sentence, the model can either attend to surrounding source words or to the target translation, encouraging the model to align the source and target representations.
Revised: To predict a word masked in an source sentence, the model can either attend to surrounding source words or to the target translation, encouraging the model to align the source and target representations.

Original: In particular, the model can leverage the target context if the source one is not sufﬁcient to infer the masked source words.
Revised: In particular, the model can leverage the target context if the source one is not sufficient to infer the masked source words.

Original: XLM also reset the positions of target sentences to facilitate the alignment.
Revised: XLM also reset the positions of target sentences to facilitate the alignment.

Original: In this work, we used the XLM 2 framework to train and evaluate our models.
Revised: In this work, we used the XLM 5 framework to train and evaluate our models.

Original:  M ETHODOLOGY 5. D ATA
Revised:  Methodology 5.1 Data

Original: YouVersion provides biblical content in 1,477 languages 3 among which many (local) African ones.
Revised: YouVersion provides biblical content in 1,756 languages 6 among which many (local) African ones.

Original: We have extracted this content in the desired languages to build our dataset.
Revised: We have extracted this content in the desired languages to build our dataset.

Original: In particular, for some countries like Cameroon, YouVersion provide more languages (22 for the latter) than JW300 (Agi´c & Vuli´c, 2019) (18 for Cameroon) that is the most used data source for work on African languages (e.g Masakhane (2020))
Revised: In particular, for some countries like Cameroon, YouVersion provide more languages (22 for the latter) than JW300 (Agi´c and Vuli´c, 2019) (18 for Cameroon) that is the most used data source for work on African languages (e.g Masakhane (2020)).

Original: We release our adaptable extraction script along with this publication.
Revised: We release our adaptable extraction script along with this publication.

Original: The number of sentences per language is given in table 3.
Revised: The number of sentences per language is given in figure 1.

Original: The Bible is one of the rare publicly available online corpora for low-resource languages such as those spoken in Africa.
Revised: The Bible is one of the rare publicly available online corpora for low-resource languages such as those spoken in Africa.

Original: Its structure (made of books, chapters and verse numbers) allows to auto- matically and unambiguously identify parallel paragraphs (verses) across all languages.
Revised: Its structure (made of books, chapters and verse numbers) allows to automatically and unambiguously identify parallel paragraphs (verses) across all languages.

Original: However, it suffers from several ﬂaws for machine translation:
Revised: However, it suffers from several flaws for machine translation:

Original: In general, YouVersion only provides the New Testament for African languages, consisting of about 7958 verses only.
Revised: In general, YouVersion only provides the New Testament for African languages, consisting of about 7958 verses only.

Original: In the case of languages for which both New and Old Testament are available, one can retrieve around 31170 verses.
Revised: In the case of languages for which both New and Old Testament are available, one can retrieve around 31170 verses.

Original: The Bible contains a lot of redundant sentences (”In truth I tell you” for example), verses made up of characters and places names etc... Making the translation task easier than if a closer to real world dataset like WMT was used.
Revised: The Bible contains a lot of redundant sentences ("Truly, I say to you/ I tell you" for example), verses made up of characters and places names etc... Making the translation task easier than if a closer to real world dataset like WMT was used.

Original: Its antiquated writing style is also far from contemporary style.
Revised: Its antiquated writing style is also far from contemporary style.

Original: The alignment information is limited to verses rather than sentences unlike the JRC-Acquis corpus (Steinberger et al.
Revised: The alignment information is limited to verses rather than sentences unlike the JRC-Acquis corpus (Steinberger et al., 2006) for instance. Christodouloupoulos

Original: for instance.
Revised: and

Original: (Christodouloupoulos & Steedman, 2015) gives more details on pros and cons of using the Bible as a machine translation corpus.
Revised: Steedman 2015) gives more details on pros and cons of using the Bible as a machine translation corpus.

Original: Following (Lample & Conneau, 2019), we process all the languages with the same shared vocabulary created using Byte Pair Encoding (BPE) (Sennrich et al., 2015).
Revised: Following Lample and Conneau (2019), we process all the languages with the same shared vocabulary created using Byte Pair Encoding (BPE) (Sennrich et al., 2015).

Original: Due to the lack of speciﬁc tokenizers for African languages, we used an english one on all of them.
Revised: Due to the lack of specific tokenizers for African languages, we used an english one on all of them (we performed tokenization before applying the BPE algorithm).

Original: 5. L ANGUAGES SIMILARITIES AND AGGREGATIONS 5.2. H ISTORICAL APPROACH FOR LANGUAGES AGGREGATION
Revised: 5.2 Languages similarities and aggregations 5.2.1 Historical approach for languages aggregation

Original: From a historical point of view, just like African borders, African languages have undergone the movements of the early inhabitants of the continent.
Revised: From a historical point of view, just like African borders, African languages have undergone the movements of the early inhabitants of the continent.

Original: These migrations have shaped and modiﬁed the languages spoken in multiple regions and led to resemblances between some of them.
Revised: These migrations have shaped and modified the languages spoken in multiple regions and led to resemblances between some of them.

Original: For these reasons, we examined and used languages aggreations proposed by Eberhard et al.
Revised: For these reasons, we examined and used languages aggreations proposed by Eberhard et al.

Original: (2020) that are based on their origins and evolutions.
Revised: (2020) that are based on their origins and evolutions.

Original: The intersection between our dataset and Eberhard et al.
Revised: The intersection between our dataset and Eberhard et al.

Original: (2020) analysis led to the following languages clusters:
Revised: (2020) analysis led to the following languages clusters:

Revised: 6 https://www.bible.com/en/languages (October 22, 2021)

Original: • Cluster 1 : (Baﬁa, Ewondo, Bulu) .
Revised: - Cluster 1 : (Bafia, Ewondo, Bulu) .

Original: These languages are all part of the Niger-Congo, Northwest 4 Narrow Bantu family and spoken in the central and southern regions of Cameroon.
Revised: These languages are all part of the Niger-Congo, Northwest 7 Narrow Bantu family and spoken in the central and southern regions of Cameroon.

Original: • Cluster 2 : (Ngiemboon, Ghomala, Limbum) .
Revised: - Cluster 2 : (Ngiemboon, Ghomala, Limbum) .

Original: They are part of the Niger-Congo, MbamNkam 5 family and spoken in the West Cameroon.
Revised: They are part of the Niger-Congo, Mbam-Nkam 8 family and spoken in the West Cameroon.

Original: • Cluster 3 : (Dii, Doyago, Samba Leko, Peere) .
Revised: - Cluster 3 : (Dii, Doyago, Samba Leko, Peere) .

Original: These languages are spoken in the Northern region of Cameroon and belong to the Niger-Congo, Adamawa 6 family • Cluster 4 : (Guiziga, Mofa, Mofu-Gudur, Kapsiki, Guidar) .
Revised: These languages are spoken in the Northern region of Cameroon and belong to the Niger-Congo, Adamawa 9 family - Cluster 4 : (Guiziga, Mofa, Mofu-Gudur, Kapsiki, Guidar) .

Original: They are also spoken in Northern Cameroon but have been classﬁed as Afro-Asiatic languages.
Revised: They are also spoken in Northern Cameroon but have been classfied as Afro-Asiatic languages.

Original: In particular, they belong to the Biu-Mandara 7 family.
Revised: In particular, they belong to the Biu-Mandara 10 family.

Original: In the rest of this work we used the ﬁrst cluster mentioned above for the experiments : it is the smallest of the 4 and is made of some of the most spoken languages in Cameroon (compared to the others listed above)
Revised: In the rest of this work we used the first cluster mentioned above for the experiments : it is the smallest of the 4 and is made of some of the most spoken languages in Cameroon (compared to the others listed above)

Original: 5.2. L ANGUAGE MODEL BASED APPROACH FOR LINGUSTIC SIMILARITIES
Revised: 5.2.2 Language model based approach for lingustic similarities

Original: Despite providing a historically grounded methodology to identify languages families, methods such as Eberhard et al.
Revised: Despite providing a historically grounded methodology to identify languages families, methods such as Eberhard et al.

Original: (2020) are time-consuming and require a lot of background knowledge on speciﬁc population migration throughout history.
Revised: (2020) are time-consuming and require a lot of background knowledge on specific population migration throughout history.

Original: Such in-depth knowledge might not be available for all languages.
Revised: Such in-depth knowledge might not be available for all languages.

Original: Besides, they don’t provide a quantitative measure for languages similarity, making it difﬁcult for one to assess if a language L i is closer to a language L j than to a language L k .
Revised: Besides, they don’t provide a quantitative measure for languages similarity, making it difficult for one to assess if a language L i is closer to a language L j than to a language L k .

Original: Quantitative metrics are necessary for well known clustering algorithms like K-Means or Hierarchical
Revised: Quantitative metrics are necessary for well known clustering algorithms like K-Means or Hierarchical

Original: Clustering.
Revised: Clustering.

Original: Multiple linguistic similarity metrics have been proposed in the past.
Revised: Multiple linguistic similarity metrics have been proposed in the past.

Original: The most popular are the ones used in lexicostatistics among which Swadesh lists (Swadesh, 1952) and Levenshtein distance (Levenshtein, 1965).
Revised: The most popular are the ones used in lexicostatistics among which Swadesh lists (Swadesh, 1952) and Levenshtein distance (Levenshtein, 1965).

Original: Nonetheless, the latter require data parallelism at word level and dictionaries might not be freely available online for all languages pairs.
Revised: Nonetheless, the latter require data parallelism at word level and dictionaries might not be freely available online for all languages pairs.

Original: Additionally, Swadesh lists  are based on a small number of words that might not even exist in all languages.
Revised: Additionally, Swadesh lists (Swadesh, 1952) are based on a small number of words that might not even exist in all languages.

Original: For example, the word ”snow” (present in the Swadesh 207 list) doesn’t have a translation in Baﬁa (likely because it never snows in this region).
Revised: For example, the word "snow" (present in the Swadesh 207 list) doesn’t have a translation in Bafia (likely because it never snows in this region).

Original: To alleviate these problems, we propose a language model based metric for linguistic similarity that requires paragraph level parallelism only.
Revised: To alleviate these problems, we propose a language model based metric for linguistic similarity that requires paragraph level parallelism only.

Original: This approach formalises (using language models) the following human intuition : a child whose native language is L 1 may think that a language L 2 is similar to L 1{+
+} if words (possibly with the same meaning) in L 2 ”sounds”/”looks” like words in L 
Revised: This approach formalises (using language models) the following human intuition : a child whose native language is L 1 may think that a language L 2 is similar to L {+
+} if words (possibly with the same meaning) in L 2 "sounds"/"looks" like words in L 1 .

Original: For example, the english word ”bilingual” is translated in french by ”bilingue” and in spanish by ”biling¨ue” but its polish translation is ”dwujezyczny”.
Revised: For example, the english word "bilingual" is translated in french by "bilingue" and in spanish by "bilingüe" but its polish translation is "dwujezyczny".

Original: A french speaker can easily think that the spanish ”biling ¨ ue” is a french word both at the spelling level because of common syllables (”bi”, ”lin” and ”gu”/”g ¨ u”) and at the pronunciation level.
Revised: A french speaker can easily think that the spanish "bilingüe" is a french word both at the spelling level because of common syllables ("bi", "lin" and "gu"/"gü") and at the pronunciation level.

Original: Nevertheless, it is unlikely for the polish ”dwujezyczny” to be seen as a french word.
Revised: Nevertheless, it is unlikely for the polish "dwujezyczny" to be seen as a french word.

Original: Following the success of recent language models like GPT-2 Radford et al.
Revised: Following the success of recent language models like GPT-2 Radford et al.

Original: (2019), the previous french speaker can be seen as a very powerful french language model lm fr .
Revised: (2019), the previous french speaker can be seen as a very powerful french language model lm fr .

Original: If we evaluate it on both ”biling ¨ ue” and ”dwujezyczny”, it should give a higher probabilty to the spanish ”biling ¨ ue” than to the polish ”dwujezyczny” because of the french ”bilingue”.
Revised: If we evaluate it on both "bilingüe" and "dwujezyczny", it should give a higher probabilty to the spanish "bilingüe" than to the polish "dwujezyczny" because of the french "bilingue".

Original: More formally, we expect the BPE encoding to capture syllables like ”bi”,”lin” and ”gu”/”g ¨ u” 8 so that if
Revised: More formally, we expect the BPE encoding to capture syllables like "bi","lin" and "gu"/"gü" 11 so that if

Original: P lm fr ( bilingue ) =
Revised: P lm fr ( bilingue ) =

Original: P lm fr ( bi ) ∗ P
Revised: P lm fr ( bi ) ∗ P

Original: lm fr ( lin | bi ) ∗ P
Revised: lm fr ( lin | bi ) ∗ P

Original: lm fr ( gu | bi, lin ) ∗ P
Revised: lm fr ( gu | bi, lin ) ∗ P

Original: lm fr ( e | bi, lin, gu ) is relatively high then P
Revised: lm fr ( e | bi, lin, gu ) is relatively high then P

Original: lm fr ( biling ¨ ue ) =
Revised: lm fr ( biling ¨ ue ) =

Original: P lm fr ( bi ) ∗ P
Revised: P lm fr ( bi ) ∗ P

Original: lm fr ( lin | bi ) ∗ P
Revised: lm fr ( lin | bi ) ∗ P

Original: lm fr ( g ¨ u | bi, lin ) ∗ P
Revised: lm fr ( g ¨ u | bi, lin ) ∗ P

Original: lm fr ( e | bi, lin, g ¨ u ) should be greater than P lm fr( dwujezyczny )
Revised: lm fr ( e | bi, lin, g ¨ u ) should be greater than P lm fr

Original: At a paragraph level, we expect that given two parallel paragraphs C spanish and C polish , lm fr gives a higher probability to C spanish i.e. P lm fr ( C spanish ) >
Revised: At a paragraph level, we expect that given two parallel paragraphs C spanish and C polish , lm fr gives a higher probability to C spanish i.e. P lm fr ( C spanish ) >

Original: P lm fr ( C polish )
Revised: P lm fr ( C polish )

Original: Given a trained language model LM 0 in language L 0 and two parallel corpora C 0 in L 0 and C 1 in language L 1 , we deﬁne the LM-similarity between L 0 and L 1 according to LM 0 and C 0 by
Revised: Given a trained language model LM 0 in language L 0 and two parallel corpora C 0 in L 0 and C 1 in language L 1 , we define the LM-similarity between L 0 and L 1 according to LM 0 and C 0 by

Original: [Equation]
Revised: [Equation]

Original: Where PPL LM 

Revised: Where PPL LM

Original: ( C i ) is the perplexity of the language model LM 0 on the corpus C i .
Revised: ( C i ) is the perplexity of the language model LM 0 on the corpus C i .

Original: It is important to precise that Accuracy (percentage of tokens correctly predicted by the language model) can be used in lieu of Perplexity (this may lead to different similarities between languages).
Revised: It is important to precise that Accuracy (percentage of tokens correctly predicted by the language model) can be used in lieu of Perplexity (this may lead to different similarities between languages).

Original: Despite the fact that this metric is not symmetric, it can be used to build clusters using a hierarchical clustering approach.
Revised: Despite the fact that this metric is not symmetric, it can be used to build clusters using a hierarchical clustering approach.

Original: We start by a set of 2 languages G= { L 1 , L 2 } , the language L ∗ that will be added to G 1 can be determined using the following formula (inspired by the unweighted average linkage clustering criteria) :
Revised: We start by a set of 2 languages G 1 = { L 1 , L 2 } , the language L ∗ that will be added to G 1 can be determined using the following formula (inspired by the unweighted average linkage clustering criteria) :

Original: [Equation]
Revised: [Equation]

Original: In other words, L ∗ is the language in the set of possible languages L (but we exclude G 1 =
Revised: In other words, L ∗ is the language in the set of possible languages L (but we exclude G 1 =

Original: { L 1 , L 2 } ) that minimises the average ”distance” (LM-similarity) from L 1 and L 2 .
Revised: { L 1 , L 2 } ) that minimises the average "distance" (LM-similarity) from L 1 and L 2 .

Original: By recursively applying this formula, one can build a cluster of any size.
Revised: By recursively applying this formula, one can build a cluster of any size.

Revised: The proposed similarity metric should be interpreted more as a dissimilarity, hence the equation normalizing a perplexity difference according to a base perplexity, to allow it to be zero on the base language and the base corpus.

Original: E XPERIMENTS AND R ESULTS
Revised: The word "similarity" here is only an abuse of language, to signify the fact that obenuous languages are similar in the sense of this metric. This method is mainly motivated by the fact that the resources needed to apply other methods such as those mentioned above are not available. M. Eberhard et al. (2019) has been easy to replicate in that it directly provides groups of similar languages: we have just made an intersection between these groups and the languages in our dataset.

Revised: On the other hand, a method like (Swadesh, 1952) requires parallelism of data at the word level, which we did not have and could not find: this is one of the limitations of this work..

Original: For each ordered language pair in the cluster of interest (Baﬁa, Ewondo, Bulu) , we examine the machine translation performance (using BLEU score) on that pair if we add a third language to the pair fo form a cluster.
Revised: For each ordered language pair in the cluster of interest (Bafia, Ewondo, Bulu) , we examine the machine translation performance (using BLEU score (Papineni et al., 2002)) on that pair if we add a third language to the pair fo form a cluster.

Original: The model will thus become a multi-task machine translation model capable of translating 3 languages rather than 2.
Revised: The model will thus become a multi-task machine translation model capable of translating 3 languages rather than 2.

Original: The language (among the 22 in our dataset) to add is chosen in 3 ways in our work: 
Revised: The language (among the 22 in our dataset) to add is chosen in 3 ways in our work: [Table]

Original: Using a historical approach like Eberhard et al.
Revised: Using a historical approach like Eberhard et al.

Original: For the pair (Baﬁa,Ewondo) we will add Bulu .
Revised: For the pair (Bafia, Ewondo) we will add Bulu .

Original: It corresponds to the column Historical in Table2.
Revised: It corresponds to the column Historical in Table2.

Original: Using the LM-similarity (as described in 5.2.2).
Revised: Using the LM-similarity (as described in 5.2.2).

Original: For the pair (Bulu,Ewondo) , it will be Mofa .
Revised: For the pair (Bulu, Ewondo) , it will be Mofa .

Original: It corresponds to the column LM in Table3.
Revised: It corresponds to the column LM in Table 3.

Original: Purely at random : it will serve as a baseline to make sure that the difference in performance is due to the particular choices of our methodology rather than multi-task learning solely.
Revised: Purely at random : it will serve as a baseline to make sure that the difference in performance is due to the particular choices of our methodology rather than multi-task learning solely.

Original: It corresponds to the column Random Table
Revised: It corresponds to the column Random Table

Original: Having identiﬁed our clusters and to further exploit multilingualism, we follow the approach of
Revised: Having identified our clusters and to further exploit multilingualism, we follow the approach of

Original: Lample & Conneau (2019) by pre-training the model’s encoder with MLM and TLM objectives  on the cluster languages, then ﬁne-tune the model on these same languages. 
Revised: Lample and Conneau (2019) by pre-training the model’s encoder with MLM and TLM objectives (without CLM) on the cluster languages, then fine-tune the model on these same languages. The

Revised: MLM pre-training was used as the languages did not have the same number of sentences : for example for the Bafia-Bulu pair, we have 7950 parallel sentences (representing the number of sentences of the

Revised: Bafia language) but 28895 sentences for Bulu (i.e. 28895 - 7950 = 20945 non-parallel sentences).

Revised: The TLM is naturally motivated by XLM : encourage the model to align source and target sentence representations (especially by taking advantage of the target context if the source context is not sufficient to infer hidden source words).

Original: We use the average BLEU score on the validation sets across all the  languages as stopping criterion.
Revised: All our models are tagged models, and the tag for a given language is its id in the list of languages used. Indeed, in addition to positional and token embedding, XLM does language embedding. This is motivated by several works. Caswell et al.

Revised: (2019) add a specific token (the < BT > tag) at the beginning of the synthetical sentences produced using back-translation to distinguish them from the available non-synthetical sentences.

Revised: Johnson et al.

Revised: (2017) do multilingual MT by adding a special token at the beginning of each sentence of the same language indicating the source language, and do not change anything about the architecture of classical MT models.

Revised: Yamagishi et al.

Original:  Due to the multi-task nature of the model, this criterion is preferred to validation BLEU score on the pair of interest : in some cases, the model can ﬁrst have low, stagnating performance on that pair for a large number of epochs (greater than the patience, threshold to stop the training) but at the same time have good performance on the other pairs (leading to an increase of average BLEU score).
Revised: (2016) add a tag specific to the voice (passive, active) of the target sentence in the source sentence so that the sentences produced by the model keep the voice of the source sentence. Kuczmarski and Johnson (2018) do the same, but more for gender (male and female gender). We use the average BLEU score on the validation sets across all the languages as stopping criterion. Due to the multi-task nature of the model, this criterion is preferred to validation BLEU score on the pair of interest : in some cases, the model can first have low, stagnating performance on that pair for a large number of epochs (greater than the patience, threshold to stop the training) but at the same time have good performance on the other pairs (leading to an increase of average BLEU score).

Original: At the end of a training with average BLEU score as stopping criterion, the pair of interest ”catches up” with the other pairs and obtain high scores.
Revised: At the end of a training with average BLEU score as stopping criterion, the pair of interest "catches up" with the other pairs and obtain high scores.

Original: We also compare the performance with model trained: 1) On the pair of interest without any form of MLM/TLM pre-training (Column None in Table 1).
Revised: We also compare the performance with model trained: 1) On the pair of interest without any form of MLM/TLM pre-training (Column None in Table 2).

Original: On the pair of interest with MLM+TLM pre-training (Column Pair in Table 1)
Revised: On the pair of interest with MLM+TLM pre-training (Column Pair in Table 2)

Original: 6. T RAINING DETAILS
Revised: 6.1 Training details

Original: We used the default settings mentioned in the documentation of the XLM framework :
Revised: We used the default settings mentioned in the documentation of the XLM framework :

Original: Transformer architectures with 1024 hidden units, 8 heads, GELU activations (Hendrycks & Gimpel, 2016),
Revised: Transformer architectures with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 2016),

Original: Dropout rate of 0.1 and learned positional embeddings, Adam optimizer (Kingma & Ba., 2014), a linear warmup (Vaswani et al., 2017) and learning rates of 0.0001 for both XLM pre-training  and machine translation.
Revised: Dropout rate of 0.1 and learned positional embeddings, Adam optimizer (Kingma and Ba., 2014), a linear warmup (Vaswani et al., 2017) and learning rates of 0.0001 for both XLM pre-training (MLM and TLM, without CLM) and machine translation.

Original: For the MLM and TLM pre-training, we used the averaged perplexity over all the languages as a measure of performance and stopping criterion for training, and for machine translation, we used the average BLEU score across all the languages pairs of the models.
Revised: For the MLM and TLM pre-training, we used the averaged perplexity over all the languages as a measure of performance and stopping criterion for training, and for machine translation, we used the average BLEU score across all the languages pairs of the models.

Original: The patience was set to 20 for all models.
Revised: The patience was set to 20 for all models.

Original: The models were trained on one 48GB NVIDIA Quadro RTX 8000 GPU for Random , Historical and LM and on a 11GB NVIDIA RTX [-
-]
Revised: The models were trained on one 48GB NVIDIA Quadro RTX 8000 GPU for Random , Historical and LM and on a 11GB NVIDIA RTX [-
-]2080 Ti for None and Pair .

Original: Ti for None and Pair .

Original: In all our experiments, we used 80% of the dataset as training data, 10% as validation data and 10% as test data.
Revised: In all our experiments, we used 80% of the dataset as training data, 10% as validation data and 10% as test data.

Original: 6. R ESULTS AND DISCUSSION
Revised: 6.2 Results, discussion and limitations

Original: Table 1 describes the results of the baselines mentioned in 6 and our models.
Revised: Table 2 describes the results of the baselines mentioned in 6 and our models.

Original: We notice that using our historical approach systematically leads to a strong boost of performance and always outperforms the None , Pair and Random baselines.
Revised: We notice that using our historical approach systematically leads to a strong boost of performance and always outperforms the None , Pair and Random baselines.

Original: As the aggregation approach that led to the best results, we evaluated its impact on translating African vernaculars to European languages.
Revised: As the aggregation approach that led to the best results, we evaluated its impact on translating African vernaculars to European languages.

Original: We chose french as it is the most spoken language in Cameroon resulting in the cluster (Baﬁa, Bulu, Ewondo, French) on which MLM + TLM pretraining and multi-task learning were done and compared to None and Pair baselines.
Revised: We chose french as it is the most spoken language in Cameroon resulting in the cluster (Bafia, Bulu, Ewondo, French) on which MLM + TLM pretraining and multi-task learning were done and compared to None and Pair baselines.

Original: Again, the historical approach outperformed all the baselines with notably a gain of22 BLEU on the pair French-Ewondo.
Revised: Again, the historical approach outperformed all the baselines with notably a gain of BLEU on the pair French-Ewondo

Original: Our LM-similarity approach systematically performs better than the None and Pair baselines, suggesting that adding a language using this metric and do multi-task learning on the resulting cluster is a good heuristic even if one only cares about a speciﬁc pair of languages.
Revised: Our LM-similarity approach systematically performs better than the None and Pair baselines, suggesting that adding a language using this metric and do multi-task learning on the resulting cluster is a good heuristic even if one only cares about a specific pair of languages.

Original: It outperforms the baseline Random for 4 out of the 6 pairs of languages.
Revised: It outperforms the baseline Random for 4 out of the 6 pairs of languages.

Original: According to this similarity, the closest language to the pair ( Baﬁa,Ewondo ) is Bulu , thus reconstituting the historical cluster identiﬁed by 5.2. without any human supervision.
Revised: According to this similarity, the closest language to the pair ( Bafia, Ewondo ) is Bulu , thus reconstituting the historical cluster identified by 5.2.1 without any human supervision.

Original: Randomly choosing this particular language would have happened with a 120 = 5% probability ( 20 = 22 − 2 , there are 22 languages in our dataset and we excludelanguages, Baﬁa and Ewondo , since we want to add another language to this pair).
Revised: Randomly choosing this particular language would have happened with a 120 = 5% probability ( 20 = 22 − 2 , there are 22 languages in our dataset and we exclude 2 languages, Bafia and Ewondo , since we want to add another language to this pair).

Original: The fact that the Random baseline performs worse than models derived from our methodologytimes out of 12 suggests that the performance boost is not solely due to the multi-task nature of the model but mostly because of the good choices of languages to add.
Revised: The fact that the Random baseline performs worse than models derived from our methodologytimes out of 12 suggests that the performance boost is not solely due to the multi-task nature of the model but mostly because of the good choices of languages to add.

Original: It is worth noticing that despite the MLM and TLM pre-training, the Random baseline performs worse than the baseline Pair 4 times out of 6.
Revised: It is worth noticing that despite the MLM and TLM pre-training, the Random baseline performs worse than the baseline Pair 4 times out of 6.

Original: This suggests that randomly adding languages may often hurt the performance compared to a simple model trained on the pair of interest
Revised: This suggests that randomly adding languages may often hurt the performance compared to a simple model trained on the pair of interest. [Table] Ew-Ba 13. Bu-Ew 10.

Revised: Ew-Bu 11.

Original: [Table]
Revised: [Table]

Revised: Pair

Revised: None

Revised: Pair

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: column None is a baseline : it represents the BLEU score of a model trained on the pair without any MLM or TLM pre-training.
Revised: [Table] [Table] [Table] Ba

Original: The column Pair is a baseline : it represents the BLEU score of a model trained on the pair with MLM and TLM pre-training.
Revised: [Table] [Table] Ba [Table]

Revised: Fr

Original: The column Random is also a baseline : it is the BLEU score of a 3 languages multi-task model where the language added was chosen purely at random.
Revised: [Table] [Table] [Table] Fr

Revised: [Table]

Revised: Ew

Original: The column Historical refers to the BLEU score of our 3 languages multi-task model where the language added was chosen using clusters identiﬁed in 5.2.1.
Revised: [Table] Fr 10. We believe that the difference between Random and Pair is not due to a disparity in data. First of all, they come from the same domain, the biblical domain.

Revised: With regard to the size of the dataset :

Original: The column LM describes the BLEU score of our 3 languages, multi-task model where the language added was chosen using the LM similarity ad described in 5.2.
Revised: • For Bafia-Bulu, the three languages added are Du_n, Ewondo and Mofa for Random (R), Historical (H) and LMS (Language Model-based approach for linguistic Similarity) respectively : this makes three clusters of almost the same size in terms of data, since only Bulu (present in all three clusters) among all these languages has 28895 sentences, the others have an average of 7939 (average of the sentences of the languages present only in the new testament). • Same thing for the Bulu-Ewondo pair with Dii, Bafia and Mofa added as languages for R, H and LMS respectively. • The pair which is an exception to this rule is Bafia-Ewondo, for which the languages added are respectively Guidar, Bulu and Bulu for R, H and LMS : H and LMS thus have an advantage over R with Bulu, but the gain is quite as good as for the other clusters.

Revised: This difference may come from the natural similarity in the Historical clusters.

Revised: In Cameroon you will have difficulty, if you are not very good at speaking these languages, in differentiating between the languages of the same historical cluster (both alphabetically and orally).

Revised: On the other hand, you can very easily, even if you are not familiar with these languages, make the difference between two languages of two different clusters.

Original: We hypothesize that a multi-task machine translation model (potentially with MLM and TLM pretraining) is capable of exploiting similarities between similar languages.
Revised: We hypothesize that a multi-task machine translation model (potentially with MLM and TLM pretraining) is capable of exploiting similarities between similar languages.

Original: A similar phenomenon can be observed with human beings : it is easier for someone that speaks French and Spanish to learn
Revised: A similar phenomenon can be observed with human beings : it is easier for someone that speaks French and Spanish to learn

Original: Italian (since they are all Latin languages) rather than polish (that is a Lechitic language). 
Revised: Italian (since they are all Latin languages) rather than polish (that is a Lechitic language). Limitations :

Revised: • Out Of Distribution (OOD) generalization impossible, as well as a practical use of our method, because of the use of data from the biblical domain • Due to the lack of specific tokenizers for African languages, we used an english.

Original: C ONCLUSION
Revised: The development of spacialized tokenizers like the one proposed by Dossou and Emezue (2021) is left as a perspective to this work. • The method used in this work to create the language clusters is only one agglomerative method among many others. We did not find it necessary to use very elaborate methods since the cluster was of size 3 (the historical cluster 1 has three languages). Negative Social Impact : The authors do not foresee negative social impacts of this work specifically.

Revised: Our theoretical work aims at showing the advantage of combining similar languages for machine translation, and the practical implications of this work (subject to the development of a more appropriate dataset) are mainly aimed at solving the problems related to language barriers in Africa

Revised:  Conclusion

Original: In this work, we studied the impact of using multilingualism to improve machine translation of
Revised: In this work, we studied the impact of using multilingualism to improve machine translation of

Original: African languages.
Revised: African languages.

Original: Our ﬁrst contribution is the creation and public release of a new dataset (as well as the code used to create it) that increases the number African languages for which parallel machine translation corpora is available.
Revised: Our first contribution is the creation and public release of a new dataset (as well as the code used to create it) that increases the number African languages for which parallel machine translation corpora is available.

Original: This study focused on pairs derived from a cluster of 3 of the most spoken languages in Cameroon.
Revised: This study focused on pairs derived from a cluster of 3 of the most spoken languages in Cameroon.

Original: Our second contribution is the proposal of a new metric to evaluate similarity between languages : this metric, based on language models, doesn’t require word level parallelism (contrary to traditional like (Levenshtein, 1965) and (Swadesh, 1952)) but only paragraph level parallelism.
Revised: Our second contribution is the proposal of a new metric to evaluate similarity between languages : this metric, based on language models, doesn’t require word level parallelism (contrary to traditional like (Levenshtein, 1965) and (Swadesh, 1952)) but only paragraph level parallelism.

Original: It also takes into account the lack of translation of words present in (Swadesh, 1952) but not in African languages (like ”snow”).
Revised: It also takes into account the lack of translation of words present in (Swadesh, 1952) but not in African languages (like "snow").

Original: Translation Language Modeling and multi-task learning were used to exploit multilingualism.
Revised: Translation Language Modeling and multi-task learning were used to exploit multilingualism.

Original: Our third contribution is the set of empirical evidences that doing multi-task learning on multiple similar languages generally improves the performances on individual pairs.
Revised: Our third contribution is the set of empirical evidences that doing multi-task learning on multiple similar languages generally improves the performances on individual pairs.

Original: The choice of the similarity is crucial since in our work, 4 times out of 6 a random choice leads to a decrease of performance.
Revised: The choice of the similarity is crucial since in our work, 4 times out of 6 a random choice leads to a decrease of performance.

Original: On the other hand, using our historical approach (derived from Eberhard et al.
Revised: On the other hand, using our historical approach (derived from Eberhard et al.

Original: (2020)) systematically led to a boost of performance on this dataset and our LM similarity approach improved them 4 times out of 6. 
Revised: (2020)) systematically led to a boost of performance on this dataset and our LM similarity approach improved them 4 times out of 6. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] The details of the dataset are given in Section 5.1, the proposed metric for evaluating linguistic similarity is detailed in Section 5.2.2, and the results obtained are presented in Table(b) Did you describe the limitations of your work?

Revised: [Yes] OOD and practical use impossible, use of an inappropriate tokenizer (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] Our theoretical work aims at showing the advantage of combining similar languages for machine translation, and the practical implications of this work (subject to the development of a more appropriate dataset) are mainly aimed at solving the problems related to language barriers in Africa (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] See 6.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [No] We did not have enough computational resources to run the experiments multiple times.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See 6. 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] The code used in this work is based on XLM 12 , the historical approach for languages aggregation is based on the work of Eberhard et al.

Revised: (2020), and the data are from Youversion(b)

Revised: Did you mention the license of the assets?

Revised: [Yes] These assets are subject to their initial licenses (XLM, YouVersion)

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [No] The data used was directly extracted from Youversion, under their Privacy Policy 14 ) (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [No] The data used was directly extracted from Youversion, under their Privacy Policy )

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: A Appendix

Revised: Additional details regarding motivations In relation to point 2 of the motivation section ( this discrepancy between spoken languages across different regions of individual African countries make the spread of misinformation easier (especially in rural regions and especially during election periods) ), we take an example relating to the election period.

Revised: During this period, the government sends people to explain to people in rural areas what the vote is going to be used for and how they should vote.

Revised: But it is common for them to do this in a way that favours certain political parties instead of simply translating the law, for example.

Revised: Let’s imagine that the text says in English "...

Revised: You must therefore choose one and only one candidate between Candidate A and Candidate

Revised: B." (... vous devez donc choisir un et un seul candidat entre le candidat A et le candidat B) and the translator says in French (to favour party A): "... vous devez donc choisir le candidat A et non le candidat B" (You must therefore choose candidate A and not candidate B).

Revised: On the other hand, the information that is broadcast on television or radio is in French or English.

Revised: If the villagers had access to a tool allowing them to translate the legal texts (which are in French or English) into their mother tongue, this could be avoided.

Revised: Someone may ask "Why learning a different unrelated language should hurt other languages I know?

Revised: " In our case, language learning takes place simultaneously, and we believe that learning several languages that are not similar (in the sense of the two similarities mentioned above) simultaneously makes the learning task more difficult.

Revised: We also believe that the gain in performance at the level of the historical clusters comes from a natural similarity in the Historical clusters.

Revised: In Cameroon you will have difficulty, if you are not very good at speaking these languages, in differentiating between the languages of the same historical cluster (both alphabetically and orally).

Revised: On the other hand, you can very easily, even if you are not familiar with these languages, make the difference between two languages of two different clusters