Original: C ROSS -L INGUAL A BILITY OF M ULTILINGUAL BERT: A N E MPIRICAL S TUDY
Revised: C ROSS -L INGUAL A BILITY OF M ULTILINGUAL BERT: A N E MPIRICAL S TUDY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent work has exhibited the surprising cross-lingual abilities of multilingual
Revised: Recent work has exhibited the surprising cross-lingual abilities of multilingual

Original: BERT ( M-BERT ) – surprising since it is trained without any cross-lingual objective and with no aligned data.
Revised: BERT ( M-BERT ) – surprising since it is trained without any cross-lingual objective and with no aligned data.

Original: In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability.
Revised: In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability.

Original: We study the impact of linguistic properties of the languages, the architecture of the model and the learning objectives.
Revised: We study the impact of linguistic properties of the languages, the architecture of the model and the learning objectives.

Original: The experimental study is done in the context of three typologically different languages – Spanish, Hindi and Russian – and using two conceptually different NLP tasks, textual entailment and named entity recognition.
Revised: The experimental study is done in the context of three typologically different languages – Spanish, Hindi and Russian – and using two conceptually different NLP tasks, textual entailment and named entity recognition.

Original: Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it
Revised: Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Embeddings of natural language text via unsupervised learning, coupled with sufﬁcient supervised training data, have been ubiquitous in NLP in recent years and have shown success in a wide range of monolingual NLP tasks, mostly in English.
Revised: Embeddings of natural language text via unsupervised learning, coupled with sufﬁcient supervised training data, have been ubiquitous in NLP in recent years and have shown success in a wide range of monolingual NLP tasks, mostly in English.

Original: Training models for other languages have been shown more difﬁcult, and recent approaches relied on bilingual embeddings that allowed the transfer of supervision in high resource languages like English to models in lower resource languages; however, inducing these bilingual embeddings required some level of supervision (Upadhyay et al., 2016).
Revised: Training models for other languages have been shown more difﬁcult, and recent approaches relied on bilingual embeddings that allowed the transfer of supervision in high resource languages like English to models in lower resource languages; however, inducing these bilingual embeddings required some level of supervision (Upadhyay et al., 2016).

Original: Multilingual BERT 1 ( M-BERT ), a Transformer-based (Vaswani et al., 2017) language model trained on raw Wikipedia text of 104 languages suggests an entirely different approach.
Revised: Multilingual BERT 1 ( M-BERT ), a Transformer-based (Vaswani et al., 2017) language model trained on raw Wikipedia text of 104 languages suggests an entirely different approach.

Original: Not only the model is contextual, but its training also requires no supervision – no alignment between the languages is done.
Revised: Not only the model is contextual, but its training also requires no supervision – no alignment between the languages is done.

Original: Nevertheless, and despite being trained with no explicit cross-lingual objective, M-BERT produces a representation that seems to generalize well across languages for a variety of downstream tasks (Wu & Dredze, 2019).
Revised: Nevertheless, and despite being trained with no explicit cross-lingual objective, M-BERT produces a representation that seems to generalize well across languages for a variety of downstream tasks (Wu & Dredze, 2019).

Original: In this work, we attempt to develop an understanding of the success of M-BERT .
Revised: In this work, we attempt to develop an understanding of the success of M-BERT .

Original: We study a range of aspects on a couple of different NLP tasks, in order to identify the key components in the success of the model.
Revised: We study a range of aspects on a couple of different NLP tasks, in order to identify the key components in the success of the model.

Original: Our study is done in the context of only two languages, source (typically English) and target (multiple, quite different languages).
Revised: Our study is done in the context of only two languages, source (typically English) and target (multiple, quite different languages).

Original: By involving only a pair of languages, we can study the performance on a given target language, ensuring that it is inﬂuenced only by the cross-lingual transfer from the source language, without having to worry about a third language interfering.
Revised: By involving only a pair of languages, we can study the performance on a given target language, ensuring that it is inﬂuenced only by the cross-lingual transfer from the source language, without having to worry about a third language interfering.

Original: We analyze the two-languages version of M-BERT ( B-BERT , from now on) in three orthogonal dimensions: (i) Linguistics properties and similarities of target and source languages; (ii) Network
Revised: We analyze the two-languages version of M-BERT ( B-BERT , from now on) in three orthogonal dimensions: (i) Linguistics properties and similarities of target and source languages; (ii) Network

Original: Architecture, and (iii) Input and Learning Objective.
Revised: Architecture, and (iii) Input and Learning Objective.

Original: One hypothesis that came up when the people thoughts about the success of M-BERT is due to some level of language similarity.
Revised: One hypothesis that came up when the people thoughts about the success of M-BERT is due to some level of language similarity.

Original: This could be lexical similarity (shared words or word-parts) or structural similarities, or both.
Revised: This could be lexical similarity (shared words or word-parts) or structural similarities, or both.

Original: We, therefore, investigate the contribution of word-piece overlap – the extent to which the same word-pieces appear in both source and target languages – and distinguish it from other similarities, which we call structural similarity between the source and target languages.
Revised: We, therefore, investigate the contribution of word-piece overlap – the extent to which the same word-pieces appear in both source and target languages – and distinguish it from other similarities, which we call structural similarity between the source and target languages.

Original: Surprisingly, as we show, B-BERT is cross-lingual even when there is absolutely no word-piece overlap.
Revised: Surprisingly, as we show, B-BERT is cross-lingual even when there is absolutely no word-piece overlap.

Original: That is, other aspects of language similarity must be contributing to the cross-lingual capabilities of the model.
Revised: That is, other aspects of language similarity must be contributing to the cross-lingual capabilities of the model.

Original: This is contrary to Pires et al.
Revised: This is contrary to Pires et al.

Original: (2019) hypothesis that M-BERT gains its power from shared word-pieces.
Revised: (2019) hypothesis that M-BERT gains its power from shared word-pieces.

Original: Furthermore, we show that the amount of word-piece overlap in
Revised: Furthermore, we show that the amount of word-piece overlap in

Original: B-BERT ’s training data contributes little to performance improvements.
Revised: B-BERT ’s training data contributes little to performance improvements.

Original: Our study of the model architecture addresses the importance of (i)
Revised: Our study of the model architecture addresses the importance of (i)

Original: The network depth, (ii) the number of attention heads, and (iii) the total number of model parameters in B-BERT .
Revised: The network depth, (ii) the number of attention heads, and (iii) the total number of model parameters in B-BERT .

Original: Our results suggest that depth and the total number of parameters of B-BERT are crucial for both monolingual and cross-lingual performance, whereas multi-head attention is not a signiﬁcant factor – a single attention head B-BERT can already give satisfactory results.
Revised: Our results suggest that depth and the total number of parameters of B-BERT are crucial for both monolingual and cross-lingual performance, whereas multi-head attention is not a signiﬁcant factor – a single attention head B-BERT can already give satisfactory results.

Original: To understand the role of the learning objective and the input representation, we study the effect of (i) the next sentence prediction objective, (ii) the language identiﬁer in the training data, and (iii) the level of tokenization in the input representation (character, word-piece, or word tokenization).
Revised: To understand the role of the learning objective and the input representation, we study the effect of (i) the next sentence prediction objective, (ii) the language identiﬁer in the training data, and (iii) the level of tokenization in the input representation (character, word-piece, or word tokenization).

Original: Our results indicate that the next sentence prediction objective actually hurts the performance of the model while identifying the language in the input does not affect B-BERT ’s performance crosslingually.
Revised: Our results indicate that the next sentence prediction objective actually hurts the performance of the model while identifying the language in the input does not affect B-BERT ’s performance crosslingually.

Original: Our experiments also show that character-level and word-level tokenization of the input results in signiﬁcantly worse performance than word-piece level tokenization.
Revised: Our experiments also show that character-level and word-level tokenization of the input results in signiﬁcantly worse performance than word-piece level tokenization.

Original: Overall, we provide an extensive set of experiments on three source-target language pairs, English–
Revised: Overall, we provide an extensive set of experiments on three source-target language pairs, English–

Original: Spanish, English–Russian, and English–Hindi.
Revised: Spanish, English–Russian, and English–Hindi.

Original: We chose these target languages since they vary in scripts and typological features.
Revised: We chose these target languages since they vary in scripts and typological features.

Original: We evaluate the performance of B-BERT on two very different downstream tasks: cross-lingual Named Entity Recognition – a sequence prediction task the requires only local context – and cross-lingual Textual Entailment Dagan et al.
Revised: We evaluate the performance of B-BERT on two very different downstream tasks: cross-lingual Named Entity Recognition – a sequence prediction task the requires only local context – and cross-lingual Textual Entailment Dagan et al.

Original: (2013) that requires more global representation of the text.
Revised: (2013) that requires more global representation of the text.

Original: Ours is not the ﬁrst study of M-BERT .
Revised: Ours is not the ﬁrst study of M-BERT .

Original: (Wu & Dredze, 2019) and (Pires et al., 2019) identiﬁed the cross-lingual success of the model and tried to understand it.
Revised: (Wu & Dredze, 2019) and (Pires et al., 2019) identiﬁed the cross-lingual success of the model and tried to understand it.

Original: The former by considering M-BERT layerwise, relating cross-lingual performance with the amount of shared word-pieces and the latter by considering the model’s ability to transfer between languages as a function of word order similarity in languages.
Revised: The former by considering M-BERT layerwise, relating cross-lingual performance with the amount of shared word-pieces and the latter by considering the model’s ability to transfer between languages as a function of word order similarity in languages.

Original: However, both works treated M-BERT as a black box and compared M-BERT ’s performance on different languages.
Revised: However, both works treated M-BERT as a black box and compared M-BERT ’s performance on different languages.

Original: This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components, along multiple aspects.
Revised: This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components, along multiple aspects.

Original: We also note that some of the architectural conclusions have been observed earlier, if not investigated, in other contexts.
Revised: We also note that some of the architectural conclusions have been observed earlier, if not investigated, in other contexts.

Original: Liu et al.
Revised: Liu et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2019) argued that the next Sentence prediction objective of BERT (the monolingual model) is not very useful; we show that this is the case in the cross-lingual setting.
Revised: (2019) argued that the next Sentence prediction objective of BERT (the monolingual model) is not very useful; we show that this is the case in the cross-lingual setting.

Original: Voita et al.
Revised: Voita et al.

Original: (2019) prunes attention heads for a transformer based machine translation model and argues that most attention heads are not important; in this work, we show that the number of attention heads is not important in the cross-lingual setting.
Revised: (2019) prunes attention heads for a transformer based machine translation model and argues that most attention heads are not important; in this work, we show that the number of attention heads is not important in the cross-lingual setting.

Original: Our contributions are threefold: (i) we provide the ﬁrst extensive study of the aspects of the multilingual BERT that give rise to its cross-lingual ability.
Revised: Our contributions are threefold: (i) we provide the ﬁrst extensive study of the aspects of the multilingual BERT that give rise to its cross-lingual ability.

Original: (ii) We develop a methodology that facilitates the analysis of similarities between languages and their impact on cross-lingual models; we do this by mapping English to a Fake-English language, that is identical in all aspects to English but shares not word-pieces with any target language.
Revised: (ii) We develop a methodology that facilitates the analysis of similarities between languages and their impact on cross-lingual models; we do this by mapping English to a Fake-English language, that is identical in all aspects to English but shares not word-pieces with any target language.

Original: Finally, (iii) we develop a set of insights into B-BERT , along linguistics, architectural, and learning dimensions, that would contribute to further understanding and to the development of more advanced cross-lingual neural models.
Revised: Finally, (iii) we develop a set of insights into B-BERT , along linguistics, architectural, and learning dimensions, that would contribute to further understanding and to the development of more advanced cross-lingual neural models.

Original:  B ACKGROUND 2. BERT
Revised:  B ACKGROUND 2. BERT

Original: BERT (Devlin et al., 2019) is Transformer (Vaswani et al., 2017) based pre-training language representation model that has been widely used in the ﬁeld of Natural Language Processing.
Revised: BERT (Devlin et al., 2019) is Transformer (Vaswani et al., 2017) based pre-training language representation model that has been widely used in the ﬁeld of Natural Language Processing.

Original: BERT is trained with Masked Language Modelling (MLM) (Taylor, 1953) and Next Sentence Prediction (NSP) objectives.
Revised: BERT is trained with Masked Language Modelling (MLM) (Taylor, 1953) and Next Sentence Prediction (NSP) objectives.

Original: Input to BERT is a pair of sentences 2 A and B, such that half of the time B comes after A in the original text and the rest of the time B is a randomly sampled sentence.
Revised: Input to BERT is a pair of sentences 2 A and B, such that half of the time B comes after A in the original text and the rest of the time B is a randomly sampled sentence.

Original: Some tokens from the input are randomly masked, and the MLM objective is to predict the masked tokens.
Revised: Some tokens from the input are randomly masked, and the MLM objective is to predict the masked tokens.

Original: The
Revised: The

Original: NSP objective is to predict whether B is the actual next sentence of A or not.
Revised: NSP objective is to predict whether B is the actual next sentence of A or not.

Original: (Devlin et al., 2019)
Revised: (Devlin et al., 2019)

Original: BERT follows two-steps 1.
Revised: BERT follows two-steps 1.

Original: Pre-training and, 2.
Revised: Pre-training and, 2.

Original: Fine-tuning.
Revised: Fine-tuning.

Original: BERT is pre-trained using the above mentioned MLM and NSP objective on BooksCorpus and English Wikipedia text, and for any supervised downstream task, BERT is initialized with the pre-trained weights and ﬁne-tuned using the labeled data.
Revised: BERT is pre-trained using the above mentioned MLM and NSP objective on BooksCorpus and English Wikipedia text, and for any supervised downstream task, BERT is initialized with the pre-trained weights and ﬁne-tuned using the labeled data.

Original: BERT uses wordpiece tokenization (Wu et al., 2016), which creates wordpiece vocabulary in a data-driven approach.
Revised: BERT uses wordpiece tokenization (Wu et al., 2016), which creates wordpiece vocabulary in a data-driven approach.

Original: 2. M ULTILINGUAL BERT
Revised: 2. M ULTILINGUAL BERT

Original: Multilingual BERT is pre-trained in the same way as monolingual BERT except using Wikipedia text from the top 104 languages.
Revised: Multilingual BERT is pre-trained in the same way as monolingual BERT except using Wikipedia text from the top 104 languages.

Original: To account for the differences in the size of Wikipedia, some languages are sub-sampled, and some are super-sampled using exponential smoothing Devlin et al.
Revised: To account for the differences in the size of Wikipedia, some languages are sub-sampled, and some are super-sampled using exponential smoothing Devlin et al.

Original: It’s worth mentioning that there are no cross-lingual objectives speciﬁcally designed nor any cross-lingual data, e.g. parallel corpus, used.
Revised: It’s worth mentioning that there are no cross-lingual objectives speciﬁcally designed nor any cross-lingual data, e.g. parallel corpus, used.

Original:  W HY M ULTILINGUAL BERT W ORKS
Revised:  W HY M ULTILINGUAL BERT W ORKS

Original: In this section, we analyze the reason for the cross-lingual ability of multilingual BERT (actually
Revised: In this section, we analyze the reason for the cross-lingual ability of multilingual BERT (actually

Original: B-BERT ) in three dimensions.
Revised: B-BERT ) in three dimensions.

Original: (i) Linguistics (ii) Architecture and (iii) Input and Learning Objective.
Revised: (i) Linguistics (ii) Architecture and (iii) Input and Learning Objective.

Original: Languages share similarities with each other.
Revised: Languages share similarities with each other.

Original: For example, English and Spanish have words that look seemingly the same; English and Russian both have a Subject-Verb-Object(SVO) order 3 ;
Revised: For example, English and Spanish have words that look seemingly the same; English and Russian both have a Subject-Verb-Object(SVO) order 3 ;

Original: English and Hindi, despite in different scripts, use the same Arabic numerals 4 .
Revised: English and Hindi, despite in different scripts, use the same Arabic numerals 4 .

Original: The similarity between languages can be a reason of M-BERT ’s cross-lingual ability.
Revised: The similarity between languages can be a reason of M-BERT ’s cross-lingual ability.

Original: In this linguistics point of view, we study the contribution of word-piece overlap — the similarity of languages arising from the same characters/words used across languages as well as code-switching data — and structure similarity, the part of linguistic similarity that is not explained by word-piece overlap, and does not rely on the script of the language
Revised: In this linguistics point of view, we study the contribution of word-piece overlap — the similarity of languages arising from the same characters/words used across languages as well as code-switching data — and structure similarity, the part of linguistic similarity that is not explained by word-piece overlap, and does not rely on the script of the language

Original: We hypothesize that the cross-lingual effectiveness of B-BERT comes from the architecture of
Revised: We hypothesize that the cross-lingual effectiveness of B-BERT comes from the architecture of

Original: BERT itself being able to extract good semantic and structural features.
Revised: BERT itself being able to extract good semantic and structural features.

Original: We study the depth, number of attention heads, and the total number of parameters of the transformer model to explore the inﬂuence of each part to the cross-lingual ability.
Revised: We study the depth, number of attention heads, and the total number of parameters of the transformer model to explore the inﬂuence of each part to the cross-lingual ability.

Original: Finally, we study the effect of learning objectives and input.
Revised: Finally, we study the effect of learning objectives and input.

Original: The Next Sentence Prediction objective is shown to be unnecessary in monolingual settings, and we try to analyze its effect in the crosslingual setting.
Revised: The Next Sentence Prediction objective is shown to be unnecessary in monolingual settings, and we try to analyze its effect in the crosslingual setting.

Original: B-BERT follows BERT and uses a word-piece vocabulary.
Revised: B-BERT follows BERT and uses a word-piece vocabulary.

Original: Word-pieces can be seen as a tradeoff between characters and words.
Revised: Word-pieces can be seen as a tradeoff between characters and words.

Original: We compare these three ways of tokenizing the input on how they affect cross-lingual transferring.
Revised: We compare these three ways of tokenizing the input on how they affect cross-lingual transferring.

Original: 3. D ATASETS AND E XPERIMENTAL S ETUP
Revised: 3. D ATASETS AND E XPERIMENTAL S ETUP

Original: In this work, we conduct all our experiments on two conceptually different downstream tasks – crosslingual Textual Entailment (TE) and cross-lingual Named Entity Recognition (NER).
Revised: In this work, we conduct all our experiments on two conceptually different downstream tasks – crosslingual Textual Entailment (TE) and cross-lingual Named Entity Recognition (NER).

Original: TE measures natural language understanding (NLU) at a sentence and sentence pair level, whereas NER measures
Revised: TE measures natural language understanding (NLU) at a sentence and sentence pair level, whereas NER measures

Original: NLU at a token level.
Revised: NLU at a token level.

Original: We use the Cross-lingual Natural Language Inference (XNLI)
Revised: We use the Cross-lingual Natural Language Inference (XNLI)

Original: (Conneau et al., 2018) dataset to evaluate cross-lingual TE performance and LORELEI dataset (Strassel & Tracey, 2016) for Cross-Lingual NER.
Revised: (Conneau et al., 2018) dataset to evaluate cross-lingual TE performance and LORELEI dataset (Strassel & Tracey, 2016) for Cross-Lingual NER.

Original: 3.1. C ROSS - LINGUAL N ATURAL L ANGUAGE I NFERENCE (XNLI)
Revised: 3.1. C ROSS - LINGUAL N ATURAL L ANGUAGE I NFERENCE (XNLI)

Original: XNLI is a standard cross-lingual textual entailment dataset that extends MultiNLI
Revised: XNLI is a standard cross-lingual textual entailment dataset that extends MultiNLI

Original: (Williams et al., 2018) dataset by creating a new dev and test set and manually translating into 14 different languages.
Revised: (Williams et al., 2018) dataset by creating a new dev and test set and manually translating into 14 different languages.

Original: Each input consists of a premise and hypothesis pair, and the task is to classify the relationship between premise and hypothesis into one of the three labels: entailment, contradiction, and neutral.
Revised: Each input consists of a premise and hypothesis pair, and the task is to classify the relationship between premise and hypothesis into one of the three labels: entailment, contradiction, and neutral.

Original: While training, both premise, and hypotheses are in English, and while testing, both are in the target language.
Revised: While training, both premise, and hypotheses are in English, and while testing, both are in the target language.

Original: XNLI uses the same set of premises and hypotheses for all the language, making the comparison across languages possible.
Revised: XNLI uses the same set of premises and hypotheses for all the language, making the comparison across languages possible.

Original: 3.1. C ROSS - LINGUAL N AMED E NTITY R ECOGNITION (NER)
Revised: 3.1. C ROSS - LINGUAL N AMED E NTITY R ECOGNITION (NER)

Original: Named Entity Recognition is the task of identifying and labeling text spans as named entities, such as people’s names and locations.
Revised: Named Entity Recognition is the task of identifying and labeling text spans as named entities, such as people’s names and locations.

Original: The NER dataset (Strassel & Tracey, 2016)
Revised: The NER dataset (Strassel & Tracey, 2016)

Original: we use consists of news and social media text labeled by native speakers following the same guideline in several languages, including English, Hindi, Spanish, and Russian.
Revised: we use consists of news and social media text labeled by native speakers following the same guideline in several languages, including English, Hindi, Spanish, and Russian.

Original: We subsample 80%, 10%, 10% of English NER data as training, development, and testing.
Revised: We subsample 80%, 10%, 10% of English NER data as training, development, and testing.

Original: We use the whole dataset of Hindi, Spanish, and Russian for testing purposes.
Revised: We use the whole dataset of Hindi, Spanish, and Russian for testing purposes.

Original: The vocabulary size is ﬁxed at 60000 and is estimated through the unigram language model in the SentencePiece library (Kudo, 2018).
Revised: The vocabulary size is ﬁxed at 60000 and is estimated through the unigram language model in the SentencePiece library (Kudo, 2018).

Original: 3.1. N OTATION AND E XPERIMENTAL SETUP
Revised: 3.1. N OTATION AND E XPERIMENTAL SETUP

Original: We denote B-BERT trained on language A and B as A-B, e.g., B-BERT trained on English (en) and
Revised: We denote B-BERT trained on language A and B as A-B, e.g., B-BERT trained on English (en) and

Original: Hindi (hi) as en-hi, similarly for Spanish (es) and Russian (ru).
Revised: Hindi (hi) as en-hi, similarly for Spanish (es) and Russian (ru).

Original: For pretraining, we subsample en, es, and ru Wikipedia to 1GB and use the entire Wikipedia for Hindi.
Revised: For pretraining, we subsample en, es, and ru Wikipedia to 1GB and use the entire Wikipedia for Hindi.

Original: Unless otherwise speciﬁed, for
Revised: Unless otherwise speciﬁed, for

Original: B-BERT training
Revised: B-BERT training

Original: , we use a batch size of 32 , the learning rate of 0 . 0001 , and 2 M training steps.
Revised: , we use a batch size of 32 , the learning rate of 0 . 0001 , and 2 M training steps.

Original: For XNLI, we use the same ﬁnetuning approach as BERT uses in English and report accuracy.
Revised: For XNLI, we use the same ﬁnetuning approach as BERT uses in English and report accuracy.

Original: For
Revised: For

Original: NER, we extract BERT representations as features and ﬁnetune a Bi-LSTM CRF model and report entity span F 1 score averaged from 5 runs with its standard deviation.
Revised: NER, we extract BERT representations as features and ﬁnetune a Bi-LSTM CRF model and report entity span F 1 score averaged from 5 runs with its standard deviation.

Original: 3. L INGUISTICS
Revised: 3. L INGUISTICS

Original: Pires et al.
Revised: Pires et al.

Original: (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages.
Revised: (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages.

Original: However, our experiments show that B-BERT is cross-lingual even when there is no word-piece overlap.
Revised: However, our experiments show that B-BERT is cross-lingual even when there is no word-piece overlap.

Original: Further, (Wu & Dredze, 2019) hypothesizes that for cross-lingual transfer learning source language should be selected such that it shares more word-pieces with the target language.
Revised: Further, (Wu & Dredze, 2019) hypothesizes that for cross-lingual transfer learning source language should be selected such that it shares more word-pieces with the target language.

Original: However, our experiment suggests that structural similarity is much more important.
Revised: However, our experiment suggests that structural similarity is much more important.

Original: Motivated by the above two hypotheses, in this section, we study the contribution of word-piece overlap and structural similarity for the cross-lingual ability of B-BERT .
Revised: Motivated by the above two hypotheses, in this section, we study the contribution of word-piece overlap and structural similarity for the cross-lingual ability of B-BERT .

Original: 3.2. W ORD - PIECE O VERLAP
Revised: 3.2. W ORD - PIECE O VERLAP

Original: M-BERT model is trained using Wikipedia text from 104 languages, and the texts from different languages share some common wordpiece vocabulary (like numbers, links, etc.. including actual words, if they have the same script), we refer to this as word-piece overlap.
Revised: M-BERT model is trained using Wikipedia text from 104 languages, and the texts from different languages share some common wordpiece vocabulary (like numbers, links, etc.. including actual words, if they have the same script), we refer to this as word-piece overlap.

Original: The previous work (Pires et al., 2019) hypothesizes that M-BERT generalizes across languages because these shared wordpieces have to be mapped to shared space forcing the other co-occurring word-pieces to be mapped to the same shared space
Revised: The previous work (Pires et al., 2019) hypothesizes that M-BERT generalizes across languages because these shared wordpieces have to be mapped to shared space forcing the other co-occurring word-pieces to be mapped to the same shared space

Original: [Table]
Revised: [Table]

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we perform experiments to compare cross-lingual performance with and without word-piece overlap.
Revised: we perform experiments to compare cross-lingual performance with and without word-piece overlap.

Original: We construct a new corpus – Fake-English (enfake) , by shifting the Unicode of each character in English Wikipedia text by a large constant so that there is strictly no character overlap with any other Wikipedia text.
Revised: We construct a new corpus – Fake-English (enfake) , by shifting the Unicode of each character in English Wikipedia text by a large constant so that there is strictly no character overlap with any other Wikipedia text.

Original: In this work, we consider Fake-English as a different language
Revised: In this work, we consider Fake-English as a different language

Original: [Table]
Revised: [Table]

Original: We measure the contribution of word-piece overlap as the drop in performance when the word-piece overlap is removed.
Revised: We measure the contribution of word-piece overlap as the drop in performance when the word-piece overlap is removed.

Original: From Table 1, we can see B-BERT is cross-lingual even when there is no wordpiece overlap .
Revised: From Table 1, we can see B-BERT is cross-lingual even when there is no wordpiece overlap .

Original: We can also see that the contribution of word-piece overlap is very small, which is quite surprising and contradictory to the hypothesis by (Pires et al., 2019; Wu & Dredze, 2019).
Revised: We can also see that the contribution of word-piece overlap is very small, which is quite surprising and contradictory to the hypothesis by (Pires et al., 2019; Wu & Dredze, 2019).

Original: 3.2. S TRUCTURAL S IMILARITY
Revised: 3.2. S TRUCTURAL S IMILARITY

Original: We deﬁne the structure of a language as every property of an individual language that is invariant to the script of the language, e.g., morphology, word-ordering, word frequency, word-pair frequency are all part of the structure of a language.
Revised: We deﬁne the structure of a language as every property of an individual language that is invariant to the script of the language, e.g., morphology, word-ordering, word frequency, word-pair frequency are all part of the structure of a language.

Original: Note that English and Fake-English don’t share any vocabulary/characters, but they have exactly the same structure.
Revised: Note that English and Fake-English don’t share any vocabulary/characters, but they have exactly the same structure.

Original: From Table 1, we can see that
Revised: From Table 1, we can see that

Original: BERT transfers very well from Fake-English to English.
Revised: BERT transfers very well from Fake-English to English.

Original: Also note that, despite not sharing any vocabulary, Fake-English transfers to Spanish, Hindi, Russian almost as well as English.
Revised: Also note that, despite not sharing any vocabulary, Fake-English transfers to Spanish, Hindi, Russian almost as well as English.

Original: On XNLI, where the scores between languages can be compared, the cross-lingual transferability from FakeEnglish to Spanish is much better than from Fake-English to Hindi/Russian.
Revised: On XNLI, where the scores between languages can be compared, the cross-lingual transferability from FakeEnglish to Spanish is much better than from Fake-English to Hindi/Russian.

Original: Since they do not share any word-pieces, this better transferability comes from the structure being closer between Spanish and Fake-English.
Revised: Since they do not share any word-pieces, this better transferability comes from the structure being closer between Spanish and Fake-English.

Original: These results suggest that we should shed more light on studying the structural similarity between languages.
Revised: These results suggest that we should shed more light on studying the structural similarity between languages.

Original: In this study, we don’t further dissect the structure of language as currently, the deﬁnition of “Structure of a Language” is fuzzy.
Revised: In this study, we don’t further dissect the structure of language as currently, the deﬁnition of “Structure of a Language” is fuzzy.

Original: Despite its amorphous deﬁnition, our experiment clearly shows that structural similarity is crucial for cross-lingual transfer.
Revised: Despite its amorphous deﬁnition, our experiment clearly shows that structural similarity is crucial for cross-lingual transfer.

Original: 3. A RCHITECTURE
Revised: 3. A RCHITECTURE

Original: From Section 3.2, we observe that B-BERT recognizes the language structure effectively, We envisage that BERT potentially gains the ability to recognize language structure because of its architecture.
Revised: From Section 3.2, we observe that B-BERT recognizes the language structure effectively, We envisage that BERT potentially gains the ability to recognize language structure because of its architecture.

Original: In this section, we study the contribution of different components of B-BERT architecture namely (i) depth, (ii) multi-head attention and, (iii) the total number of parameters.
Revised: In this section, we study the contribution of different components of B-BERT architecture namely (i) depth, (ii) multi-head attention and, (iii) the total number of parameters.

Original: The motivation is to understand which components are crucial for its cross-lingual ability.
Revised: The motivation is to understand which components are crucial for its cross-lingual ability.

Original: We perform all our cross-lingual experiments on the XNLI dataset with Fake-English as the source and Russian as the target language; we measure cross-lingual ability by the difference between the performance of Fake-English and Russian (lesser the difference better the cross-lingual ability)
Revised: We perform all our cross-lingual experiments on the XNLI dataset with Fake-English as the source and Russian as the target language; we measure cross-lingual ability by the difference between the performance of Fake-English and Russian (lesser the difference better the cross-lingual ability)

Original: . 3.3. D EPTH
Revised: . 3.3. D EPTH

Original: We presume the ability of B-BERT to extract good semantic and structural features is a crucial reason for its cross-lingual effectiveness, and the deepness of B-BERT helps it extract good language features.
Revised: We presume the ability of B-BERT to extract good semantic and structural features is a crucial reason for its cross-lingual effectiveness, and the deepness of B-BERT helps it extract good language features.

Original: In this section, we study the effect of depth on both the monolingual and cross-lingual performance of B-BERT .
Revised: In this section, we study the effect of depth on both the monolingual and cross-lingual performance of B-BERT .

Original: We ﬁx the number of attention heads and change the size of hidden units and intermediate units such that the total number of parameters are almost the same (size of intermediate units is always 4 × size of hidden units)
Revised: We ﬁx the number of attention heads and change the size of hidden units and intermediate units such that the total number of parameters are almost the same (size of intermediate units is always 4 × size of hidden units)

Original: [Table]
Revised: [Table]

Original: From Table
Revised: From Table

Original: [Table]
Revised: [Table]

Original: we can see that a deeper model not only perform better on English but are also better cross-lingual( ∆ ).
Revised: we can see that a deeper model not only perform better on English but are also better cross-lingual( ∆ ).

Original: We can also see a strong correlation between performance on English and crosslingual ability ( ∆ ), which further supports our assumption that the ability to extract good semantic and structural features is a crucial reason for its cross-lingual effectiveness.
Revised: We can also see a strong correlation between performance on English and crosslingual ability ( ∆ ), which further supports our assumption that the ability to extract good semantic and structural features is a crucial reason for its cross-lingual effectiveness.

Original: 3.3. M ULTI - HEAD ATTENTION
Revised: 3.3. M ULTI - HEAD ATTENTION

Original: In this section, we study the effect of multi-head attention on the cross-lingual ability of B-BERT .
Revised: In this section, we study the effect of multi-head attention on the cross-lingual ability of B-BERT .

Original: We ﬁx the depth and the total number of parameters – which is a function of depth and size of hidden and intermediate and study the performance for the different number of attention heads.
Revised: We ﬁx the depth and the total number of parameters – which is a function of depth and size of hidden and intermediate and study the performance for the different number of attention heads.

Original: From Table 3, we can see that the number of attention heads doesn’t have a signiﬁcant effect on cross-lingual ability( ∆ ) – B-BERT is satisfactorily cross-lingual even with a single attention head, which is in agreement with the recent study on monolingual BERT (Voita et al., 2019; Clark et al., 2019).
Revised: From Table 3, we can see that the number of attention heads doesn’t have a signiﬁcant effect on cross-lingual ability( ∆ ) – B-BERT is satisfactorily cross-lingual even with a single attention head, which is in agreement with the recent study on monolingual BERT (Voita et al., 2019; Clark et al., 2019).

Original: [Table]
Revised: [Table]

Original: 3.3. T OTAL N UMBER OF P ARAMETERS
Revised: 3.3. T OTAL N UMBER OF P ARAMETERS

Original: Similar to the depth, we also anticipate that a large number of parameters could potentially help
Revised: Similar to the depth, we also anticipate that a large number of parameters could potentially help

Original: B-BERT extract good semantic and structural features.
Revised: B-BERT extract good semantic and structural features.

Original: We study the effect of the total number of parameters on cross-lingual performance by ﬁxing the number of attention heads and depth; we change the number of parameters by changing the size of hidden and intermediate units (size of intermediate units is always 4 × size of hidden units).
Revised: We study the effect of the total number of parameters on cross-lingual performance by ﬁxing the number of attention heads and depth; we change the number of parameters by changing the size of hidden and intermediate units (size of intermediate units is always 4 × size of hidden units).

Original: From Table
Revised: From Table

Original: [Table]
Revised: [Table]

Original: we can see that the total number of parameters is not as signiﬁcant as depth; however, below a threshold, the number of parameters seems signiﬁcant, which suggests that B-BERT requires a certain minimum number parameters to extract good semantic and structural feature.
Revised: we can see that the total number of parameters is not as signiﬁcant as depth; however, below a threshold, the number of parameters seems signiﬁcant, which suggests that B-BERT requires a certain minimum number parameters to extract good semantic and structural feature.

Original: [Table]
Revised: [Table]

Original: 3. I NPUT AND L EARNING O BJECTIVE
Revised: 3. I NPUT AND L EARNING O BJECTIVE

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we study the effect of input representation and learning objectives on the crosslingual ability of B-BERT .
Revised: we study the effect of input representation and learning objectives on the crosslingual ability of B-BERT .

Original: BERT is a Transformer model trained with MLM and NSP objectives.
Revised: BERT is a Transformer model trained with MLM and NSP objectives.

Original: XLM (Lample & Conneau, 2019) shows that the Transformer model trained with Causal Language
Revised: XLM (Lample & Conneau, 2019) shows that the Transformer model trained with Causal Language

Original: Modeling (CLM) objective is also cross-lingual; however, it also observes that pre-training with
Revised: Modeling (CLM) objective is also cross-lingual; however, it also observes that pre-training with

Original: MLM objective consistently outperforms the one with CLM.
Revised: MLM objective consistently outperforms the one with CLM.

Original: In this work, we don’t study further the effect of MLM objective.
Revised: In this work, we don’t study further the effect of MLM objective.

Original: Recent works (Lample & Conneau, 2019; Joshi et al., 2019; Liu et al., 2019) show that the NSP objective hurts the performance of several monolingual tasks; in this work, we verify if the NSP objective helps or hurts cross-lingual performance.
Revised: Recent works (Lample & Conneau, 2019; Joshi et al., 2019; Liu et al., 2019) show that the NSP objective hurts the performance of several monolingual tasks; in this work, we verify if the NSP objective helps or hurts cross-lingual performance.

Original: M-BERT (Devlin et al., 2018) states that they intentionally do not use any marker to identify language so that cross-lingual transfer works, however, our experiments suggest that adding a language identity marker to the input doesn’t hurt the cross-lingual performance of BERT .
Revised: M-BERT (Devlin et al., 2018) states that they intentionally do not use any marker to identify language so that cross-lingual transfer works, however, our experiments suggest that adding a language identity marker to the input doesn’t hurt the cross-lingual performance of BERT .

Original: We are also interested in studying the effect of characters and words vocabulary instead of word-pieces.
Revised: We are also interested in studying the effect of characters and words vocabulary instead of word-pieces.

Original: Characters provide handling unseen words better than words, words carry more semantic and syntactic information inside it
Revised: Characters provide handling unseen words better than words, words carry more semantic and syntactic information inside it

Original: [Table]
Revised: [Table]

Original: and word-pieces is more of a middle ground of these two.
Revised: and word-pieces is more of a middle ground of these two.

Original: 3.4. N EXT SENTENCE P REDICTION (NSP)
Revised: 3.4. N EXT SENTENCE P REDICTION (NSP)

Original: The input to the BERT is a pair of sentences separated by a special token such that half the time the second sentence is the next and rest half the time, it is a random sentence.
Revised: The input to the BERT is a pair of sentences separated by a special token such that half the time the second sentence is the next and rest half the time, it is a random sentence.

Original: The NSP objective of
Revised: The NSP objective of

Original: BERT ( B-BERT ) is to predict whether the second sentence comes after the ﬁrst one in the original text.
Revised: BERT ( B-BERT ) is to predict whether the second sentence comes after the ﬁrst one in the original text.

Original: We study the effect of NSP objective by comparing the performance of B-BERT pre-trained with and without this objective.
Revised: We study the effect of NSP objective by comparing the performance of B-BERT pre-trained with and without this objective.

Original: From Table
Revised: From Table

Original: [Table]
Revised: [Table]

Original: we can see that the NSP objective hurts the crosslingual performance even more than monolingual performance
Revised: we can see that the NSP objective hurts the crosslingual performance even more than monolingual performance

Original: [Table]
Revised: [Table]

Original: 3.4. L ANGUAGE I DENTITY M ARKER
Revised: 3.4. L ANGUAGE I DENTITY M ARKER

Original: In this work
Revised: In this work

Original: [Table]
Revised: [Table]

Original: we argue that B-BERT is cross-lingual because of its ability to recognize language structure and semantics, and hence we presume adding a language identity marker doesn’t affect its cross-lingual ability.
Revised: we argue that B-BERT is cross-lingual because of its ability to recognize language structure and semantics, and hence we presume adding a language identity marker doesn’t affect its cross-lingual ability.

Original: Even if we don’t add a language identity marker, BERT learns language identity (Wu & Dredze, 2019).
Revised: Even if we don’t add a language identity marker, BERT learns language identity (Wu & Dredze, 2019).

Original: To incorporate language identity in the input, we add a different end of string tokens([SEP]) for different languages (i
Revised: To incorporate language identity in the input, we add a different end of string tokens([SEP]) for different languages (i

Original: [Table]
Revised: [Table]

Original: e., our input format is [CLS] SENT1 [SEP-M]
Revised: e., our input format is [CLS] SENT1 [SEP-M]

Original: SENT2 [SEP-N], where M and N are languages corresponding to SENT1 and SENT2 respectively)
Revised: SENT2 [SEP-N], where M and N are languages corresponding to SENT1 and SENT2 respectively)

Original: [Table]
Revised: [Table]

Original: From Table
Revised: From Table

Original: [Table]
Revised: [Table]

Original: we can observe that adding language identity marker doesn’t affect cross-lingual performance.
Revised: we can observe that adding language identity marker doesn’t affect cross-lingual performance.

Original: [Table]
Revised: [Table]

Original: 3.4. C HARACTER VS . W ORD -P IECE VS . WORD
Revised: 3.4. C HARACTER VS . W ORD -P IECE VS . WORD

Original: In this section we compare the performance of B-BERT with character, word-piece, and word tokenized input.
Revised: In this section we compare the performance of B-BERT with character, word-piece, and word tokenized input.

Original: For character B-BERT , we use all the characters as vocabulary and for word
Revised: For character B-BERT , we use all the characters as vocabulary and for word

Original: B-BERT , we use the most frequent 100000 words as vocabulary.
Revised: B-BERT , we use the most frequent 100000 words as vocabulary.

Original: From Table 7, we can see that both monolingual and cross-lingual performance of B-BERT with word-piece tokenized input is better than the character as well as word tokenized input.
Revised: From Table 7, we can see that both monolingual and cross-lingual performance of B-BERT with word-piece tokenized input is better than the character as well as word tokenized input.

Original: We believe that this is because wordpieces carry much more information than characters, and word-pieces address unseen words better than words.
Revised: We believe that this is because wordpieces carry much more information than characters, and word-pieces address unseen words better than words.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION AND F UTURE WORK
Revised:  D ISCUSSION AND F UTURE WORK

Original: This paper provides a systematic empirical study addressing the cross-lingual ability of B-BERT .
Revised: This paper provides a systematic empirical study addressing the cross-lingual ability of B-BERT .

Original: The analysis presented here covers three dimensions: (1) Linguistics properties and similarities of the source and target languages, (2) Neural Architecture, and (3) Input representation and Learning
Revised: The analysis presented here covers three dimensions: (1) Linguistics properties and similarities of the source and target languages, (2) Neural Architecture, and (3) Input representation and Learning

Original: Objective.
Revised: Objective.

Original: In order to gauge the language similarity aspect needed to make B-BERT successful, we created a new language – Fake-English – and this allows us to study the effect of word-piece overlap while maintaining all other properties of the source language.
Revised: In order to gauge the language similarity aspect needed to make B-BERT successful, we created a new language – Fake-English – and this allows us to study the effect of word-piece overlap while maintaining all other properties of the source language.

Original: Our experiments reveal some interesting and surprising results like the fact that word-piece overlap on the one hand, and multi-head attention on the other, are both not signiﬁcant, whereas structural similarity and the depth of B-BERT are crucial for its cross-lingual ability.
Revised: Our experiments reveal some interesting and surprising results like the fact that word-piece overlap on the one hand, and multi-head attention on the other, are both not signiﬁcant, whereas structural similarity and the depth of B-BERT are crucial for its cross-lingual ability.

Original: While, in order to better control interference among languages, we studied the cross-lingual ability of B-BERT instead of those of M-BERT , it would be interesting now to extend this study, allowing for more interactions among languages.
Revised: While, in order to better control interference among languages, we studied the cross-lingual ability of B-BERT instead of those of M-BERT , it would be interesting now to extend this study, allowing for more interactions among languages.

Original: We leave it to future work to study these interactions.
Revised: We leave it to future work to study these interactions.

Original: In particular, one important question is to understand the extent to which adding to M-BERT languages that are related to the target language, helps the model’s cross-lingual ability.
Revised: In particular, one important question is to understand the extent to which adding to M-BERT languages that are related to the target language, helps the model’s cross-lingual ability.

Original: We introduced the term Structural Similarity , despite its obscure deﬁnition, and show its signiﬁcance in cross-lingual ability.
Revised: We introduced the term Structural Similarity , despite its obscure deﬁnition, and show its signiﬁcance in cross-lingual ability.

Original: Another interesting future work could be to develop a better deﬁnition and, consequently, a ﬁner set of experiments, to better understand the Structural similarity and study its individual components.
Revised: Another interesting future work could be to develop a better deﬁnition and, consequently, a ﬁner set of experiments, to better understand the Structural similarity and study its individual components.

Original: Finally, we note an interesting observation made in Table 8.
Revised: Finally, we note an interesting observation made in Table 8.

Original: We observe a drastic drop in the entailment performance of B-BERT when the premise and hypothesis are in different languages.
Revised: We observe a drastic drop in the entailment performance of B-BERT when the premise and hypothesis are in different languages.

Original: (This data was created using XNLI when in the original form the languages contain same premise and hypothesis pair).
Revised: (This data was created using XNLI when in the original form the languages contain same premise and hypothesis pair).

Original: One of the possible explanations could be that BERT is learning to make textual entailment decisions by matching words or phrases in the premise to those in the hypothesis.
Revised: One of the possible explanations could be that BERT is learning to make textual entailment decisions by matching words or phrases in the premise to those in the hypothesis.

Original: This question, too, is left as a future direction.
Revised: This question, too, is left as a future direction.

Original: [Table]
Revised: [Table]

Original: A A PPENDIX A. F URTHER DISCUSSIONS ON S TRUCTURAL S IMILARITY
Revised: A A PPENDIX A. F URTHER DISCUSSIONS ON S TRUCTURAL S IMILARITY

Original: In the main text, we deﬁned structural similarity as all the properties of a language that is invariant to the script of the language, like morphology, word-ordering, word-frequency, etc..
Revised: In the main text, we deﬁned structural similarity as all the properties of a language that is invariant to the script of the language, like morphology, word-ordering, word-frequency, etc..

Original: Here, we analyze 2 sub-components of structural similarity – word-ordering similarity and word-frequency (Unigram frequency) similarity to understand the concept of structural similarity better.
Revised: Here, we analyze 2 sub-components of structural similarity – word-ordering similarity and word-frequency (Unigram frequency) similarity to understand the concept of structural similarity better.

Original: A.1. W ORD -O RDERING S IMILARITY
Revised: A.1. W ORD -O RDERING S IMILARITY

Original: Words are ordered differently between languages.
Revised: Words are ordered differently between languages.

Original: For example, English has a Subject-Verb-Object order, while Hindi has a Subject-Object-Verb order.
Revised: For example, English has a Subject-Verb-Object order, while Hindi has a Subject-Object-Verb order.

Original: We analyze whether similarity in how words are ordered affects learning cross-lingual transferability.
Revised: We analyze whether similarity in how words are ordered affects learning cross-lingual transferability.

Original: We study the effect of word-ordering similarity – one component of structural similarity – by destroying the word-ordering structure by shufﬂing some percentage of random words in sentences during pretraining.
Revised: We study the effect of word-ordering similarity – one component of structural similarity – by destroying the word-ordering structure by shufﬂing some percentage of random words in sentences during pretraining.

Original: We shufﬂe both source (fakeEnglish) and target language (permuting any one of them would also be sufﬁcient).
Revised: We shufﬂe both source (fakeEnglish) and target language (permuting any one of them would also be sufﬁcient).

Original: This way, the similarity of word-ordering is hidden from B-BERT.
Revised: This way, the similarity of word-ordering is hidden from B-BERT.

Original: We model how much we permute each sentence, for example, when the sentence is 100% permuted, each sentence can be treated as a Bag of Words.
Revised: We model how much we permute each sentence, for example, when the sentence is 100% permuted, each sentence can be treated as a Bag of Words.

Original: For each word, the words that appear in its context (other words in the sentence) is not changed.
Revised: For each word, the words that appear in its context (other words in the sentence) is not changed.

Original: Note that during ﬁne-tuning on source language (and testing on target language), we do not permute, as the cross-lingual ability is gained only during the pretraining process.
Revised: Note that during ﬁne-tuning on source language (and testing on target language), we do not permute, as the cross-lingual ability is gained only during the pretraining process.

Original: From Table 9, we can see that the performance drops signiﬁcantly when we destroy word-order similarity.
Revised: From Table 9, we can see that the performance drops signiﬁcantly when we destroy word-order similarity.

Original: However, the cross-lingual performance is still quite good, which indicates that there are other components of structural similarity, which could contribute to the cross-lingual ability of
Revised: However, the cross-lingual performance is still quite good, which indicates that there are other components of structural similarity, which could contribute to the cross-lingual ability of

Original: B-BERT .
Revised: B-BERT .

Original: [Table]
Revised: [Table]

Original: A.1. W ORD - FREQUENCY S IMILARITY
Revised: A.1. W ORD - FREQUENCY S IMILARITY

Original: Here
Revised: Here

Original: [Table]
Revised: [Table]

Original: we study whether only word frequency allows for good cross-lingual representations – no, it is not much useful.
Revised: we study whether only word frequency allows for good cross-lingual representations – no, it is not much useful.

Original: We collect the frequency of words in the target language and generate a new monolingual corpus by sampling words based on the frequency,
Revised: We collect the frequency of words in the target language and generate a new monolingual corpus by sampling words based on the frequency,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., each sentence is a set of random words sampled from the same unigram frequency as the original target language.
Revised: e., each sentence is a set of random words sampled from the same unigram frequency as the original target language.

Original: The only information about target language BERT has is its unigram frequency and sub-word level information.
Revised: The only information about target language BERT has is its unigram frequency and sub-word level information.

Original: We train B-BERT using Fake-English and this newly generated target corpus.
Revised: We train B-BERT using Fake-English and this newly generated target corpus.

Original: From Table 10, we can see that the performance is very low, although non-trivial (random performance is 33.33%).
Revised: From Table 10, we can see that the performance is very low, although non-trivial (random performance is 33.33%).

Original: Therefore, unigram frequencies alone don’t contain enough information for cross-lingual learning (bi-gram or tri-gram frequencies might be useful)
Revised: Therefore, unigram frequencies alone don’t contain enough information for cross-lingual learning (bi-gram or tri-gram frequencies might be useful)

Original: [Table]
Revised: [Table]

Original: A. F URTHER DISCUSSIONS ON A RCHITECTURE A.2. M ULTILINGUAL BERT
Revised: A. F URTHER DISCUSSIONS ON A RCHITECTURE A.2. M ULTILINGUAL BERT

Original: All the experiments in the main text are conducted on Bilingual BERT.
Revised: All the experiments in the main text are conducted on Bilingual BERT.

Original: However, the results hold even for the multilingual case; to further illustrate this, we experiment on four language BERT (en, es, hi, ru).
Revised: However, the results hold even for the multilingual case; to further illustrate this, we experiment on four language BERT (en, es, hi, ru).

Original: From table 11, we can see that the performance on XNLI is comparable even with just 15% of parameters, and just 1, 3 attention heads when the depth is good enough, which is in agreement with our observations in the main text.
Revised: From table 11, we can see that the performance on XNLI is comparable even with just 15% of parameters, and just 1, 3 attention heads when the depth is good enough, which is in agreement with our observations in the main text.

Original: Also, Currently, there is a lot of interest in reducing the size of BERT( (Anonymous, 2020)), using various pruning techniques.
Revised: Also, Currently, there is a lot of interest in reducing the size of BERT (Anonymous, 2020), using various pruning techniques.

Original: Our analysis shows that we can get comparable performance by maintaining (or increasing) depth.
Revised: Our analysis shows that we can get comparable performance by maintaining (or increasing) depth.

Original: Indeed, by changing the number of parameters from 132.78M to 20.05M (approx 85% less) by reducing only the hidden layer sizes, the English performance drops only from 79.0 to 75.0 (about 4%), which is comparable to the drop in MNLI performance of (Anonymous, 2020), where the performance drops by -4.7% for 88.4% reduction of parameters (English XNLI is similar to MNLI).
Revised: Indeed, by changing the number of parameters from 132.78M to 20.05M (approx 85% less) by reducing only the hidden layer sizes, the English performance drops only from 79.0 to 75.0 (about 4%), which is comparable to the drop in MNLI performance of (Anonymous, 2020), where the performance drops by -4.7% for 88.4% reduction of parameters (English XNLI is similar to MNLI).

Original: We believe that these pruning techniques could be combined with the insights from the paper to get much better results.
Revised: We believe that these pruning techniques could be combined with the insights from the paper to get much better results.

Original: [Table]
Revised: [Table]

Original: A.2. N UMBER OF PARAMETERS
Revised: A.2. N UMBER OF PARAMETERS

Original: Here we show more results on the effect of the number of parameters to get more insights on the threshold on the number of parameters.
Revised: Here we show more results on the effect of the number of parameters to get more insights on the threshold on the number of parameters.

Original: The experimental setting was similar to that of table 4.
Revised: The experimental setting was similar to that of table 4.

Original: From table
Revised: From table

Original: [Table]
Revised: [Table]

Original: we can notice a drastic change in the performance of Russian when the number of parameters is decreased from 11.83M to 7.23M, so we can consider this range to be the required minimum number of parameters, at least for the 12 layer and 12 attention heads situation
Revised: we can notice a drastic change in the performance of Russian when the number of parameters is decreased from 11.83M to 7.23M, so we can consider this range to be the required minimum number of parameters, at least for the 12 layer and 12 attention heads situation

Original: [Table]
Revised: [Table]