Original: M ODEL S ELECTION FOR C ROSS -L INGUAL T RANSFER USING A L EARNED S CORING F UNCTION
Revised: M ODEL S ELECTION FOR C ROSS -L INGUAL T RANSFER USING A L EARNED S CORING F UNCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers that are pre-trained on multilingual text corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer learning results.
Revised: Transformers that are pre-trained on multilingual text corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer learning results.

Original: In the zero-shot cross-lingual transfer setting, only English training data is assumed, and the ﬁne-tuned model is evaluated on another target language.
Revised: In the zero-shot cross-lingual transfer setting, only English training data is assumed, and the ﬁne-tuned model is evaluated on another target language.

Original: No target-language validation data is assumed in this setting, however substantial variance has been observed in target language performance between different ﬁne-tuning runs.
Revised: No target-language validation data is assumed in this setting, however substantial variance has been observed in target language performance between different ﬁne-tuning runs.

Original: Prior work has relied on English validation/development data to select among models that are ﬁne-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices.
Revised: Prior work has relied on English validation/development data to select among models that are ﬁne-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices.

Revised: In this paper, we show that it is possible to select consistently better models when small amounts of annotated data are available in an auxiliary pivot language.

Original: To address this challenge, we propose a meta-learning approach to model selection that uses the ﬁne-tuned model’s own internal representations to predict its cross-lingual capabilities.
Revised: We propose a machine learning approach to model selection that uses the ﬁne-tuned model’s own internal representations to predict its cross-lingual capabilities.

Original: In extensive experiments we ﬁnd that our approach consistently selects better models than English validation data across ﬁve languages and ﬁve well-studied NLP tasks, achieving results that are comparable to small amounts of target language development data.
Revised: In extensive experiments we ﬁnd that our approach consistently selects better models than English validation data across ﬁve languages and ﬁve well-studied NLP tasks, achieving results that are comparable to small amounts of target language development data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pre-trained Transformers (Vaswani et al., 2017; Devlin et al., 2019) have lead to state-of-the-art results on a wide range of NLP tasks, for example, named entity recognition, relation extraction and question answering, often approaching human inter-rater agreement (Joshi et al., 2020).
Revised: Pre-trained Transformers (Vaswani et al., 2017; Devlin et al., 2019) have lead to state-of-the-art results on a wide range of NLP tasks, for example, named entity recognition, relation extraction and question answering, often approaching human inter-rater agreement (Joshi et al., 2020a).

Original: These models have also been demonstrated to learn effective cross-lingual representations, even without access to parallel text or bilingual lexicons (Wu & Dredze, 2019; Pires et al., 2019).
Revised: These models have also been demonstrated to learn effective cross-lingual representations, even without access to parallel text or bilingual lexicons (Wu & Dredze, 2019; Pires et al., 2019).

Original: Multilingual pre-trained Transformers, such as mBERT and XLM-RoBERTa (Conneau et al., 2019), support surprisingly effective zero-shot cross-lingual transfer, where training and development data are only assumed in a high resource source language (e.g. English), and performance is evaluated on another target language.
Revised: Multilingual pre-trained Transformers, such as mBERT and XLM-RoBERTa (Conneau et al., 2019), support surprisingly effective zero-shot cross-lingual transfer, where training and development data are only assumed in a high resource source language (e.g. English), and performance is evaluated on another target language.

Original: Because no target language annotations are assumed in this setting, source language data is typically used to select among models that are ﬁne-tuned with different hyperparameters and random seeds.
Revised: Because no target language annotations are assumed in this setting, source language data is typically used to select among models that are ﬁne-tuned with different hyperparameters and random seeds.

Original: However, recent work has shown that English dev accuracy does not always correlate well with target language performance (Keung et al., 2020).
Revised: However, recent work has shown that English dev accuracy does not always correlate well with target language performance (Keung et al., 2020).

Original: In this paper, we propose an alternative strategy for model selection in a zero-shot setting.
Revised: In this paper, we propose an alternative strategy for model selection in a zero-shot setting.

Original: Our approach, dubbed Learned Model Selection (LMS), learns a function that scores the compatibility between a ﬁne-tuned multilingual transformer, and a target language.
Revised: Our approach, dubbed Learned Model Selection (LMS), learns a function that scores the compatibility between a ﬁne-tuned multilingual transformer, and a target language.

Original: The compatibility score is calculated based on features of the multilingual model’s learned representations and the target language.
Revised: The compatibility score is calculated based on features of the multilingual model’s learned representations and the target language.

Original: A model’s features arecomputed based on its own internal representations; this is done by aggregating representations over an unlabeled target language text corpus.
Revised: A model’s features are based on its own internal representations; this is done by aggregating representations over an unlabeled target language text corpus.

Original: These model-speciﬁc features capture information about how the cross-lingual representations transfer to the target language after ﬁne-tuning on source language data.
Revised: These model-speciﬁc features capture information about how the cross-lingual representations transfer to the target language after ﬁne-tuning on source language data.

Original: In addition to model-speciﬁc representations, we also make use of learned language embeddings from the lang2vec package (Malaviya et al., 2017), 2 which have been shown to encode typological information, for example, whether a language has prepositions or postpositions. 
Revised: In addition to modelspeciﬁc representations, we also make use of learned language embeddings from the lang2vec package (Malaviya et al., 2017), 2 which have been shown to encode typological information, for example, whether a language has prepositions or postpositions. To measure compatibility between [Figure] [Figure]

Original: To measure compatibility between a multilingual model’s ﬁne-tuned representations and a target language, the modeland language- speciﬁc representations are combined in a bilinear layer.
Revised: [Figure] [Figure] [Figure]

Original: Parameters of the scoring
Revised: En

Original: [Figure]
Revised: [Figure]

Original: Unlabeled
Revised: mBERT

Original: [Figure]
Revised: [Figure]

Original:  Language Data 0.
Revised: Development [Figure] Unlabeled Target Language Data 0.

Original: Scores mBERT [CLS
Revised: Scores mBERT [CLS]

Original: [Figure]

Original: S c
Revised: S c

Original: o r
Revised: o r

Original: i ng
Revised: i ng

Original: F un c t i on
Revised: F un c t i on

Original: [Figure]
Revised: [Figure]

Revised: a multilingual model’s ﬁne-tuned representations and a target language, the model- and languagespeciﬁc representations are combined in a bilinear layer.

Original:  function are optimized to minimize a pairwise ranking loss on a set of held-out models, where the gold ranking is calculated using standard performance metrics, such as accuracy or F 1 , on a set of pivot languages (not including the target language).
Revised: Parameters of the scoring function are optimized to minimize a pairwise ranking loss on a set of held-out models, where the gold ranking is calculated using standard performance metrics, such as accuracy or F 1 , on a set of pivot languages (not including the target language).

Original:  LMS does not rely on any annotated data in the target language for meta-learning or hyperparameter tuning, yet it is effective in learning to predict whether a multilingualmodel’s representations are a good match for a speciﬁc target language.
Revised: Our method assumes training data in English, and small amounts of annotated data in one or more pivot languages (not the target language). This corresponds to a scenario where a new multilingual NLP task needs to be quickly applied to a new language. LMS does not rely on any annotated data in the target language, yet it is effective in learning to predict whether ﬁne-tuned multilingual representations are a good match.

Original: In experiments on ﬁve well-studied NLP tasks (part of speech tagging, named entity recognition, question answering, relation extraction and event argument role labeling), we ﬁnd LMS consistently selects models with better target-language performance than those chosen using English dev data.
Revised: In experiments on ﬁve well-studied NLP tasks (part of speech tagging, named entity recognition, question answering, relation extraction and event argument role labeling), we ﬁnd LMS consistently selects models with better target-language performance than those chosen using English dev data.

Original: Appendix A.4 demonstrates that our framework supports multi-task learning, which can be helpful in settings where some target-language annotations are available, but not for the desired task.
Revised: Appendix A.5 demonstrates that our framework supports multi-task learning, which can be helpful in settings where some target-language annotations are available, but not for the desired task.

Original: Finally, we show that LMS generalizes to both mBERT and XLM-RoBERTa in Appendix A.3.
Revised: Finally, we show that LMS generalizes to both mBERT and XLM-RoBERTa in Appendix A.4.

Original:  B ACKGROUND : Z ERO -S HOT C ROSS L INGUAL T RANSFER
Revised:  B ACKGROUND : Z ERO -S HOT C ROSS L INGUAL T RANSFER

Original: The zero-shot setting considered in this paper works as follows.
Revised: The zero-shot setting considered in this paper works as follows.

Original: A transformerlanguage model is ﬁrst pre-trained using a standard masked language model objective.
Revised: A transformer model is ﬁrst pretrained using a standard masked language model objective.

Original: The only difference from the monolingual approach to contextual word representations (Peters et al., 2018; Devlin et al., 2019) is the pre-training corpus, which contains text written in multiple languages.
Revised: The only difference from the monolingual approach to contextual word representations (Peters et al., 2018; Devlin et al., 2019) is the pretraining corpus, which contains text written in multiple languages.

Original: For example, mBERT is trained on text written in 104 languages from Wikipedia.
Revised: For example, mBERT is trained on text written in 104 languages from Wikipedia.

Original: After pre-training on a multilingual corpus, the resulting transformer encodes language-independent representations that support surprisingly effective cross-lingual transfer, simply by ﬁne-tuning the pre-trained parameters using English training data.
Revised: After pre-training on a multilingual corpus, the resulting transformer encodes language-independent representations that support surprisingly effective cross-lingual transfer, simply by ﬁne-tuning the pre-trained parameters using English training data.

Original: For example, after ﬁne-tuning mBERT using the English portion of the CoNLL Named Entity
Revised: For example, after ﬁne-tuning mBERT using the English portion of the CoNLL Named Entity

Original: Recognition dataset, the resulting model can be used to perform inference directly on Spanish text, achieving an F 1 score around 75, and outperforming prior work using cross-lingual word embeddings (Xie et al., 2018; Mikolov et al., 2013).
Revised: Recognition dataset, the resulting model can be used to perform inference directly on Spanish text, achieving an F 1 score around 75, and outperforming prior work using cross-lingual word embeddings (Xie et al., 2018; Mikolov et al., 2013).

Original: A challenge with this approach, however, is that there is a relatively high variance in this performance across training runs.
Revised: A challenge with this approach, however, is that there is a relatively high variance in this performance across training runs.

Original: Although the mean F 1 score on Spanish is 75, the performance of 60 models ﬁne-tuned with different learning rates and random seeds ranges from around 70 F 1 to 78 (Figure 3).
Revised: Although the mean F 1 score on Spanish is 75, the performance of 60 models ﬁne-tuned with different learning rates and random seeds ranges from around 70 F 1 to 78 (Figure 3).

Original: In zero-shot learning, no validation/development data is assumed in the target language, motivating the need for a meta-learning approach to model selection.
Revised: In zero-shot learning, no validation/development data is assumed in the target language, motivating the need for a machine learning approach to model selection.

Original:  R ANKING M ODEL C OMPATIBILITY WITH A T ARGET L ANGUAGE
Revised:  R ANKING M ODEL C OMPATIBILITY WITH A T ARGET L ANGUAGE

Original: Given a set of multilingual BERT-based models, M = m 1 , m 2 , ..., m n that are ﬁne-tuned on an
Revised: Given a set of multilingual BERT-based models, M = m 1 , m 2 , ..., m n that are ﬁne-tuned on an

Original: English training set using different hyperparameters and random seeds, our goal is to select the model that performs the best on a target language, l target .
Revised: English training set using different hyperparameters and random seeds, our goal is to select the model that performs the best on a target language, l target .

Original: Our approach (LMS) learns to rank a set of models based on two sources of information: (1) the models’ own internal representations, and (2) lang2vec representations of the target language (Malaviya et al., 2017).
Revised: Our approach (LMS) learns to rank a set of models based on two sources of information: (1) the models’ own internal representations, and (2) lang2vec representations of the target language (Malaviya et al., 2017).

Original: By integrating a model’s target language representations together with lang2vec embeddings, our approach predicts which model will perform best.
Revised: By integrating a model’s target language representations together with lang2vec embeddings, LMS predicts which model will perform best.

Original: We adopt a pairwise approach to learning to rank (Burges et al., 2005; K¨oppel et al., 2019).
Revised: We adopt a pairwise approach to learning to rank (Burges et al., 2005; K¨oppel et al., 2019).

Original: The learned ranking is computed using a scoring function, s ( m, l ) =
Revised: The learned ranking is computed using a scoring function, s ( m, l ) =

Original: f ( g mBERT ( m ) , g lang2vec ( l )) , where g mBERT ( m ) is a feature vector for model m , which is computed from the model’s own hidden state representations, and g lang2vec ( l ) is the lang2vec representation of language l .
Revised: f ( g mBERT ( m ) , g lang2vec ( l )) , where g mBERT ( m ) is a feature vector for model m , which is computed from the model’s own hidden state representations, and g lang2vec ( l ) is the lang2vec representation of language l .

Original: The model and language features are each passed through a feed-forward neural network and then combined using a bilinear layer to calculate a ﬁnal score as follows:
Revised: The model and language features are each passed through a feed-forward neural network and then combined using a bilinear layer to calculate a ﬁnal score as follows:

Original: [Equation]
Revised: [Equation]

Original: Using the above score, we can represent the probability that model
Revised: Using the above score, we can represent the probability that model

Original: m i performs better than m j on language l :
Revised: m i performs better than m j on language l :

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) is the sigmoid function.
Revised: where σ ( · ) is the sigmoid function.

Original: To tune the parameters of the scoring function, which include the feed-forward and bilinear layers, we minimize cross-entropy loss:
Revised: To tune the parameters of the scoring function, which include the feed-forward and bilinear layers, we minimize cross-entropy loss:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here 1[ m j (cid:46) l m i ] is an indicator function that has the value 1 if m j outperforms m i , as evaluated using labeled development data in language l .
Revised: Here 1 [ m j (cid:46) l m i ] is an indicator function that has the value 1 if m j outperforms m i , as evaluated using labeled development data in language l .

Original: The ﬁrst sum in Equation 1 ranges over all languages where development data is available, excluding the target language.
Revised: The ﬁrst sum in Equation 1 ranges over all languages where development data is available, excluding the target language.

Original: After tuning parameters to minimize cross-entropy loss on these languages, the models are ranked based on their scores for the target language, and the highest scoring model, ˆ m = arg max
Revised: After tuning parameters to minimize cross-entropy loss on these languages, the models are ranked based on their scores for the target language, and the highest scoring model, ˆ m = arg max

Original: m s ( m, l target ) , is selected.
Revised: m s ( m, l target ) , is selected.

Original:  T ASKS AND D ATASETS
Revised:  T ASKS AND D ATASETS

Original: Section 3 presented an approach to selecting a model for zero-shot cross-lingual transfer.
Revised: Section 3 presented an approach to selecting a model for zero-shot cross-lingual transfer.

Original: To evaluate its effectiveness, a set of ﬁne-tuned models is split into meta-train/dev/test sets.
Revised: To evaluate its effectiveness, a set of ﬁne-tuned models is split into meta-train/dev/test sets.

Original: We then use models in the meta-train set to tune parameters of the scoring function, s ( · ) , develop input features with models in the meta-dev set (see § 7 for details), and report our main results using models in the meta-test set in Table 2.
Revised: We then use models in the meta-train set to tune parameters of the scoring function, s ( · ) , develop input features with models in the meta-dev set (see § 7 for details), and report our main results using models in the meta-test set in Table 2.

Original: We experiment with ﬁve well-studied NLP tasks in the zero-shot transfer setting: named entity recognition (NER), part-of-speech (POS) tagging, question answering (QA), relation extraction (RE), and event-argument role labeling (ARL).
Revised: We experiment with ﬁve well-studied NLP tasks in the zero-shot transfer setting: named entity recognition (NER), part-of-speech (POS) tagging, question answering (QA), relation extraction (RE), and event-argument role labeling (ARL).

Original: Labeled training data for each task is assumed in
Revised: Labeled training data for each task is assumed in

Original: English and trained models are evaluated on the target language.
Revised: English and trained models are evaluated on the target language.

Original: The task/language combinations covered by our experiments are summarized in Table 1.
Revised: The task/language combinations covered by our experiments are summarized in Table 1.

Original: The following subsections describe the tasks and datasets used in our experiments; more details are presented in Appendix A.1. 
Revised: The following subsections describe the tasks and datasets used in our experiments; more details are presented in Appendix A.1. [Table] bg da de es fa hi hu it

Revised: nl pt ro

Revised: sk sl sv vi zh

Original: 4. N AMED E NTITY R ECOGNITION
Revised: [Table] (cid:88) (cid:88) (cid:88) (cid:88) [Table] (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)

Revised: [Table]

Revised: (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)

Revised: [Table]

Revised: (cid:88) (cid:88)

Original:  We use data from the CoNLL 2002 and 2003 (Sang, 2002; Sang & Meulder, 2003) shared tasks (four target languages: en, de, es, and nl), in addition to a Chinese NER dataset (Levow, 2006), following Wu & Dredze (2019).
Revised: [Table] (cid:88) (cid:88) 4. N AMED E NTITY R ECOGNITION We use data from the CoNLL 2002 and 2003 (Sang, 2002; Sang & Meulder, 2003) shared tasks (four target languages: en, de, es, and nl), in addition to a Chinese NER dataset (Levow, 2006), following Wu & Dredze (2019).

Original: The data is tagged using the BIO tagging scheme with four types of named entities.
Revised: The data is tagged using the BIO tagging scheme with four types of named entities.

Original: We formulate this as a tagging task and add a linear classiﬁcation layer with a softmax function to obtain word-level predictions.
Revised: We formulate this as a tagging task and add a linear classiﬁcation layer with a softmax function to obtain word-level predictions.

Original: Because the labels are at the word level, but mBERT operates on the subword level, representation of the ﬁrst subword is used to make wordlevel predictions, following (Devlin et al., 2019).
Revised: Because the labels are at the word level, but mBERT operates on the subword level, representation of the ﬁrst subword is used to make wordlevel predictions, following (Devlin et al., 2019).

Original: The CoNLL evaluation script is used to compute
Revised: The CoNLL evaluation script is used to compute

Original: F1 scores.
Revised: F1 scores.

Original: ar de es nl zh

Original: NER (cid:88) (cid:88) (cid:88) (cid:88)

Original: POS (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)

Original: QA (cid:88) (cid:88) (cid:88) (cid:88)

Original: RE (cid:88) (cid:88)

Original: ARL (cid:88) (cid:88)

Original: 4. P ART - OF -S PEECH T AGGING
Revised: 4. P ART - OF -S PEECH T AGGING

Original: A subset of Universal Dependency (UD) Treebanks (v1.4) (Nivre et al., 2018) (ﬁve target languages: ar,  de, es,  nl,  zh) is used, following the setup of Kim et al.
Revised: A subset of Universal Dependency (UD) Treebanks (v1.4) (Nivre et al., 2016) (ﬁfteen target languages: ar, bg, da, de, es, fa, hu, it, nl, pt, ro, sk, sl, sv, zh) is used, following the setup of Kim et al.

Original: There are eighteen POS tags in total across all languages.
Revised: There are eighteen POS tags in total across all languages.

Original: We formulate this as a tagging task, following the same approach as described for named entity recognition, and report the accuracy of the predicted POS tags (ACC).
Revised: We formulate this as a tagging task, following the same approach as described for named entity recognition, and report the accuracy of the predicted POS tags (Acc).

Original: 4. Q UESTION A NSWERING
Revised: 4. Q UESTION A NSWERING

Original: The Multilingual Question Answering (MLQA) dataset (Lewis et al., 2020) is used to evaluate model selection for cross-lingual transfer in the context of question answering.
Revised: The Multilingual Question Answering (MLQA) dataset (Lewis et al., 2020) is used to evaluate model selection for cross-lingual transfer in the context of question answering.

Original: We usea subset of the MLQA dataset, consisting of four target languages: ar, de, es,  zh.
Revised: We use the MLQA dataset, consisting of six target languages: ar, de, es, hi, vi, zh.

Original: Following the setup of Lewis et al.
Revised: Following the setup of Lewis et al.

Original: (2020), models are trained using English data from SQuAD v1.1 (Rajpurkar et al., 2016), and  MLQA English development set is used for the En-Dev baseline.
Revised: (2020), models are trained using English data from SQuAD v1.1 (Rajpurkar et al., 2016), and the MLQA English development set is used for the En-Dev baseline.

Original: We use the standard BERT approach for question answering Devlin et al. (2019), where a (question, passage) pair is encoded using special tokens as follows:
Revised: We use the standard BERT approach for question answering (Devlin et al., 2019), where a (question, passage) pair is encoded using special tokens as follows:

Original: [BOS] Question [SEP] Passage [SEP] .
Revised: [BOS] Question [SEP] Passage [SEP] .

Original: The classiﬁcation layer predicts both the start and end index of the answer span from the text passage.
Revised: The classiﬁcation layer predicts both the start and end index of the answer span from the text passage.

Original: The formal evaluation script of MLQA is adopted to compute F1 scores.
Revised: The formal evaluation script of MLQA is adopted to compute F1 scores.

Original: 4. R ELATION E XTRACTION
Revised: 4. R ELATION E XTRACTION

Original: Given a pair of entity mentions, RE aims to identify a relation between them and classify its type.
Revised: Given a pair of entity mentions, RE aims to identify a relation between them and classify its type.

Original: We use the ACE 2005 corpus (Walker et al., 2006), which contains two target languages, Arabic and Chinese, using the same preprocessed dataset as Subburathinam et al.
Revised: We use the ACE 2005 corpus (Walker et al., 2006), which contains two target languages, Arabic and Chinese, using the same preprocessed dataset as Subburathinam et al.

Original: ACE contains eighteen relation types and an additional label indicating no relation (19-way classiﬁcation).
Revised: ACE contains eighteen relation types and an additional label indicating no relation (19-way classiﬁcation).

Original: We adopt the best performing model from Soares et al.
Revised: We adopt the best performing model from Soares et al.

Original: (2019), [E NTITY M ARKERS - E NTITY S TART ], which adds special tokens [ E1 start ] , [ E1 end ] and [ E2 start ] , [ E2 end ] surrounding entity mention pairs in a sentence.
Revised: (2019), [E NTITY M ARKERS - E NTITY S TART ], which adds special tokens [ E1 start ] , [ E1 end ] and [ E2 start ] , [ E2 end ] surrounding entity mention pairs in a sentence.

Original: The modiﬁed sentence is then fed into mBERT and representations of the starting markers [ E1 start ] , [ E2 start ] are concatenated as inputs to a linear classiﬁcation layer.
Revised: The modiﬁed sentence is then fed into mBERT and representations of the starting markers [ E1 start ] , [ E2 start ] are concatenated as inputs to a linear classiﬁcation layer.

Original: We follow evaluation protocols established in prior work, using gold entities as inputs to control for differences in performance due to named entity recognition errors (Subburathinam et al., 2019).
Revised: We follow evaluation protocols established in prior work, using gold entities as inputs to control for differences in performance due to named entity recognition errors (Subburathinam et al., 2019).

Original: A predicted relation mention is considered correct if its type and the heads of both entities match the gold data.
Revised: A predicted relation mention is considered correct if its type and the heads of both entities match the gold data.

Original: 4.4. E VENT -A RGUMENT R OLE L ABELING
Revised: 4.4. E VENT -A RGUMENT R OLE L ABELING

Original: Event-Argument Role Labeling (ARL) aims to identify event triggers and their arguments in texts.
Revised: Event-Argument Role Labeling (ARL) aims to identify event triggers and their arguments in texts.

Original: For example, in the sentence Facebook aquired Instagram , an event trigger is aquired , whose arguments are Facebook and Instagram , with roles B UYER and S ELLER .
Revised: For example, in the sentence Facebook aquired Instagram , an event trigger is aquired , whose arguments are Facebook and Instagram , with roles B UYER and S ELLER .

Original: Again we draw on the
Revised: Again we draw on the

Original: ACE 2005 corpus (Walker et al., 2006) (two target languages: ar, zh), using the same preprocessed dataset as Subburathinam et al.
Revised: ACE 2005 corpus (Walker et al., 2006) (two target languages: ar, zh), using the same preprocessed dataset as Subburathinam et al.

Original: There are thirty-ﬁve role labels in ACE.
Revised: There are thirty-ﬁve role labels in ACE.

Original: Following a similar approach as was described for relation extraction, we add special markers [ T start ] , [ T end ] and [ E start ] , [ E end ] around the trigger and candidate entity mentions.
Revised: Following a similar approach as was described for relation extraction, we add special markers [ T start ] , [ T end ] and [ E start ] , [ E end ] around the trigger and candidate entity mentions.

Original: The modiﬁed sentence is fed into mBERT, and the marker representations are concatenated as input to a classiﬁcation layer that predicts the entity’s role with respect to the trigger word.
Revised: The modiﬁed sentence is fed into mBERT, and the marker representations are concatenated as input to a classiﬁcation layer that predicts the entity’s role with respect to the trigger word.

Original:  E XPERIMENTAL D ESIGN
Revised:  E XPERIMENTAL D ESIGN

Original: For a downstream task with n languages L : { l 1 , ..., l n } , our goal is to select the model that performs best on a target language, l target ∈ L .
Revised: For a downstream task with n languages L : { l 1 , ..., l n } , our goal is to select the model that performs best on a target language, l target ∈ L .

Original: We assume the available resources are English training and development data, in addition to pivot language dev data, where pivot languages are all those excluding the target language,
Revised: We assume the available resources are English training and development data, in addition to a small development set in one or more pivot languages (that do not include the target language).

Original: l pivot ∈ L \ l target . First, a set of mBERT models, M , are ﬁne-tuned on an English training set using different hyperparameters and random seeds and shufﬂed into metatrain/dev/test sets.
Revised: First, a set of mBERT models, M , are ﬁne-tuned on an English training set using different hyperparameters and random seeds and shufﬂed into meta-train/dev/test sets.

Original: We then evaluate each model, m i , on the pivot languages’ dev sets to calculate a gold ranking, (cid:46) l , that is used in the cross-entropy loss (Equation 1).
Revised: We then evaluate each model, m i , on the pivot languages’ dev sets to calculate a gold ranking, (cid:46) l , that is used in the cross-entropy loss (Equation 1).

Original: Model-speciﬁc features are then extracted from the ﬁne-tuned mBERTs, by feeding unlabeled pivot-language text  as input.
Revised: Model-speciﬁc features are then extracted from the ﬁne-tuned mBERTs, by feeding unlabeled pivot language text (dev set examples) as input.

Original: Development and Evaluation mBERT models in the meta-dev set are used to experiment with different model and language features.
Revised: Development and Evaluation mBERT models in the meta-dev set are used to experiment with different model and language features.

Original: Target language dev data is used to experiment with a handful of model and language features (see § 7 for details).
Revised: Target language dev data was used to experiment with a handful of model and language features, as described in § 7.

Original: Evaluation is performed using the meta-test set.
Revised: Evaluation is performed using the metatest set.

Original: For each target language, we rank models using the learned scoring function, select the highest scoring model, and report results in Tables 2.
Revised: For each target language, we rank models using the learned scoring function, select the highest scoring model, and report results in Table 2.

Original: 5. B ASELINES
Revised: 5. B ASELINES

Original: En-Dev is our main baseline following standard practice for model selection in the zero-shot setting (Wu & Dredze, 2019; Pires et al., 2019).
Revised: En-Dev is our main baseline following standard practice for model selection in the zero-shot setting (Wu & Dredze, 2019; Pires et al., 2019).

Revised: Model selection with the pivot language dev set is included as another baseline, where the pivot language is selected based on the highest similarity with the target language, as measured using cosine similarity between lang2vec embeddings among candidates in { ar, de, es, nl, zh } .

Original: In addition, we compare our results with models selected using 100 sentences annotated in the target language to understand how our approach compares to the more costly alternative of annotating a small amount of target language development data.
Revised: Also, we compare our results with models selected using 100 sentences annotated in the target language to understand how our approach compares to the more costly alternative of annotating a small amount of target language development data.

Original: Finally, a Full-Target that picks the best model based on target language development data is adopted.
Revised: Finally, an All-Target oracle that picks the best model based on target language development data is adopted.

Original: These baselines and oracles are summarized below:
Revised: These baselines and oracles are summarized below:

Original: • En-Dev (baseline): choose an mBERT with the English dev set.
Revised: • En-Dev (baseline): choose an mBERT with the English dev set.

Original: •  100-Target (oracle): choose an mBERT with 100 target language dev set instances.• Full-Target
Revised: • Pivot-Dev (baseline): choose an mBERT with the pivot language dev set. • 100-Target (oracle): choose an mBERT with 100 target language dev set instances.

Original:  (oracle): choose an mBERT with the full target language dev set.
Revised: • All-Target (oracle): choose an mBERT with the full target language dev set.

Original: 5. H YPERPARAMETERS AND O THER S ETTINGS
Revised: 5. H YPERPARAMETERS AND O THER S ETTINGS

Original: To train the scoring function, s ( · ) , we use Adam (Kingma & Ba, 2015), and select the batch size among { 16, 32, 64, 128 } , learning rate λ among { 1 × 10 − 4 , 5 × 10 − 5 , 1 × 10 − 5 , 5 × 10 − 6 , 1 × 10 − 6 } , and train for { 3 } epochs.
Revised: To train the scoring function, s ( · ) , we use Adam (Kingma & Ba, 2015), and select the batch size among { 16, 32, 64, 128 } , learning rate λ among { 1 × 10 − 4 , 5 × 10 − 5 , 1 × 10 − 5 , 5 × 10 − 6 , 1 × 10 − 6 } , and train for { 3 } epochs.

Original: The scoring function, s ( · ) , contains a 2-layer FFNN with 1024 hidden units and ReLU activation (Glorot et al., 2011).
Revised: The scoring function, s ( · ) , contains a 2-layer FFNN with 1024 hidden units and ReLU activation (Glorot et al., 2011).

Original: The base cased mBERT has 179M parameters and a vocabulary of around 120k wordpieces.
Revised: The base cased mBERT has 179M parameters and a vocabulary of around 120k wordpieces.

Original: Both the pre-trained transformer layers and taskspeciﬁc layers are ﬁne-tuned using Adam, with β 1 = 0.9, β 2 = 0.999, and an L2 weight decay of 0.01.
Revised: Both the pre-trained transformer layers and task-speciﬁc layers are ﬁne-tuned using Adam, with β 1 = 0.9, β 2 = 0.999, and an L2 weight decay of 0.01.

Original: Model candidates are ﬁne-tuned with varying learning rates and number of epochs with the following settings: learning rate ∈ { 3 × 10 − 5 , 5 × 10 − 5 , 7 × 10 − 5 } ; number of epochs ∈ { 3 , 4 , 5 , 6 } ; batch size ∈ { 32 }  .
Revised: Model candidates are ﬁne-tuned with varying learning rates and number of epochs with the following settings: learning rate ∈ { 3 × 10 − 5 , 5 × 10 − 5 , 7 × 10 − 5 } ; number of epochs ∈ { 3 , 4 , 5 , 6 } ; batch size ∈ { 32 } ; random seeds ∈ { 0 , 1 , ..., 239 } .

Original: 240 mBERT models  are ﬁne-tuned and then split into meta- train/dev/test sets (120/60/60).
Revised: 240 mBERT models with different random seeds are ﬁnetuned with 12 different hyperparameter settings (20 random seeds for each set of hyperparameters), and then split into meta-train/dev/test sets (120/60/60).

Original: All models are trained on a RTX 2080 Ti.
Revised: All models are trained on an RTX 2080 Ti.

Original:  E VALUATION
Revised:  E VALUATION

Original: Below we report model selection results on mBERTs in the meta-test set for each of the ﬁve tasks. 
Revised: Below we report model selection results on mBERTs in the meta-test set for each of the ﬁve tasks. [Table] [Table] 87.05 (es) 87.

Original: NER As illustrated in Table 2, our method selects models with a higher F1 score than En-Dev except in the case of German.
Revised: da 88. 88.87 (nl) 88. fa 72. 71.63 (es) 73.

Revised: hu

Revised: [Table]

Revised: 82.05 (de) 83.

Revised: it 84.

Revised: 84.89 (es) 85.

Revised: pt 82.

Revised: 81.88 (es) 82.

Revised: ro 84.

Revised: 84.19 (es) 84.

Revised: sk 83.

Revised: [Table]

Revised: sl 84.

Revised: [Table]

Revised: [Table]

Revised: sv

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ar 45.

Revised: [Table]

Revised: de 57.

Revised: 55.83 (ar) 55.

Revised: es 64.

Revised: 64.74 (ar) 64.

Revised: zh 57.

Revised: 58.09 (de) 58.

Revised: hi 43.

Revised: 42.05 (es) 42.

Revised: vi 57.

Revised: [Table]

Revised: NER As illustrated in Table

Original: In addition, it outperforms model selection using small amounts of targetlanguage annotations (100-Target) on Dutch  and selects a model that performs as well on Spanish.
Revised: [Table] our method selects models with a higher F1 score than En-Dev except in the case of German (de). Besides, it outperforms model selection using small amounts of target-language annotations (100-Target) on Dutch (nl) and selects a model that performs as well on Spanish (es).

Revised: On average, LMS achieves 0.86 point increase in F1 score relative to Pivot-Dev.

Original: We use (Wu & Dredze,2019; Pires et al., 2019) as references for zero-shot cross-lingual transfer with mBERT.
Revised: We use (Wu & Dredze, 2019) as references for zero-shot cross-lingual transfer with mBERT.

Original: POS Table 2 presents POS accuracies on the test set, using various approaches to model selection for the ﬁve target languages.
Revised: POS Table 2 presents POS accuracies on the test set, using various approaches to model selection for the ﬁﬁteen target languages.

Original: Our method selects a model with higher accuracy compared to both En-Dev and 100-Target across all ﬁve languages.
Revised: LMS outperforms En-Dev and Pivot-Dev except in the case of Swedish (sv) and Dutch (nl).

Original:  QA In Table 2, our method selects a model with higher F1 across four languages. In addition, it outperforms 100-Target on Spanish. [Table]
Revised: Interestingly, model selection for Italian with Spanish dev set does not outperform LMS QA Our method selects a model with higher F1 across all languages compared with En-Dev, although we ﬁnd that Pivot-Dev performs slightly better on Arabic (ar).

Original: [Figure]
Revised: We use (Lewis et al., 2020) as references for zero-shot cross-lingual transfer with mBERT.

Original: ARL and RE In Tables 2, our method selects models with higher F1 score compared to En-Dev.
Revised: ARL and RE In Table 2, our method selects models with higher F1 scores compared to En-Dev.

Original: It also outperforms 100-Target across both languages.
Revised: It also outperforms 100-Target across both languages.

Revised: We hypothesize this is because 100 targetlanguage examples is not sufﬁcient for effective model selection, as the dataset contains a large proportion of negative examples (no relation).

Revised: Also, RE and ARL have large label sizes (18 and 35) so a random sample of 100 instances might not cover every label.

Revised: In contrast, the full dev set contains thousands of examples.

Original: We use GCN ReImp as a reference in the imbalanced dataset (see Appendix A.1 for details).
Revised: We use GCN ReImp as a reference in the imbalanced dataset (see Appendix A.1 for details).

Original: Reference models were selected using the English dev 
Revised: Reference models were selected using the English dev set.

Original: Model Score Distributions Figure 2 visualizes the En-Dev and LMS results on the test set in the context of the score distributions of the 60 models in the meta-test set, using kernel density estimation.
Revised: Model Score Distributions Figure 2 visualizes the En-Dev and LMS results on the test set in the context of the score distributions of the 60 models in the meta-test set, using kernel density estimation.

Original: English development data tends to select models that perform only slightly better than average, whereas LMS does signiﬁcantly better.
Revised: English development data tends to select models that perform only slightly better than average, whereas LMS does signiﬁcantly better.

Original: Similar visualizations for NER, POS, and QA are presented in Appendix A.
Revised: Similar visualizations for NER, POS, and QA are presented in Appendix A.2.

Revised: In addition to the model score, we run a statistical analysis of the results for POS and QA in Appendix A.3.

Revised: LMS is statistically signiﬁcantly higher than En-Dev while Pivot-Dev fails in three target languages.

Original: [Figure]
Revised: [Figure]

Original:  A NALYSIS
Revised:  A NALYSIS

Original: In Section 6, we empirically demonstrated that our learned scoring function, s ( · ) , consistently selects better models than the standard approach (EN-Dev) [Figure] and is comparable to small amounts of labeled target language data.
Revised: In Section 6, we empirically demonstrated that our learned scoring function, s ( · ) , consistently selects better models than the standard approach (En-Dev), and is comparable to small amounts of labeled target language data.

Original: Section 7 presents additional analysis of our approach, exploring the impact of various modeling choices.
Revised: Section 7 presents additional analysis of our approach, exploring the impact of various modeling choices with { ar, de, es, nl, zh } .

Original: In addition, analysis of generalization beyond mBERT, across tasks capability, and size of training are present in AppendicesA.3, A.4,  and A.5.
Revised: In addition, analysis of generalization beyond mBERT, across tasks capability, and size of training are present in Appendices A.4, A.5, and A.6.

Original: 7. M ODEL AND L ANGUAGE F EATURES
Revised: 7. M ODEL AND L ANGUAGE F EATURES

Original: This section explores the impact of different choices for model and language representations, during meta-learning.
Revised: This section explores the impact of different choices for model and language representations for LMS.

Original: Four types of model features and two language embeddings are explored.
Revised: Four types of model features and two language embeddings are explored.

Original: We start by delineating possible choices for representations, then describe the details of our experiments, results, and the ﬁnal choices used in § 6.
Revised: We start by delineating possible choices for representations, then describe the details of our experiments, results, and the ﬁnal choices used in § 6.

Original: Four model-speciﬁc features are described below.
Revised: Four model-speciﬁc features are described below.

Original: Note [CLS] vectors are extracted from mBERT by feeding unlabeled text as input.
Revised: Note [CLS] vectors are extracted from mBERT by feeding unlabeled text as input.

Original: • [Eng] : Averaged [CLS] vectors computed over an unlabeled English text corpus are used for both training and testing.• [Pivot] : During training, [CLS] vectors are averaged over an unlabeled text corpus in the pivot language.
Revised: • [Eng] : Averaged [CLS] vectors computed over an unlabeled English text corpus are used for both training and testing.• [Pivot] : During training, [CLS] vectors are averaged over an unlabeled text corpus in the pivot language.

Original: At test time, [CLS] embeddings are averaged over an unlabeled corpus in the target language.
Revised: At test time, [CLS] embeddings are averaged over an unlabeled corpus in the target language.

Revised: We use the target-language development set (ignoring labels) for this purpose in our experiments.

Original: • [Target] : [CLS] vectors are averaged over a text corpus in the target language (for both training and testing).
Revised: • [Target] : [CLS] vectors are averaged over a text corpus in the target language (for both training and testing).

Original: • Fusion: A linear combination of the above features.
Revised: • Fusion: A linear combination of the above features.

Original: Weights on each representation are learned during training.
Revised: Weights on each representation are learned during training.

Original: Two types of language embeddings are examined.
Revised: Two types of language embeddings are examined.

Original: • lang2vec : 512-dimensional vectors learned by a neural network trained for typological prediction (Malaviya et al., 2017).
Revised: • lang2vec : 512-dimensional vectors learned by a neural network trained for typological prediction (Malaviya et al., 2017).

Original: • syntax : 103-dimensional binary vectors, which capture syntax features from the URIEL knowledge base (Littell et al., 2017). 
Revised: • syntax : 103-dimensional binary vectors, which capture syntax features from the URIEL knowledge base (Littell et al., 2017). First [Table]

Original: First, we determine the choice of model-speciﬁc features by averaging performance across both language embeddings.
Revised: we determine the choice of model-speciﬁc features by averaging performance across both language embeddings.

Original: Table 3 reports averaged evaluation metrics for each model-speciﬁc representation across all target languages with En-Dev as a baseline.
Revised: Table 3 reports averaged evaluation metrics for each model-speciﬁc representation across all target languages with En-Dev as a baseline.

Original: Averaged evaluation metrics across all target languages for each language embedding are reported in Table 4.
Revised: Averaged evaluation metrics across all target languages for each language embedding are reported in Table 4.

Original: In addition to evaluating the effectiveness of each language embedding, we also experimented with a variant of our scoring function that does not include any language embeddings as input.
Revised: In addition to evaluating the effectiveness of each language embedding, we also experimented with a variant of our scoring function that does not include any language embeddings as input.

Original: Results are reported on mBERT models in the meta-dev set and the target languages’ dev sets for all experiments in this section.
Revised: Results are reported on mBERT models in the meta-dev set and the target languages’ dev sets for all experiments in this section

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We use mBERT models in the meta-dev set for analysis.
Revised: We use mBERT models in the meta-dev set analysis

Original: Each number represents average of scores across all the target languages in a particular task.

Original: In Table 3, [PIVOT] features achieve top-2 performance in all ﬁve tasks.
Revised: In Table 3, [PIVOT] features achieve top-2 performance in all ﬁve tasks.

Original: [Eng] and [Target] achieve mixed results, and the fusion of three features does not effectively incorporate the advantages of each representation, except in the case of ARL.
Revised: [Eng] and [Target] achieve mixed results, and the fusion of three features does not effectively incorporate the advantages of each representation, except in the case of ARL.

Original: Table 4 shows that lang2vec outperforms syntax for all tasks but ARL and also outperforms our approach when language embeddings are not included.
Revised: Table 4 shows that lang2vec outperforms syntax for all tasks but ARL and also outperforms our approach when language embeddings are not included.

Original: Thus, lang2vec and [PIVOT] are used for all experiments in Section 6.
Revised: Thus, lang2vec and [PIVOT] are used for all experiments in Section 6.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent work has explored hyper-parameter optimization Klein et al.
Revised: Recent work has explored hyper-parameter optimization Klein et al.

Original: (2019), and model selection for a new task.
Revised: (2019), and model selection for a new task.

Original: task2vec
Revised: task2vec

Original: (Achille et al., 2019) presents a meta-learning approach to selecting a pre-trained feature extractor from a library for a new visual task.
Revised: (Achille et al., 2019) presents a meta-learning approach to selecting a pre-trained feature extractor from a library for a new visual task.

Original: More concretely, task2vec represents tasks in a vector space and is capable of predicting task similarities and taxonomic relations.
Revised: More concretely, task2vec represents tasks in a vector space and is capable of predicting task similarities and taxonomic relations.

Original: It encodes a new task and selects the best feature extractor trained on the most similar task.
Revised: It encodes a new task and selects the best feature extractor trained on the most similar task.

Original: Unlike task2vec , we select a trained model for a speciﬁc task, and we represent a trained model with model-speciﬁc features on a target language.
Revised: Unlike task2vec , we select a trained model for a speciﬁc task, and we represent a trained model with model-speciﬁc features on a target language.

Original: MAML (Finn et al., 2017; Rajeswaran et al., 2019) takes another approach, pre-training a single model with a meta-loss to initialize a set of parameters that can be quickly ﬁne-tuned for related tasks. 
Revised: MAML (Finn et al., 2017; Rajeswaran et al., 2019) is another approach to meta-learning, pre-training a single model with a meta-loss to initialize a set of parameters that can be quickly ﬁne-tuned for related tasks. Nooralahzadeh et al.

Original: In contrast to MAML, which is designed to support few-shot learning through better initialization of model parameters, our approach improves model selection for zero-shot cross-lingual transfer, where the training and test sets contain text in different languages.
Revised: (2020) explore the use of MAML in the cross-lingual transfer setting. MAML is designed to support few-shot learning through better initialization of model parameters and does not address the problem of model selection. In contrast, our approach improves model selection in the zero-shot cross-lingual transfer setting.

Original: Most relevant to our work, Xia et al.
Revised: Most relevant to our work, Xia et al.

Original: (2020) use regression methods to predict a model’s performance on an NLP task.
Revised: (2020) use regression methods to predict a model’s performance on an NLP task.

Original: They formulate this as a regression problem based on features of the task (dataset size, average sentence length, etc.), incorporating a discrete feature to represent the choice of model.
Revised: They formulate this as a regression problem based on features of the task (dataset size, average sentence length, etc.), incorporating a discrete feature to represent the choice of model.

Original: In contrast, LMS inspects a model’s internal representations, thus it is suitable for predicting which out of a set of ﬁne-tuned models will best transfer to a target language. 
Revised: In contrast, LMS inspects a model’s internal representations, thus it is suitable for predicting which out of a set of ﬁne-tuned models will best transfer to a target language. Also relevant is prior work on learning to select the best language to transfer from Lin et al.

Revised: Finally, we note that there is a need for more NLP research on low-resource languages (Joshi et al., 2020b).

Revised: Lauscher et al.

Revised: (2020) present a number of challenges in transferring to languages with few resources using pre-trained transformers.

Revised: The languages used in our experiments could be considered high-resource, however, our experiments do cover a fairly diverse set of languages, including

Original:   C ONCLUSION
Revised: Arabic and Chinese. We believe that there is still a need for more research on multilingual NLP for high-resource languages as well, as this is not a solved problem. Finally, we note that there are several other prominent benchmarks for evaluating cross-lingual transfer including XTERME (Hu et al., 2020) and XGLUE (Liang et al., 2020), both of which include some datasets used in this work.  C ONCLUSION

Revised: In this paper, we presented a machine learning approach to model selection for zero-shot crosslingual transfer, which is appropriate when small amounts of development data are available in one or more pivot languages, but not in the target language.

Revised: We showed that our approach improves over the standard practice of model selection using source language development data.

Revised: Experiments on ﬁve well-studied NLP tasks show that by inspecting internal representations, our method consistently selects better models.

Revised: LMS also achieves comparable results to the more expensive alternative of annotating small amounts of target-language development data.

Original: A A PPENDIX A. I MBALANCED D ATA FOR RE AND ARL
Revised: A A PPENDIX A. I MBALANCED D ATA FOR RE AND ARL

Original: In this section, we describe details of the imbalanced dataset for RE and ARL.
Revised: In this section, we describe details of the imbalanced dataset for RE and ARL.

Original: Table 5 reports the statistics of the dataset and Table 6 summarizes references and baseline results.
Revised: Table 5 reports the statistics of the dataset and Table 6 summarizes references and baseline results.

Original: The number of non-relation entity pairs or trigger-entity pairs (negative instances) far exceeds positive instances, which negatively affects a model’s performance.
Revised: The number of non-relation entity pairs or trigger-entity pairs (negative instances) far exceeds positive instances, which negatively affects a model’s performance.

Original: However, the preprocessed dataset from (Subburathinam et al., 2019) is artiﬁcially balanced, which does not reﬂect a practical setting.
Revised: However, the preprocessed dataset from (Subburathinam et al., 2019) is artiﬁcially balanced, which does not reﬂect a practical setting.

Original: Therefore, we create an additional imbalanced dataset using the ACE2005 corpus Walker et al.
Revised: Therefore, we create an additional imbalanced dataset using the ACE2005 corpus Walker et al.

Original: (2006), which more closely replicates the setting a model will be faced with in a real-world information extraction scenario.
Revised: (2006), which more closely replicates the setting a model will be faced with in a real-world information extraction scenario.

Original: First, we shufﬂe documents into 80%/10%/10% splits for train/dev/test, then extract candidate entity-pairs from each document.
Revised: First, we shufﬂe documents into 80%/10%/10% splits for train/dev/test, then extract candidate entity-pairs from each document.

Original: For RE, the ﬁrst approach in (Ye et al., 2019) is adopted to extract negative instances.
Revised: For RE, the ﬁrst approach in (Ye et al., 2019) is adopted to extract negative instances.

Original: Negative instances whose entity-type combination has never appeared as a positive example in the training data are ﬁltered out.
Revised: Negative instances whose entity-type combination has never appeared as a positive example in the training data are ﬁltered out.

Original: This results in a positive/negative ratio of 1:8.9 within 80,256 mention pairs for English.
Revised: This results in a positive/negative ratio of 1:8.9 within 80,256 mention pairs for English.

Original: We repeat the same process for Chinese/Arabic and end up with a positive/negative ratio of 1:9.4/1:8.9 and 73,082/42, instances.
Revised: We repeat the same process for Chinese/Arabic and end up with a positive/negative ratio of 1:9.4/1:8.9 and 73,082/42, instances.

Original: For ARL, we create negative instances by pairing each trigger with every entity in a sentence.
Revised: For ARL, we create negative instances by pairing each trigger with every entity in a sentence.

Original: This leads to a positive/negative ratio of 2.6/2.7/2.9 and 27,823/19,338/14,376 instances in
Revised: This leads to a positive/negative ratio of 2.6/2.7/2.9 and 27,823/19,338/14,376 instances in

Original: English/Chinese/Arabic.
Revised: English/Chinese/Arabic.

Original: Details on the two datasets are summarized in Table 5.
Revised: Details on the two datasets are summarized in Table 5.

Original: As a baseline for the imbalanced dataset, we reimplement the Graph Convolutional Network (GCN) model of (Subburathinam et al., 2019) using multilingual embeddings learned by fastText (Bojanowski et al., 2017) on Wikipedia (GCN ReImp
Revised: As a baseline for the imbalanced dataset, we reimplement the Graph Convolutional Network (GCN) model of (Subburathinam et al., 2019) using multilingual embeddings learned by fastText (Bojanowski et al., 2017) on Wikipedia (GCN ReImp

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: de70.
Revised: de

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: es   nl 50.0 52.5 55.0 57.
Revised: es   nl 50.0 52.5 55.0 57.

Original: zh
Revised: zh

Original: En-Dev
Revised: En-Dev

Original: LMS
Revised: LMS

Original: NER F1 Distribution
Revised: NER F1 Distribution

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: ar  de   es   nl   zh
Revised: ar  de   es   nl   zh

Original: En-Dev LMS
Revised: En-Dev LMS

Original: POS Accuracy Distribution
Revised: POS Accuracy Distribution

Original: [Figure]
Revised: [Figure]

Original: de    es   zh
Revised: de    es   zh

Original: En-Dev
Revised: En-Dev

Original: LMS
Revised: LMS

Original: [Figure]
Revised: [Figure]

Original: A. M ODEL S CORE D ISTRIBUTION
Revised: A. M ODEL S CORE D ISTRIBUTION

Original: Details of model score distribution are explained and analyzed in this section.
Revised: Details of model score distribution are explained and analyzed in this section.

Original: Distribution ﬁgures of NER, POS, and QA are shown in Figure 3.
Revised: Distribution ﬁgures of NER, POS, and QA are shown in Figure 3.

Original: To better interpret model selection results, we visualize the distribution of model scores in the metatest set with the kernel density estimate (KDE) plot.
Revised: To better interpret model selection results, we visualize the distribution of model scores in the metatest set with the kernel density estimate (KDE) plot.

Original: KDE represents the data using a continuous probability density curve in one or more dimensions.
Revised: KDE represents the data using a continuous probability density curve in one or more dimensions.

Original: LMS (Red) and En-Dev (Blue) selection results are marked in the model score distribution.
Revised: LMS (Red) and En-Dev (Blue) selection results are marked in the model score distribution.

Original: Majority model scores are concentrated at the center of the distribution but the variations are large.
Revised: The majority model scores are concentrated at the center of the distribution but the variations are large.

Original: For example, Arabic POS accuracy in Figureranging from 48 to 53, where the majority  models achieve a score around 50.
Revised: For example, Arabic POS accuracy in Figure 3 ranging from 48 to 53, where the majority of models achieve a score around 50.

Original: En-Dev fails to select a model from the middle of the distribution, which leads to a score below average models.
Revised: En-Dev fails to select a model from the middle of the distribution, which leads to a score below average models.

Original: However, LMS is able to not only select a model better than the En-Dev but also select a model that is better than the majority of candidates.
Revised: However, LMS is able to not only select a model better than the En-Dev but also select a model that is better than the majority of candidates.

Original: In fact, En-Dev is not a good model selection strategy when it comes to linguistically different languages such as Arabic and Chinese.
Revised: In fact, En-Dev is not a good model selection strategy when it comes to linguistically different languages such as Arabic and Chinese.

Original: For ArabicPOS, Arabic
Revised: For Arabic

Original:  QA, and Chinese POS, En-Dev selects a model that performs worse than an average model from the candidates (left side of the distribution).
Revised: POS, Arabic QA, and Chinese POS, En-Dev selects a model that performs worse than an average model from the candidates (left side of the distribution).

Original: This might suggest the model is over-ﬁtting to English data but fails to transfer to a distanced language.
Revised: This might suggest the model is over-ﬁtting to English data but fails to transfer to a distanced language.

Original: Our observations motivate us to develop a better model selection method - LMS.
Revised: Our observations motivate us to develop a better model selection method - LMS.

Original: A. D OES THIS A PPROACH G ENERALIZE TO XLM-R O BERT A ?
Revised: A. S TATISTICAL TEST OF LMS PERFORMANCE

Revised: In this section, we present a statistical analysis of model selection results for POS and QA. Tableshows LMS on average improve a point of 0.74 relative to and a point of 0.44 relative to Pivot-Dev. We train a single LMS with pivot languages in { ar, de, es, nl, zh } for POS and { ar, de, es, zh } for

Revised: QA, following by testing it on all the target languages.

Revised: All the results are reported with mean and standard deviation with ﬁve runs (different random seeds).

Revised: A z-test is performed to the differences between LMS/Pivot-Dev and En-Dev.

Revised: LMS is statistically signiﬁcantly ( p -value ≤ 0 . 05 ) higher than En-Dev baseline across all languages and two tasks while Pivot-Dev fails in three languages.

Revised: LMS also obtains a lower standard deviation for the model scores except for Swedish (sv) and Vietnamese (vi).

Revised: [Table]

Revised: 85.14 ± 0.

Revised: 82.14 ± 0.

Revised: 84.78 ± 0.

Revised: 84.07 ± 0.

Revised: 84.23 ± 0.

Revised: 91.58 ± 0.

Revised: 40.04 ± 0.

Revised: 58.83 ± 0.90 All-

Revised: [Table]

Revised: 88.09 ± 0.

Revised: 89.01 ± 0.

Revised: 73.80 ± 0.

Revised: 83.18 ± 0.

Revised: 85.61 ± 0.

Revised: 82.36 ± 0.

Revised: 85.39 ± 0.

Revised: 84.90 ± 0.

Revised: 85.41 ± 0.

Revised: 91.73 ± 0.

Revised: 42.56 ± 0.

Revised: 59.12 ± 0.92 # ofAll-Target

Revised: A. D OES THIS A PPROACH G ENERALIZE TO XLM-R O BERT A ?

Original: In Section 6, we showed that our approach consistently selects better ﬁne-tuned models than those chosen using English development data.
Revised: In Section 6, we showed that our approach consistently selects better ﬁne-tuned models than those chosen using English development data.

Original: To test the robustness of our approach with a different multilingual pre-trained transformer, we  evaluate it on XLM-RoBERTa-base (Conneau et al., 2019),  the same setting used for mBERT in Section 6 when testing RE and ARL with imbalance dataset.
Revised: To test the robustness of our approach with a different multilingual pre-trained transformer, we re-train and evaluate using XLM-RoBERTa-base (Conneau et al., 2019), with the same settings used for mBERT in Section 6 when testing RE and ARL with imbalance dataset.

Original: RE In the left section of Table 7, our approach selects a model with a higher F1 score compared to
Revised: RE In the left section of Table 8, our approach selects a model with a higher F1 score compared to

Original: En-Dev in Chinese and on par with En-Dev in Arabic.
Revised: En-Dev in Chinese and on par with En-Dev in Arabic.

Original: ARL In the right section of Table 7, our approach selects a model with a higher F1 score compared to En-Dev in Arabic but performs worse on Chinese.
Revised: ARL In the right section of Table 8, our approach selects a model with a higher F1 score compared to En-Dev in Arabic but performs worse on Chinese.

Original: Overall, our approach appears to be effective when used with XLM-RoBERTa. 
Revised: Overall, our approach appears to be effective when used with XLM-RoBERTa. [Equation]

Original: [Table]
Revised: [Table]

Original: A. C AN M ULTI - TASK L EARNING H ELP ?
Revised: A. C AN M ULTI - TASK L EARNING H ELP ?

Original: Our setting does not assume access to the labeled data in the target language for a particular task.
Revised: Our setting does not assume access to the labeled data in the target language for a particular task.

Original: However, labeled data in the target language may be available for a relevant auxiliary task, which could help the scoring function learn to better estimate whether a model is a good match for the target language.
Revised: However, labeled data in the target language may be available for a relevant auxiliary task, which could help the scoring function learn to better estimate whether a model is a good match for the target language.

Original: To test whether an auxiliary task in the target language might help to select a better model for the target task, we ﬁne-tune sets of mBERT models for ARL and RE.
Revised: To test whether an auxiliary task in the target language might help to select a better model for the target task, we ﬁne-tune sets of mBERT models for ARL and RE.

Original: Gold rankings on the models are then computed for each language using the pivot languages’ dev sets.
Revised: Gold rankings on the models are then computed for each language using the pivot languages’ dev sets.

Original: In addition, another “silver” ranking is computed for each language using the auxiliary task.
Revised: Also, another “silver” ranking is computed for each language using the auxiliary task.

Original: The scoring function is then trained to rank mBERT models for both tasks.
Revised: The scoring function is then trained to rank mBERT models for both tasks.

Original: To differentiate the two tasks, a variant of  scoring function, s ( m, l, t ) , which concatenates a randomly initialized task embedding with the language embedding is adopted.
Revised: To differentiate the two tasks, a variant of the scoring function, s ( m, l, t ) , which concatenates a randomly initialized task embedding with the language embedding is adopted.

Original: Following Section 7, we use model selection results from mBERT models in the metadev set and report average target language dev-set results across Chinese and Arabic.
Revised: Following Section 7, we use model selection results from mBERT models in the metadev set and report average target language dev-set results across Chinese and Arabic.

Original: In Table 8, our approach is able to select a model with  higher F1 score for RE.
Revised: In Table 9, our approach can select a model with a higher F1 score for RE.

Original: However, multi-task does not beneﬁt ARL but still outperforms En-Dev.
Revised: However, multi-task does not beneﬁt ARL but still outperforms En-Dev.

Revised: As for future direction, we believe an LMS that is

Original: [Table]
Revised: [Figure]

Original: We mBERT models in the
Revised: n d [Figure] n [Figure]

Revised: C

Revised: [Figure]

Revised: rr e l a t i o n

Revised: C

Revised: [Figure]

Revised: e ff i c

Revised: i

Revised: [Figure]

Revised: trained on an auxiliary dataset can be transferred to the target dataset, hence release the requirement of a small amount of pivot language development data in the target dataset.

Revised: [Figure]

Original:  A. H OW MANY MODELS DO WE NEED TO TRAIN LMS?
Revised: [Table] [Figure] [Table] A. H OW MANY MODELS DO WE NEED TO TRAIN LMS?

Original: We analyze how practical LMS is in this section.
Revised: We analyze how practical LMS is in this section.

Original: In particular, we aim to answer the question of how many models do we need in order to train LMS.
Revised: In particular, we aim to explore the question of can the LMS be trained without large amount models.

Original: In fact, LMS can be trained with only six models.
Revised: LMS can be trained with only six models

Original: LMS outperforms the En-Dev in three out of four target languages by training on only six models (Table 9).
Revised: LMS outperforms the En-Dev in three out of four target languages by training on only six models (Table 10).

Original: Varying the number of models in the meta-training set from 6 to 120 for QA, we see LMS perform consistently across  different numbers of training sizes.
Revised: Varying the number of models in the meta-training set from 6 to 120 for QA, we see LMS performs consistently across a different number of training sizes.

Original: It outperforms the En-Dev when trained with 72 models.
Revised: It outperforms the En-Dev when trained with 72 models.

Original: To better understand how LMS performs compared to En-Dev, we use Kendall Rank Correlation
Revised: To better understand how LMS performs compared to En-Dev, we use Kendall Rank Correlation

Original: Coefﬁcient (KRCC) (Kendal, 1938) to measure the similarity of the orderings.
Revised: Coefﬁcient (KRCC) (Kendal, 1938) to measure the similarity of the orderings.

Original: KRCC (-1 to 1) between two model rankings will be high when have a similar rank.
Revised: KRCC (-1 to 1) between two model rankings will be high when having a similar rank.

Original: In Figure 4, KRCC is calculated between En-Dev/LMS and golden model rankings for 60 models in the QA meta-test set.
Revised: In Figure 4, KRCC is calculated between En-Dev/LMS and golden model rankings for 60 models in the QA meta-test set.

Original: LMS outperforms En-Dev with only six models and the KRCC improves as more models are used to train
Revised: LMS outperforms En-Dev with only six models and the KRCC improves as more models are used to train

Original: LMS.
Revised: LMS.

Original: It is interesting to see LMS has the highest KRCC in Arabic.
Revised: It is interesting to see LMS has the highest KRCC in Arabic.

Original: We hypothesis this is because of the high variation of F1 score in Arabic - the standard deviation of the Arabic F1 score is 3.3 as opposed to 2.4 and 1.3 in German and Spanish.
Revised: We hypothesis this is because of the high variation of F1 score in Arabic - the standard deviation of the Arabic F1 score is 3.3 as opposed to 2.4 and 1.3 in German and Spanish.

Original: [Table]
Revised: [Table]

Original: [Figure]