Original: Learning Cross-Lingual IR from an English Retriever
Revised: Learning Cross-Lingual IR from an English Retriever

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Revised: We present D R .D

Original: We present a new cross-lingual information retrieval ( CLIR ) system trained using multi-stageknowledge distillation ( KD ).
Revised: ECR ( D ense R etrieval with D istillation- E nhanced C ross-Lingual R epresentation), a new cross-lingual information retrieval ( CLIR ) system trained using multi-stage knowledge distillation ( KD ).

Original: The teacher  relies on a highly effective but  expensive twostage process consisting of query translationand monolingual IR , while the student executes a single CLIR step.
Revised: The teacher of D R .D ECR relies on a highly effective but computationally expensive two-stage inference process consisting of query translation and monolingual IR , while the student, D R .D ECR , executes a single CLIR step.

Original: We teach the student powerful multilingual encoding aswellas CLIR by optimizing two corresponding KDobjectives.
Revised: We teach D R .D ECR powerful multilingual representations as well as CLIR by optimizing two corresponding KD objectives.

Original: Learning usefulnon-English representations  from an English-only retriever isaccomplished through a cross-lingual tokenalignment algorithm that relies on the representation capabilities of the underlying multilingual language model.
Revised: Learning useful representations of non-English text from an English-only retriever is accomplished through a cross-lingual token alignment algorithm that relies on the representation capabilities of the underlying multilingual encoders.

Original: In both in-domainand zero-shot  evaluation, the proposed methoddemonstrates far superior accuracy over directﬁne-tuning with labeled CLIR data.
Revised: In both in-domain and zero-shot out-of-domain evaluation, D R .D ECR demonstrates far superior accuracy over direct ﬁne-tuning with labeled CLIR data.

Original: One of oursystems is also thecurrent best single-modelsystem on the XOR-TyDi leaderboard.
Revised: It is also the best single-model retriever on the XOR-TyDi benchmark at the time of this writing.

Original:  Introduction
Revised:  Introduction

Original: Multilingual models are critical for the democratization of AI.
Revised: Multilingual models are critical for the democratization of AI.

Original: Cross-lingual information retrieval( CLIR ) (Braschler et al., 1999; Shakery and Zhai,2013; Jiang et al., 2020; Asai et al., 2021a), for example, can ﬁnd relevant text in a high-resource language such as English even when the query is posedin a different, possibly low-resource, language.
Revised: Cross-lingual information retrieval ( CLIR ) (Braschler et al., 1999; Shakery and Zhai, 2013; Jiang et al., 2020; Asai et al., 2021a), for example, can ﬁnd relevant text in a high-resource language such as English even when the query is posed in a different, possibly low-resource, language.

Original: Inthis work, we develop useful CLIR models for thisconstrained, yet important, setting where a retrievalcorpus is available only in a single high-resourcelanguage (English in our experiments).
Revised: In this work, we develop useful CLIR models for this constrained, yet important, setting where a retrieval corpus is available only in a single high-resource language (English in our experiments).

Original: A straightforward solution to this problem canbe based on machine translation ( MT ) of the queryinto English, followed by English IR (Asai et al.,2021a).
Revised: A straightforward solution to this problem could use machine translation ( MT ) to translate the query into English, and then perform English IR (Asai et al., 2021a).

Original: While this two-stage process is capableof providing accurate predictions, an alternativeend-to-end approach that can tackle the problempurely cross-lingually, i.e. , without involving MT ,would clearly be more efﬁcient and cost-effective.
Revised: While such a two-stage process can produce reasonably accurate predictions, an alternative end-to-end approach that can tackle the problem purely cross-lingually, i.e. , without involving MT for inference, would clearly be more efﬁcient and cost-effective.

Original: Pre-trained multilingual masked language models( PLM s) such as multilingual BERT (Devlin et al., 2019) or XLM - R o BERT a ( XLM - R ) (Conneau et al., 2020) can provide the foundation for such an approach, asone can simply ﬁne-tune a PLM with labeled CLIR data  (Asai et al., 2021b).
Revised: Pre-trained multilingual masked language models ( PLM s) such as multilingual BERT (Devlin et al., 2019) or XLM - R o BERT a ( XLM - R ) (Conneau et al., 2020) can provide the foundation for such a one-step solution, as simply ﬁne-tuning a PLM with labeled CLIR data would yield a crosslingual retriever (Asai et al., 2021b).

Original: Here we ﬁrst run an empirical evaluation of these two approaches on a public CLIR benchmark (Asai et al., 2021a), which includes both in-domain and zero-shot out-of-domain tests.
Revised: Here we ﬁrst run an empirical evaluation of these two approaches on a public CLIR benchmark (Asai et al., 2021a), which involves both in-domain and zero-shot out-of-domain tests.

Original: We use Col BERT (Khattab and Zaharia, 2020; Khattab et al., 2021) as  our IR architecture1 and XLM - R as the underlying PLM for both methods (§2).
Revised: We use Col BERT (Khattab and Zaharia, 2020; Khattab et al., 2021)— a state-of-the-art ( SOTA ) neural IR model that has been shown to outperform other recent methods such as DPR (Karpukhin et al., 2020)—as our IR architecture and XLM - R as the underlying PLM for both methods (§2).

Original: Results indicate that the MT -based solution can be vastly more effective than  CLIR ﬁne-tuning, with observed differencesin Recall@5kt of 22.2–28.6 points (§3).
Revised: Results indicate that the MTbased solution can be vastly more effective than direct EN + CLIR ﬁne-tuning, with observed differences of 22.2–28.6 Recall@5k-tokens (§3).

Original: Crucially, the modular design of the former allows it to leverage additional English-only training data for its IR component, providing signiﬁcant boosts to its results.
Revised: Crucially, the modular design of the former allows it to leverage additional English-only training data for its IR component, providing signiﬁcant boosts to its performance.

Original: The above ﬁndings lead naturally to the central research question of this paper: Can a highperformance CLIR model be trained that can operate without having to rely on MT ?
Revised: The above ﬁndings lead naturally to the central research question of this paper: Can a highperformance CLIR model be trained that can operate without having to rely on MT ?

Original: To answer the question, instead of viewing the MT -based approach as a competing one, we propose to leverage its strength via knowledge distillation ( KD ) into an end-to-end CLIR model.
Revised: To answer the question, instead of viewing the MT -based approach as a competing one, we propose to leverage its strength via knowledge distillation ( KD ) into an end-to-end CLIR model, which we call D R .D

Revised: ECR ( D ense R etrieval with D istillationE nhanced C ross-lingual R epresentation).

Original: KD (Hinton et al., 2014) is a powerful supervision technique typically used to distill the knowledge of a large teacher model about some task into a smaller student model (Mukherjee and Awadallah, 2020; Turc et al., 2020).
Revised: KD (Hinton et al., 2014) is a powerful supervision technique typically used to distill the knowledge of a large teacher model about some task into a smaller student model (Mukherjee and Awadallah, 2020; Turc et al., 2020).

Original: Here we propose to use it in a slightly different context, where the teacher and the student retriever are identical in size, but the former has superior performance simply due to utilizing MT output and consequently operating in a high-resource and lowdifﬁculty monolingual environment.
Revised: Here we propose to use it in a slightly different context, where the teacher and the student retriever are identical in size, but the former has superior performance simply due to utilizing MT output and consequently operating in a high-resource and low-difﬁculty monolingual environment.

Original: We run two independent KD operations (§2.2).
Revised: We run two independent KD operations (§2.2).

Original: One directly optimizes an IR objective by utiliz- XLM-R (T) Linear Trans. XLM-R (T)
Revised: One directly optimizes an IR objective by utilizing labeled CLIR data: parallel questions (English and non-English) and corresponding relevant and non-relevant English passages.

Original: Linear Trans.

Original: Σ

Original: XLM-R (T)

Original: Linear Trans.

Original: Σ

Original: Paris was founded in3rd century BC... Where is the capital of France? Rome was founded in753 B.C… XLM-R (S)
Revised: The teacher and the student, D R .D ECR , are shown the English and nonEnglish versions of the questions, respectively; the training objective is for D R .D

Original: Linear Trans. XLM-R (S) Linear Trans. Σ
Revised: ECR to match the soft query-passage relevance predictions of the teacher.

Original: XLM-R (S) Linear Trans. Σ Paris was founded in3rd century BC...
Revised: The second KD task is representation learning from parallel text, where D R .D

Original: 法国的首都在哪里

Original: Rome was founded in753 B.C… 𝑑 !

Original: 𝑞 𝑑 " 𝑆 !,# !

Original: 𝑆 !,# "

Original: Distillation

Original: Teacher Student 𝑑 ! 𝑞 𝑑 " Target: 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 $ (𝑆 !,# ! ,𝑆 !,# " ) 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 ing labeled CLIR data: parallel questions (Englishand non-English) and corresponding relevant andnon-relevant English passages.
Revised: ECR learns to encode a non-English text in a way that matches the teacher’s encoding of the aligned English text, at the token level .

Original: The teacher andthe student are shown the English and non-Englishversions of the questions, respectively; the training objective is for the student to match the softquery-passage relevance predictions of the teacher.

Original: The second KD task is representation learning fromparallel text, where the student learns to encode anon-English text in a way that matches the teacher’sencoding of the aligned English text, at the tokenlevel .

Original: The cross-lingual token alignment needed tocreate the training data for this task is generatedusing a greedy alignment process that exploits thePLM ’s multilingual representations.
Revised: The cross-lingual token alignments needed to create the training data for this task are generated using a greedy alignment process, which exploits the multilingual representation capabilities of the underlying PLM encoders.

Original: In our experiments on the XOR-TyDi dataset(Asai et al., 2021a), the KD student outperforms theﬁne-tuned Col BERT baseline by 25.4 (in-domain)and 14.9 (zero-shot) Recall@5kt, recovering muchof the performance loss from the MT -based solution.
Revised: In our evaluation on the XOR-TyDi benchmark (Asai et al., 2021a), D R .D ECR outperforms the ﬁne-tuned Col BERT baseline by 25.4 (in-domain) and 14.9 (zero-shot) Recall@5k-tokens, recovering much of the performance loss from the MT -based solution.

Original: It is also the best single-model  system on the XOR-TyDi leaderboard 2 at the time of this writing.
Revised: It is also the best single-model IR system on the XOR-TyDi leaderboard 1 at the time of this writing.

Original: Ablation studies show that each of our two KDprocesses contribute signiﬁcantly towards the ﬁnalperformance of the student model.
Revised: Ablation studies show that each of our two KD processes contribute signiﬁcantly towards the ﬁnal performance of D R .D ECR .

Original: Our contributions can be summarized as follows:(1)
Revised: Our contributions can be summarized as follows:

Original:  We present an empirical study of the effectiveness of a SOTA IR method (Col BERT ) on crosslingual IR with and without MT .
Revised: (1) We present an empirical study of the effectiveness of a SOTA IR method (Col BERT ) on crosslingual IR with and without MT , (2)

Revised: We

Original: (2) We propose anovel end-to-end cross-lingual solution that usesknowledge distillation to learn both improved textrepresentation and retrieval. (3) We demonstratewith a new cross-lingual alignment algorithm thatdistillation using parallel text can strongly augment cross-lingual IR training. (4) We achieve newsingle-model SOTA results on XOR-TyDi.
Revised: propose a novel end-to-end cross-lingual solution that uses knowledge distillation to learn both improved text representation and retrieval, (3) We demonstrate with a new cross-lingual alignment algorithm that distillation using parallel text can strongly augment cross-lingual IR training, and (4) We achieve new single-model SOTA results on XOR-TyDi.

Original:  Method
Revised:  Method

Original: Here we ﬁrst describe our base IR architecture (Col BERT ) and then the proposed KD -based crosslingual training algorithms.
Revised: Here we ﬁrst describe our base IR architecture (Col BERT ) and then the proposed KD -based crosslingual training algorithms.

Original: 2. The Col BERT Model
Revised: 2. The Col BERT Model

Original: Col BERT (Khattab and Zaharia, 2020) employs a transformer-based encoder to separately encode the input query and document, followed by a linear compression layer.
Revised: Col BERT (Khattab and Zaharia, 2020) employs a transformer-based encoder to separately encode the input query and document, followed by a linear compression layer.

Original: Each training instance is a < q , d + , d − > triple, where q is a query, d + is a positive (relevant) document and d − is a negative (non-relevant) document.
Revised: Each training instance is a < q , d + , d − > triple, where q is a query, d + is a positive (relevant) document and d − is a negative (non-relevant) document.

Original: A relevance score S q,d for the pair ( q, d ) is ﬁrst computed using Eq. 1, where d ∈ { d + , d − } and E q i and E d j are the output embeddings of query token q i and document token d j , respectively.
Revised: A relevance score S q,d for the pair ( q, d ) is ﬁrst computed using Eq. 1, where d ∈ { d + , d − } and E q i and E d j are the output embeddings of query token q i and document token d j , respectively.

Original: For a given training triple, a cross-entropy loss is minimized for the softmax over S q,d + and S q,d − .
Revised: For a given training triple, a cross-entropy loss is minimized for the softmax over S q,d + and S q,d − .

Original: [Equation]
Revised: [Equation]

Original: For inference, the embeddings of all documents are calculated a priori , while the query embeddings and the relevance score are computed at runtime.
Revised: For inference, the embeddings of all documents are calculated a priori , while the query embeddings and the relevance score are computed at runtime.

Original: 2. Knowledge Distillation
Revised: 2. Knowledge Distillation

Original: Our teacher and student are both Col BERT models that ﬁne-tune the same underlying multilingual PLM for IR .
Revised: Our teacher and D R .D ECR are both Col BERT models that ﬁne-tune the same underlying multilingual PLM for IR .

Original: The teacher is ﬁrst trained with all-English triples using the procedure of §2.1.
Revised: The teacher is ﬁrst trained with all-English triples using the procedure of §2.1.

Original: The goal of the subsequent KD training is to teach the student how to reproduce the behavior of this teacher, provided that it sees non-English questions while the teacher sees English questions.
Revised: The goal of the subsequent KD training is to teach D R .D ECR to reproduce the behavior of this teacher when it sees non-English translations of the teacher’s English questions.

Original: We apply KD at two different stages of the
Revised: We apply KD at two different stages of the

Original: Col BERT workﬂow: (1) relevance score computa-
Revised: Col BERT workﬂow: ( a ) relevance score computation ( S q,d in Eq. 1), and ( b ) encoding ( e.g. , E q i ). Figure 1 depicts ( a ) in detail, where training minimizes the KL divergence between the D R .D ECR ’s and the teacher’s output softmax distributions (with temperature) over S q,d + and S q,d − . Labeled training data for CLIR are scarce, whereas MT , being a more established area of research, has produced a large amount of parallel text over the years. We seek to exploit existing parallel corpora in our second KD training, where we teach D R .D

Revised: ECR to compute representations of nonEnglish texts that closely match the teacher’s representations of aligned English texts.

Revised: Importantly, since Col BERT computes a single vector for each

Original: Input : v T : Teacher’s output representation of tokenized
Revised: XLM-R (T) Linear Trans. XLM-R (T) Linear Trans.

Original: English (EN) text.
Revised: Σ XLM-R (T) Linear Trans.

Original: v
Revised: Σ Paris was founded in3rd century BC... Where is the capital of France? Rome was founded in753 B.C…

Original: S : Student’s representation of parallel non-EN text.
Revised: XLM-R (S) Linear Trans. XLM-R (S) Linear Trans.

Revised: Σ

Revised: XLM-R (S)

Revised: Linear Trans.

Revised: Σ

Revised: Paris was founded in3rd century BC...

Revised: 法国的首都在哪里

Revised: Rome was founded in753 B.C… 𝑑 !

Revised: 𝑞 𝑑 " 𝑆 !,# !

Revised: 𝑆 !,# "

Revised: Distillation

Original: Output : v ( a ) T : Reordered teacher output embeddings to reﬂect position-wise alignment with v S .
Revised: Teacher Student 𝑑 ! 𝑞 𝑑 " Target: 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 $ (𝑆 !,# ! ,𝑆 !,# " ) 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 position

Original: Procedure :
Revised: Procedure :

Original: DM ← cosine _ distance ( v T , v S ) //matrix
Revised: DM ← cosine _ distance ( v T , v S ) //matrix

Original: //get index pairs to swap in v T swaps ← [ ] for row in rows(DM) do //loop runs | v T | times minV alue ← min ( DM ) i, j ← index _ of ( minV alue ) //swap rows i and j
Revised: //get index pairs to swap in v T swaps ← [ ] for row in rows(DM) do //loop runs | v T | times minV alue ← min ( DM ) i, j ← index _ of ( minV alue ) //swap rows i and j

Original: DM [[ i, j ] , :] =
Revised: DM [[ i, j ] , :] =

Original: DM [[ j, i ] , :] //set row j and column j to + ∞
Revised: DM [[ j, i ] , :] //set row j and column j to + ∞

Original: DM [ j, :] ← + ∞ DM
Revised: DM [ j, :] ← + ∞ DM

Original: [: , j ] ← + ∞ swaps.append (( i, j )) end //swap
Revised: [: , j ] ← + ∞ swaps.append (( i, j )) end //swap

Original: teacher’s output tokens v ( a ) T ← v T for s in swaps do v ( a ) T [ s [ , s ]] ← v T  , s 
Revised: teacher’s output tokens v ( a ) T ← v T for s in swaps do v ( a ) T [ s [0] , s [1]] ← v T [ s [1] , s [0]] end individual input token ( i.e. , a PLM vocabulary item) and not for the entire input text, our algorithm must support distillation at the token level.

Original: ]] end

Original: Algorithm 1: Cross-lingual alignment.

Original: tion ( S q,d in Eq. 1), and (2) encoding ( e.g. , E q i ).

Original: Figure 1 depicts the former in detail, where training minimizes the KL divergence between the student’s and the teacher’s output softmax distributions (with temperature) over S q,d + and S q,d − .

Original: Labeled training data for CLIR are of limitedavailability, whereas MT , being a more establishedarea of research, has produced a large amount ofparallel text over the years.

Original: We seek to exploit parallel corpora in our second KD training stage, wherewe train the student to compute representations fornon-English texts that closely match the teacher’srepresentations of aligned English texts.

Original: Crucially,since Col BERT computes a single vector for eachindividual input token ( i.e. , a PLM vocabulary item)and not for the entire input text, our algorithm mustsupport distillation at the token level.

Original: To achieve this, we apply an unsupervisedcross-lingual alignment algorithm.
Revised: To achieve this, we design an unsupervised cross-lingual token alignment algorithm.

Original: Assuming( ne 1 , ..., ne S ) to be the ordered tuple of tokens in anon-English text and ( e 1 , ..., e T ) the correspondingtuple from the aligned English text, each iterationof this algorithm greedily aligns the next ( ne i , e j )pair with the highest cosine similarity of their output embeddings.
Revised: Assuming ( ne 1 , ..., ne S ) to be the ordered tuple of tokens in a non-English text and ( e 1 , ..., e T ) the corresponding tuple from the parallel English text, each iteration of this algorithm greedily picks the next ( ne i , e j ) pair with the highest cosine similarity of their output embeddings.

Original: Algorithm 1 implements this ideaby repositioning the teacher’s tokens so that theyare position-wise aligned with the correspondingstudent tokens.
Revised: Algorithm 1 implements this idea by repositioning the teacher’s tokens so that they are position-wise aligned with the corresponding D R .D

Original:  Note that the design choice of usKD over indices
Revised: ECR tokens. Note that the design choice of [Figure]

Original: KD over aligned tokens
Revised: KD over aligned tokens

Original: Teacher
Revised: Teacher

Original: Student Student where where 在哪里
Revised: Student Student where where 在哪里

Original: XLM-R (S)
Revised: XLM-R (S)

Original: Linear Trans.
Revised: Linear Trans.

Original: 法国的首都在哪里
Revised: 法国的首都在哪里

Original: XLM-R (S)
Revised: XLM-R (S)

Original: Linear Trans.
Revised: Linear Trans.

Original: Where is the capital of France? XLM-R (T) Linear Trans. Where is the capital of France?
Revised: [Figure]

Original: ing a common multilingual PLM in the teacher and the student, even though the former is taskedonly with  handling English content, is key forthe operation of this algorithm as it relies on the PLM ’s multilingual representation capabilities.
Revised: ﬁne-tuning a common multilingual PLM for the teacher and the D R .D ECR , even though the former is tasked with only handling English content, is key for this algorithm as it relies on the PLM s’ multilingual representation capabilities.

Revised: See Appendix A. for details on our parallel corpora used for training.

Original: In addition to cross-lingual alignment, we also perform a similar KD procedure in which both the teacher and the student are shown the same English text.
Revised: In addition to cross-lingual alignment, we also perform a similar KD procedure in which both the teacher and the D R .D ECR are shown the same English text.

Original: This step is useful because Col BERT uses a shared encoder for the query and the document, necessitating a student that is able to effectively encode text from both English documents and nonEnglish queries.
Revised: This step is useful because Col BERT uses a shared encoder for the query and the document, necessitating a student that is able to effectively encode text from both English documents and nonEnglish queries.

Original: Using the alignment information, we train the student by minimizing the Euclidean distance between its representation of a token (English or nonEnglish) and the teacher’s representation of the corresponding English token.
Revised: Using the alignment information, we train D R .D ECR by minimizing the Euclidean distance between its representation of a token (English or non-English) and the teacher’s representation of the corresponding English token.

Original: Figure 2 shows the KD process for representation learning.
Revised: Figure 2 shows the KD process for representation learning.

Revised: [Table]

Revised: [Table]

Original:  Experiments 
Revised:  Experiments 3. Setup

Original: Setup.

Original: Our primary CLIR dataset is XOR-TyDi (Asai et al., 2021a), which contains examples in seven typologically diverse languages: Arabic (Ar), Bengali (Bn), Finnish (Fi), Japanese (Ja), Korean (Ko), Russian (Ru) and Telugu (Te).
Revised: Our primary CLIR dataset is XOR-TyDi (Asai et al., 2021a), which contains examples in seven typologically diverse languages: Arabic (Ar), Bengali (Bn), Finnish (Fi), Japanese (Ja), Korean (Ko), Russian (Ru) and Telugu (Te).

Original: For standard in-domain experiments, we use a train-dev-test split of this dataset.
Revised: For standard indomain experiments, we use a train-dev-test split of this dataset.

Original: There are 2,113 questions in the test set.
Revised: There are 2,113 questions in the test set.

Original: For zero-shot experiments, we use the
Revised: For zero-shot experiments, we use the

Original: MKQA (Longpre et al., 2020) dataset for training and validation, and the following shared languages in the XOR-TyDi test set for evaluation:
Revised: MKQA (Longpre et al., 2020) dataset for training and validation, and the following shared languages in the XOR-TyDi test set for evaluation:

Original: Ar, Fi, Ja, Ko and Ru.
Revised: Ar, Fi, Ja, Ko and Ru.

Original: Both training sets contain
Revised: Both training sets contain

Original: English questions and their human translations in the other languages, their short answers and corresponding relevant (positive) and non-relevant (negative) Wikipedia snippets.Additionally, we use the [Table] [Table]
Revised: English questions and their human translations in the other languages, their short answers and corresponding relevant (positive) and non-relevant (negative) Wikipedia snippets.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] OpenNQ (Kwiatkowski et al., 2019)training set for English pre -training of the baseline model.
Revised: Additionally, we use training examples from the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019) for English pretraining of the baseline model.

Original: Furtherdetails on data pre-processing, the ﬁnal trainingsets and the optimal hyperparameter conﬁgurationsare provided in Appendix A.1 and A.2.
Revised: Further details on data pre-processing and the ﬁnal training sets are provided in Appendix A.1.

Original: The CLIR baseline used in our experiments isa
Revised: The CLIR baseline used in our experiments is

Original: Col BERTmodel with an  XLM - R PLM , which weiteratively ﬁne-tune ﬁrst on English and then oncross-lingual IR triples for optimal performance.
Revised: Col BERT with an underlying XLM - R PLM , which we iteratively ﬁne-tune ﬁrst on English and then on cross-lingual IR triples for optimal performance.

Original: Our student model is initialized with the parameterweights of this baseline, and is further ﬁne-tunedusing the two KD objectives.
Revised: Our D R .D ECR model is initialized with the parameter weights of this baseline, and is further ﬁne-tuned using the two KD objectives.

Original: As the KD teacher, wetrain a  model on only English triples, as statedbefore.
Revised: The KD teacher is a Col BERT model ﬁne-tuned with only English triples, as stated before.

Original: During evaluation, it is given machinetranslated questions that come with the XOR-TyDidataset.
Revised: During evaluation, it is given machine-translated questions that come with the XOR-TyDi dataset.

Original: Appendix A.1 contains more details on thesupervision of these models.
Revised: Appendices A.1 and A.2 contain additional details on the supervision of these models and the optimal hyperparameter conﬁgurations.

Original: Our evaluation metric is Recall at  5000 tokens(R@5kt) (Asai et al., 2021a), which computes thefraction of questions for which the ground truthshort answer is contained within the top 5,000 tokens of the retrieved passages.Appendix A.3 alsopresents R@2kt results.
Revised: We evaluate using Recall at t tokens for t ∈ [Table] { 2000 , 5000 } , i.e. , R@2kt and R@5kt (Asai et al., 2021a), which compute the fraction of questions for which the ground truth short answer is contained within the top t tokens of the retrieved passages.

Original: Evaluation.
Revised: 3. Evaluation

Original: Table 1 compares the performanceof our models measured by their R@5kt scores.
Revised: Table 1 compares the performance of our different models.

Original: First,  we observe that pre-training the baselinemodel on English IR triples from the OpenNQdataset (Kwiatkowski et al., 2019) substantiallyboosts its performance in both in-domain and zeroshot settings.
Revised: First, looking at the R@5kt results, we observe that pre-training the baseline model with English IR triples from the NQ train set (rows 2, 6) substantially boosts its performance in both indomain and zero-shot settings.

Original: However, it still underperforms the MT + English IR pipeline  by 28.6 and 26.5 points,respectively.
Revised: However, it still underperforms the MT + English IR pipeline (rows 3, 7) by 28.6 and 22.2 points, respectively.

Original: The proposed KD -based procedure,ﬁrst with the parallel corpus (for representationlearning) and then with the IR triples (for CLIR ),achieves anoverall improvement of 25.4 pointsfor the baseline model in in-domain evaluation,which, quite impressively, is only 3.2 pointsbeSystem R@5kt
Revised: By distilling ﬁrst with the parallel corpus (for representation learning) and then with the IR triples (for CLIR ), D R .D ECR (row 4) yields an improvement of 25. points over the baseline model in in-domain evaluation, which, quite impressively, is within 3.2 points

Original: With target-domain supervision:

Original: [Table]
Revised: [Table]

Original: hind the teacher’s score.
Revised: of the teacher’s score.

Original: In the zero-shot setting, we again observe a sizable gain of 14.9 points.
Revised: A sizable gain of 14.9 points is also observed in zero-shot evaluation (row 8).

Original: We present more ﬁne-grained per-language results in
Revised: Finally, the R@2kt numbers show a very similar pattern. Table 2 shows the performance (R@5kt) of D R .D ECR and the baseline on each individual language: the former outperforms the latter both with and without target domain supervision, yielding large gains across all languages. These results demonstrate the robustness of our approach, which stems from combining the individual strengths of

Original: Appendix A.4.
Revised: MT , English IR and KD in a single model.

Original:  Leaderboard Submission.
Revised: 3. Leaderboard Submission

Original: Our KD student of Table 4 row 4, trained onXOR-TyDi and submitted under the name CHIMAERAS B, is the best singlemodel system on theofﬁcial XOR-TyDi leaderboard  at the time of this writing.
Revised: The D R .D ECR model trained on the XOR-TyDi training set, shown in Table 1 row 4, is the best single-model retriever on the XOR-TyDi leaderboard 2 at the time of this writing.

Original: Please see Appendix A.5 for further details.
Revised: Since our parallel corpus extraction process relies on in-house source code that is not publicly available, we submitted to the “Systems using External APIs” category. Crucially, all other submitted systems under the External APIs category rely on MT at decoding time, avoiding which is one of the primary goals of our work. We also created parallel corpora purely from public available sources. 3 Our model distilled with these instances also achieved top position on the white-box systems leaderboard of XOR-TyDi.

Original:  Ablation Study.
Revised: 3. Ablation Study

Original: We train two more student models,each of which goes through only one of the two KD training steps.
Revised: We experiment with two more student models, one distilled with only CLIR examples and the other with only the parallel corpus.

Original: Table 2 summarizes the results: KD with only CLIR examples and with only the parallel corpus improves the system’s score by 15. and 20.9 points, respectively with target-domain supervision.
Revised: As the results in Table 3 show, each has a substantial impact on system performance.

Original: Interestingly, although the parallel corpus does not provide any IR signal, it contributes more to the ﬁnal performance of the model.
Revised: Interestingly, although the parallel corpus does not provide any IR signal, it contributes more to the model’s accuracy.

Original: These results also conﬁrm that our cross-lingual alignment algorithm  indeed produces useful alignments.
Revised: These results also conﬁrm that our cross-lingual alignment algorithm does indeed produce useful alignments.

Original:  Conclusion
Revised:  Conclusion

Original: We train highly effective end-to-end cross-lingual
Revised: We train highly effective end-to-end cross-lingual

Original: IR models by distilling the knowledge of an English retriever.
Revised: IR models by distilling the knowledge of an English retriever.

Original: We propose separate processes to teach IR and multilingual text representations, and present for the latter a cross-lingual alignment algorithm that only relies on the underlying masked language model’s multilingual representation capabilities.
Revised: We propose separate processes to teach IR and multilingual text representations, and present for the latter a cross-lingual alignment algorithm that only relies on the underlying masked language model’s multilingual representation capabilities.

Original: Supervised and zero-shot evaluations show that our model recovers much of the performance lost due to operating in an efﬁcient crosslingual mode.
Revised: Supervised and zero-shot evaluations show that our model recovers much of the performance lost due to operating in an efﬁcient crosslingual mode.

Original: Our KD -based method also yields new single-model SOTA results on the XOR-TyDi benchmark.
Revised: Our KD -based method also yields new single-model SOTA results on the XOR-TyDi benchmark.

Original: Future work will explore IR on unseen languages and evaluation on additional datasets.
Revised: Future work will explore IR on unseen languages and evaluation on additional datasets.

Original:  Ethics 
Revised:  Ethics 5. Limitations

Original: 5. Limitations

Original: We show the effectiveness of multi-stage knowledge distillation and cross-lingual token alignmentin training a cross-lingual information retrieval system.
Revised: We show the effectiveness of multi-stage knowledge distillation and cross-lingual token alignment in training a cross-lingual information retrieval system.

Original: We believe that it can be transferred to moredatasets and languages, but here we only showproof of concept for the XOR-TyDi and MKQAdatasets and the seven languages mentioned in themanuscript.
Revised: We believe that it can be transferred to more datasets and languages, but here we only show proof of concept for the XOR-TyDi and MKQA datasets and the seven languages mentioned in the paper.

Original: 5. Risks
Revised: 5. Risks

Original: The intent of this work is to develop a new methodfor high-performance cross-lingual information retrieval.
Revised: The intent of this work is to develop a new method for high-performance cross-lingual information retrieval.

Original: It is possible that a malicious user couldtry to attack the system by providing poor or offensive training data.
Revised: It is possible that a malicious user could try to attack the system by providing poor or offensive training data.

Original: We do not support it being usedin such a manner.
Revised: We do not support it being used in such a manner.

Original: The risks of our system are thesame as other NLP systems and we do not believewe introduce any additional risk.
Revised: The risks of our system are the same as other NLP systems and we do not believe we introduce any additional risk.

Revised: Acknowledgements

Revised: We thank Graeme Blackwood and Christoph Tillmann for providing the in-house parallel corpora.

Revised: We also thank Akari Asai for her help submitting

Revised: D R .D

Revised: ECR to the XOR-TyDi leaderboard

Original: A Appendix 
Revised: A Appendix A. Data Pre-processing

Original: A. Data Pre-processing

Original: The ofﬁcial XOR-TyDi training set consists of15,221 natural language queries, their short answers, and examples of corresponding relevant(positive) and non-relevant (negative) Wikipediasnippets.
Revised: The ofﬁcial XOR-TyDi training set consists of 15,221 natural language queries, their short answers, and examples of corresponding relevant (positive) and non-relevant (negative) Wikipedia snippets.

Original: For most queries, there are one positiveand three negative examples.
Revised: For most queries, there are one positive and three negative examples.

Original: We remove the 1,699(11%) questions that have no answers in the dataset.
Revised: We remove the 1, (11%) questions that have no answers in the dataset.

Original: A random selection of 90% of the remaining examples is used for training and the rest for validation.
Revised: A random selection of 90% of the remaining examples is used for training and the rest for validation.

Original: Following the original XOR-TyDi process, wealso obtain additional training examples by running
Revised: Following the original XOR-TyDi process, we also obtain additional training examples by running

Original: BM25-based retrieval against a Wikipedia corpusand using answer string match as the relevance criterion.
Revised: BM25-based retrieval against a Wikipedia corpus and using answer string match as the relevance criterion.

Original: These examples are added to the original setto obtain three positive and 100 negative examplesper query.
Revised: These examples are added to the original set to obtain three positive and 100 negative examples per query.

Original: As the blind test set for ﬁnal evaluation,we use the 2,113 questions in the ofﬁcial XORTyDi dev set.
Revised: As the blind test set for ﬁnal evaluation, we use the 2,113 questions in the ofﬁcial XORTyDi dev set.

Original: Our monolingual (English) training data ofabout 17.5 M triples are derived from the thirdﬁne-tuning round (Col BERT -QA3) of Col BERTrelevance-guided supervision (Khattab et al., 2021)with OpenNQ data (Kwiatkowski et al., 2019).
Revised: Our monolingual (English) training data containing about 17.5 M triples are derived from the third ﬁne-tuning round (Col BERT -QA3) of Col BERT relevance-guided supervision (Khattab et al., 2021) with NQ examples (Kwiatkowski et al., 2019).

Original: Theparallel corpus used in our KD experiments forrepresentation learning is constructed from threedifferent sources: (1) an in-house crawl of Korean,(2) LDC releases (Arabic), and (3) OPUS 3 .
Revised: The parallel corpus used in our KD experiments (Table 1) for representation learning is constructed from three different sources: (1) an in-house crawl of Korean, (2) LDC releases (Arabic), and (3) OPUS.

Original: Thecorpus has a total of 6.9M passage pairs which include .9M pairs in Telugu and 1M pairs in each ofthe other six languages.
Revised: The corpus has a total of 6.9M passage pairs which include .9M pairs in Telugu and 1M pairs in each of the other six languages.

Revised: The parallel corpus used in our white-box system was created purely from OPUS.

Revised: The statistics and sources are shown in the table below.

Original:  For zero-shot experiments, the training examplesare derived from MKQA (Longpre et al. , 2020),which consists of 10k queries selected from NQ,human translated into 25 additional languages, ﬁveof which overlap with XOR-TyDI: Ar, Fi, Ja, Koand Ru.
Revised: [Table] For zero-shot experiments, the training examples are derived from MKQA (Longpre et al., 2020), which consists of 10k queries selected from NQ, human translated into 25 additional languages, ﬁve of which overlap with XOR-TyDI: Ar, Fi, Ja, Ko and Ru.

Original: We construct training data (triples) from2,037 queries translated into these ﬁve languagesfor which there are corresponding positive and negative passages in the OpenNQ dataset.
Revised: We construct training data (triples) from 2,037 queries translated into these ﬁve languages for which there are corresponding positive and negative passages in the NQ dataset.

Original: For each ofthe ﬁve languages, there are 519k triples for a totalof 2.6M triples.
Revised: For each of the ﬁve languages, there are 519k triples for a total of 2.6M triples.

Original: We set aside 200 queries translatedinto the 5 languages for a total of 1,000 queries asa development set.
Revised: We set aside 200 queries translated into the 5 languages for a total of 1,000 queries as a development set.

Original: We remove all MKQA queriesfrom the OpenNQ training data for these experiments.
Revised: We remove all MKQA queries from the NQ training data for these experiments.

Original: The CLIR baseline for our experiments is a
Revised: The CLIR baseline for our experiments is a

Original: Col BERT model with an XLM - R PLM , which we ﬁrst ﬁne-tune with 17.5M NQ examples for one epoch
Revised: Col BERT model with an XLM - R PLM , which we ﬁrst ﬁne-tune with 17.5M NQ examples for one epoch

Original: and then 2.9M XOR-TyDi triples for ﬁve epochs.
Revised: and then 2.9M XOR-TyDi triples for ﬁve epochs.

Original: Our student model is initialized with the parameter weights of the baseline, and is further ﬁne-tuned using the two KD objectives.
Revised: Our D R .D ECR model is initialized with the parameter weights of the baseline, and is further ﬁne-tuned using the two KD objectives.

Original: The monolingual teacher model—also a Col BERT model running on top of the pre-trained XLM - R —is trained with only the 17.5M NQ triples for one epoch.
Revised: The monolingual teacher model—also a Col BERT model running on top of the pre-trained XLM - R —is trained with only the 17.5M NQ triples for one epoch.

Original: A. Model Selection
Revised: A. Model Selection

Original: All the models were trained with single Nvidia
Revised: All the models were trained with single Nvidia

Original: A100 GPU.
Revised: A100 GPU.

Original: The longest training time for a single model was less than 200 hours.
Revised: The longest training time for a single model was less than 200 hours.

Original: Following are the ﬁnal hyperparameter conﬁgurations of our different models.
Revised: Following are the ﬁnal hyperparameter conﬁgurations of our different models.

Original: They were selected based on the respective validation sets performance.
Revised: They were selected based on the respective validation sets performance.

Original: Hyperparameter

Original: Value

Original: Standard ColBERT hyperparameters: batch size

Original: [Table]

Original: steps

Original: [Figure]

Original: [Table]

Original: dim 128 query

Original: [Table]

Original: [Table]

Original: Target-domain supervision

Original: Baseline model: lr (NQ) 1.5e-6 lr (XOR)

Original: [-6e-
-]
Revised: [-6e-
-]

Original: [-# Epochs (NQ) 
-]
Revised: [-# Epochs (NQ) 
-]

Original: # Epochs (XOR)

Original: [Table]

Original: A. R@2kt Results

Original: In Table

Original: [Table]

Original: we present results for R@2kt which evaluates a smaller subset of the top retrievals than

Original: R@5kt.

Original: The general pattern in these results is very

Original: [Table]

Original: [Table]

Original: similar to that for R@5kt: the student model closesmuch of the performance gap between the teacherand the baseline in both settings.

Original: Table 5 also showsthe R@2kt results of the ablation study, which isagain similar to the R@5kt results of Table 2.

Original: A. Results on Individual Languages

Original: Table 6 compares the performance (R@5kt) of our

Original: KD student with that of the baseline CLIR model oneach individual language.

Original: In both settings, i.e. , withand without target-domain supervision, the studentconsistently outperforms the baseline, with largegains across all languages.

Original: These results point tothe robustness of our approach, stemming from theindividual strengths of MT , English IR and KD .

Original: A. Leaderboard Submission

Original: We anonymously submitted the ofﬁcial XOR-TyDitest set predictions of the KD student of Table 1row 4 to the XOR-TyDi leaderboard 4 , under themodel name CHIMAERAS B.

Original: Since our parallelcorpus extraction process relies on in-house sourcecode that is not publicly available, we submittedto the “Systems using External APIs” category.

Original: Itshould be noted here though that our approach isnot dependent on any speciﬁc extraction algorithm;

Original: [Table]

Original: we used an existing parallel corpus originally created for a different purpose simply for convenience.

Original: CHIMAERAS B, at the time of this writing, is the best single-model system on the leaderboard, outperformed only by a Col BERT ensemble that also uses MT .

Original: Crucially, all other submitted systems under the External APIs category rely on MT at decoding time, avoiding which is one of the primary goals of our work.

Original: A. Qualitative Analysis
Revised: A. Qualitative Analysis

Original: To ﬁnd out what exact weaknesses of the baseline model the proposed method helps to address, we examine thirty random zero-shot test examples where the baseline fails to retrieve the correct answer in the top 5k tokens, but the KD student succeeds within the top 3 passages.
Revised: To ﬁnd out what exact weaknesses of the baseline model the proposed method helps to address, we examine thirty random zero-shot test examples where the baseline fails to retrieve the correct answer in the top 5k tokens, but D R .D ECR succeeds within the top 3 passages.

Original: We show four examples in Table 7 with human translations of the original non-English questions.
Revised: We show four examples in Table 6 with human translations of the original non-English questions.

Original: The vast majority of our observed cases are related to weak cross-lingual encoding on the baseline model’s part, where at least one important non-English word/entity in the question seems to be incorrectly matched with a similar but different English entity in the passage (e.g., the name of a different place).
Revised: The vast majority of our observed cases are related to weak cross-lingual encoding on the baseline model’s part, where at least one important non-English word/entity in the question seems to be incorrectly matched with a similar but different English entity in the passage (e.g., the name of a different place).

Original: For the Korean,
Revised: For the Korean,

Original: Russian and Arabic queries in the table, we observe the presence of such topically similar entities ( e.g. , microwave ↔ gamma-ray , Germany ↔ places in
Revised: Russian and Arabic queries in the table, we observe the presence of such topically similar entities ( e.g. , microwave ↔ gamma-ray , Germany ↔ places in

Original: North America).
Revised: North America).

Original: Much more rarely, we see cases similar to the Japanese query where the retrieved passage is completely off-topic.
Revised: Much more rarely, we see cases similar to the Japanese query where the retrieved passage is completely off-topic.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Who
Revised: Who

Original: [Table]
Revised: [Table]

Original: Baseline Top Passage: History of gamma-ray burst research The history of gamma-ray began with the serendipi- tous detection of a gamma-ray burst (GRB) on July 2, 1967, by the U.S. Vela satellites.
Revised: Baseline Top Passage: History of gamma-ray burst research The history of gamma-ray began with the serendipi- tous detection of a gamma-ray burst (GRB) on July 2, 1967, by the U.S. Vela satellites.

Original: After these satellites detected ﬁfteen other GRBs, Ray Klebesadel of the Los Alamos National Laboratory published the ﬁrst paper on the subject, "Observations of Gamma-Ray Bursts of Cosmic Origin".
Revised: After these satellites detected ﬁfteen other GRBs, Ray Klebesadel of the Los Alamos National Laboratory published the ﬁrst paper on the subject, "Observations of Gamma-Ray Bursts of Cosmic Origin".

Original: As more and more research was done on these mysterious events, hundreds of models were developed in an attempt to explain their origins.
Revised: As more and more research was done on these mysterious events, hundreds of models were developed in an attempt to explain their origins.

Original: Gamma-ray bursts were discovered in the late 1960s by the U.S. Vela nuclear test detection satellites.
Revised: Gamma-ray bursts were discovered in the late 1960s by the U.S. Vela nuclear test detection satellites.

Original: The Velas were built
Revised: The Velas were built

Original: KD Student Top Passage: project to date, it consists of more than 66 dishes and was built in an international collaboration by Europe, North America, East Asia and Chile.
Revised: D R .D ECR Top Passage: project to date, it consists of more than 66 dishes and was built in an international collaboration by Europe, North America, East Asia and Chile.

Original: A major recent focus of microwave radio astronomy has been mapping the cosmic microwave background radiation (CMBR) discovered in 1964 by radio astronomers Arno Penzias and Robert Wilson.
Revised: A major recent focus of microwave radio astronomy has been mapping the cosmic microwave background radiation (CMBR) discovered in 1964 by radio astronomers Arno Penzias and Robert Wilson.

Original: This faint background radiation, which ﬁlls the universe and is almost the same in all directions, is "relic radiation" from the Big Bang, and is one of the few sources of information about conditions in the early universe.
Revised: This faint background radiation, which ﬁlls the universe and is almost the same in all directions, is "relic radiation" from the Big Bang, and is one of the few sources of information about conditions in the early universe.

Original: Due to the expansion and thus cooling of the
Revised: Due to the expansion and thus cooling of the

Original: Russian Query :
Revised: Russian Query :

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Translation
Revised: Translation

Original: [Table]
Revised: [Table]

Original: What
Revised: What

Original: [Table]
Revised: [Table]

Original: did the
Revised: did the

Original: [Table]
Revised: [Table]

Original:  take partin
Revised: Aurora take part

Original: [Table]
Revised: [Table]

Original: Baseline Top Passage: Battle of the Arar
Revised: Baseline Top Passage: Battle of the Arar

Original: The Battle of the Arar was fought between the migrating tribes of the Helvetii, and four Roman legions (Legions VII, VIII, IX "Hispana" and X "Equestris"), under the command of
Revised: The Battle of the Arar was fought between the migrating tribes of the Helvetii, and four Roman legions (Legions VII, VIII, IX "Hispana" and X "Equestris"), under the command of

Original: Gaius Julius Caesar, in 58 BC.
Revised: Gaius Julius Caesar, in 58 BC.

Original: This was the ﬁrst major battle of the Gallic Wars.
Revised: This was the ﬁrst major battle of the Gallic Wars.

Original: The Helvetii were a tribe that originated from what is now Switzerland.
Revised: The Helvetii were a tribe that originated from what is now Switzerland.

Original: Just prior to the battle with Caesar, they had commenced on a mass migration through Roman Gaul towards the Atlantic coast.
Revised: Just prior to the battle with Caesar, they had commenced on a mass migration through Roman Gaul towards the Atlantic coast.

Original: At Geneva, the Romans destroyed the wooden bridge across the Rhone and constructed 19 mi of fortiﬁcations.
Revised: At Geneva, the Romans destroyed the wooden bridge across the Rhone and constructed 19 mi of fortiﬁcations.

Original: KD Student Top Passage: ordered back to Asia as part of the Russian 2nd Paciﬁc Squadron, a collection formed from the Russian Baltic Fleet, under the command of Vice-Admiral Zinovy Rozhestvensky.
Revised: D R .D ECR Top Passage: ordered back to Asia as part of the Russian 2nd Paciﬁc Squadron, a collection formed from the Russian Baltic Fleet, under the command of Vice-Admiral Zinovy Rozhestvensky.

Original: On the way to the
Revised: On the way to the

Original: Far East, "Aurora" sustained light damage from confused friendly ﬁre, which killed the ship’s chaplain and a sailor, in the Dogger Bank incident.
Revised: Far East, "Aurora" sustained light damage from confused friendly ﬁre, which killed the ship’s chaplain and a sailor, in the Dogger Bank incident.

Original: On 27 and 28 May 1905 "Aurora" took part in the Battle of Tsushima, along with the rest of the Russian squadron.
Revised: On 27 and 28 May 1905 "Aurora" took part in the Battle of Tsushima, along with the rest of the Russian squadron.

Original: During the battle her captain, Captain 1st rank Eugene R. Yegoryev, and 14 crewmen were killed.
Revised: During the battle her captain, Captain 1st rank Eugene R. Yegoryev, and 14 crewmen were killed.

Original: The executive ofﬁcer, Captain of 2nd rank
Revised: The executive ofﬁcer, Captain of 2nd rank

Original: Japanese Query : [Table] 誰
Revised: Japanese Query : 抑 された 記 憶 を 提 唱 したのは 誰

Original: English Translation : Who proposed repressed memory Answer : Sigmund Freud
Revised: English Translation : Who proposed repressed memory Answer : Sigmund Freud

Original: Baseline Top Passage: 14 on the charts on another reissue at the end of 1979, and is now regarded as the Moody Blues signature song by British audiences.
Revised: Baseline Top Passage: 14 on the charts on another reissue at the end of 1979, and is now regarded as the Moody Blues signature song by British audiences.

Original: In the US, "Nights in White Satin" did not make the "Billboard"
Revised: In the US, "Nights in White Satin" did not make the "Billboard"

Original: Hot 100 in 1968, although it reached No. 2 on re-release in 1972; "Tuesday Afternoon" was more successful on initial release stateside, peaking at No. 24 on the Billboard Hot 100.
Revised: Hot 100 in 1968, although it reached No. 2 on re-release in 1972; "Tuesday Afternoon" was more successful on initial release stateside, peaking at No. 24 on the Billboard Hot 100.

Original: The 1968 follow-up LP, "In Search of the
Revised: The 1968 follow-up LP, "In Search of the

Original: Lost Chord" included "Legend of a Mind", a song written by Ray Thomas in tribute to LSD guru Timothy Leary which encompassed a ﬂute
Revised: Lost Chord" included "Legend of a Mind", a song written by Ray Thomas in tribute to LSD guru Timothy Leary which encompassed a ﬂute

Original: KD Student Top Passage: literary example of traumatic memory that has been repressed by an otherwise healthy individual, and then recovered."
Revised: D R .D ECR Top Passage: literary example of traumatic memory that has been repressed by an otherwise healthy individual, and then recovered."

Original: Pope claimed that no entrant had satisﬁed the criteria.
Revised: Pope claimed that no entrant had satisﬁed the criteria.

Original: Ross Cheit, a political scientist at Brown University, cited Nina, a 1786 opera by the French composer Nicolas Dalayrac.
Revised: Ross Cheit, a political scientist at Brown University, cited Nina, a 1786 opera by the French composer Nicolas Dalayrac.

Original: The concept of repressed memory originated with Sigmund Freud in his 1896 essay "Zur Ätiologie der Hysterie" ("On the etiology of hysteria").
Revised: The concept of repressed memory originated with Sigmund Freud in his 1896 essay "Zur Ätiologie der Hysterie" ("On the etiology of hysteria").

Original: One of the studies published in his essay involved a young woman by the name of Anna O. Among her many ailments, she suffered from stiff paralysis on the right side of her
Revised: One of the studies published in his essay involved a young woman by the name of Anna O. Among her many ailments, she suffered [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: only exception being the Japanese query where the issue is a weaker understanding of the passage content.