Original: Unsupervised Cross-Task Generalization via Retrieval Augmentation
Revised: Unsupervised Cross-Task Generalization via Retrieval Augmentation

Original: Abstract
Revised: Abstract

Original: Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all.
Revised: Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all.

Original: In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting.
Revised: In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting.

Original: We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization.
Revised: We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization.

Original: ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking.
Revised: ReCross is a straightforward yet effective retrieval method that combines both efﬁcient dense retrieval and effective pair-wise reranking.

Original: Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.
Revised: Our results and analysis show that it signiﬁcantly outperforms both non-retrieval methods and other baseline methods.

Original:  Introduction
Revised:  Introduction

Original: Advances in pre-training techniques for large language models (LMs) have considerably improved natural language processing (NLP) models on various important tasks via fine-tuning with labeled data.
Revised: Advances in pre-training techniques for large language models (LMs) have considerably improved natural language processing (NLP) models on various important tasks via ﬁne-tuning with labeled data.

Original: While these fine-tuned models are impressive in their target tasks, they can hardly generalize to unseen tasks.
Revised: While these ﬁne-tuned models are impressive in their target tasks, they can hardly generalize to unseen tasks.

Original: This thus makes it difficult to approach the general linguistic intelligence that we ultimately want an NLP model to enjoy.
Revised: This thus makes it difﬁcult to approach the general linguistic intelligence that we ultimately want an NLP model to enjoy.

Original: A promising avenue is to train a massively multi-task model that learns a large set of NLP tasks.
Revised: A promising avenue is to train a massively multi-task model that learns a large set of NLP tasks.

Original: However, in real-world applications, users often expect a multi-task NLP model can also perform unseen tasks that they are interested in.
Revised: However, in real-world applications, users often expect a multi-task NLP model can also perform unseen tasks that they are interested in.

Original: These users may only be able to provide a few unlabeled examples (i e , the input-only data) of the target tasks with natural-language instructions.
Revised: These users may only be able to provide a few unlabeled examples (i e., the input-only data) of the target tasks with natural-language instructions.

Original: How can we generalize the multi-task model to unseen tasks without labels?
Revised: How can we generalize the multi-task model to unseen tasks without labels?

Original: This desirable ability is dubbed “unsupervised cross-task generalization.”
Revised: This desirable ability is dubbed “unsupervised cross-task generalization.”

Original: Recent studies show that multi-task prompted training makes language models better in cross-task generalization, especially when natural-language instructions are used for formatting the training data (Ye et al., 2021; Sanh et al., 2021; Wei et al., 2021).
Revised: Recent studies show that multi-task prompted training makes language models better in cross-task generalization, especially when natural-language instructions are used for formatting the training data (Ye et al., 2021; Sanh et al., 2021; Wei et al., 2021).

Original: The general recipe is to first fine-tune a textto-text language model such as T5 (Raffel et al., 2020) on a multi-task mixture of diverse NLP datasets that are converted to sequence-to-sequence formats.
Revised: The general recipe is to ﬁrst ﬁne-tune a textto-text language model such as T5 (Raffel et al., 2020) on a multi-task mixture of diverse NLP datasets that are converted to sequence-to-sequence formats.

Original: We use the term upstream learning to refer to this multi-task training stage.
Revised: We use the term upstream learning to refer to this multi-task training stage.

Original: Given a target task that is unseen during upstream learning, we want the upstream multi-task model to also perform well on it via reusing the previously acquired knowledge.
Revised: Given a target task that is unseen during upstream learning, we want the upstream multi-task model to also perform well on it via reusing the previously acquired knowledge.

Original: FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021)
Revised: FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021)

Original: both use natural language (NL) instructions as prompts to reformat the data of various NLP tasks for upstream learning and generalization.
Revised: both use natural language (NL) instructions as prompts to reformat the data of various NLP tasks for upstream learning and generalization.

Original: Their results suggest that NL instructions are keys to unsupervised cross-task generalization.
Revised: Their results suggest that NL instructions are keys to unsupervised cross-task generalization.

Original: Despite of the exciting results from Wei et al.
Revised: Despite of the exciting results from Wei et al.

Original: and Sanh et al.
Revised: and Sanh et al.

Original: (2021), their studies are limited to the analysis of the task generalization performance of the frozen, target-agnostic upstream models (i.e., FLAN and T0).
Revised: (2021), their studies are limited to the analysis of the task generalization performance of the frozen, target-agnostic upstream models (i.e., FLAN and T0).

Original: We argue that the generalization performance can be further improved if we can exploit the unlabeled data of target tasks as hints for adjusting the upstream model to a more upstream tasks multi-task
Revised: We argue that the generalization performance can be further improved if we can exploit the unlabeled data of target tasks as hints for adjusting the upstream model to a more upstream tasks multi-task

Original: NLP Model
Revised: NLP Model

Original: Unsupervised
Revised: Unsupervised

Original: Cross-Task
Revised: Cross-Task

Original: Generalization 𝓤 𝒊 : an unseen task w/ a few
Revised: Generalization ! ! : an unseen task w/ a few

Original: [Figure]
Revised: [Figure]

Original: dedicated, target-aware model.
Revised: dedicated, target- model.

Original: Intuitively, the upstream examples thatshare similar skills with thetarget task should help the task generalization if theupstream model could recapthese skills via retrieving.
Revised: Intuitively, the upstream examples that the if the model could recap via retrieving.

Original: Motivated by this idea, we propose to further improve the cross-task generalization ability of upstream models via retrieval augmentation from the upstream data.
Revised: Motivated by this idea, we propose to further improve the cross-task generalization ability of upstream models via retrieval augmentation from the upstream data.

Original: The key challenge of such retrieval augmentation is to predict the example-level utility for cross-task generalization, which we introduce with details in Sec. 2.
Revised: The key challenge of such retrieval augmentation is to predict the example-level utility for cross-task generalization, which we introduce with details in Sec. 2.

Original: To address the challenges, we present a two-stage retrieval-augmentation framework, ReCross, for unsupervised cross-task generalization in Section 3.
Revised: To address the challenges, we present a two-stage retrieval-augmentation framework, ReCross, for unsupervised cross-task generalization in Section 3.

Original: Specifically, we pre-compute a dense index by encoding all upstream data as dense vectors.
Revised: Speciﬁcally, we pre-compute a dense index by encoding all upstream data as dense vectors.

Original: Given a set of unlabeled examples, we first use them to retrieve an initial set of upstream data by using encoded queries to efficiently search over the dense index.
Revised: Given a set of unlabeled examples, we ﬁrst use them to retrieve an initial set of upstream data by using encoded queries to efﬁciently search over the dense index.

Original: Then, we apply the reranking module to carefully analyze the utility of each candidate example.
Revised: Then, we apply the reranking module to carefully analyze the utility of each candidate example.

Original: To get such a reranker, we finetune a cross-encoder model with distant supervision mined by a novel algorithm.
Revised: To get such a reranker, we ﬁnetune a cross-encoder model with distant supervision mined by a novel algorithm.

Original: Finally, we take top-ranking retrieved data to fine-tune the upstream model for a few steps and use this updated model for inference on the target task in the future (i.e., the retrieval augmentation and model update is a one-time procedure for each unseen task).
Revised: Finally, we take top-ranking retrieved data to ﬁne-tune the upstream model for a few steps and use this updated model for inference on the target task in the future (i.e., the retrieval augmentation and model update is a one-time procedure for each unseen task).

Original: To more efficiently evaluate generalization methods without losing the generality, we train a variant of T0-like models, named BART0, which has comparable performance with T0-3B yet is 8x smaller.
Revised: To more efﬁciently evaluate generalization methods without losing the generality, we train a variant of T0-like models, named BART0, which has comparable performance with T0-3B yet is 8x smaller.

Original: Our extensive experiments show that the proposed ReCross outperforms the baseline methods by a large margin.
Revised: Our extensive experiments show that the proposed ReCross outperforms the baseline methods by a large margin.

Original: For example, ReCross improves the non-retrieval methods by 4 points on the overall performance of 10 target tasks and similarly on a few BigBench tasks.
Revised: For example, ReCross improves the non-retrieval methods by 4 points on the overall performance of 10 target tasks and similarly on a few BigBench tasks.

Original: We also analyze the distribution of the retrieved data to understand the behavior of retrieval-augmentation methods better and find that ReCross has a very different distribution compared to semantic retrieval baselines.
Revised: We also analyze the distribution of the retrieved data to understand the behavior of retrieval-augmentation methods better and ﬁnd that ReCross has a very different distribution compared to semantic retrieval baselines.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Massively Multi-Task Language Models.
Revised: Massively Multi-Task Language Models.

Original: To build a general NLP model that can serve a wide range of real-world downstream applications, it is important to train a massively multi-task upstream model.
Revised: To build a general NLP model that can serve a wide range of real-world downstream applications, it is important to train a massively multi-task upstream model.

Original: We assume there are N different upstream tasks (e.g., sentiment analysis of IMDB reviews), dubbed as {T 1 , . . .
Revised: We assume there are N different upstream tasks (e.g., sentiment analysis of IMDB reviews), dubbed as {T 1 , . . .

Original: , T N } .
Revised: , T N } .

Original: We use D to denote the collection of all labeled data for these upstream tasks (i.e., the upstream data ), which are then used for training a massive multi-task model M (e.g., BART, T5, and other Transformer-based models).
Revised: We use D to denote the collection of all labeled data for these upstream tasks (i.e., the upstream data ), which are then used for training a massive multi-task model M (e.g., BART, T5, and other Transformer-based models).

Original: The datasets of these upstream tasks are all converted to a shared text-to-text format using natural-language instruction templates such as
Revised: The datasets of these upstream tasks are all converted to a shared text-to-text format using natural-language instruction templates such as

Original: PromptSource (Bach et al., 2022) to reformat data of different NLP tasks.
Revised: PromptSource (Bach et al., 2022) to reformat data of different NLP tasks.

Original: This pipeline has become a common approach, adopted by several recent massive multi-task models for NLP, such as T0 (Sanh et al., 2021), FLAN (Wei et al., 2021), and CrossFit (Ye et al., 2021).
Revised: This pipeline has become a common approach, adopted by several recent massive multi-task models for NLP, such as T0 (Sanh et al., 2021), FLAN (Wei et al., 2021), and CrossFit (Ye et al., 2021).

Original: Unsupervised Cross-Task Generalization.
Revised: Unsupervised Cross-Task Generalization.

Original: In real-world scenarios, it is very common that users to want a general multi-task model to perform tasks of their interest, even if their target tasks are never seen before by the upstream model.
Revised: In real-world scenarios, it is very common that users to want a general multi-task model to perform tasks of their interest, even if their target tasks are never seen before by the upstream model.

Original: For these unseen target tasks, users usually can provide only a few unlabeled examples (i.e., the input-only data) of them for specifying the task instructions.
Revised: For these unseen target tasks, users usually can provide only a few unlabeled examples (i.e., the input-only data) of them for specifying the task instructions.

Original: This is the reason why we need to study how to generalize a multi-task LM to unseen tasks with only a few
Revised: This is the reason why we need to study how to generalize a multi-task LM to unseen tasks with only a few

Original: [Figure]
Revised: [Figure]

Original: Reranker
Revised: Reranker

Original: [Figure]
Revised: [Figure]

Original: 𝑅′ 𝑅 t o p K 𝑄 ! 
Revised: t o p K # ! "

Original: 𝐷 Task Instances distant sup.
Revised: Task Instances distant sup.

Original: Final
Revised: Final

Original: RetrievedData
Revised: RetrievedData

Original: Alg.query cand.
Revised: Alg.query cand.

Original: Reranker pair-wise score
Revised: Reranker pair-wise score

Original: [Figure]
Revised: [Figure]

Original: learn
Revised: learn

Original: [Figure]
Revised: [Figure]

Revised: unlabeled examples, i.e., unsupervised cross-task generalization .

Revised: For instance, in Fig.

Revised: 1, the unseen task U

Revised: i is a coreference-resolution task that is not covered by the upstream training (the top right in Fig. 1).

Revised: We have only a few inputs for it as the “ hints ” for cross-task generalization, which we call query examples Q i .

Revised: Our objective is to use the query examples Q i to enhance the performance of upstream model M on the unseen task U i .

Revised: For evaluating such unsupervised cross-task generalization methods, we test the enhanced model with a held-out labeled data of each target task

Original: Challenges.
Revised: Challenges.

Original: Standard fine-tuning approaches (with or without meta-learning designs) for few-shot cross-task generalization are not feasible here.
Revised: Standard ﬁne-tuning approaches (with or without meta-learning designs) for few-shot cross-task generalization are not feasible here.

Original: We have to adjust the upstream model based on only a few input-only examples for the unseen task.
Revised: We have to adjust the upstream model based on only a few input-only examples for the unseen task.

Original: Intuitively, upstream examples that share similar skills with the target task
Revised: Intuitively, upstream examples that share similar skills with the target task

Original: U i should be more beneficial than other upstream data.
Revised: U i should be more beneﬁcial than other upstream data.

Original: Thus, one naive idea is to first estimate the utility of each upstream example for U i and then re-train a dedicated model M i via a weighted learning method (e.g., examples of more utility are trained with larger loss).
Revised: Thus, one naive idea is to ﬁrst estimate the utility of each upstream example for U i and then re-train a dedicated model M i via a weighted learning method (e.g., examples of more utility are trained with larger loss).

Original: However, such a target-aware weighted re-training method cannot scale, because the upstream data is usually very large and there can be a large number of unseen tasks from users in real-world applications.
Revised: However, such a target-aware weighted re-training method cannot scale, because the upstream data is usually very large and there can be a large number of unseen tasks from users in real-world applications.

Original: In addition, it is particularly challenging to estimate the utility scores of upstream data for a given unseen task, as we do not have ground-truth annotations for learning this.
Revised: In addition, it is particularly challenging to estimate the utility scores of upstream data for a given unseen task, as we do not have ground-truth annotations for learning this.

Original: Although there are some existing studies on task-to-task relatedness and transferability (Vu et al., 2020; Lange et al., 2021; Padmakumar et al., 2022), most of them are not designed for unsupervised settings and few are done with multi-task (prompted) upstream models.
Revised: Although there are some existing studies on task-to-task relatedness and transferability (Vu et al., 2020; Lange et al., 2021; Padmakumar et al., 2022), most of them are not designed for unsupervised settings and few are done with multi-task (prompted) upstream models.

Original: Moreover, these prior analyses are mainly limited to the task-level analysis and they may not directly generalize to studying example-level utility, which is particularly important for the problem setup of this work.
Revised: Moreover, these prior analyses are mainly limited to the task-level analysis and they may not directly generalize to studying example-level utility, which is particularly important for the problem setup of this work.

Original:  ReCross: Retrieval Augmentation for Cross-Task Generalization 3.1 Overview
Revised:  ReCross: Retrieval Augmentation for Cross-Task Generalization 3.1 Overview

Original: To address the above challenges for unsupervised cross-task generalization, we propose a retrievalaugmentation method named ReCross.
Revised: To address the above challenges for unsupervised cross-task generalization, we propose a retrievalaugmentation method named ReCross.

Original: The ReCross method is also based on the simple idea that we should exploit the upstream examples that enjoy better utility for a given unseen target task.
Revised: The ReCross method is also based on the simple idea that we should exploit the upstream examples that enjoy better utility for a given unseen target task.

Original: Instead of costly re-training from scratch, our method first retrieves a small subset of the upstream data for each unseen task.
Revised: Instead of costly re-training from scratch, our method ﬁrst retrieves a small subset of the upstream data for each unseen task.

Original: It then uses them to efficiently fine-tune the upstream model such that the updated model is generalized.
Revised: It then uses them to efﬁciently ﬁne-tune the upstream model such that the updated model is generalized.

Original: This can ensure scalability to a great extent and benefit upstream models from re-learning target-specific acquired knowledge for cross-task generalization.
Revised: This can ensure scalability to a great extent and beneﬁt upstream models from re-learning target-speciﬁc acquired knowledge for cross-task generalization.

Original: Ideally, we aim to retrieve the upstream examples that are the most beneficial ones for generalizing the upstream model towards a particular unseen task — ranking the upstream data by their example-level utility.
Revised: Ideally, we aim to retrieve the upstream examples that are the most beneﬁcial ones for generalizing the upstream model towards a particular unseen task — ranking the upstream data by their example-level utility.

Original: To achieve this goal while preserving the efficiency, we first use the query examples to retrieve initial candidates via efficient maximum inner product search (MIPS) over a dense index, which consists of embedding vectors of all upstream examples (Section 3.2).
Revised: To achieve this goal while preserving the efﬁciency, we ﬁrst use the query examples to retrieve initial candidates via efﬁcient maximum inner product search (MIPS) over a dense index, which consists of embedding vectors of all upstream examples (Section 3.2).

Original: Based on the candidates from dense retrieval, we learn a reranking module for further improving the retrieval results (Section 3.3).
Revised: Based on the candidates from dense retrieval, we learn a reranking module for further improving the retrieval results (Section 3.3).

Original: The reranker is based on the cross-encoder architecture that takes a query-candidate pair of examples and outputs a more curated score of utility.
Revised: The reranker is based on the cross-encoder architecture that takes a query-candidate pair of examples and outputs a more curated score of utility.

Original: Recall that we do not have any annotation for such example-level utility scores, and the only allowed resources are the upstream data and model.
Revised: Recall that we do not have any annotation for such example-level utility scores, and the only allowed resources are the upstream data and model.

Original: Therefore, we propose an algorithm to mine distant supervision from the upstream data for learning the reranker (Section 3.4).
Revised: Therefore, we propose an algorithm to mine distant supervision from the upstream data for learning the reranker (Section 3.4).

Original: The overview of ReCross is shown in Fig.
Revised: The overview of ReCross is shown in Fig.

Original: 3.2 Dense Retrieval
Revised: 3.2 Dense Retrieval

Original: To efficiently estimate the example-level utility for generalization, we propose to first employ a dense retrieval module that ensures high scalability.
Revised: To efﬁciently estimate the example-level utility for generalization, we propose to ﬁrst employ a dense retrieval module that ensures high scalability.

Original: Specifically, we build a matrix D ∈ R | D |× d , where each upstream example in D is encoded with a dense vector.
Revised: Speciﬁcally, we build a matrix D 2 R | D | ⇥ d , where each upstream example in D is encoded with a dense vector.

Original: Based on this dense index, we can now estimate the utility of an upstream example with its cosine distances to the encoded query examples in Q .
Revised: Based on this dense index, we can now estimate the utility of an upstream example with its cosine distances to the encoded query examples in Q .

Original: That is to say, the upstream examples that are the nearest neighbors of query examples, are more likely to be beneficial for generalizing the upstream model M to the unseen target task.
Revised: That is to say, the upstream examples that are the nearest neighbors of query examples, are more likely to be beneﬁcial for generalizing the upstream model M to the unseen target task.

Original: To retrieve the candidate set R ′ , we use MIPS to search for the top- K examples for each query example in Q , so K = ⌈| R ′ | / | Q |⌉ .
Revised: To retrieve the candidate set R 0 , we use MIPS to search for the top- K examples for each query example in Q , so K = d | R 0 | / | Q | e .

Original: (We introduce the details and other aggregation strategies in Appendix.)
Revised: (We introduce the details and other aggregation strategies in Appendix.)

Original: This dense-retrieval process is very efficient as we pre-compute the upstream index and perform MIPS for querying the candidates over the index on-the-fly during the generalization stage.
Revised: This dense-retrieval process is very efﬁcient as we pre-compute the upstream index and perform MIPS for querying the candidates over the index on-the-ﬂy during the generalization stage.

Original: We use the FAISS library (Johnson et al., 2019) in our implementation.
Revised: We use the FAISS library (Johnson et al., 2019) in our implementation.

Original: Instance embeddings.
Revised: Instance embeddings.

Original: The example encoder is a key component of the dense-retrieval pipeline.
Revised: The example encoder is a key component of the dense-retrieval pipeline.

Original: An ideal example encoder is supposed to represent the underlying skills behind an example such that we can use the distances in the result embedding space to estimate utility for cross-task generalization.
Revised: An ideal example encoder is supposed to represent the underlying skills behind an example such that we can use the distances in the result embedding space to estimate utility for cross-task generalization.

Original: As we do not have annotations of utility scores for training an encoder, one may want to use pre-trained sentence embedding models such as SentenceBERT (Reimers and Gurevych, 2019).
Revised: As we do not have annotations of utility scores for training an encoder, one may want to use pre-trained sentence embedding models such as SentenceBERT (Reimers and Gurevych, 2019).

Original: Our empirical results show that such semantics-based encoders cannot lead to much improvement over random retrieval results.
Revised: Our empirical results show that such semantics-based encoders cannot lead to much improvement over random retrieval results.

Original: We think there are two reasons for this failure.
Revised: We think there are two reasons for this failure.

Original: First, the semantic similarities between examples are not suitable for estimating the utility for generalization.
Revised: First, the semantic similarities between examples are not suitable for estimating the utility for generalization.

Original: Second, the external encoding modules do not reflect the nature of the upstream model which we want to generalize.
Revised: Second, the external encoding modules do not reﬂect the nature of the upstream model which we want to generalize.

Original: To address these two issues, we propose to use the encoding layers of upstream model M for computing the example embeddings.
Revised: To address these two issues, we propose to use the encoding layers of upstream model M for computing the example embeddings.

Original: Without loss of generality, let us assume M to be a text-to-text
Revised: Without loss of generality, let us assume M to be a text-to-text

Original: Transformer that has multiple layers for both encoders and decoders such as BART.
Revised: Transformer that has multiple layers for both encoders and decoders such as BART.

Original: We encode an example by first obtaining the hidden representation of each token at the last encoder layer (i.e., a sequence of token vectors), and then performing mean-pooling over them to get a single dense vector to represent this example.
Revised: We encode an example by ﬁrst obtaining the hidden representation of each token at the last encoder layer (i.e., a sequence of token vectors), and then performing mean-pooling over them to get a single dense vector to represent this example.

Original: By doing this, the produced example embeddings reflect the internal features of the upstream model, which are more relevant to the “thinking process” of the upstream model for the examples instead of the shallow semantic information.
Revised: By doing this, the produced example embeddings reﬂect the internal features of the upstream model, which are more relevant to the “thinking process” of the upstream model for the examples instead of the shallow semantic information.

Original: 3.3 Reranking Module
Revised: 3.3 Reranking Module

Original: Weakness of the dense retrieval.
Revised: Weakness of the dense retrieval.

Original: Although dense retrieval is very efficient thanks to the MIPS support, the retrieval performance is limited by its two major weakness.
Revised: Although dense retrieval is very efﬁcient thanks to the MIPS support, the retrieval performance is limited by its two major weakness.

Original: First, it is a dual-encoder architecture that encodes the candidate example and the query example separately , which ignores informative features behind token-to-token attention across a pair of examples.
Revised: First, it is a dual-encoder architecture that encodes the candidate example and the query example separately , which ignores informative features behind token-to-token attention across a pair of examples.

Original: Second, it is too costly to frequently update the example encoder, which prevents us from learning to refine the retrieval results with distant supervision (if any).
Revised: Second, it is too costly to frequently update the example encoder, which prevents us from learning to reﬁne the retrieval results with distant supervision (if any).

Original: Therefore, we design a re-ranking stage where we train a cross-encoder to further enhance the dense-retrieval results with mined distant supervision (Sec. 3.4).
Revised: Therefore, we design a re-ranking stage where we train a cross-encoder to further enhance the dense-retrieval results with mined distant supervision (Sec. 3.4).

Original: Encoding query-candidate pairs.
Revised: Encoding query-candidate pairs.

Original: The cross-encoder architecture has been widely used in sentencepair classification tasks such as natural language inference and paraphrase detection.
Revised: The cross-encoder architecture has been widely used in sentencepair classiﬁcation tasks such as natural language inference and paraphrase detection.

Original: We here use a cross-encoder to encode the concatenation of a query example and a candidate example.
Revised: We here use a cross-encoder to encode the concatenation of a query example and a candidate example.

Original: Specifically, we fine-tune a RoBERTa
Revised: Speciﬁcally, we ﬁne-tune a RoBERTa

Original: (Liu et al., 2019) model to classify whether an example pair is a positive or negative match.
Revised: (Liu et al., 2019) model to classify whether an example pair is a positive or negative match.

Original: The confidence of classifying such a pair to be positive can thus be used as the utility score of the candidate upstream example for this query example.
Revised: The conﬁdence of classifying such a pair to be positive can thus be used as the utility score of the candidate upstream example for this query example.

Original: On top of this, we then develop a reranking module for further improving retrieval performance as follows.
Revised: On top of this, we then develop a reranking module for further improving retrieval performance as follows.

Original: Scoring paired data.
Revised: Scoring paired data.

Original: To re-rank the initially retrieved data by the dense retriever, we apply the cross-encoder on all pairs of query examples Q and candidate retrieved examples R ′ , producing scores of all | Q | ∗ | R | ′ query-candidate pairs.
Revised: To re-rank the initially retrieved data by the dense retriever, we apply the cross-encoder on all pairs of query examples Q and candidate retrieved examples R 0 , producing scores of all | Q | ⇤ | R | 0 query-candidate pairs.

Original: For each candidate example r ∈ R ′ , we use the average of all cross-encoder scores involving r as its utility score.
Revised: For each candidate example r 2 R 0 , we use the average of all cross-encoder scores involving r as its utility score.

Original: Finally, we take the top- K examples based on this new ranking of candidate examples in R ′ as the final retrieved data R .
Revised: Finally, we take the top- K examples based on this new ranking of candidate examples in R 0 as the ﬁnal retrieved data R .

Original: We use upsampling ratio µ to denote the ratio between R ′ and R , i.e., µ = | R ′ | / | R | .
Revised: We use upsampling ratio µ to denote the ratio between R 0 and R , i.e., µ = | R 0 | / | R | .

Original: 3.4 Mining Distant Supervision for Reranking
Revised: 3.4 Mining Distant Supervision for Reranking

Original: How do we train such a re-ranking module?
Revised: How do we train such a re-ranking module?

Original: Recall that we only have access to the upstream data
Revised: Recall that we only have access to the upstream data

Original: D and must not use any data from the unseen tasks at this stage.
Revised: D and must not use any data from the unseen tasks at this stage.

Original: Inspired by meta-learning works, we propose an algorithm (Alg. 1) to mine distant supervision data for creating a training-as-testing environment for learning the reranker.
Revised: Inspired by meta-learning works, we propose an algorithm (Alg. 1) to mine distant supervision data for creating a training-as-testing environment for learning the reranker.

Original: Our key motivation is to examine the utility scores of candidate examples by assessing the generalization performance of updated models that are fine-tuned with these candidates as if we use them for real unseen tasks.
Revised: Our key motivation is to examine the utility scores of candidate examples by assessing the generalization performance of updated models that are ﬁne-tuned with these candidates as if we use them for real unseen tasks.

Original: Such more realistic estimation of utility scores can thus help us train a reranker to predict.
Revised: Such more realistic estimation of utility scores can thus help us train a reranker to predict.

Original: Algorithm
Revised: Algorithm

Original: Input:M ; D ;T q Output: Z =
Revised: Input: ; D ; q Output

Original:  (Z q , Z p , Z n )
Revised: = (

Original: DT q ←− { x ∈ D | x is anexample of T q } Z q ←− Sample( D T q ) ; H q ←− Sample( D T q ) R 
Revised: D is an of  Sample( D T q ) ; H q  Sample( D T q ) R Z

Original: Z ←− DenseRetrieve( Z q , D ) /
Revised: DenseRetrieve ( Z q , D ) /

Original: * Delete retrieved examples from the same task as queries.
Revised: * Delete retrieved examples from the same task as queries.

Original: R Z ←− R Z .
Revised: R Z  R Z .

Original: discard( D T q ) foreach round do
Revised: discard( D T q ) foreach round do

Original: R Z .
Revised: R Z .

Original: shuffle() / * Split retrieved examples into n groups */ { G 1 , ..., G n } ←− R Z . split() foreach G i in { G 1 , ..., G n } do M ′ ←− M .
Revised: shu ✏ e() /* Split retrieved examples into n groups */ { G 1 , ..., G n }  R Z . split() foreach G i in { G 1 , ..., G n } do M 0  M .

Original: copy()
Revised: copy()

Original: M ′ .
Revised: M 0 .

Original: fine _ tune( G i ) ℓ ←− M ′ .
Revised: ﬁne _ tune( G i ) `  M 0 .

Original: calc _ loss( H q ) foreach x ∈ G i do scores [ x ] .
Revised: calc _ loss( H q ) foreach x 2 G i do scores [ x ] .

Original: append( ℓ ) /
Revised: append( ` ) /

Original: * Score each in the group w/ the loss.
Revised: * Score each in the group w/ the loss.

Original: */ /* Use mean group score as score for single examples */ foreach x ∈ R Z do score [ x ] ←− mean( scores [ x ]) / * Sort R Z by score in increasing order.
Revised: */ /* Use mean group score as score for single examples */ foreach x 2 R Z do score [ x ]  mean( scores [ x ]) /* [Table]

Original: */ R
Revised: */ R

Original: Z . sort(key: score , order: increasing)
Revised: Z . [Table] (key: score , order: increasing)

Original: Z p
Revised: Z p

Original: [Table]
Revised: [Table]

Original: First W items of R Z Z n
Revised: First W items of R Z Z n

Original: [Table]
Revised: [Table]

Original: Last W items of 
Revised: Last W items of R

Original: [Table]
Revised: [Table]

Original: Specifically, we define a data point of such distant supervision as a tuple Z =
Revised: Speciﬁcally, we deﬁne a data point of such distant supervision as a tuple Z =

Original: ( Z q , Z p , Z n ) : 1) Z q is a set of query examples of a particular task T q ; 2) Z p is the set of positive examples from other tasks; 3) Z n is the set of negative examples from other tasks.
Revised: ( Z q , Z p , Z n ) : 1) Z q is a set of query examples of a particular task T q ; 2) Z p is the set of positive examples from other tasks; 3) Z n is the set of negative examples from other tasks.

Original: We expect that Z p is of more utility for generalization than Z n if Z q would be a query set for the target task T q .
Revised: We expect that Z p is of more utility for generalization than Z n if Z q would be a query set for the target task T q .

Original: To this end, we first randomly sample an upstream task T q and use a small subset of its training data as the Z q .
Revised: To this end, we ﬁrst randomly sample an upstream task T q and use a small subset of its training data as the Z q .

Original: Here, we also sample a larger held-out set H q examples of task T q to facilitate utility estimation.
Revised: Here, we also sample a larger held-out set H q examples of task T q to facilitate utility estimation.

Original: Then, we apply the dense retriever using Z q as the query examples and get the retrieval results R Z .
Revised: Then, we apply the dense retriever using Z q as the query examples and get the retrieval results R Z .

Original: This R Z is thus the candidate pool where we create Z p and Z n .
Revised: This R Z is thus the candidate pool where we create Z p and Z n .

Original: That is,
Revised: That is,

Original: Z p ⊂ R Z and Z n ⊂ R Z .
Revised: Z p ⇢ R Z and Z n ⇢ R Z .

Original: We discard examples that are from the T q , so that the generated tuples are closer to the real scenarios where we use the reranker on the query sets of unseen tasks.
Revised: We discard examples that are from the T q , so that the generated tuples are closer to the real scenarios where we use the reranker on the query sets of unseen tasks.

Original: Our criteria to select Z p and Z n from R Z is motivated by the hypothesis that a more suitable set of retrieved examples should improve the performance M on T i after fine-tuning with it.
Revised: Our criteria to select Z p and Z n from R Z is motivated by the hypothesis that a more suitable set of retrieved examples should improve the performance M on T i after ﬁne-tuning with it.

Original: Therefore, we iteratively sample a small subset from R Z , then fine-tune M with it, and finally, use the fine-tuned model to evaluate on Z ′ q .
Revised: Therefore, we iteratively sample a small subset from R Z , then ﬁne-tune M with it, and ﬁnally, use the ﬁne-tuned model to evaluate on Z 0 q .

Original: The performance of such a temporarily fine-tuned model can be seen as the utility score—how well this subset can help generalize M to the unseen task T q .
Revised: The performance of such a temporarily ﬁne-tuned model can be seen as the utility score—how well this subset can help generalize M to the unseen task T q .

Original: Through multiple rounds of such sample-train-test procedures, we can thus score each example in
Revised: Through multiple rounds of such sample-train-test procedures, we can thus score each example in

Original: R Z by taking the average of all test results where it is involved.
Revised: R Z by taking the average of all test results where it is involved.

Original: With such a new ranking of examples in R Z , we take the best W examples as Z p and the worst W as Z n .
Revised: With such a new ranking of examples in R Z , we take the best W examples as Z p and the worst W as Z n .

Original: With such distant supervision, we then can create pair of query-positive instances and query-negative instances via pairing Z q with Z p and Z n respectively.
Revised: With such distant supervision, we then can create pair of query-positive instances and query-negative instances via pairing Z q with Z p and Z n respectively.

Original: Now we can fine-tune a RoBERTa-base model by concatenating each pair and learning a binary-classification objective.
Revised: Now we can ﬁne-tune a RoBERTa-base model by concatenating each pair and learning a binary-classiﬁcation objective.

Original: The output logits of this trained model will be used for the reranking procedure as shown in Sec. 3.3.
Revised: The output logits of this trained model will be used for the reranking procedure as shown in Sec. 3.3.

Original: 3.5 Re-learning via Fine-Tuning with Retrieved Data
Revised: 3.5 Re-learning via Fine-Tuning with Retrieved Data

Original: When we have the final retrieved data R i for a certain query set Q i , we can now enhance the upstream model M for the unseen task U i .
Revised: When we have the ﬁnal retrieved data R i for a certain query set Q i , we can now enhance the upstream model M for the unseen task U i .

Original: We use a small learning rate to continually fine-tune M with the retrieved upstream examples R i for a small number of steps.
Revised: We use a small learning rate to continually ﬁne-tune M with the retrieved upstream examples R i for a small number of steps.

Original: We find that the learning rate has to be very small so that this step can be seen as a natural continuation of the finished upstream training and avoid overfitting the retrieved data.
Revised: We ﬁnd that the learning rate has to be very small so that this step can be seen as a natural continuation of the ﬁnished upstream training and avoid overﬁtting the retrieved data.

Original: We acknowledge that there could be more effective methods to reuse the query examples Q as guidance for fine-tuning, and we leave this as future work.
Revised: We acknowledge that there could be more effective methods to reuse the query examples Q as guidance for ﬁne-tuning, and we leave this as future work.

Original: Please find more discussion on the hyper-parameter selection and configuration in our appendix.
Revised: Please ﬁnd more discussion on the hyper-parameter selection and conﬁguration in our appendix.

Original:  Evaluation
Revised:  Evaluation

Original: In this section, we first introduce the experimental setups, including the task distribution, upstream learning details, and the configurations of the main experiments.
Revised: In this section, we ﬁrst introduce the experimental setups, including the task distribution, upstream learning details, and the conﬁgurations of the main experiments.

Original: We present experimental results and reveal some non-trivial findings with extensive analysis that justify the effectiveness of ReCross.
Revised: We present experimental results and reveal some non-trivial ﬁndings with extensive analysis that justify the effectiveness of ReCross.

Original: 4.1 Evaluating Unsupervised Cross-Task Generalization
Revised: 4.1 Evaluating Unsupervised Cross-Task Generalization

Original: We follow Sanh et al.
Revised: We follow Sanh et al.

Original: (2021) to use the templates from PromptSource (Bach et al., 2022) for converting data of different types of NLP tasks to text-to-text formats.
Revised: (2021) to use the templates from PromptSource (Bach et al., 2022) for converting data of different types of NLP tasks to text-to-text formats.

Original: In total, we have 36 upstream tasks and 10 target unseen tasks for our main experiments.
Revised: In total, we have 36 upstream tasks and 10 target unseen tasks for our main experiments.

Original: The upstream tasks are the same as the ones
Revised: The upstream tasks are the same as the ones

Original: that
Revised: that

Original: the
Revised: the

Original: T0 models used for upstream learning.
Revised: T0 models used for upstream learning.

Original: We follow the evaluation protocol proposed by Sanh et al.
Revised: We follow the evaluation protocol proposed by Sanh et al.

Original: (2021) and select the target tasks that are significantly different from the upstream tasks.
Revised: (2021) and select the target tasks that are signiﬁcantly different from the upstream tasks.

Original: Besides, we also include 5 additional tasks from the BIG-bench project (Srivastava et al., 2022) to create an even more out-of-distribution set of unseen tasks for analysis.
Revised: Besides, we also include 5 additional tasks from the BIG-bench project (Srivastava et al., 2022) to create an even more out-of-distribution set of unseen tasks for analysis.

Original: Metric.
Revised: Metric.

Original: When we apply the natural-language templates for the test examples, we only keep the templates that can be evaluated with an exact match (classification, question answering, answer selection, etc.)
Revised: When we apply the natural-language templates for the test examples, we only keep the templates that can be evaluated with an exact match (classiﬁcation, question answering, answer selection, etc.)

Original: so that it is feasible to use exact-match for evaluating all tasks.
Revised: so that it is feasible to use exact-match for evaluating all tasks.

Original: To allow a smoother grading, our metric also counts the cases when outputs and truths are sub-strings of each other, which we call SoftEM .
Revised: To allow a smoother grading, our metric also counts the cases when outputs and truths are sub-strings of each other, which we call SoftEM .

Original: The only difference between SoftEM and the standard EM is that it also counts the sub-string matches.
Revised: The only difference between SoftEM and the standard EM is that it also counts the sub-string matches.

Original: We observe that sometimes even though T0-like models (including ours) answer the input questions correctly, their raw outputs are not exactly the same as the truth outputs generated by the PromptSource templates.
Revised: We observe that sometimes even though T0-like models (including ours) answer the input questions correctly, their raw outputs are not exactly the same as the truth outputs generated by the PromptSource templates.

Original: In particular, the ground-truth outputs for multiple-choice
Revised: In particular, the ground-truth outputs for multiple-choice

Original: QA tasks are often in the form of “[A/B/C/D]: [answer]”, while the models often only output the id of the correct choice (e.g., “A”) or the text of the answer.
Revised: QA tasks are often in the form of “[A/B/C/D]: [answer]”, while the models often only output the id of the correct choice (e.g., “A”) or the text of the answer.

Original: We also find that the model can output some noise (such as additional punctuation) after the answer (e.g., “True” vs “True.”).
Revised: We also ﬁnd that the model can output some noise (such as additional punctuation) after the answer (e.g., “True” vs “True.”).

Original: The standard
Revised: The standard

Original: EM will discard such matches and cause inaccurate measurements.
Revised: EM will discard such matches and cause inaccurate measurements.

Original: Although SoftEM might add false positives due to substring matches, we found it is very rare according to our manual inspection of the 10 tasks.
Revised: Although SoftEM might add false positives due to substring matches, we found it is very rare according to our manual inspection of the 10 tasks.

Original: Therefore, we choose to use SoftEM for a more precise evaluation.
Revised: Therefore, we choose to use SoftEM for a more precise evaluation.

Original: We report the results with the standard EM in Table 7 that also supports our findings.
Revised: We report the results with the standard EM in Table 7 that also supports our ﬁndings.

Original: 4.2 BART0: Upstream Learning with a Smaller LM
Revised: 4.2 BART0: Upstream Learning with a Smaller LM

Original: The T0(pp) models are all very huge, and the smallest version, T0-3B (3 billion parameters), is still too large to be fine-tuned on popular affordable GPUs.
Revised: The T0(pp) models are all very huge, and the smallest version, T0-3B (3 billion parameters), is still too large to be ﬁne-tuned on popular affordable GPUs.

Original: We need a parameter-efficient alternative that makes the study on cross-task generalization more accessible to a broader community while keeping the generality.
Revised: We need a parameter-efﬁcient alternative that makes the study on cross-task generalization more accessible to a broader community while keeping the generality.

Original: Thus, we fine-tune a BART-large (Lewis et al., 2020a) (0.4 billion parameters) following the recipe of training T0.
Revised: Thus, we ﬁne-tune a BART-large (Lewis et al., 2020a) (0.4 billion parameters) following the recipe of training T0.

Original: Specifically, we sample 50k examples at most from each upstream task to build a large upstream dataset consisting of 1.7 million examples (i.e., | D | = 1.7m), and then we fine-tune a BART-large with 22k steps with this upstream dataset.
Revised: Speciﬁcally, we sample 50k examples at most from each upstream task to build a large upstream dataset consisting of 1.7 million examples (i.e., | D | = 1.7m), and then we ﬁne-tune a BART-large with 22k steps with this upstream dataset.

Original: Finally, we use the fine-tuned checkpoint as our upstream model M and name it BART0 .
Revised: Finally, we use the ﬁne-tuned checkpoint as our upstream model M and name it BART0 .

Original: Surprisingly, we find that BART0 and
Revised: Surprisingly, we ﬁnd that BART0 and

Original: T0-3B have comparable zero-shot performance on the unseen target tasks, even though T0-3B is about 8x larger than BART0.
Revised: T0-3B have comparable zero-shot performance on the unseen target tasks, even though T0-3B is about 8x larger than BART0.

Original: More implementation details are shown in Appendix.
Revised: More implementation details are shown in Appendix.

Original: 4.3 Setup and Configurations
Revised: 4.3 Setup and Conﬁgurations

Original: In our main experiments, we use | Q i | = 16 query examples for each unseen task U i and retrieve | R i | = 512 examples for augmenting BART0.
Revised: In our main experiments, we use | Q i | = 16 query examples for each unseen task U i and retrieve | R i | = 512 examples for augmenting BART0.

Original: In the fine-tuning stage, we use a learning rate of 1e-6 and a
Revised: In the ﬁne-tuning stage, we use a learning rate of 1e-6 and a

Original: [Table]
Revised: [Table]

Original: batch size of 4 to continually fine-tune all layers of BART0 for 2 epochs.
Revised: batch size of 4 to continually ﬁne-tune all layers of BART0 for 2 epochs.

Original: As for re-ranking, we set the upsampling ratio µ = 2 , meaning that we first retrieve 1024 examples for reranking and use the top 512 ones as the final retrieved data.
Revised: As for re-ranking, we set the upsampling ratio µ = 2 , meaning that we ﬁrst retrieve 1024 examples for reranking and use the top 512 ones as the ﬁnal retrieved data.

Original: To obtain more convincing evaluation results, we average the scores of all target tasks to show the general zero-shot performance.
Revised: To obtain more convincing evaluation results, we average the scores of all target tasks to show the general zero-shot performance.

Original: For each task
Revised: For each task

Original: U
Revised: U

Original: i , we use five different query sets, { Q (1) i , . .
Revised: i , we use ﬁve different query sets, { Q (1) i , . .

Original: , Q (5) i } , to conduct five individual rounds of retrieval , thus resulting in five average scores for all tasks.
Revised: , Q (5) i } , to conduct ﬁve individual rounds of retrieval , thus resulting in ﬁve average scores for all tasks.

Original: To get a comprehensive assessment, we report the mean, std, median, min, and max of these five overall scores in the lower part of Table 1.
Revised: To get a comprehensive assessment, we report the mean, std, median, min, and max of these ﬁve overall scores in the lower part of Table 1.

Original: We present an ablation study on hyper-parameter configurations in Table 3 and include more details in Appendix.
Revised: We present an ablation study on hyper-parameter conﬁgurations in Table 3 and include more details in Appendix

Original: 4.4 Experimental Results
Revised: 4.4 Experimental Results

Original: BART0 vs T0-3B. As mentioned earlier, we find that BART0 is comparable with the much larger
Revised: BART0 vs T0-3B. As mentioned earlier, we ﬁnd that BART0 is comparable with the much larger

Original: T0-3B in terms of their zero-shot performance on our unseen tasks (41.33 vs 40.38).
Revised: T0-3B in terms of their zero-shot performance on our unseen tasks (41.33 vs 40.38).

Original: As we use BART as our base model for testing different retrieval-augmentation methods, its overall performance 40.
Revised: As we use BART as our base model for testing different retrieval-augmentation methods, its overall performance 40.

Original: is what we want retrieval-augmentation methods to beat.
Revised: is what we want retrieval-augmentation methods to beat.

Original: Note that when using BART0 and T0-3B for non-retrieval zero-shot inference, they do not use any information from the query examples, so their mean, median, min, and max are always the same.
Revised: Note that when using BART0 and T0-3B for non-retrieval zero-shot inference, they do not use any information from the query examples, so their mean, median, min, and max are always the same.

Original: Random Retrieval.
Revised: Random Retrieval.

Original: The Random column shows the results when we randomly sample R i from the upstream data D without using any information from Q i . .
Revised: The Random column shows the results when we randomly sample R i from the upstream data D without using any information from Q i . .

Original: SBERT and ReCross † .
Revised: SBERT and ReCross † .

Original: We use SentenceBERT (SBERT) as a strong baseline method to create a dense index of the upstream data, compared with our proposed indexing method, ReCross † (i.e.,
Revised: We use SentenceBERT (SBERT) as a strong baseline method to create a dense index of the upstream data, compared with our proposed indexing method, ReCross † (i.e.,

Original: ReCross without reranking).
Revised: ReCross without reranking).

Original: We can see that ReCross † always outperforms the other methods.
Revised: We can see that ReCross † always outperforms the other methods.

Original: Even its minimum performance in the five rounds ( 42.65 ) is better than the maximum of the SBERT ( 41.76) .
Revised: Even its minimum performance in the ﬁve rounds ( 42.65 ) is better than the maximum of the SBERT ( 41.76) .

Original: Besides, the standard deviation also becomes much smaller (1.61 → 0.68), which means that improvement by the ReCross † is more consistent under different query sets.
Revised: Besides, the standard deviation also becomes much smaller (1.61 ! 0.68), which means that improvement by the ReCross † is more consistent under different query sets.

Original: The SBERT indexing relies mainly on the semantic similarities between a query example and the upstream data.
Revised: The SBERT indexing relies mainly on the semantic similarities between a query example and the upstream data.

Original: Instead, our proposed ReCross † uses the hidden representations inside the upstream model M for representing examples.
Revised: Instead, our proposed ReCross † uses the hidden representations inside the upstream model M for representing examples.

Original: We believe using such an indexing method can better help us find examples that share similar reasoning skills acquired by the upstream model.
Revised: We believe using such an indexing method can better help us ﬁnd examples that share similar reasoning skills acquired by the upstream model.

Original: ReCross =
Revised: ReCross =

Original: ReCross † + Reranking.
Revised: ReCross † + Reranking.

Original: The full version of our ReCross with reranking can further improve the performance substantially on multiple dimensions.
Revised: The full version of our ReCross with reranking can further improve the performance substantially on multiple dimensions.

Original: Both all@mean and median are improved by 1 point from the ReCross † , and the std is also reduced from 0.68 to 0.42.
Revised: Both all@mean and median are improved by 1 point from the ReCross † , and the std is also reduced from 0.68 to 0.42.

Original: The last column ( ∆ ) in Table 1 shows its improvement compared to the base model BART0, and we can see that ReCross consistently outperforms non-retrieval methods (e.g., BART0) by a significant gap.
Revised: The last column (  ) in Table 1 shows its improvement compared to the base model BART0, and we can see that ReCross consistently outperforms non-retrieval methods (e.g., BART0) by a signiﬁcant gap.

Original: To explore the potential benefits of retrieval-augmentation methods such as our ReCross, we also conduct the same experiments on five tasks selected from the BIG-Bench project.
Revised: To explore the potential beneﬁts of retrieval-augmentation methods such as our ReCross, we also conduct the same experiments on ﬁve tasks selected from the BIG-Bench project.

Original: The results are shown in Table 2, where we can see that ReCross still outperforms the non-retrieval methods.
Revised: The results are shown in Table 2, where we can see that ReCross still outperforms the non-retrieval methods.

Original: An interesting case is the movie_dialog task, where the prompt in the template requires a model to output “same” or “different.”
Revised: An interesting case is the movie_dialog task, where the prompt in the template requires a model to output “same” or “different.”

Original: However, both T0-3B and BART0 fail to follow the prompt instruction, and can only output “yes/no.”
Revised: However, both T0-3B and BART0 fail to follow the prompt instruction, and can only output “yes/no.”

Original: Only when we use retrieval-augmentation methods, there are performance improvement on this task.
Revised: Only when we use retrieval-augmentation methods, there are performance improvement on this task.

Original: 4.5 Analysis & More Findings.
Revised: 4.5 Analysis & More Findings.

Original: [Table]
Revised: [Table]

Original: More configurations.
Revised: More conﬁgurations.

Original: We have used a particular configuration in our main experiments that are in Table 1, which is | Q | =16, | R | =512, and | u | =2.
Revised: We have used a particular conﬁguration in our main experiments that are in Table 1, which is | Q | =16, | R | =512, and | u | =2.

Original: In Table 3, we explore more configurations as ablation studies.
Revised: In Table 3, we explore more conﬁgurations as ablation studies.

Original: The “Main Exp.” row refers to the results shown in Table 1, and the configurations of other rows are only changed with one factor at a time.
Revised: The “Main Exp.” row refers to the results shown in Table 1, and the conﬁgurations of other rows are only changed with one factor at a time.

Original: Even using a single query example, ReCross is better than BART0.
Revised: Even using a single query example, ReCross is better than BART0.

Original: However, when increasing the query size to 32, we find that the performance starts to decrease, meaning that there could be an optimal query size for a certain | R | =512.
Revised: However, when increasing the query size to 32, we ﬁnd that the performance starts to decrease, meaning that there could be an optimal query size for a certain | R | =512.

Original: We find that increasing | R | is generally beneficial, while the all@mean decreases when | R | is changed from 512 to 1024, although the max and the median slightly increased.
Revised: We ﬁnd that increasing | R | is generally beneﬁcial, while the all@mean decreases when | R | is changed from 512 to 1024, although the max and the median slightly increased.

Original: Finally, we see that increasing µ increases the std.
Revised: Finally, we see that increasing µ increases the std.

Original: and does not improve the overall performance.
Revised: and does not improve the overall performance.

Original: [Table]
Revised: [Table]

Original: Retrieved

Original: data distribution.
Revised: distribution.

Original: Figure presents the difference between the methods in terms of their retrieved data.
Revised: Figure presents the difference between the methods in terms of their retrieved data.

Original: We draw the distribution of the retrieved data among different upstream tasks for each unseen task individually.
Revised: We draw the distribution of the retrieved data among different upstream tasks for each unseen task individually.

Original: From the heatmap, we can see that ReCross tends to have more dominant retrieved tasks (i.e., darker cells), while
Revised: From the heatmap, we can see that ReCross tends to have more dominant retrieved tasks (i.e., darker cells), while

Original: SBERT’s results are more sparse.
Revised: SBERT’s results are more sparse.

Original: They both can identify that squad is most similar to the adversarial_qa tasks.
Revised: They both can identify that squad is most similar to the adversarial_qa tasks.

Original: Their behaviors are very different too.
Revised: Their behaviors are very different too.

Original: Taking the unseen task winogrande (wngrnd) as an example, we can see that the SBERT retrieves from multiple upstream tasks such as paws-x and cosmosQA , but the ReCross mainly retrieves from social-iqa , wiki-qa , and cos-e .
Revised: Taking the unseen task winogrande (wngrnd) as an example, we can see that the SBERT retrieves from multiple upstream tasks such as paws-x and cosmosQA , but the ReCross mainly retrieves from social-iqa , wiki-qa , and cos-e .

Original: The experimental results in Table 1 show that ReCross produces a better performance than SBERT (i.e., 55.46 vs 52.18), while it is not clear how we can predict such task correlation in advance.
Revised: The experimental results in Table 1 show that ReCross produces a better performance than SBERT (i.e., 55.46 vs 52.18), while it is not clear how we can predict such task correlation in advance.

Original: This suggests that we should explore more about the utility of instances and tasks in future work.
Revised: This suggests that we should explore more about the utility of instances and tasks in future work.

Original: More analysis.
Revised: More analysis.

Original: In the appendix, we further presented some analysis to help understand “how” and “when” the retrieval augmentation works: Table 4, Table 5, Appendix A.1 A.2, and Appendix
Revised: In the appendix, we further presented some analysis to help understand “how” and “when” the retrieval augmentation works: Table 4, Table 5, Appendix A.1 A.2, and Appendix

Original: B. We investigate whether the utility of upstream examples in retrieval augmentation is related to the similarity in terms of the task formats.
Revised: B. We investigate whether the utility of upstream examples in retrieval augmentation is related to the similarity in terms of the task formats.

Original: From Appendix A.1, we found some counterintuitive results.
Revised: From Appendix A.1, we found some counterintuitive results.

Original: For example, if removing MCQA upstream tasks from the upstream index, then the ARC target task can have an even better performance, although it is an MCQA-formatted task.
Revised: For example, if removing MCQA upstream tasks from the upstream index, then the ARC target task can have an even better performance, although it is an MCQA-formatted task.

Original: Thus, we hypothesize that similarity in terms of reasoning types is more important than format similarity for retrieval augmentation.
Revised: Thus, we hypothesize that similarity in terms of reasoning types is more important than format similarity for retrieval augmentation.

Original: After all, the upstream model has been already trained to work with these basic task formats.
Revised: After all, the upstream model has been already trained to work with these basic task formats.

Original: Re-learning the tasks of the same format might lead the model to overfit the seen domains.
Revised: Re-learning the tasks of the same format might lead the model to overﬁt the seen domains.

Original: Additionally, to provide a more concrete analysis, we also present case studies with two specific tasks (CB and SQUADv2) in Appendix B.
Revised: Additionally, to provide a more concrete analysis, we also present case studies with two speciﬁc tasks (CB and SQUADv2) in Appendix B.

Original: Moreover, we conjecture the natural language instructions in the templates are necessary for ReCross to get impressive results.
Revised: Moreover, we conjecture the natural language instructions in the templates are necessary for ReCross to get impressive results.

Original: Therefore, we investigated two ways of perturbing the instructions and monitoring the performance changes in Appendix A.2.
Revised: Therefore, we investigated two ways of perturbing the instructions and monitoring the performance changes in Appendix A.2.

Original: We find it is indeed true that perturbations of the instructions will lead to much worse performance.
Revised: We ﬁnd it is indeed true that perturbations of the instructions will lead to much worse performance.

Original: We believe that a rigorous, principled way of analyzing the correlation between query and retrieval examples will be a great future direction, given the strong evidence that ReCross works so well as such a simple method.
Revised: We believe that a rigorous, principled way of analyzing the correlation between query and retrieval examples will be a great future direction, given the strong evidence that ReCross works so well as such a simple method.

Original:  More Discussion 5.1 Practicality of unsupervised setting.
Revised:  More Discussion 5.1 Practicality of unsupervised setting.

Original: Cost of obtaining task labels The unsupervised setting in the paper does not require any human annotation of labels.
Revised: Cost of obtaining task labels The unsupervised setting in the paper does not require any human annotation of labels.

Original: For some tasks (NLG tasks in particular, e.g., summarization), the expected output (label) are open-ended and possibly lengthy and thus human annotation is much more expensive and time-consuming.
Revised: For some tasks (NLG tasks in particular, e.g., summarization), the expected output (label) are open-ended and possibly lengthy and thus human annotation is much more expensive and time-consuming.

Original: Also, few-shot learning must ask humans to label examples for each new task, and it is thus less practical when there are a large number of emerging tasks from the users.
Revised: Also, few-shot learning must ask humans to label examples for each new task, and it is thus less practical when there are a large number of emerging tasks from the users.

Original: Meanwhile,
Revised: Meanwhile,

Original: ReCross requires only a natural-language task template, which does not require potentially expensive manual annotation or domain expertise.
Revised: ReCross requires only a natural-language task template, which does not require potentially expensive manual annotation or domain expertise.

Original: Scalability & Real-Time response Deploying the ReCross pipeline is a one-time process.
Revised: Scalability & Real-Time response Deploying the ReCross pipeline is a one-time process.

Original: All we need to do is to pre-compute the upstream index with LM and configure the reranker (a simple masked
Revised: All we need to do is to pre-compute the upstream index with LM and conﬁgure the reranker (a simple masked

Original: LM) by running our script.
Revised: LM) by running our script.

Original: In production, once the users input the examples with NL instructions, we do not need to wait for any human annotations anymore, so it is much more efficient in the long run.
Revised: In production, once the users input the examples with NL instructions, we do not need to wait for any human annotations anymore, so it is much more efﬁcient in the long run.

Original: In the scenarios where users only provide one query example and want to get its label from the model, ReCross also shows great performance (i.e., |Q|=1 in Table 1).
Revised: In the scenarios where users only provide one query example and want to get its label from the model, ReCross also shows great performance (i.e., |Q|=1 in Table 1).

Original: It is then impractical to assume there are a few labeled data from the users too in such cases.
Revised: It is then impractical to assume there are a few labeled data from the users too in such cases.

Original: 5.2 Empirical studies
Revised: 5.2 Empirical studies

Original: The unsupervised ReCross performance is comparable to few-shot learning with label annotations.
Revised: The unsupervised ReCross performance is comparable to few-shot learning with label annotations.

Original: In Appendix D.2, we report the performance of directly fine-tuning BART0 with the labeled query examples.
Revised: In Appendix D.2, we report the performance of directly ﬁne-tuning BART0 with the labeled query examples.

Original: Although it is an unfair comparison with our previous ReCross results, we found that they are comparable.
Revised: Although it is an unfair comparison with our previous ReCross results, we found that they are comparable.

Original: More importantly, the ReCross framework does not conflict with the few-shot setting.
Revised: More importantly, the ReCross framework does not conﬂict with the few-shot setting.

Original: Given a labeled query set for a target task, retrieved examples from the ReCross can still improve few-shot learning as additional training data.
Revised: Given a labeled query set for a target task, retrieved examples from the ReCross can still improve few-shot learning as additional training data.

Original: We designed two simple methods for applying
Revised: We designed two simple methods for applying

Original: ReCross under the few-shot setting and report the empirical results in Appendix D.2.
Revised: ReCross under the few-shot setting and report the empirical results in Appendix D.2.

Original: It turns out that
Revised: It turns out that

Original: ReCross can also boost the performance under the few-shot setting by about 3 points.
Revised: ReCross can also boost the performance under the few-shot setting by about 3 points.

Original:  Related Work
Revised:  Related Work

Original: Multi-task training for task generalization.
Revised: Multi-task training for task generalization.

Original: Text-to-text Transformer language models such as
Revised: Text-to-text Transformer language models such as

Original: T5 enable us to train a multi-task NLP model with a more straightforward recipe: mixing the data of multiple tasks into a unified seq2seq format, and then fine-tuning text-to-text LMs for implicit multi-task learning.
Revised: T5 enable us to train a multi-task NLP model with a more straightforward recipe: mixing the data of multiple tasks into a uniﬁed seq2seq format, and then ﬁne-tuning text-to-text LMs for implicit multi-task learning.

Original: UnifiedQA (Khashabi et al., 2020) is among the first works in this direction.
Revised: UniﬁedQA (Khashabi et al., 2020) is among the ﬁrst works in this direction.

Original: Although it shows great generalization performance within QA tasks, it can hardly generalize to other NLP tasks.
Revised: Although it shows great generalization performance within QA tasks, it can hardly generalize to other NLP tasks.

Original: Recent works, such as CrossFit (Ye et al., 2021), ExT5 (Aribandi et al., 2022),
Revised: Recent works, such as CrossFit (Ye et al., 2021), ExT5 (Aribandi et al., 2022),

Original: FLAN (Wei et al., 2021), T0 (Sanh et al., 2021), and InstructGPT Ouyang et al.
Revised: FLAN (Wei et al., 2021), T0 (Sanh et al., 2021), and InstructGPT Ouyang et al.

Original: (2022) focus on how to generalize a massively multi-task model across task boundaries in a much broader context.
Revised: (2022) focus on how to generalize a massively multi-task model across task boundaries in a much broader context.

Original: Particularly, in the CrossFit framework (Ye et al., 2021), cross-task generalization requires a small number of labeled instances of the target task for fine-tuning.
Revised: Particularly, in the CrossFit framework (Ye et al., 2021), cross-task generalization requires a small number of labeled instances of the target task for ﬁne-tuning.

Original: It is because the templates of CrossFit use the task names as the hard prefixes.
Revised: It is because the templates of CrossFit use the task names as the hard preﬁxes.

Original: Therefore, it is necessary to fine-tune the upstream model with a few examples that have the target task names as prefixes (i.e., few-shot learning), but this largely limits the application scenarios of these multi-task NLP models in practice.
Revised: Therefore, it is necessary to ﬁne-tune the upstream model with a few examples that have the target task names as preﬁxes (i.e., few-shot learning), but this largely limits the application scenarios of these multi-task NLP models in practice.

Original: We instead focus on unsupervised cross-task generalization, where there is no labeled data of an unseen task (i.e., zero-shot learning).
Revised: We instead focus on unsupervised cross-task generalization, where there is no labeled data of an unseen task (i.e., zero-shot learning).

Original: Using natural-language instructions as prompts, both FLAN and T0 show that it is promising to perform zero-shot cross-task generalization.
Revised: Using natural-language instructions as prompts, both FLAN and T0 show that it is promising to perform zero-shot cross-task generalization.

Original: In this work, we also focus on such an unsupervised setting for cross-task generalization, while our problem setup is a bit different from the ones used in T0 and FLAN.
Revised: In this work, we also focus on such an unsupervised setting for cross-task generalization, while our problem setup is a bit different from the ones used in T0 and FLAN.

Original: As for the assumption about the unlabeled data, their setups can be seen as a special case of ours when | Q | = 1 for all unseen tasks.
Revised: As for the assumption about the unlabeled data, their setups can be seen as a special case of ours when | Q | = 1 for all unseen tasks.

Original: The evaluation protocols of T0 and FLAN assess the generalization performance of the upstream model as it is, and thus their evaluation is more about the quality of templates and the upstream training tricks.
Revised: The evaluation protocols of T0 and FLAN assess the generalization performance of the upstream model as it is, and thus their evaluation is more about the quality of templates and the upstream training tricks.

Original: In contrast, our evaluation protocol can also study how to efficiently adjust the upstream model such that the updated models can generalize to new tasks without labeled data.
Revised: In contrast, our evaluation protocol can also study how to efﬁciently adjust the upstream model such that the updated models can generalize to new tasks without labeled data.

Original: Thus, we believe ours is a more general setup for studying unsupervised cross-task generalization.
Revised: Thus, we believe ours is a more general setup for studying unsupervised cross-task generalization.

Original: Retrieval augmentation in NLP.
Revised: Retrieval augmentation in NLP.

Original: We aim to retrieve useful examples from the upstream data and re-learning them for cross-task generalization.
Revised: We aim to retrieve useful examples from the upstream data and re-learning them for cross-task generalization.

Original: The proposed ReCross pipeline is inspired by open-ended QA methods such as DPR (Karpukhin et al., 2020),
Revised: The proposed ReCross pipeline is inspired by open-ended QA methods such as DPR (Karpukhin et al., 2020),

Original: DrFact (Lin et al., 2021), and
Revised: DrFact (Lin et al., 2021), and

Original: RAG (Lewis et al., 2020b).
Revised: RAG (Lewis et al., 2020b).

Original: Retrieval augmentation also shows great performance in pre-training
Revised: Retrieval augmentation also shows great performance in pre-training

Original: LMs (Guu et al., 2020).
Revised: LMs (Guu et al., 2020).

Original: Besides, Wang et al.
Revised: Besides, Wang et al.

Original: (2022) shows that learning with similar data via retrieval augmentation can improve the performance of a task-specific model.
Revised: (2022) shows that learning with similar data via retrieval augmentation can improve the performance of a task-speciﬁc model.

Original: Rubin et al.
Revised: Rubin et al.

Original: (2022) show that retrieving better demonstration examples is also helpful for in-context few-shot learning of GPTstyle language models (Brown et al., 2020).
Revised: (2022) show that retrieving better demonstration examples is also helpful for in-context few-shot learning of GPTstyle language models (Brown et al., 2020).

Original: The key challenge in the problem setup of this work is to predict the utility of the examples for unseen tasks with the consideration of efficiency and scalability.
Revised: The key challenge in the problem setup of this work is to predict the utility of the examples for unseen tasks with the consideration of efﬁciency and scalability.

Original: We have discussed more details about this challenge and related works in Sec. 2.
Revised: We have discussed more details about this challenge and related works in Sec. 2.

Original:  Conclusion & Future Directions
Revised:  Conclusion & Future Directions

Original: We demonstrate that retrieval augmentation can largely improve the cross-task generalization ability to multitask LMs in unsupervised settings.
Revised: We demonstrate that retrieval augmentation can largely improve the cross-task generalization ability to multitask LMs in unsupervised settings.

Original: Our proposed method, ReCross, is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking.
Revised: Our proposed method, ReCross, is a straightforward yet effective retrieval method that combines both efﬁcient dense retrieval and effective pair-wise reranking.

Original: Our empirical results show that it significantly outperforms both non-retrieval methods and other baseline methods.
Revised: Our empirical results show that it signiﬁcantly outperforms both non-retrieval methods and other baseline methods.

Original: We perform ablation studies showing the impact of changing query sizes, retrieval sizes, upsampling ratios, etc.
Revised: We perform ablation studies showing the impact of changing query sizes, retrieval sizes, upsampling ratios, etc.

Original: We also find the distribution of retrieved data for analyzing the behavior differences between ReCross and others.
Revised: We also ﬁnd the distribution of retrieved data for analyzing the behavior differences between ReCross and others.

Original: We believe that our paper will spur further research on retrieval-augmentation methods for cross-task generalization.
Revised: We believe that our paper will spur further research on retrieval-augmentation methods for cross-task generalization.

Original: Interesting future directions include:
Revised: Interesting future directions include:

Original: 1) improve the re-learning stage by including more information from query examples,2) extend the distant supervision mining process as a self-training procedure, 3) rigorously analyze the correlation between upstream data and target tasks, etc.
Revised: ) improve the re-learning stage by including more information from query examples, extend the distant supervision mining process as a self

Revised: -training

Original:  Acknowledgments
Revised: procedure, rigorously analyze the correlation between upstream data and target tasks, etc. Acknowledgments

Original: This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence
Revised: This research is supported in part by the Ofﬁce of the Director of National Intelligence (ODNI), Intelligence

Original: Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033, the Defense Advanced Research Projects Agency with awards W911NF19-20271, NSF IIS 2048211, and gift awards from Google, Amazon, JP Morgan, and Sony.
Revised: Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033, the Defense Advanced Research Projects Agency with awards W911NF19-20271, NSF IIS 2048211, and gift awards from Google, Amazon, JP Morgan, and Sony.

Original: We thank all collaborators in USC and the NeurIPS 2022 reviewers for their constructive feedback on the work.
Revised: We thank all collaborators in USC and the NeurIPS 2022 reviewers for their constructive feedback on the work.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Please see the experiment analysis sections and the appendix.
Revised: [Yes] Please see the experiment analysis sections and the appendix.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Please check the appendix.
Revised: [Yes] Please check the appendix.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We include our code, data, and instructions in our supplemental material for reproducing all results presented in the paper.
Revised: [Yes] We include our code, data, and instructions in our supplemental material for reproducing all results presented in the paper.

Original: As the data is quite large, we also include a script to download it.
Revised: As the data is quite large, we also include a script to download it.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We have specified the details of the base model (i.e., BART0) and the configurations for our retrieval-augmentation methods.
Revised: [Yes] We have speciﬁed the details of the base model (i.e., BART0) and the conﬁgurations for our retrieval-augmentation methods.

Original: Please check the experiment section and the appendix respectively.
Revised: Please check the experiment section and the appendix respectively.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] All our results are based on five different rounds, where each we use a different set of query examples.
Revised: [Yes] All our results are based on ﬁve different rounds, where each we use a different set of query examples.

Original: We also report multiple dimensions of the statistics in our table (e.g., the median, min, max, std, and mean).
Revised: We also report multiple dimensions of the statistics in our table (e.g., the median, min, max, std, and mean).

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We show these details in our appendix.
Revised: [Yes] We show these details in our appendix.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] The data we used are all open-source and publicly available via the “datasets” library from HuggingFace.
Revised: [Yes] The data we used are all open-source and publicly available via the “datasets” library from HuggingFace.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] Please refer to our appendix and the links to the used datasets.
Revised: [Yes] Please refer to our appendix and the links to the used datasets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] Appendix (i.e., Supplementary Material)
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: of “Unsupervised

Original: Cross

Original: -Task Generalization via Retrieval Augmentation”

Original: (Submission # 2811)

Original: This appendix include more implementation details, additional experimental results for ablation studies, and more analysis as well as findings.

Original: Please note that we have uploaded our code too (named “ReCross” folder).

Original: We first further analyze the performance of ReCross with more ablation analysis in Sec.

Original: A, and present detailed case studies for specific datasets in Sec.

Original: B, and introduce more implementation details in Sec.

Original: C.

Original: A Additional analysis

Original: A.1 Utility analysis by grouping upstream tasks.

Original: Table 4 shows the results of ReCorss under the scenarios where one specific group of upstream tasks are excluded from the index.

Original: This allows us to evaluate the impact of various upstream task categories on each downstream task

Original: [Table]

Original: Our key findings are as follows:

Original: • (1) Using all upstream tasks leads to the best overall performance, although for many target tasks there are some particular groups that are less useful than others.

Original: The last row shows this result and the summarization is the least useful group of upstream tasks.

Original: The potential best performance of retrieval-augmentation methods can be even higher.

Original: That is, if we have an enhanced version of ReCross that can avoid examples from less useful groups, then the final performance can be even higher.

Original: For example, if ReCross were able to ignore MCQA examples for ARC task during retrieval augmentation, then the overall performance of ReCross can be even higher.

Original: • (3) The utility analysis via grouping upstream tasks by their original task formulations does not align with general intuition.

Original: For example, people may think that MCQA (multiple-choice QA) should be more useful than other groups for the task of ARC, which is also a multiplechoice QA dataset.

Original: However, removing MCQA doesn’t hurt the performance of ARC.

Original: Instead, it actually improves the performance by 1 point.

Original: We argue that the example-based utility is of more importance for analysis.

Original: A.2 Template Perturbation

Original: To investigate the importance of templates in retrieval quality, we investigated two methods of perturbing the templates of query examples Q : 1.

Original: Simply concatenate the elements in the raw data.

Original: Change the words in the templates to random words to remove the semantic meaning.

Original: See figurefor an example.

Original: We than used these updated query examples and the same setup and configurations described in Section 4.3 to perform unsupervised cross-task generalization.

Original: [Table]
Revised: [N/A]

Original: Table 5 shows that when we simply concatenate the elements in raw data, the performance degrades to a level close to random retrieval.

Original: On the other hand, if we construct the query examples as specified by the templates, even if we break the semantics of the template, the performance boost is largely preserved.

Original: This might mean that the formatting of input, for example the existence of parallel choices in some form, potentially plays an important role in the performance gain

Original: [Table]

Original: A.3 Re-ranking for Random and SBERT

Original: [Table]

Original: We evaluated training re-rankers for random and

Original: SentenceBERT retrievers.

Original: Specifically, we applied the same distant supervision mining methods introduced in Section 3.4 on data retrieved by Random and SBERT .

Original: Table 6 shows the results.

Original: We can see that reranking does not improve the results for both Random and SBERT retriever.

Original: We believe it is because that the initial retrieval results are not good enough, so that the distant supervision mined from them are thus also not of good quality.

Original: A.4 Mining distant supervision for multiple iterations.

Original: The algorithm that we proposed in Alg.

Original: 1 can be extended to an iterative process.

Original: That is, we can continually update the reranker module and uses the retrieved results from the latest reranker to mine the training data for the next iteration.

Original: Although this self-training style process sounds promising, our empirical results show that the overall performance starts to saturate after the first iteration and using the 2nd-iteration re-ranker won’t improve the overall performance anymore.

Original: We think there can be better methods of continual learning to obtain a reranker module for better performance, while it is beyond the scope of this work.

Original: We hope this can be a promising future direction.

Original: A.5 Transferring ReCross for Larger Base Models.

Original: Recall that we choose to use BART0 as our base model for its smaller size and comparable results.

Original: People may wonder what if we transfer the ReCross methods for larger base models.

Original: Therefore, we conduct a pilot study on this.

Original: Considering the size of T0, we choose to only fine-tune its last few layers of T0-3B and still use the prior materials from BART0 (i.e., the BART0-based index and the trained reranker).

Original: We found that the performance is not improved over simply using T0-3B for zero-shot inference.

Original: We conjecture there are two major reasons for this: 1) the parameter-efficient tuning method need to added here to improve the training efficiency, 2) the BART0-based index and the associated reranker do not align with the other models such as T0-3B.

Original: We admit this could be one limitation of our methods – i.e., the index and reranker are specific to the base model that is used to generate them.

Original: In order to address these challenges, we argue that studying the common space of the index created by different encoders will be an important direction.

Original: B Case studies

Original: In this section, we discuss two specific datasets with detailed analysis as they have quite special results in Table 1 and Table 4.

Original: B.1 SuperGLUE CommitmentBank (cb)

Original: For the SuperGLUE CommitmentBank dataset, instances retrieved by the BART retriever are predominantly multiple choice question-answering.

Original: However, heat map and remove-one-group analysis shows that re-training on instances from multiple choice question-answering seems to undermine the model’s zero-shot performance on this dataset.

Original: We examined the output of the model and discovered that the model tends to make one type of error a lot more often when re-trained using multiple choice question-answering: instead of answering yes, no, possible, or impossible, it picks part of the discourse as its prediction.

Original: For example:

Original: Input: “Suppose A: I’m like, I’ll get a job some day and my boss will pay for it, I’ll be needed.

Original: B:

Original: Yeah.

Original: A: Because, um, I didn’t want to go do it myself because I didn’t think I was really going to use it.

Original: Can we infer that “he was really going to use it”?

Original: Yes, no, or maybe?”

Original: Output: “A: I didn’t want to go do it myself because I didn’t think I was really going to use it.”

Original: We believe this is because the model misunderstood the people having the discourse (A and B) to be the options for answers.

Original: The abundance of the template of “A:xxx, B:xxx” in the SuperGLUE

Original: CommitmentBank dataset might be the reason why the BART retriever retrieved mostly from multiple choice question-answering in the first place.

Original: B.2 SQuAD V

Original: For the dataset SQuAD V2, the retriever typically finds upstream examples from extractive question answering datasets, which match the format of SQuAD V2 inputs closely.

Original: However, we find that when we exclude extractive question answering examples from the upstream dataset, performance on

Original: SQuAD V2 improves.

Original: To explain this unexpected result, we note that the majority of our test examples for SQuAD V2, despite being formatted as extractive question answering tasks, are examples which expect the model to output whether or not the question is answerable.

Original: The ‘context and question’ format of the SQuAD V2 examples causes the retriever to focus on extractive question answering examples, but because most of the examples focus on answerability (a distinct task from extractive question answering), these examples are not helpful.

Original: We speculate that by excluding extractive question answering from the upstream dataset, the model avoids these misleading irrelevant examples and is able to retrieve more related examples for determining if a question is answerable.

Original: For example, our results show that when extractive question answering examples are excluded, the retriever finds examples from tasks such as Wiki QA, which asks whether or not a proposed answer is a valid answer to a given question (a more relevant task to determining if a question is answerable).

Original: C Implementation details C.1 Retrieval aggregation.

Original: Note that the target size of our retrieved data is | R | and we have | Q | query examples.

Original: To retrieve | R | examples, we search for the top- K examples for each query example, where K = ⌈ | R | | Q | ⌉ , and then take the first | R | of them when K | Q | > | R | .

Original: Our results have shown that this method is more effective than other strategies, such as combining the distance scores generated for each query example.

Original: Note that by retrieving the top- K examples, we may repeat examples that are close to multiple query vectors.

Original: This effect is desirable because it allows us to naturally focus more on the especially relevant upstream examples in re-learning.

Original: C.2 Upstream learning.

Original: Upstream tasks.

Original: Here we refer to the T0’s paper (cited in our main paper) for Figure ??

Original: , which shows the list of upstream tasks and their categories.

Original: We use this taxonomy to conduct ablation study.

Original: Please find the link to download these datasets from huggingface/dataset from our submitted code.

Original: All datasets are publicly available and their license are suitable for open-source research.

Original: We do not see any ethical concerns from using such datasets for learning a model and developing the ReCross method to further improve their task generalization performance.

Original: Training details.

Original: We specify the hyper-parameters and the concrete for training the BART models in our submitted code.

Original: Please read the “Readme.md” file where we point to the script and configurations for training BART0.

Original: Our GPU type is Quadro RTX 6000 and 8000.

Original: C.3 Retrieval Methods.

Original: Similarly, we leave the details such as the hyper-parameters and the concrete pipeline for running the retrieval augmentation methods (i.e., ReCross and the other baseline methods) in a unified framework that is presented in our code.

Original: D Others D.1 Evaluation metrics.

Original: Results with the standard EM.

Original: In Table 7, we report our main experimental results (the equivalent results to those in Table 1) with the standard EM metric instead of the SoftEM metric used in Table 1.

Original: We can see that the relative performance from the ReCross framework is about the same as in Table 1, although the absolute numbers are mostly smaller due to a more strict matching by

Original: We show the empirical results related to the few-shot setting in Table 8.

Original: Experimental setup.

Original: We assume that the labels of the examples in the query set are available, and directly use them to fine-tune the upstream model for learning the target task.

Original: We tune the hyper-parameters (epochs and learning rates) such that they do not overfit the few-shot data and lead to a better performance over BART0.

Original: Note that the real performance of few-shot learning performance may be lower than the ones in the table because there is not enough development data for us to tune hyper-parameters for each target task.

Original: Few-shot learning is not even better than the unsupervised ReCross.

Original: Although FS can outperform ReCross in some target tasks, the two approaches have very similar overall performance ontasks.

Original: Even in such an unfair setting, ReCross shows great benefits to the users.

Original: ReCross and Few-Shot together can produce better performance.

Original: We attempted to use both the few-shot data and the retrieved data for generalization.

Original: The FS+RC(mix) method simply merge the 16 labeled query examples (i.e., few-shot) and the 512 retrieved data (by ReCross) to get a larger

Original: [Table]

Original: [Table]

Original: [Table]

Original: dataset for fine-tuning BART0.

Original: The FS+RC(2-stage) method updates the model firstly with theretrieved data and then train the fine-tuned model with the 16 FS data.

Original: Both methods show a great enhancement over FS and RC used separately.

Original: This is to say, RC is still beneficial in the FS setting.