Original: C OMPOSITIONAL T ASK R EPRESENTATIONS FOR L ARGE L ANGUAGE M ODELS
Revised: C OMPOSITIONAL T ASK R EPRESENTATIONS FOR L ARGE L ANGUAGE M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: ABSTRACT
Revised: A

Revised: BSTRACT

Original: Large language models have shown a remarkable cross-task generalization ability.
Revised: Large language models have shown a remarkable cross-task generalization ability.

Original: Most prior works assumed that prompts effectively extract knowledge from language models to facilitate generalization to new tasks.
Revised: Most prior works assumed that prompts effectively extract knowledge from language models to facilitate generalization to new tasks.

Original: This perspective led to numerous studies on improving prompts.
Revised: This perspective led to numerous studies on improving prompts.

Original: In contrast, we introduce a new perspective, compositional generalization, that views each task as a composition of latent codes and generalizes to test tasks by a new composition of seen codes.
Revised: In contrast, we introduce a new perspective, compositional generalization that views each task as a composition of latent codes and generalizes to test tasks by a new composition of seen codes.

Original: To this end, we propose a novel prompt-free approach, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.
Revised: To this end, we propose a novel prompt-free approach, Compositional Task Representations (CTR) that employs multi-task training to learn a discrete, compositional codebook.

Original: Empirically, our CTR substantially outperforms prompt-based methods in zero-label learning on average.
Revised: Empirically, our CTR substantially outperforms prompt-based methods in zero-label learning on average.

Original: According to our analysis, some of the learned CTR codes are interpretable to humans and demonstrate a certain degree of controllability.
Revised: According to our analysis, some of the learned CTR codes are interpretable to humans and demonstrate a certain degree of controllability

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large language models (LLMs) have shown remarkable performance in cross-task generalization.
Revised: Large language models (LLMs) have shown remarkable performance in cross-task generalization.

Original: Without using any labeled data for the target task, GPT-3 (Brown et al., 2020) obtains reasonable performance on a wide range of tasks.
Revised: Without using any labeled data for the target task, GPT-3 (Brown et al., 2020) obtains reasonable performance on a wide range of tasks.

Original: Later extensions such as FLAN (Wei et al., 2022) and T0 (Sanh et al., 2022) continue training the LLMs on a large number of supervised tasks, which further improves cross-task generalization performance.
Revised: Later extensions such as FLAN (Wei et al., 2022) and T0 (Sanh et al., 2022) continue training the LLMs on a large number of supervised tasks, which further improves cross-task generalization performance.

Original: The aforementioned studies have used an important assumption that natural language prompts extract knowledge from LLMs to facilitate generalization to new tasks.
Revised: The aforementioned studies have used an important assumption that natural language prompts extract knowledge from LLMs to facilitate generalization to new tasks.

Original: In this direction, numerous studies have focused on different aspects of improving prompt-based learning, such as designing better prompts (Xu et al., 2022), increasing the number of prompts (Wang et al., 2022;
Revised: In this direction, numerous studies have focused on different aspects of improving prompt-based learning, such as designing better prompts (Xu et al., 2022), increasing the number of prompts (Wang et al., 2022;

Original: Aribandi et al., 2022), and improving the training efficiency of prompts (Lester et al., 2021).
Revised: Aribandi et al., 2022), and improving the training efficiency of prompts (Lester et al., 2021).

Original: In contrast, we explore an alternative perspective for cross-task generalization, i.e., compositional generalization.
Revised: In contrast, we explore an alternative perspective for cross-task generalization, i.e., compositional generalization.

Original: Specifically, we explore whether it is possible to represent tasks using discrete compositions of latent codes.
Revised: Specifically, we explore whether it is possible to represent tasks using discrete compositions of latent codes.

Original: This perspective enjoys several potential benefits.
Revised: This perspective enjoys several potential benefits.

Original: First, since the latent codes have been trained for seen tasks, we expect the LLMs to have strong cross-task generalization abilities because new tasks can also be represented as a composition of these trained codes.
Revised: First, since the latent codes have been trained for seen tasks, we expect the LLMs to have strong cross-task generalization abilities because new tasks can also be represented as a composition of these trained codes.

Original: Second, it provides a way to analyze and understand cross-task generalization by investigating the association between tasks and the learned representations.
Revised: Second, it provides a way to analyze and understand cross-task generalization by investigating the association between tasks and the learned representations.

Original: Third, it has the potential of being more controllable than prompts for task generalization due to its built-in compositionality.
Revised: Third, it has the potential of being more controllable than prompts for task generalization due to its built-in compositionality.

Original: Motivated by the aforementioned potentials, we propose a new method, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.
Revised: Motivated by the aforementioned potentials, we propose a new method, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.

Original: Specifically, given a large number of training tasks, we use an encoder to map each randomlyinitialized task embedding to a fixed-length sequence of query vectors.
Revised: Specifically, given a large number of training tasks, we use an encoder to map each randomlyinitialized task embedding to a fixed-length sequence of query vectors.

Original: Each query vector is used to retrieve a code from a codebook, which is formulated as an embedding lookup table.
Revised: Each query vector is used to retrieve a code from a codebook, which is formulated as an embedding lookup table.

Original: This produces  a sequence of codes, a compositional representation of the current task.
Revised: This produces [Table] [Figure] a sequence of codes, a compositional representation of the current task.

Original: These compositional codes are fed as the input to an LLM in place of prompts to make predictions.
Revised: These compositional codes are fed as the input to an LLM in place of prompts to make predictions.

Original: At test time, given a new task, we use unlabeled data to search for a high-performing composition of codes, which enables zero-label cross-task generalization.
Revised: At test time, given a new task, we use unlabeled data to search for a high-performing composition of codes, which enables zero-label cross-task generalization.

Original: CTR is also applicable to the few-shot setting where the few labeled examples are used for code search.
Revised: CTR is also applicable to the few-shot setting where the few labeled examples are used for code search.

Original: Empirically, we demonstrate improved performance under both the settings of zero-label learning and few-shot learning, outperforming strong baselines including prompt tuning, model tuning, and[Table] [Figure] genetic prompt search (Xu et al., 2022).
Revised: Empirically, we demonstrate improved performance under both the settings of zero-label learning and few-shot learning, outperforming strong baselines including prompt tuning, model tuning, and genetic prompt search (Xu et al., 2022).

Original: Importantly, we analyze the learned task representations and show that they demonstrate a certain degree of interpretability and controllability.
Revised: Importantly, we analyze the learned task representations and show that they demonstrate a certain degree of interpretability and controllability.

Original: For example, as shown in Figure 1, CTR learns to generalize to a new task by a new composition of existing codes.
Revised: For example, as shown in Figure 1, CTR learns to generalize to a new task by a new composition of existing codes.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Language Model Prompting.
Revised: Language Model Prompting.

Original: Brown et al.
Revised: Brown et al.

Original: (2020) showed that GPT-3 performs well in the fewshot setting if properly handcrafted prompts are provided.
Revised: (2020) showed that GPT-3 performs well in the fewshot setting if properly handcrafted prompts are provided.

Original: Other works (Shoeybi et al., 2019; Rae et al., 2021; Schick & Sch ¨ utze, 2021) also report promising zero-shot or few-shot performances of
Revised: Other works (Shoeybi et al., 2019; Rae et al., 2021; Schick & Sch¨utze, 2021) also report promising zero-shot or few-shot performances of

Original: LLMs.
Revised: LLMs.

Original: Besides, Wei et al.
Revised: Besides, Wei et al.

Original: and Sanh et al.
Revised: and Sanh et al.

Original: (2022) collect a set of labeled datasets and use manual templates to transform them into a sequence-to-sequence style.
Revised: (2022) collect a set of labeled datasets and use manual templates to transform them into a sequence-to-sequence style.

Original: Such a formulation makes it possible to continue training LLMs on these labeled datasets and improves cross-task generalization.
Revised: Such a formulation makes it possible to continue training LLMs on these labeled datasets and improves cross-task generalization.

Original: Wang et al.
Revised: Wang et al.

Original: ; Mishra et al.
Revised: ; Mishra et al.

Original: (2022) introduced a benchmark of over 1600 tasks and their expertwritten instructions Gao et al.
Revised: (2022) introduced a benchmark of over 1600 tasks and their expertwritten instructions Gao et al.

Original: Shin et al.
Revised: Shin et al.

Original: (2020) studied automating the search process of discrete prompts.
Revised: (2020) studied automating the search process of discrete prompts.

Original: Li & Liang (2021); Liu et al. (2021) propose continuous soft prompts with gradient-based optimization.
Revised: Li & Liang (2021); Liu et al. (2021) propose continuous soft prompts with gradient-based optimization.

Original: Compared to these approaches, we study a different approach that learns compositional task representations, which benefits cross-task generalization.
Revised: Compared to these approaches, we study a different approach that learns compositional task representations, which benefits cross-task generalization.

Original: Compositional Architecture for LLMs.
Revised: Compositional Architecture for LLMs.

Original: Previous work has explored designing compositional architectures.
Revised: Previous work has explored designing compositional architectures.

Original: Sparsely Gated Mixture of Expert (MoE) (Lepikhin et al., 2021) activates a subset of a network given the input data.
Revised: Sparsely Gated Mixture of Expert (MoE) (Lepikhin et al., 2021) activates a subset of a network given the input data.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2021) trained an MoE model with 1.1T parameters, which is shown to outperform a dense model with similar computational cost.
Revised: (2021) trained an MoE model with 1.1T parameters, which is shown to outperform a dense model with similar computational cost.

Original: SkillNet-NLU (Tang et al., 2022) and SkillNet-NLG (Liao et al., 2022b) employed a similar sparsely activated mechanism to handle different NLU or NLG tasks.
Revised: SkillNet-NLU (Tang et al., 2022) and SkillNet-NLG (Liao et al., 2022b) employed a similar sparsely activated mechanism to handle different NLU or NLG tasks.

Original: Different from these approaches, our approach focuses on learning compositional task representations using a discrete codebook. 
Revised: Different from these approaches, our approach focuses on learning compositional task representations using a discrete codebook. [Figure]

Original:  C OMPOSITIONAL T ASK R EPRESENTATIONS
Revised:  C OMPOSITIONAL T ASK R EPRESENTATIONS

Original: The motivation of CTR is to explore the cross-task generalization ability of LLMs from a brandnew perspective—compositional generalization, and to further improve the performance of crosstask generalization.
Revised: The motivation of CTR is to explore the cross-task generalization ability of LLMs from a brandnew perspective—compositional generalization, and to further improve the performance of crosstask generalization.

Original: Specifically, our main hypothesis is that being trained on a variety of natural language tasks, LLMs will be able to learn to represent each task as a composition of discrete latent codes, where each latent code is associated with certain aspects of a task.
Revised: Specifically, our main hypothesis is that being trained on a variety of natural language tasks, LLMs will be able to learn to represent each task as a composition of discrete latent codes, where each latent code is associated with certain aspects of a task.

Original: As a result, CTR poten- [Figure] tially enjoys the advantages of better cross-task generalization since it could represent new tasks by forming new compositions.
Revised: As a result, CTR potentially enjoys the advantages of better cross-task generalization since it could represent new tasks by forming new compositions.

Original: This section introduces the overall architecture of CTR and how it is trained to overcome optimization challenges.
Revised: This section introduces the overall architecture of CTR and how it is trained to overcome optimization challenges.

Original: As Figure 2 shows, CTR consists of the CTR learning module and an LLM, where the CTR learning module contains an encoder, a decoder, task embeddings, and a codebook.
Revised: As Figure 2 shows, CTR consists of the CTR learning module and an LLM, where the CTR learning module contains an encoder, a decoder, task embeddings, and a codebook.

Original: 3. D ISCRETE L ATENT T ASK C ODEBOOK
Revised: 3. D ISCRETE L ATENT T ASK C ODEBOOK

Original: We define a latent task codebook embedding space C ∈ R S × D where S is the size of the latent codebook embedding space (i.e., each task code can take either of the S categorical values), and D is the dimension of each latent code embedding C i ∈ R D , i ∈
Revised: We define a latent task codebook embedding space C ∈ R S × D where S is the size of the latent codebook embedding space (i.e., each task code can take either of the S categorical values), and D is the dimension of each latent code embedding C i ∈ R D , i ∈

Original: [1 , 2 , · · · , S ] .
Revised: [1 , 2 , · · · , S ] .

Original: This is analogous to the idea of VQ-VAE (van den Oord et al., 2017) where a discrete latent codebook is also employed.
Revised: This is analogous to the idea of VQ-VAE (van den Oord et al., 2017) where a discrete latent codebook is also employed.

Original: As Figure 2 shows, given a training task as the input (assuming the task id is k ), CTR first obtains its task embedding E k ∈ R D by retrieving from a randomly-initialized task embedding lookup table
Revised: As Figure 2 shows, given a training task as the input (assuming the task id is k ), CTR first obtains its task embedding E k ∈ R D by retrieving from a randomly-initialized task embedding lookup table

Original: R N × D where N is the number of training tasks.
Revised: R N × D where N is the number of training tasks.

Original: The task embedding E k is then passed to the encoder module and is mapped into a fixed-length sequence of query vectors Q ∈ R L × D where L is the length of the sequence.
Revised: The task embedding E k is then passed to the encoder module and is mapped into a fixed-length sequence of query vectors Q ∈ R L × D where L is the length of the sequence.

Original: For each query vector Q l , l ∈ [1 , 2 , · · · , L ] , it is used to retrieve a task code embedding from the codebook embedding space.
Revised: For each query vector Q l , l ∈ [1 , 2 , · · · , L ] , it is used to retrieve a task code embedding from the codebook embedding space.

Original: Specifically, it calculates the l 2 distance with each of the latent code embedding C i ∈ R D , i ∈
Revised: Specifically, it calculates the l 2 distance with each of the latent code embedding C i ∈ R D , i ∈

Original: [1 , 2 , · · · , S ] , and find the nearest neighbor,
Revised: [1 , 2 , · · · , S ] , and find the nearest neighbor,

Revised: CTR l = C z l , where z l = arg min i

Original: [Equation]
Revised: [Equation]

Original: In this way, all query vectors together produce a L -length sequence of code embeddings, denoted as the compositional task representations (CTR).
Revised: In this way, all query vectors together produce a L -length sequence of code embeddings, denoted as the compositional task representations (CTR).

Original: They are then passed through a decoder, followed by being used as the input to an LLM in place of prompts to make predictions.
Revised: They are then passed through a decoder, followed by being used as the input to an LLM in place of prompts to make predictions.

Original: Let z be a vector of L latent codes z 1 , z 2 , . .
Revised: Let z be a vector of L latent codes z 1 , z 2 , . .

Original: z L .
Revised: z L .

Original: We consider each latent code as describing an attribute or a necessary skill of a certain task, such as task type, output space, etc.
Revised: We consider each latent code as describing an attribute or a necessary skill of a certain task, such as task type, output space, etc.

Original: Intuitively, we need multiple codes to fully describe a task, and each task is formulated as a composition of codes.
Revised: Intuitively, we need multiple codes to fully describe a task, and each task is formulated as a composition of codes.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: Training can be challenging due to the existence of discrete latent variables.
Revised: Training can be challenging due to the existence of discrete latent variables.

Original: Moreover, during the initial training phase, the CTR learning module is randomly initialized.
Revised: Moreover, during the initial training phase, the CTR learning module is randomly initialized.

Original: As a result, the codebook embeddings C will have a very different distribution to the query vectors Q , which increases the difficulty of optimization.
Revised: As a result, the codebook embeddings C will have a very different distribution to the query vectors Q , which increases the difficulty of optimization.

Original: We decouple training into two phases.
Revised: We decouple training into two phases.

Original: In the first phase, we freeze the
Revised: In the first phase, we freeze the

Original: LLM and only update the CTR learning module.
Revised: LLM and only update the CTR learning module.

Original: This is followed by tuning all parameters.
Revised: This is followed by tuning all parameters.

Original: In terms of the loss function, following van den Oord et al.
Revised: In terms of the loss function, following van den Oord et al.

Original: (2017), we employ two separate losses, an embedding loss and a commitment loss, to match the query vectors with the compositional task representations.
Revised: (2017), we employ two separate losses, an embedding loss and a commitment loss, to match the query vectors with the compositional task representations.

Original: These losses are used in combination with a standard language modeling loss,
Revised: These losses are used in combination with a standard language modeling loss,

Original: [Equation]
Revised: [Equation]

Original: where L LM is a standard language modeling loss for solving the target task, sg denotes stopping the gradients, and β is a hyperparameter.
Revised: where L LM is a standard language modeling loss for solving the target task, sg denotes stopping the gradients, and β is a hyperparameter.

Original: 3. I NFERENCE
Revised: 3. I NFERENCE

Original: We consider two settings: zero-label learning and few-shot learning, and describe how we apply
Revised: We consider two settings: zero-label learning and few-shot learning, and describe how we apply

Original: CTR in these settings.
Revised: CTR in these settings.

Original: Code Ensemble for Zero-Label Learning.
Revised: Code Ensemble for Zero-Label Learning.

Original: In the zero-label setting, we are given a new task along with a set of unlabeled data.
Revised: In the zero-label setting, we are given a new task along with a set of unlabeled data.

Original:  The question is how to decide the code for this new task without labeled data.
Revised: * The question is how to decide the code for this new task without labeled data.

Original: Our main idea is to select one of the training tasks (319 in total in our experiments) and use its learned code for this new task.
Revised: Our main idea is to select one of the training tasks (319 in total in our experiments) and use its learned code for this new task.

Original: We first obtain a set of candidate codes by examining how much a code gives predictions that deviate from a uniform distribution on the unlabeled data (Zhao et al., 2021).
Revised: We first obtain a set of candidate codes by examining how much a code gives predictions that deviate from a uniform distribution on the unlabeled data (Zhao et al., 2021).

Original: The candidate set is formed by N (set to 60 in our experiments) codes with the lowest deviations.
Revised: The candidate set is formed by N (set to 60 in our experiments) codes with the lowest deviations.

Original: We then ensemble the candidate set of codes to predict pseudo labels on unlabeled data, and select the code with the highest pseudo-label accuracy.
Revised: We then ensemble the candidate set of codes to predict pseudo labels on unlabeled data, and select the code with the highest pseudo-label accuracy.

Original: Bitwise Search for Few-Shot Learning.
Revised: Bitwise Search for Few-Shot Learning.

Original: In the few-shot setting, we are given a new task along with a set of labeled data.
Revised: In the few-shot setting, we are given a new task along with a set of labeled data.

Original: We use this set of labeled data as our validation set to search for a high-performing code.
Revised: We use this set of labeled data as our validation set to search for a high-performing code.

Original: Our preliminary study shows that one can control the output of CTR by changing one bit in the code vector z .
Revised: Our preliminary study shows that one can control the output of CTR by changing one bit in the code vector z .

Original: Inspired by this, we first examine the validation-set accuracy for codes of training tasks, and select the code with the best accuracy as an initialization.
Revised: Inspired by this, we first examine the validation-set accuracy for codes of training tasks, and select the code with the best accuracy as an initialization.

Original: Then we iteratively change a single bit of the selected code and evaluate the validation-set performance.
Revised: Then we iteratively change a single bit of the selected code and evaluate the validation-set performance.

Original: At each iteration, we keep the updated code with higher performance.
Revised: At each iteration, we keep the updated code with higher performance.

Original: Finally, the code vector that obtains the best result on the validation set is taken as the test-task code.
Revised: Finally, the code vector that obtains the best result on the validation set is taken as the test-task code.

Original: Our preliminary study shows that a certain code value usually occurs in a specific position of the code vector, and each position usually only has a small set of code values.
Revised: Our preliminary study shows that a certain code value usually occurs in a specific position of the code vector, and each position usually only has a small set of code values.

Original: Motivated by this, we only changes a small set of valid code candidates for each position.
Revised: Motivated by this, we only changes a small set of valid code candidates for each position.

Original: In both the zero-label and few-shot settings, after we obtain a code vector z for the new task, we use the vector to obtain a composition task representation by indexing z in the codebook C .
Revised: In both the zero-label and few-shot settings, after we obtain a code vector z for the new task, we use the vector to obtain a composition task representation by indexing z in the codebook C .

Original: The task representations are then passed through the decoder and the LLM to perform the new task, similar to training time.
Revised: The task representations are then passed through the decoder and the LLM to perform the new task, similar to training time.

Original: This is also illustrated in the right part of Figure 2.
Revised: This is also illustrated in the right part of Figure 2.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: We conduct extensive quantitative experiments to validate the performance of cross-task generalization of CTR.
Revised: We conduct extensive quantitative experiments to validate the performance of cross-task generalization of CTR.

Original: We mainly consider two settings, the zero-label setting and the few-shot setting.
Revised: We mainly consider two settings, the zero-label setting and the few-shot setting.

Original: Aside from quantitative experiments, we perform qualitative analysis to understand the cross-task generalization ability by investigating the association between the discrete latent codes and the tasks.
Revised: Aside from quantitative experiments, we perform qualitative analysis to understand the cross-task generalization ability by investigating the association between the discrete latent codes and the tasks.

Original: 4.1. D ATASETS
Revised: 4.1. D ATASETS

Original: CTR requires a large multi-task set for training, and a held-out set of tasks whose types are never seen during training for evaluation.
Revised: CTR requires a large multi-task set for training, and a held-out set of tasks whose types are never seen during training for evaluation.

Original: We follow the T0 benchmark.
Revised: We follow the T0 benchmark.

Original: The training part consists of 39 tasks of 8 task types, including closed-book question answering (QA), multiple-choice QA, extractive QA, sentiment analysis, paraphrase identification, topic classification, summarization, and structure to text.
Revised: The training part consists of 39 tasks of 8 task types, including closed-book question answering (QA), multiple-choice QA, extractive QA, sentiment analysis, paraphrase identification, topic classification, summarization, and structure to text.

Original: The test part consists of 11 tasks of 4 task types, including natural language inference (RTE (Dagan et al., 2006), CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020)), sentence completion (COPA (Roemmele et al., 2011),
Revised: The test part consists of 11 tasks of 4 task types, including natural language inference (RTE (Dagan et al., 2006), CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020)), sentence completion (COPA (Roemmele et al., 2011),

Original: StoryCloze (Mostafazadeh et al., 2017), Hellaswag (Zellers et al., 2019)), and word sense disambiguation (WiC (Pilehvar & Camacho-Collados, 2019)).
Revised: StoryCloze (Mostafazadeh et al., 2017), Hellaswag (Zellers et al., 2019)), and word sense disambiguation (WiC (Pilehvar & Camacho-Collados, 2019)).

Original: Both the training and test parts are disjoint in task types, ensuring the zero-label setting.
Revised: Both the training and test parts are disjoint in task types, ensuring the zero-label setting.

Original: We follow T0 to use the accuracy on the validation split of test tasks as the metric.
Revised: We follow T0 to use the accuracy on the validation split of test tasks as the metric.

Original: 4.1. B ASELINES
Revised: 4.1. B ASELINES

Original: For the zero-label setting, we primarily compare CTR with the following baselines.
Revised: For the zero-label setting, we primarily compare CTR with the following baselines.

Original: It is noteworthy that all baselines share a similar model size as CTR (i.e., about 770M), thus are comparable.
Revised: It is noteworthy that all baselines share a similar model size as CTR (i.e., about 770M), thus are comparable.

Original: We provide implementation details of baselines in Appendix A.7.
Revised: We provide implementation details of baselines in Appendix A.7.

Original: • T0 (Sanh et al., 2022) shares similar goals as CTR, which uses prompted multi-task training to improve the generalization performance.
Revised: • T0 (Sanh et al., 2022) shares similar goals as CTR, which uses prompted multi-task training to improve the generalization performance.

Original: T0 reports the average results over multiple prompts.
Revised: T0 reports the average results over multiple prompts.

Original: • Self-Training Since unlabeled data are accessed, we consider the self-training (Schick & Sch¨utze, 2020) method as one of the baselines.
Revised: • Self-Training Since unlabeled data are accessed, we consider the self-training (Schick & Sch ¨ utze, 2020) method as one of the baselines.

Original: Starting from T0, it uses T0 to label the unlabeled data, and further finetunes T0 with the pseudo-labeled data.
Revised: Starting from T0, it uses T0 to label the unlabeled data, and further finetunes T0 with the pseudo-labeled data.

Original: It also reports average performance over prompts.
Revised: It also reports average performance over prompts.

Original: • Manual-Code uses artificial feature vectors in place of CTR vector.
Revised: • Manual-Code uses artificial feature vectors in place of CTR vector.

Original: Specifically, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, and whether it is a classification task, etc.
Revised: Specifically, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, and whether it is a classification task, etc.

Original: Each task can be represented as an discrete feature vector, each dimension associated with one of the aspects.
Revised: Each task can be represented as an discrete feature vector, each dimension associated with one of the aspects.

Original: • ZPS (Liao et al., 2022a) is method for zero-label prompt selection.
Revised: • ZPS (Liao et al., 2022a) is method for zero-label prompt selection.

Original: It first labels a set of unlabeled data through prompt ensemble, and use the pseudo-labeled data to select the best natural language prompt for the test task.
Revised: It first labels a set of unlabeled data through prompt ensemble, and use the pseudo-labeled data to select the best natural language prompt for the test task.

Original: We apply ZPS to multi-task T as a baseline
Revised: We apply ZPS to multi-task T0 as a baseline.

Original: For the few-shot setting where there are 32 test-set labeled examples, based on the multi-task T0, we experimented the following five baseline methods.
Revised: For the few-shot setting where there are 32 test-set labeled examples, based on the multi-task T0, we experimented the following five baseline methods.

Original: • ModelTuning directly finetunes the pretrained model using the test-set labeled data.
Revised: • Model Tuning directly finetunes the pretrained model using the test-set labeled data.

Original: Specifically, we follow the few-shot setting in Zheng et al.
Revised: Specifically, we follow the few-shot setting in Zheng et al.

Original: (2021) to use 16 labeled data for finetuning and another 16 labeled samples for model selection.
Revised: (2021) to use 16 labeled data for finetuning and another 16 labeled samples for model selection.

Original: • PromptTuning (Lester et al., 2021) introduces additional continuous prompts to the backbone model (i
Revised: • Prompt Tuning (Lester et al., 2021) introduces additional continuous prompts to the backbone model (i

Original: e., T0) and trains the continuous prompts using the few labeled data.
Revised: e., T0) and trains the continuous prompts using the few labeled data.

Original: • GPS (Xu et al., 2022) is a genetic prompt search method.
Revised: • GPS (Xu et al., 2022) is a genetic prompt search method.

Original: Based on T0, GPS gradually mutates the prompts with a generative model and uses the few labeled data to selects prompt candidates.
Revised: Based on T0, GPS gradually mutates the prompts with a generative model and uses the few labeled data to selects prompt candidates.

Original: • GRIPS (Prasad et al., 2022) is a gradient-free edit-based method for optimal prompt search.
Revised: • GRIPS (Prasad et al., 2022) is a gradient-free edit-based method for optimal prompt search.

Original: • Black-Box Tuning (BBT) (Sun et al., 2022) is a gradient-free few-shot prompt selection method.
Revised: • Black-Box Tuning (BBT) (Sun et al., 2022) is a gradient-free few-shot prompt selection method.

Original: Unlike GPS and GRIPS, it searches for the best soft prompt embedding in a continuous space.
Revised: Unlike GPS and GRIPS, it searches for the best soft prompt embedding in a continuous space.

Original: 4.1. T RAINING D ETAILS
Revised: 4.1. T RAINING D ETAILS

Original: We instantiate our CTR with T5-Large (Raffel et al., 2019) being the LLM.
Revised: We instantiate our CTR with T5-Large (Raffel et al., 2019) being the LLM.

Original: We implement both the encoder and the decoder in our CTR model as two linear networks.
Revised: We implement both the encoder and the decoder in our CTR model as two linear networks.

Original: We have experimented with various architectures (see Appendix A.4 for more details).
Revised: We have experimented with various architectures (see Appendix A.4 for more details).

Original: For the first stage of training
Revised: For the first stage of training

Original: where the
Revised: where the

Original: LLM is frozen, we use the Adam optimizer with a learning rate of 1e-2, a decay rate of 0.1, and a batch size of 2048.
Revised: LLM is frozen, we use the Adam optimizer with a learning rate of 1e-2, a decay rate of 0.1, and a batch size of 2048.

Original: We use a codebook embedding dimension of 1024, which is the same as the hidden dimension.
Revised: We use a codebook embedding dimension of 1024, which is the same as the hidden dimension.

Original: The CTR length is set at 10 and each position can be assigned values ranging from 0 to 127; i.e., the codebook size is 128.
Revised: The CTR length is set at 10 and each position can be assigned values ranging from 0 to 127; i.e., the codebook size is 128.

Original: The hyperparameter β is set at 0.1.
Revised: The hyperparameter β is set at 0.1.

Original: We experimented with different codebook sizes and CTR lengths, and detailed results are provided in Appendix A.5.
Revised: We experimented with different codebook sizes and CTR lengths, and detailed results are provided in Appendix A.5.

Original: For the second training phase where all parameters are updated, we use the Adam optimizer with a learning rate of 1e-4 and a batch size of 1024.
Revised: For the second training phase where all parameters are updated, we use the Adam optimizer with a learning rate of 1e-4 and a batch size of 1024.

Original: We follow the training recipe as T0 (Sanh et al., 2022) for the rest of the hyperparameters. 
Revised: We follow the training recipe as T0 (Sanh et al., 2022) for the rest of the hyperparameters. [Table]

Original: [Table]
Revised: [Table]

Original: 4. M AIN R ESULTS AND A NALYSIS
Revised: 4. M AIN R ESULTS AND A NALYSIS

Original: The performance of cross-task generalization, respectively under the zero-label setting and the fewshot setting, are shown in Table 1.
Revised: The performance of cross-task generalization, respectively under the zero-label setting and the fewshot setting, are shown in Table 1.

Original: Our CTR outperforms all baseline methods on average under both settings.
Revised: Our CTR outperforms all baseline methods on average under both settings.

Original: Comparing CTR with T0-Large and its variant (i.e., self-training), CTR outperforms them respectively by more than 4 points and by almost 1.5 points on average.
Revised: Comparing CTR with T0-Large and its variant (i.e., self-training), CTR outperforms them respectively by more than 4 points and by almost 1.5 points on average.

Original: The improvements potentially originate from two aspects—(a) the learned compositional task representations benefit from better generalization abilities than the discrete manual prompts used by T0/self-training; (b)
Revised: The improvements potentially originate from two aspects—(a) the learned compositional task representations benefit from better generalization abilities than the discrete manual prompts used by T0/self-training; (b)

Original: our
Revised: our

Original: CTR can select a high-performing compositional task representation for the unseen task.
Revised: CTR can select a high-performing compositional task representation for the unseen task.

Original: Compared with Manual-Code, our CTR demonstrate significant advantages of more than 4.5 points, proving that artificially-designed features of tasks are unreliable, and CTR provides an effective way of automatically training such data-driven compositional task representations using multi-tasks.
Revised: Compared with Manual-Code, our CTR demonstrate significant advantages of more than 4.5 points, proving that artificially-designed features of tasks are unreliable, and CTR provides an effective way of automatically training such data-driven compositional task representations using multi-tasks.

Original: Compared with Model Tuning, Prompt Tuning, and BBT, where they require parameter updates over the test-task data, CTR shows better performance of cross-task generalization on average.
Revised: Compared with Model Tuning, Prompt Tuning, and BBT, where they require parameter updates over the test-task data, CTR shows better performance of cross-task generalization on average.

Original: Compared with baselines that do not tune parameters (i.e., GPS, GRIPS), CTR shows even larger and more consistent advantages on most of the test tasks; i.e., on 9/11 tasks CTR shows dominating performance.
Revised: Compared with baselines that do not tune parameters (i.e., GPS, GRIPS), CTR shows even larger and more consistent advantages on most of the test tasks; i.e., on 9/11 tasks CTR shows dominating performance.

Original: On co-reference tasks , CTR performs worse under the zero-label setting and better under the fewshot setting.
Revised: On co-reference tasks , CTR performs worse under the zero-label setting and better under the fewshot setting.

Original: The zero-label setting uses pseudo-labeled data to select test codes while the few-shot setting uses real-labeled data to select codes.
Revised: The zero-label setting uses pseudo-labeled data to select test codes while the few-shot setting uses real-labeled data to select codes.

Original: The reason for the decreased performance lies in that the pseudo data of the co-reference task were of low quality and did not select effective task codes.
Revised: The reason for the decreased performance lies in that the pseudo data of the co-reference task were of low quality and did not select effective task codes.

Original: Please refer to Appendix A.3 for generated zero-label cases from CTR .
Revised: Please refer to Appendix A.3 for generated zero-label cases from CTR .

Original: 4. A BLATION S TUDY 4.3. A RE M ANUAL P ROMPTS N ECESSARY ?
Revised: 4. A BLATION S TUDY 4.3. A RE M ANUAL P ROMPTS N ECESSARY ?

Original: We are interested in whether CTR will be further improved when combined with discrete manual prompts.
Revised: We are interested in whether CTR will be further improved when combined with discrete manual prompts.

Original: We conduct comparative experiments of CTR respectively with and without manual prompts.
Revised: We conduct comparative experiments of CTR respectively with and without manual prompts.

Original: Specifically, for CTR without manual prompts, the inputs are a direct concatenation of multiple text fields, with compositional task representations appended in front of it.
Revised: Specifically, for CTR without manual prompts, the inputs are a direct concatenation of multiple text fields, with compositional task representations appended in front of it.

Original: For CTR with manual prompts, the inputs are constructed by leveraging T0 prompts , with CTR placed in front.
Revised: For CTR with manual prompts, the inputs are constructed by leveraging T0 prompts , with CTR placed in front.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 2.

Original: We report both the CTR results as well as its “upper-bound” results.
Revised: We report both the CTR results as well as its “upper-bound” results.

Original: It is noteworthy that the “upper-bound” results are post-hoc, which are obtained by the best code/prompt with observing the test task performance, and are merely given for estimating the potential of the two methods.
Revised: It is noteworthy that the “upper-bound” results are post-hoc, which are obtained by the best code/prompt with observing the test task performance, and are merely given for estimating the potential of the two methods.

Original: We shall observe that adding additional discrete manual prompts does not improve the CTR performance as well as the “upper-bound”.
Revised: We shall observe that adding additional discrete manual prompts does not improve the CTR performance as well as the “upper-bound”.

Original: From the results, we shall conclude that (1) our CTR does not rely on manual prompts to obtain optimal performance.
Revised: From the results, we shall conclude that (1) our CTR does not rely on manual prompts to obtain optimal performance.

Original: CTR can work as an alternative to the prompt-based methods.
Revised: CTR can work as an alternative to the prompt-based methods.

Original: By comparing the CTR performance and the CTR upper-bound, there is still room to optimize the code-searching algorithm, and as a result to further improve the generalization performance.
Revised: By comparing the CTR performance and the CTR upper-bound, there is still room to optimize the code-searching algorithm, and as a result to further improve the generalization performance.

Original: [Table]
Revised: [Table]

Original: 4.3. T RAINING O BJECTIVES
Revised: 4.3. T RAINING O BJECTIVES

Original: Our CTR consists of three training loss items.
Revised: Our CTR consists of three training loss items.

Original: Intuitively, the LM loss is optimized to predict the correct answer of the tasks.
Revised: Intuitively, the LM loss is optimized to predict the correct answer of the tasks.

Original: The embedding loss and the commitment loss are optimized to minimize the distance between the query embeddings and the CTR that are lookup from the codebook embeddings.
Revised: The embedding loss and the commitment loss are optimized to minimize the distance between the query embeddings and the CTR that are lookup from the codebook embeddings.

Original: To study the effectiveness of each loss item, we conduct an ablation study on them, and the results are shown in Table 3.
Revised: To study the effectiveness of each loss item, we conduct an ablation study on them, and the results are shown in Table 3.

Original: Results show that removing either of the loss items will drastically hurt the zero-label performance , proving that either loss item is indispensable for the training of CTR .
Revised: Results show that removing either of the loss items will drastically hurt the zero-label performance , proving that either loss item is indispensable for the training of CTR .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. I N -D EPTH A NALYSIS 4.4. I NTERPRETABILITY
Revised: 4. I N -D EPTH A NALYSIS 4.4. I NTERPRETABILITY

Original: Since CTR is trained to represent tasks with compositional codes, each code associated with one of the key aspects of tasks, CTR demonstrates a certain degree of interpretability.
Revised: Since CTR is trained to represent tasks with compositional codes, each code associated with one of the key aspects of tasks, CTR demonstrates a certain degree of interpretability.

Original: Table 4 presents examples of CTR that shows how each compositional task code is possibly interpreted.
Revised: Table 4 presents examples of CTR that shows how each compositional task code is possibly interpreted.

Original: Interestingly
Revised: Interestingly

Original: [Table]
Revised: [Table]

Original: it frequently occurred that tasks that share similar features indeed have the same compositional code.
Revised: it frequently occurred that tasks that share similar features indeed have the same compositional code.

Original: For example, the code 52 occurs in tasks that require extracting information from the given contexts, including samsum * (summarization task), wiki bio * (structured data to text task)
Revised: For example, the code 52 occurs in tasks that require extracting information from the given contexts, including samsum * (summarization task), wiki bio * (structured data to text task)

Original: [Table]
Revised: [Table]

Original: and paws labeled final paraphrase task (paraphrase generation) etc.
Revised: and paws labeled final paraphrase task (paraphrase generation) etc.

Original: Another example is that code 111 exists in most of the tasks that require generating long setences, including multi news * and samsum * (both are summarization tasks).
Revised: Another example is that code 111 exists in most of the tasks that require generating long setences, including multi news * and samsum * (both are summarization tasks).

Original: We also validate these explanations on unseen test tasks.
Revised: We also validate these explanations on unseen test tasks.

Original: Results show that these possible explanations still hold.
Revised: Results show that these possible explanations still hold.

Original: For example, the COPA task generates long answers given two candidate choices, and its compositional code has the code 111 (indicating sentence generation).
Revised: For example, the COPA task generates long answers given two candidate choices, and its compositional code has the code 111 (indicating sentence generation).

Original: 4.4. C ONTROLLABILITY
Revised: 4.4. C ONTROLLABILITY

Original: CTR also demonstrates a certain degree of controllability.
Revised: CTR also demonstrates a certain degree of controllability.

Original: Specifically, by modifying one bit of the compositional task code, our CTR will exhibit a different task behavior.
Revised: Specifically, by modifying one bit of the compositional task code, our CTR will exhibit a different task behavior.

Original: Table 5 shows several examples of controlling CTR.
Revised: Table 5 shows several examples of controlling CTR.

Original: We shall observe that, given the same inputs, by simply changing one
Revised: We shall observe that, given the same inputs, by simply changing one

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: bit of the compositional task code, the task behavior of CTR turns from DIALOGUEGENERATION to
Revised: bit of the compositional task code, the task behavior of CTR turns from DIALOGUE to

Original: TOPIC CLASSIFICATION , from to

Original: , from We shall conclude— (1) Since the inputs are randomly sampled from all tasks, some of which are quite different in data distribution, CTR performs well on them, proving that CTR indeed learns the ability to perform different tasks, instead of simply memorizing/overfitting to a certain dataset.
Revised: We shall conclude— (1) Since the inputs are randomly sampled from all tasks, some of which are quite different in data distribution, CTR performs well on them, proving that CTR indeed learns the ability to perform different tasks, instead of simply memorizing/overfitting to a certain dataset.

Original: (2) CTR is capable of switching between different tasks by simply changing one bit of the compositional code, proving that CTR effectively encodes the “task behavior” factor into the bits of the compositional code while disentangling other factors.
Revised: (2) CTR is capable of switching between different tasks by simply changing one bit of the compositional code, proving that CTR effectively encodes the “task behavior” factor into the bits of the compositional code while disentangling other factors.

Original: 4.4. H OW CTR G ENERALIZES TO N EW T ASKS
Revised: 4.4. H OW CTR G ENERALIZES TO N EW T ASKS

Original: To reveal how CTR essentially works, we use Figure 1 to explain how CTR generalizes from training tasks to unseen tasks such as COPA.
Revised: To reveal how CTR essentially works, we use Figure 1 to explain how CTR generalizes from training tasks to unseen tasks such as COPA.

Original: During the training phase, CTR learns the compositional task code for each task, each code associated with certain features of the task.
Revised: During the training phase, CTR learns the compositional task code for each task, each code associated with certain features of the task.

Original: For example, codeis associated with many of the multiple-choice/reasoning-based QA tasks; code 111 is associated with most of the long-sentence-generation tasks, etc.
Revised: For example, codeis associated with many of the multiple-choice/reasoning-based QA tasks; code 111 is associated with most of the long-sentence-generation tasks, etc.

Original: For a new task, its task code is a re-arranged composition of the code existing in training tasks.
Revised: For a new task, its task code is a re-arranged composition of the code existing in training tasks.

Original: If the new task shares the same code snippet as the training tasks, it inherits corresponding task features as well.
Revised: If the new task shares the same code snippet as the training tasks, it inherits corresponding task features as well.

Original: In the example of Figure 1, a real case produced by our model, the new task COPA is performed by combining the abilities of reasoning-based QA, sentence generation, and multi-choice selection from training tasks.
Revised: In the example of Figure 1, a real case produced by our model, the new task COPA is performed by combining the abilities of reasoning-based QA, sentence generation, and multi-choice selection from training tasks.

Original: These observations are consistent with our motivation of learning CTR for cross-task generalization.
Revised: These observations are consistent with our motivation of learning CTR for cross-task generalization.

Original: 4.4. G ENERALIZING TO NEW UNSEEN DOMAINS / CLASSES
Revised: 4.4. G ENERALIZING TO NEW UNSEEN DOMAINS / CLASSES

Original: Another interesting finding is that, for tasks with limited categorical classes, e.g., topic classification tasks, CTR is capable of generalizing to new unseen classes/domains.
Revised: Another interesting finding is that, for tasks with limited categorical classes, e.g., topic classification tasks, CTR is capable of generalizing to new unseen classes/domains.

Original: Table 6 shows several examples.
Revised: Table 6 shows several examples.

Original: Given different inputs that are randomly sampled from a large set of tasks and the compositional task code of AGNews, CTR not only predicts topic classes that are originally defined
Revised: Given different inputs that are randomly sampled from a large set of tasks and the compositional task code of AGNews, CTR not only predicts topic classes that are originally defined

Original: [Table] [Table]
Revised: [Table] [Table]

Original: by AGNews, but also predicts new topic classes that never occur in AGNews.
Revised: by AGNews, but also predicts new topic classes that never occur in AGNews.

Original: Please refer to Appendix A.1 and Appendix A.2 for more cases.
Revised: Please refer to Appendix A.1 and Appendix A.2 for more cases.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we explore cross-task generalization from a new perspective—compositional generalization.
Revised: In this paper, we explore cross-task generalization from a new perspective—compositional generalization.

Original: We propose the Compositional Task Representations (CTR) method that learns a discrete compositional codebook for tasks and generalizes to new unseen tasks by forming new compositions of the task codes.
Revised: We propose the Compositional Task Representations (CTR) method that learns a discrete compositional codebook for tasks and generalizes to new unseen tasks by forming new compositions of the task codes.

Original: For the inference of CTR , we propose two algorithms — Code Ensemble and
Revised: For the inference of CTR , we propose two algorithms — Code Ensemble and

Original: Bitwise Search, respectively for zero-label and few-shot settings.
Revised: Bitwise Search, respectively for zero-label and few-shot settings.

Original: Experiments demonstrate that our
Revised: Experiments demonstrate that our

Original: CTR significantly outperforms existing prompted-based methods on both zero-label and few-shot settings.
Revised: CTR significantly outperforms existing prompted-based methods on both zero-label and few-shot settings.

Original: Analysis of the learned compositional task codes proves that some of the CTR codes show certain degrees of interpretability and controllability.
Revised: Analysis of the learned compositional task codes proves that some of the CTR codes show certain degrees of interpretability and controllability.

Original: A A PPENDIX A. E XAMPLES OF AGN EWS
Revised: A A PPENDIX A. E XAMPLES OF AGN EWS

Original: [Table]
Revised: [Table]

Original: Fears
Revised: Turner Newall say they are

Original: [Table]
Revised: [Table]

Original: T N pension after talks Unions representing workers at Turner Newall say they are ’disappointed’ after talks with strickenparent firm Federal Mogul.
Revised: stricken

Original: [Table]
Revised: [Table]

Original: Q
Revised: firm Federal Mogul

Original: [Table]
Revised: [Table]

Original: : Prediction Unit Helps Forecast Wildfires (AP) AP - It’s barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring.
Revised: : Prediction Unit Helps Forecast Wildfires (AP) AP - It’s barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring.

Original: Lightning will strike in places he expects.
Revised: [Table] will [Table] in places he expects.

Original: Winds will pick up, moist places will dry and flames will roar.
Revised: Winds will pick up, moist places will dry and flames will roar

Original: [Table]
Revised: [Table]

Original: Science and technology

Original: I
Revised: There

Original: [Table]
Revised: [Table]

Original: How did serfdom develop in and then leave Russia?

Original: [Table]
Revised: [Table]

Original: I
Revised: on

Original: [Table]
Revised: [Table]

Original: : There are 10 apples on an apple tree.
Revised: tree.

Original: Three fall off.
Revised: Three [Table] off.

Original: Now there are X apples.
Revised: Now there are X apples.

Revised: [Table]

Revised: I

Revised: [Table]

Revised: the rock is destined to be the 21st

Original: [Table]
Revised: [Table]

Original: I NPUT : the rock is destined to be the 21st century’s new “conan” and that he’s going to make a splash even greater than arnold schwarzenegger, jean-claud van damme or steven segal.
Revised: new “conan” and that he’s going to make a splash even greater than arnold [Table] , jean-claud van damme or steven [Table]

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: Film I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.
Revised: I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.

Original: A major part of their output was under sub-contract to motor vehicle manufacturers.
Revised: A major part of their output was under sub-contract to motor vehicle manufacturers.

Original: Their business closed in 1972.
Revised: Their business closed in 1972.

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Revised: O

Original: O UTPUT ( OUT OF ORIGINAL CLASSES ): Company
Revised: [Table] OUT OF [Table] CLASSES ): Company

Original: I NPUT : Stuning even for the non-gamer This sound track was beautiful!
Revised: I NPUT : Stuning even for the non-gamer This sound track was beautiful!

Original: It paints the senery in your mind so well I would recomend it even to people who hate vid.
Revised: It paints the senery in your mind so well I would recomend it even to people who hate vid.

Original: game music!
Revised: game music!

Original: I have played the game Chrono Cross but out of all of the games I have ever played it has the best music!
Revised: I have played the game Chrono Cross but out of all of the games I have ever played it has the best music!

Original: It backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras.
Revised: It backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras.

Original: It would impress anyone who cares to listen!
Revised: It would impress anyone who cares to listen!

Original: M

Original: ODEL O

Original: [Table]
Revised: [Table]

Original: [Table] I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.
Revised: I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.

Original: A major part of their output was under sub-contract to motor vehicle manufacturers.
Revised: A major part of their output was under sub-contract to motor vehicle manufacturers.

Original: Their business closed in 1972.
Revised: Their business closed in 1972.

Original: M

Original: ODEL

Original:  O UTPUT ( OUT OF ORIGINAL CLASSES ): Company
Revised: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ): Company

Original: Q UESTION : Slack (2003) compares three groups that conducted biological research at Yale during overlapping periods between 1910 and 1970.
Revised: Q UESTION : Slack (2003) compares three groups that conducted biological research at Yale during overlapping periods between 1910 and 1970.

Original: Yale proved important as a site for this research.
Revised: Yale proved important as a site for this research.

Original: The leaders of these groups were Ross Granville Harrison, Grace E. Pickford, and G. Evelyn Hutchinson, and their members included both graduate students and more experienced scientists.
Revised: The leaders of these groups were Ross Granville Harrison, Grace E. Pickford, and G. Evelyn Hutchinson, and their members included both graduate students and more experienced scientists.

Original: All produced innovative research, including the opening of new subfields in embryology, endocrinology, and ecology, respectively, over a long period of time.
Revised: All produced innovative research, including the opening of new subfields in embryology, endocrinology, and ecology, respectively, over a long period of time.

Original: Harrison’s group is shown to have been a classic research school; Pickford’s and Hutchinson’s were not.
Revised: Harrison’s group is shown to have been a classic research school; Pickford’s and Hutchinson’s were not.

Original: Pickford’s group was successful in spite of her lack of departmental or institutional position or power.
Revised: Pickford’s group was successful in spite of her lack of departmental or institutional position or power.

Original: Hutchinson and his graduate and postgraduate students were extremely productive, but in diverse areas of ecology rather than one focused area of research or the use of one set of research tools.
Revised: Hutchinson and his graduate and postgraduate students were extremely productive, but in diverse areas of ecology rather than one focused area of research or the use of one set of research tools.

Original: Hutchinson’s example shows that new models for research groups are needed, especially for those that include extensive field research.
Revised: Hutchinson’s example shows that new models for research groups are needed, especially for those that include extensive field research.

Original: M ODEL A NSWER ( OUT OF ORIGINAL CLASSES ): Ecology
Revised: M ODEL A NSWER ( OUT OF ORIGINAL CLASSES ): Ecology

Original: A. E XAMPLES OF DBPEDIA
Revised: A. E XAMPLES OF DBPEDIA

Original: Q UESTION : Federation of International Trade Associations - The Federation of International Trade
Revised: Q UESTION : Federation of International Trade Associations - The Federation of International Trade

Original: Associations (FITA) based in Reston Virginia and New York New York USA was founded in 1984.
Revised: Associations (FITA) based in Reston Virginia and New York New York USA was founded in 1984.

Original: It fosters international trade by seeking to strengthen the role of associations in the United States
Revised: It fosters international trade by seeking to strengthen the role of associations in the United States

Original: Mexico and Canada.
Revised: Mexico and Canada.

Original: FITA is the strategic partner of the United States Commercial Service for ecommerce.
Revised: FITA is the strategic partner of the United States Commercial Service for ecommerce.

Original: company, educational institution, artist, athlete, office holder, mean of transportation, building, natural place, village, animal, plant, album, film or written work
Revised: company, educational institution, artist, athlete, office holder, mean of transportation, building, natural place, village, animal, plant, album, film or written work

Original: M ODEL A NSWER  WITHIN THE ORIGINAL CLASSES 
Revised: M ODEL A NSWER ( WITHIN THE ORIGINAL CLASSES ):

Original: ): Company
Revised: Company

Original: Q UESTION : Furian knife
Revised: Q UESTION : Furian knife

Original: In this movie sequel, Vin Diesel returns as Riddick, an escaped convict with a price on his head.
Revised: In this movie sequel, Vin Diesel returns as Riddick, an escaped convict with a price on his head.

Original: Riddick has been hiding on a snow planet for the last five years, when a group of mercenaries try to capture him.
Revised: Riddick has been hiding on a snow planet for the last five years, when a group of mercenaries try to capture him.

Original: Riddick returns to the planet Helion, and finds out that his friend Jack, is in prison on Crematoria, a very hot planet.
Revised: Riddick returns to the planet Helion, and finds out that his friend Jack, is in prison on Crematoria, a very hot planet.

Original: While on Helion, the planet is invaded by the Necromongers, led by Lord Marshal (Colm Feore), who wants to rule the universe.
Revised: While on Helion, the planet is invaded by the Necromongers, led by Lord Marshal (Colm Feore), who wants to rule the universe.

Original: Riddick is captured by the mercs and flown to the same unsavoury and possibly illegal prison Kyra is in.
Revised: Riddick is captured by the mercs and flown to the same unsavoury and possibly illegal prison Kyra is in.

Original: Turns out Jack is now known as Kyra (Alexa Davalos) and is tough as nails.M ODEL
Revised: Turns out Jack is now known as Kyra (Alexa Davalos) and is tough as nails.

Original: A
Revised: M

Original:  NSWER ( WITHIN THE ORIGINAL CLASSES ):
Revised: ODEL A NSWER ( WITHIN THE ORIGINAL CLASSES ):

Original: Film
Revised: Film

Original: Q UESTION : lila abu-lughod’s: - name - alma mater - website - known for - birth date - employer - nationality - occupation Bio: lila abu-lughod -lrb- born 1952 -rrb- is an american with palestinian and jewish ancestry who is professor of anthropology and women’s and gender studies at columbia university in new york city.
Revised: Q UESTION : lila abu-lughod’s: - name - alma mater - website - known for - birth date - employer - nationality - occupation Bio: lila abu-lughod -lrb- born 1952 -rrb- is an american with palestinian and jewish ancestry who is professor of anthropology and women’s and gender studies at columbia university in new york city.

Original: a specialist of the arab world, her seven books, most based on long term ethnographic research, cover topics from sentiment and poetry to nationalism and media, from gender politics to the politics of memory.
Revised: a specialist of the arab world, her seven books, most based on long term ethnographic research, cover topics from sentiment and poetry to nationalism and media, from gender politics to the politics of memory.

Original: M ODEL A NSWER ( WITHIN THE ORIGINAL CLASSES ): Educational Institution
Revised: M ODEL A NSWER ( WITHIN THE ORIGINAL CLASSES ): Educational Institution

Original: I NPUT : when he dug his massive body into the batter’s box tuesday night at legends field during an exhibition game against the new york yankees, adam dunn not only wasn’t hitting his weight, he wasn’t hitting the weight of an UNK bag of baseballs.
Revised: I NPUT : when he dug his massive body into the batter’s box tuesday night at legends field during an exhibition game against the new york yankees, adam dunn not only wasn’t hitting his weight, he wasn’t hitting the weight of an UNK bag of baseballs.

Original: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ): Sports
Revised: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ): Sports

Original: I NPUT :
Revised: I NPUT :

Original: We know that, thanks to our DNA, each of us is a little bit different.
Revised: We know that, thanks to our DNA, each of us is a little bit different.

Original: Some of those differences are obvious, like eye and hair color.
Revised: Some of those differences are obvious, like eye and hair color.

Original: Others are not so obvious, like how our bodies react to medication.
Revised: Others are not so obvious, like how our bodies react to medication.

Original: Researchers are beginning to look at how to tailor medical treatments to our genetic profiles, in a relatively new field called pharmacogenomics.
Revised: Researchers are beginning to look at how to tailor medical treatments to our genetic profiles, in a relatively new field called pharmacogenomics.

Original: Some of the biggest breakthroughs have been in cancer treatment.
Revised: Some of the biggest breakthroughs have been in cancer treatment.

Original: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES 
Revised: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ): Science

Original: ): Science

Original: I NPUT : how did athenians make money?
Revised: I NPUT : how did athenians make money?

Original: Other Greek cities set up democracies, and even though most followed an Athenian model, none were as powerful, stable, nor as well-documented as that of
Revised: Other Greek cities set up democracies, and even though most followed an Athenian model, none were as powerful, stable, nor as well-documented as that of

Original: Athens.
Revised: Athens.

Original: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ) City
Revised: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ): City

Original: I NPUT : Great food, portions could be smaller.
Revised: I NPUT : Great food, portions could be smaller.

Original: A little pricey for Middleton.
Revised: A little pricey for Middleton.

Original: Would try somewhere else next time but would also not hesitate to return
Revised: Would try somewhere else next time but would also not hesitate to return

Original: M ODEL O UTPUT  OUT OF ORIGINAL CLASSES 
Revised: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ):

Original: ): Restaurant
Revised: Restaurant

Original: I NPUT : Facts: - name: mr. marcus - spelling: us - caption: mr. october 2007 marcus at a porn star karaoke event, - height: 5 8 - alias: mister marcus, mr. marquis - image: mr marcus, 2007.jpg - birth date: 4 september 1970 - birth name: jesse spencer - weight: 200 lb kg on - ethnicity: black - birth place: pomona, california, usa - hair color: black - number of films: 1,782 -lrb- per iafd -rrb- as a performer & 36 as a director - eye color: brown mr. marcus.
Revised: I NPUT : Facts: - name: mr. marcus - spelling: us - caption: mr. october 2007 marcus at a porn star karaoke event, - height: 5 8 - alias: mister marcus, mr. marquis - image: mr marcus, 2007.jpg - birth date: 4 september 1970 - birth name: jesse spencer - weight: 200 lb kg on - ethnicity: black - birth place: pomona, california, usa - hair color: black - number of films: 1,782 -lrb- per iafd -rrb- as a performer & 36 as a director - eye color: brown mr. marcus.

Original: M ODEL O UTPUT  OUT OF ORIGINAL CLASSES ): Actor
Revised: M ODEL O UTPUT ( OUT OF ORIGINAL CLASSES ): Actor

Original: [Table]
Revised: [Table]

Revised: ABEL

Original: [Table]
Revised: [Table]

Original: C OMPONENT A RCHITECTURE

Original: [Table]
Revised: [Table]

Original: study on different architectures of the CTR encoder and decoder.
Revised: study on different architectures of the CTR encoder and decoder.

Original: All experiments are conducted under the zero-label setting.
Revised: All experiments are conducted under the zero-label setting.

Original: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).
Revised: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).

Original: Results show that the simple “Linear - Linear” combination achieves the best performance.
Revised: Results show that the simple “Linear - Linear” combination achieves the best performance.

Original: Table 10 shows the zero-label results when using different architecture of the encoder and the decoder.
Revised: Table 10 shows the zero-label results when using different architecture of the encoder and the decoder.

Original: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).
Revised: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).

Original: Results show that simply using a linear network for both the encoder and the decoder performs best, while complicated architectures, e.g., Transformer and MLP, yield poor performance.
Revised: Results show that simply using a linear network for both the encoder and the decoder performs best, while complicated architectures, e.g., Transformer and MLP, yield poor performance.

Original: This could because that the learning of codebook embeddings as well as the code mapping does not necessarily need complicated transformation.
Revised: This could because that the learning of codebook embeddings as well as the code mapping does not necessarily need complicated transformation.

Original: When using complicated architectures, it instead increases the difficulties of the learning process.
Revised: When using complicated architectures, it instead increases the difficulties of the learning process.

Original: A. C ODEBOOK S IZE A ND CTR L ENGTH
Revised: A. C ODEBOOK S IZE A ND CTR L ENGTH

Original: There are two critical hyper-parameters for CTR —the codebook size and the CTR length.
Revised: There are two critical hyper-parameters for CTR —the codebook size and the CTR length.

Original: We conduct experiments of different selection of the two hyper-parameters
Revised: We conduct experiments of different selection of the two hyper-parameters

Original: [Table]
Revised: [Table]

Original: and see how it influence the performance of zero-label generalization.
Revised: and see how it influence the performance of zero-label generalization.

Original: Table 11 presents the results.
Revised: Table 11 presents the results.

Original: We shall observe that when the codebook size decreases below 64, the performance decreases to a large degree.
Revised: We shall observe that when the codebook size decreases below 64, the performance decreases to a large degree.

Original: We conjecture that it is due to the capacity of a codebook size below 64 is not sufficient for representing the multiple aspects of the tasks.
Revised: We conjecture that it is due to the capacity of a codebook size below 64 is not sufficient for representing the multiple aspects of the tasks.

Original: In addition, experimenting with a CTR length of 10 generally outperforms those with larger CTR length
Revised: In addition, experimenting with a CTR length of 10 generally outperforms those with larger CTR length

Original: [Table]
Revised: [Table]

Original: 11: Ablation study on different codebook size and CTR length.
Revised: : Ablation study on different codebook size and CTR length.

Original: All experiments are conducted under the zero-label setting.
Revised: All experiments are conducted under the zero-label setting.

Original: We experiment three of the codebook size, respectively 128,and 48, and two of the CTR length, including 10 and 20.
Revised: We experiment three of the codebook size, respectively 128,and 48, and two of the CTR length, including 10 and 20.

Original: Results show that ”128 - 10” combination achieves the best performance.
Revised: Results show that ”128 - 10” combination achieves the best performance.

Original: A. P ERFORMANCE S ENSITIVITY TO DIFFERENT SELECTION OF CODES
Revised: A. P ERFORMANCE S ENSITIVITY TO DIFFERENT SELECTION OF CODES

Original: We want to verify whether different selections of codes will affect the performance of the model.
Revised: We want to verify whether different selections of codes will affect the performance of the model.

Original: Intuitively, if a code is merely noise that gets ignored by the model eventually, the model will not sensitive to different selections of codes and vice versa.
Revised: Intuitively, if a code is merely noise that gets ignored by the model eventually, the model will not sensitive to different selections of codes and vice versa.

Original: Therefore, we randomly sampled several codes of training tasks and evaluate their performance on different test tasks.
Revised: Therefore, we randomly sampled several codes of training tasks and evaluate their performance on different test tasks.

Original: The results are shown in Table 12.
Revised: The results are shown in Table 12.

Original: As shown in the table, the model is highly sensitive to a different selection of codes.
Revised: As shown in the table, the model is highly sensitive to a different selection of codes.

Original: For example, the code of task amazon polarity user satisfied can perform well on RTE and significantly better than the performance of code of task cnn dailymail 3.0.0 news card view
Revised: For example, the code of task amazon polarity user satisfied can perform well on RTE and significantly better than the performance of code of task cnn dailymail 3.0.0 news card view

Original: [Table]
Revised: [Table]

Original: A. E XPERIMENTAL D ETAILS
Revised: A. E XPERIMENTAL D ETAILS

Original: For the data preprocessing of all experiments, to balance the number of data for different tasks, we restrict the maximum data examples for each training task to be 50,000, which empirically yields better results.
Revised: For the data preprocessing of all experiments, to balance the number of data for different tasks, we restrict the maximum data examples for each training task to be 50,000, which empirically yields better results.

Original: Training details of each baseline method under the zero-label setting are illustrated as follows.
Revised: Training details of each baseline method under the zero-label setting are illustrated as follows.

Original: T0-Large Based on T5-Large-LM-Adapted, it performs multi-task training for 10000 steps.
Revised: T0-Large Based on T5-Large-LM-Adapted, it performs multi-task training for 10000 steps.

Original: We set the maximum length of input and target sequences to 384 and 32 respectively.
Revised: We set the maximum length of input and target sequences to 384 and 32 respectively.

Original: We use the Adam optimizer with a learning rate of 1e-4, a dropout rate of 0.1, and a batch size of 1024.
Revised: We use the Adam optimizer with a learning rate of 1e-4, a dropout rate of 0.1, and a batch size of 1024.

Original: Following
Revised: Following

Original: T0 Sanh et al.
Revised: T0 Sanh et al.

Original: (2022), we use the same task prompts from PromptSource(Bach et al., 2022).
Revised: (2022), we use the same task prompts from PromptSource(Bach et al., 2022).

Original: We report the average accuracy of multiple prompts for each test task.
Revised: We report the average accuracy of multiple prompts for each test task.

Original: Note that our reproduced T0Large results are much better than those reported in the original paper Sanh et al.
Revised: Note that our reproduced T0Large results are much better than those reported in the original paper Sanh et al.

Original: (2022), which sets a much stronger baseline for comparison.
Revised: (2022), which sets a much stronger baseline for comparison.

Original: We report the average accuracy of prompts for each test task.
Revised: We report the average accuracy of prompts for each test task.

Original: We believe our baseline is well-optimized.
Revised: We believe our baseline is well-optimized.

Original: Because the performance on test tasks is comparable to the results of T0-3B reported in Sanh et al.
Revised: Because the performance on test tasks is comparable to the results of T0-3B reported in Sanh et al.

Original: , even our baseline only contains 770M parameters.
Revised: , even our baseline only contains 770M parameters.

Original: Self-Training For a fair comparison, we randomly sample 32 unlabeled data for self-training, which is the same as CTR.
Revised: Self-Training For a fair comparison, we randomly sample 32 unlabeled data for self-training, which is the same as CTR.

Original: The Self-Training method trains from the T0-Large with these pseudo-labeled examples for 5 epochs and reports average performance over prompts.
Revised: The Self-Training method trains from the T0-Large with these pseudo-labeled examples for 5 epochs and reports average performance over prompts.

Original: For the training, we use a batch size of 32 and the Adam optimizer with a learning rate of 1e-4.
Revised: For the training, we use a batch size of 32 and the Adam optimizer with a learning rate of 1e-4.

Original: Manual-Code
Revised: Manual-Code

Original: In practice, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, whether it includes options into inputs, and whether it is a classification task, etc.
Revised: In practice, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, whether it includes options into inputs, and whether it is a classification task, etc.

Original: Each task can be represented as an artificially-defined discrete feature vector, each dimension associated with one of the aspects.
Revised: Each task can be represented as an artificially-defined discrete feature vector, each dimension associated with one of the aspects.

Original: Manual-Code follows exact the same training recipe as the second training phase of our CTR.
Revised: Manual-Code follows exact the same training recipe as the second training phase of our CTR.

Original: Training details are presented in Section 4.1.3.
Revised: Training details are presented in Section 4.1.3.

Original: Zero-Label Prompt Selection (ZPS)
Revised: Zero-Label Prompt Selection (ZPS)

Original: (Liao et al., 2022a)
Revised: (Liao et al., 2022a)

Original: For each task, we use 32 unlabeled data for producing pseudo-labeled data.
Revised: For each task, we use 32 unlabeled data for producing pseudo-labeled data.

Original: Finally, we report the accuracy of the selected prompt.
Revised: Finally, we report the accuracy of the selected prompt.

Original: For the few-shot setting, we consider the following five baseline methods.
Revised: For the few-shot setting, we consider the following five baseline methods.

Original: All few-shot baselines are based on our reproduced T0-Large.
Revised: All few-shot baselines are based on our reproduced T0-Large.

Original: Model Tuning We use the Adam optimizer with a batch size of 256 and a learning rate of 1e-4.
Revised: Model Tuning We use the Adam optimizer with a batch size of 256 and a learning rate of 1e-4.

Original: We combine all training data of the test tasks for training.
Revised: We combine all training data of the test tasks for training.

Original: The maximum training step is 100.
Revised: The maximum training step is 100.

Original: We use a validation set for model selection.
Revised: We use a validation set for model selection.

Original: Finally, we report the average accuracy of the selected best checkpoint.
Revised: Finally, we report the average accuracy of the selected best checkpoint.

Original: Prompt Tuning (Lester et al., 2021)
Revised: Prompt Tuning (Lester et al., 2021)

Original: We use the Adam optimizer with a batch size of 128 and a learning rate of 0.05.
Revised: We use the Adam optimizer with a batch size of 128 and a learning rate of 0.05.

Original: We combine all training data of the test tasks for training.
Revised: We combine all training data of the test tasks for training.

Original: The maximum training step is 100.
Revised: The maximum training step is 100.

Original: The length of continuous prompts for each task is 20.
Revised: The length of continuous prompts for each task is 20.

Original: We use a validation set for model selection.
Revised: We use a validation set for model selection.

Original: Finally, we report the average accuracy of the selected best checkpoint.
Revised: Finally, we report the average accuracy of the selected best checkpoint.

Original: GPS (Xu et al., 2022)
Revised: GPS (Xu et al., 2022)

Original: We follow hyper-parameters reported in the original paper.
Revised: We follow hyper-parameters reported in the original paper.

Original: Specifically, we run the GPS for 6 steps.
Revised: Specifically, we run the GPS for 6 steps.

Original: At each step, new prompts are generated by a T5-xxl-lm-adapted model.
Revised: At each step, new prompts are generated by a T5-xxl-lm-adapted model.

Original: GRIPS In our experiments, we set max patience P = 2 , candidate l = 5 , and step m = 5 .
Revised: GRIPS In our experiments, we set max patience P = 2 , candidate l = 5 , and step m = 5 .

Original: For the rest of the hyper-parameters, we follow the original GRIPS
Revised: For the rest of the hyper-parameters, we follow the original GRIPS

Original: paper(Prasad et al., 2022).
Revised: paper(Prasad et al., 2022).

Original: Black-Box Tuning (BBT) (Sun et al., 2022)
Revised: Black-Box Tuning (BBT) (Sun et al., 2022)

Original: In practice, we use the Adam optimizer with a learning rate of 0.05.
Revised: In practice, we use the Adam optimizer with a learning rate of 0.05.

Original: For each test task, we train the soft prompt for 200 steps.
Revised: For each test task, we train the soft prompt for 200 steps.

Original: We set the prompt length
Revised: We set the prompt length

Original: L = 10 , subspace dimension d = 500 , and cma budget 1000.
Revised: L = 10 , subspace dimension d = 500 , and cma budget 1000.

Original: For the rest of the hyper-parameters, we follow the original BBT(Sun et al., 2022).
Revised: For the rest of the hyper-parameters, we follow the original BBT(Sun et al., 2022).