Original: C OMPOSITIONAL T ASK R EPRESENTATIONS FOR L ARGE L ANGUAGE M ODELS
Revised: C OMPOSITIONAL T ASK R EPRESENTATIONS FOR L ARGE L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large language models have shown a remarkable cross-task generalization ability.
Revised: Large language models have shown a remarkable cross-task generalization ability.

Original: Most prior work assumed that prompts effectively extract knowledge from language models to facilitate generalization to new tasks.
Revised: Most prior works assumed that prompts effectively extract knowledge from language models to facilitate generalization to new tasks.

Original: This perspective led to numerous studies on improving prompts.
Revised: This perspective led to numerous studies on improving prompts.

Original: In contrast, we introduce a new perspective, compositional generalization, that views each task as a composition of latent codes and generalizes to test tasks by a new composition of seen codes.
Revised: In contrast, we introduce a new perspective, compositional generalization, that views each task as a composition of latent codes and generalizes to test tasks by a new composition of seen codes.

Original: To this end, we propose a novel prompt-free approach, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.
Revised: To this end, we propose a novel prompt-free approach, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.

Original: Empirically, our CTR substantially outperforms prompt-based methods in zero-label learning on average.
Revised: Empirically, our CTR substantially outperforms prompt-based methods in zero-label learning on average.

Original: According to our analysis, some of the learned CTR codes are interpretable to human and demonstrate a certain degree of controllability.
Revised: According to our analysis, some of the learned CTR codes are interpretable to humans and demonstrate a certain degree of controllability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large language models (LLMs) have shown remarkable performance in cross-task generalization.
Revised: Large language models (LLMs) have shown remarkable performance in cross-task generalization.

Original: Without using any labeled data for the target task, GPT-3 (Brown et al., 2020) obtains reasonable performance on a wide range of tasks.
Revised: Without using any labeled data for the target task, GPT-3 (Brown et al., 2020) obtains reasonable performance on a wide range of tasks.

Original: Later extensions such as FLAN (Wei et al., 2022) and T0 (Sanh et al., 2022) continue training the LLMs on a large number of supervised tasks, which further improves cross-task generalization performance.
Revised: Later extensions such as FLAN (Wei et al., 2022) and T0 (Sanh et al., 2022) continue training the LLMs on a large number of supervised tasks, which further improves cross-task generalization performance.

Original: The aforementioned studies have used an important assumption that natural language prompts extract knowledge from LLMs to facilitate generalization to new tasks.
Revised: The aforementioned studies have used an important assumption that natural language prompts extract knowledge from LLMs to facilitate generalization to new tasks.

Original: In this direction, numerous studies have focused on different aspects of improving prompt-based learning, such as designing better prompts (Xu et al., 2022), increasing the number of prompts (Wang et al., 2022;
Revised: In this direction, numerous studies have focused on different aspects of improving prompt-based learning, such as designing better prompts (Xu et al., 2022), increasing the number of prompts (Wang et al., 2022;

Original: Aribandi et al., 2022), and improving the training efficiency of prompts (Lester et al., 2021).
Revised: Aribandi et al., 2022), and improving the training efficiency of prompts (Lester et al., 2021).

Original: In contrast, we explore an alternative perspective for cross-task generalization, i.e., compositional generalization.
Revised: In contrast, we explore an alternative perspective for cross-task generalization, i.e., compositional generalization.

Original: Specifically, we explore whether it is possible to represent tasks using discrete compositions of latent codes.
Revised: Specifically, we explore whether it is possible to represent tasks using discrete compositions of latent codes.

Original: This perspective enjoys several potential benefits.
Revised: This perspective enjoys several potential benefits.

Original: First, since the latent codes have been trained for seen tasks, we expect the LLMs to have strong cross-task generalization abilities because new tasks can also be represented as a composition of these trained codes.
Revised: First, since the latent codes have been trained for seen tasks, we expect the LLMs to have strong cross-task generalization abilities because new tasks can also be represented as a composition of these trained codes.

Original: Second, it provides a way to analyze and understand cross-task generalization by investigating the association between tasks and the learned representations.
Revised: Second, it provides a way to analyze and understand cross-task generalization by investigating the association between tasks and the learned representations.

Original: Third, it has the potential of being more controllable than prompts for task generalization due to its built-in compositionality.
Revised: Third, it has the potential of being more controllable than prompts for task generalization due to its built-in compositionality.

Original: Motivated by the aforementioned potentials, we propose a new method, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.
Revised: Motivated by the aforementioned potentials, we propose a new method, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook.

Original: Specifically, given a large number of training tasks, we use an encoder to map each randomlyinitialized task embedding to a fixed-length sequence of query vectors.
Revised: Specifically, given a large number of training tasks, we use an encoder to map each randomlyinitialized task embedding to a fixed-length sequence of query vectors.

Original: Each query vector is used to retrieve a code from a codebook, which is formulated as an embedding lookup table.
Revised: Each query vector is used to retrieve a code from a codebook, which is formulated as an embedding lookup table.

Original: This produces a sequence of codes, a compositional representation for the current task.
Revised: This produces a sequence of codes, a compositional representation of the current task.

Original: These compositional codes are fed as the input to an LLM in place of prompts to make predictions.
Revised: These compositional codes are fed as the input to an LLM in place of prompts to make predictions.

Original: At test time, given a new task, we use unlabeled data to search for a high-performing composition of codes, which enables zero-label cross-task generalization.
Revised: At test time, given a new task, we use unlabeled data to search for a high-performing composition of codes, which enables zero-label cross-task generalization.

Original: CTR is also applicable to the few-shot setting where the few labeled examples are used for code search.
Revised: CTR is also applicable to the few-shot setting where the few labeled examples are used for code search.

Original: Empirically, we demonstrate improved performance under both the settings of zero-label learning and few-shot learning, outperforming strong baselines including prompt tuning, model tuning, and
Revised: Empirically, we demonstrate improved performance under both the settings of zero-label learning and few-shot learning, outperforming strong baselines including prompt tuning, model tuning, and

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: genetic prompt search(Xu et al., 2022).
Revised: genetic prompt search (Xu et al., 2022).

Original: Importantly, we analyze the learned task representations and show that they demonstrate a certain degree of interpretability and controllability.
Revised: Importantly, we analyze the learned task representations and show that they demonstrate a certain degree of interpretability and controllability.

Original: For example, as shown in Figure 1, CTR learns to generalize to a new task by a new composition of existing codes.
Revised: For example, as shown in Figure 1, CTR learns to generalize to a new task by a new composition of existing codes.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Language Model Prompting.
Revised: Language Model Prompting.

Original: Brown et al.
Revised: Brown et al.

Original: (2020) showed that GPT-3 performs well in the fewshot setting if properly handcrafted prompts are provided.
Revised: (2020) showed that GPT-3 performs well in the fewshot setting if properly handcrafted prompts are provided.

Original: Other works (Shoeybi et al., 2019; Rae et al., 2021; Schick & Sch ¨ utze, 2021) also report promising zero-shot or few-shot performances of
Revised: Other works (Shoeybi et al., 2019; Rae et al., 2021; Schick & Sch ¨ utze, 2021) also report promising zero-shot or few-shot performances of

Original: LLMs.
Revised: LLMs.

Original: Besides, Wei et al.
Revised: Besides, Wei et al.

Original: and Sanh et al.
Revised: and Sanh et al.

Original: (2022) collect a set of labeled datasets and use manual templates to transform them into a sequence-to-sequence style.
Revised: (2022) collect a set of labeled datasets and use manual templates to transform them into a sequence-to-sequence style.

Original: Such a formulation makes it possible to continue training LLMs on these labeled datasets and improves cross-task generalization.
Revised: Such a formulation makes it possible to continue training LLMs on these labeled datasets and improves cross-task generalization.

Original: Wang et al.
Revised: Wang et al.

Original: ; Mishra et al.
Revised: ; Mishra et al.

Original: (2022) introduced a benchmark of over 1600 tasks and their expertwritten instructions Gao et al.
Revised: (2022) introduced a benchmark of over 1600 tasks and their expertwritten instructions Gao et al.

Original: Shin et al.
Revised: Shin et al.

Original: (2020) studied automating the search process of discrete prompts.
Revised: (2020) studied automating the search process of discrete prompts.

Original: Li & Liang (2021); Liu et al. (2021) propose continuous soft prompts with gradient-based optimization.
Revised: Li & Liang (2021); Liu et al. (2021) propose continuous soft prompts with gradient-based optimization.

Original: Compared to these approaches, we study a different approach that learns compositional task representations, which benefits cross-task generalization.
Revised: Compared to these approaches, we study a different approach that learns compositional task representations, which benefits cross-task generalization.

Original: Compositional Architecture for LLMs.
Revised: Compositional Architecture for LLMs.

Original: Previous work has explored designing compositional architectures.
Revised: Previous work has explored designing compositional architectures.

Original: Sparsely Gated Mixture of Expert (MoE) (Lepikhin et al., 2021) activates a subset of a network given the input data.
Revised: Sparsely Gated Mixture of Expert (MoE) (Lepikhin et al., 2021) activates a subset of a network given the input data.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2021) trained an MoE model with 1.1T parameters that is shown to outperform a dense model with similar compute.
Revised: (2021) trained an MoE model with 1.1T parameters, which is shown to outperform a dense model with similar computational cost.

Original: SkillNet-NLU (Tang et al., 2022) and SkillNet-NLG (Liao et al., 2022) employed a similar sparsely activated mechanism to handle different NLU or NLG tasks.
Revised: SkillNet-NLU (Tang et al., 2022) and SkillNet-NLG (Liao et al., 2022b) employed a similar sparsely activated mechanism to handle different NLU or NLG tasks.

Original: Different from these approaches, our approach focuses on learning compositional task representations using a discrete codebook.
Revised: Different from these approaches, our approach focuses on learning compositional task representations using a discrete codebook.

Original:  C OMPOSITIONAL T ASK R EPRESENTATIONS
Revised:  C OMPOSITIONAL T ASK R EPRESENTATIONS

Original: The motivation of CTR is to explore the cross-task generalization ability of LLMs from a brandnew perspective—compositional generalization, and to further improve the performance of crosstask generalization.
Revised: The motivation of CTR is to explore the cross-task generalization ability of LLMs from a brandnew perspective—compositional generalization, and to further improve the performance of crosstask generalization.

Original: Specifically, our main hypothesis is that being trained on a variety of natural 
Revised: Specifically, our main hypothesis is that being trained on a variety of natural language tasks, LLMs will be able to learn to represent each task as a composition of discrete latent codes, where each latent code is associated with certain aspects of a task. As a result, CTR poten-

Original: [Figure]
Revised: [Figure]

Original: language tasks, LLMs will be able to learn to represent each task as a composition of discrete latent codes, where each latent code is associated with certain aspects of a task.

Original: As a result, CTR potentially enjoys the advantages of better cross-task generalization since it could represent new tasks by forming new compositions.
Revised: tially enjoys the advantages of better cross-task generalization since it could represent new tasks by forming new compositions.

Original: This section introduces the overall architecture of CTR and how it is trained to overcome optimization challenges.
Revised: This section introduces the overall architecture of CTR and how it is trained to overcome optimization challenges.

Original: As Figure 2 shows, CTR consists of the CTR learning module and an LLM, where the CTR learning module contains an encoder, a decoder, task embeddings, and a codebook.
Revised: As Figure 2 shows, CTR consists of the CTR learning module and an LLM, where the CTR learning module contains an encoder, a decoder, task embeddings, and a codebook.

Original: 3. D ISCRETE L ATENT T ASK C ODEBOOK
Revised: 3. D ISCRETE L ATENT T ASK C ODEBOOK

Original: We define a latent task codebook embedding space C ∈ R S × D where S is the size of the latent codebook embedding space (i.e., each task code can take either of the S categorical values), and D is the dimension of each latent code embedding C i ∈ R D , i ∈
Revised: We define a latent task codebook embedding space C ∈ R S × D where S is the size of the latent codebook embedding space (i.e., each task code can take either of the S categorical values), and D is the dimension of each latent code embedding C i ∈ R D , i ∈

Original: [1 , 2 , · · · , S ] .
Revised: [1 , 2 , · · · , S ] .

Original: This is analogous to the idea of VQ-VAE (van den Oord et al., 2017) where a discrete latent codebook is also employed.
Revised: This is analogous to the idea of VQ-VAE (van den Oord et al., 2017) where a discrete latent codebook is also employed.

Original: As Figure 2 shows, given a training task as the input (assuming the task id is k ), CTR first obtains its task embedding E k ∈ R D by retrieving from a randomly-initialized task embedding lookup table
Revised: As Figure 2 shows, given a training task as the input (assuming the task id is k ), CTR first obtains its task embedding E k ∈ R D by retrieving from a randomly-initialized task embedding lookup table

Original: R N × D where N is the number of training tasks.
Revised: R N × D where N is the number of training tasks.

Original: The task embedding E k is then passed to the encoder module and is mapped into a fixed-length sequence of query vectors Q ∈ R L × D where L is the length of the sequence.
Revised: The task embedding E k is then passed to the encoder module and is mapped into a fixed-length sequence of query vectors Q ∈ R L × D where L is the length of the sequence.

Original: For each query vector Q l , l ∈ [1 , 2 , · · · , L ] , it is used to retrieve a task code embedding from the codebook embedding space.
Revised: For each query vector Q l , l ∈ [1 , 2 , · · · , L ] , it is used to retrieve a task code embedding from the codebook embedding space.

Original: Specifically, it calculates the l 2 distance with each of the latent code embedding C i ∈ R D , i ∈
Revised: Specifically, it calculates the l 2 distance with each of the latent code embedding C i ∈ R D , i ∈

Original: [1 , 2 , · · · , S ] , and find the nearest neighbor,
Revised: [1 , 2 , · · · , S ] , and find the nearest neighbor,

Original: CTR l = C

Original: [Equation]

Original: l

Original: [Equation]
Revised: [Equation]

Original: where z l = arg min i [Equation] In this way, all query vectors together produce a L -length sequence of code embeddings, denoted as the compositional task representations (CTR).
Revised: In this way, all query vectors together produce a L -length sequence of code embeddings, denoted as the compositional task representations (CTR).

Original: They are then optionally pass through a decoder, followed by being used as the input to an LLM in place of prompts to make predictions.
Revised: They are then passed through a decoder, followed by being used as the input to an LLM in place of prompts to make predictions.

Original: Let z be a vector of L latent codes z 1 , z 2 , . .
Revised: Let z be a vector of L latent codes z 1 , z 2 , . .

Original: z L .
Revised: z L .

Original: We consider each latent code as describing an attribute or a necessary skill of a certain task, such as task type, output space, etc.
Revised: We consider each latent code as describing an attribute or a necessary skill of a certain task, such as task type, output space, etc.

Original: Intuitively, we need multiple codes to fully describe a task, and each task is formulated as a composition of codes.
Revised: Intuitively, we need multiple codes to fully describe a task, and each task is formulated as a composition of codes.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: Training can be challenging due to the existence of discrete latent variables.
Revised: Training can be challenging due to the existence of discrete latent variables.

Original: Moreover, during the initial training phase, the CTR learning module is randomly initialized.
Revised: Moreover, during the initial training phase, the CTR learning module is randomly initialized.

Original: As a result, the codebook embeddings C will have a very different distribution to the query vectors Q , which increases the difficulty of optimization.
Revised: As a result, the codebook embeddings C will have a very different distribution to the query vectors Q , which increases the difficulty of optimization.

Original: We decouple training into two phases.
Revised: We decouple training into two phases.

Original: In the first phase, we freeze the LLMparameters and only update the CTR learning module.
Revised: In the first phase, we freeze the LLM and only update the CTR learning module.

Original: This is followed by tuning all parameters.
Revised: This is followed by tuning all parameters.

Original: In terms of the loss function, following van den Oord et al.
Revised: In terms of the loss function, following van den Oord et al.

Original: (2017), we employ two separate losses, an embedding loss and a commitment loss, to match the query vectors with the compositional task representations.
Revised: (2017), we employ two separate losses, an embedding loss and a commitment loss, to match the query vectors with the compositional task representations.

Original: These losses are used in combination with a standard language modeling loss,
Revised: These losses are used in combination with a standard language modeling loss,

Original: [Equation]
Revised: [Equation]

Original: where L LM is a standard language modeling loss for solving the target task, sg denotes stopping the gradients, and β is a hyperparameter.
Revised: where L LM is a standard language modeling loss for solving the target task, sg denotes stopping the gradients, and β is a hyperparameter.

Original: 3. I NFERENCE
Revised: 3. I NFERENCE

Original: We consider two settings: zero-label learning and few-shot learning, and describe how we apply
Revised: We consider two settings: zero-label learning and few-shot learning, and describe how we apply

Original: CTR in these settings.
Revised: CTR in these settings.

Original: Code Ensemble for Zero-Label Learning.
Revised: Code Ensemble for Zero-Label Learning.

Original: In the zero-labellearning setting, we are given a new task along with a set of unlabeled data.
Revised: In the zero-label setting, we are given a new task along with a set of unlabeled data.

Original: The question is how to decide the code for this new task without labeled data.
Revised: The question is how to decide the code for this new task without labeled data.

Original: Our main idea is to select one of the training tasks (319 in total in our experiments) and use its learned code for this new task.
Revised: Our main idea is to select one of the training tasks (319 in total in our experiments) and use its learned code for this new task.

Original: We first obtain a set of candidate codes by examining how much a code gives predictions that deviate from a uniform distribution on the unlabeled data (Zhao et al., 2021).
Revised: We first obtain a set of candidate codes by examining how much a code gives predictions that deviate from a uniform distribution on the unlabeled data (Zhao et al., 2021).

Original: The candidate set is formed by N (set to 60 in our experiments) codes with the lowest deviations.
Revised: The candidate set is formed by N (set to 60 in our experiments) codes with the lowest deviations.

Original: We then ensemble the candidate set of codes to predict pseudo labels on unlabeled data, and select the code with the highest pseudo-label accuracy.
Revised: We then ensemble the candidate set of codes to predict pseudo labels on unlabeled data, and select the code with the highest pseudo-label accuracy.

Original: Bitwise Search for Few-Shot Learning.
Revised: Bitwise Search for Few-Shot Learning.

Original: In the few-shotlearning setting, we are given a new task along with a set of labeled data.
Revised: In the few-shot setting, we are given a new task along with a set of labeled data.

Original: We use this set of labeled data as our validation set to search for a high-performing code.
Revised: We use this set of labeled data as our validation set to search for a high-performing code.

Original: Our preliminary study shows that one can control the output of CTR by changing one bit in the code vector z .
Revised: Our preliminary study shows that one can control the output of CTR by changing one bit in the code vector z .

Original: Inspired by this, we use the  codes of training tasks as an initialization and iteratively change a single bit of the code.
Revised: Inspired by this, we first examine the validation-set accuracy for codes of training tasks, and select the code with the best accuracy as an initialization. Then we iteratively change a single bit of the selected code and evaluate the validation-set performance. At each iteration, we keep the updated code with higher performance.

Original: We then select the code  that obtains the best result on the validation set.
Revised: Finally, the code vector that obtains the best result on the validation set is taken as the test-task code.

Revised: Our preliminary study shows that a certain code value usually occurs in a specific position of the code vector, and each position usually only has a small set of code values.

Revised: Motivated by this, we only changes a small set of valid code candidates for each position.

Original: In both the zero-label and few-shot settings, after we obtain a code vector z for the new task, we use the vector to obtain a composition task representation by indexing z in the codebook C .
Revised: In both the zero-label and few-shot settings, after we obtain a code vector z for the new task, we use the vector to obtain a composition task representation by indexing z in the codebook C .

Original: The task representations are then passed through the decoder and the LLM to perform the new task, similar to training time.
Revised: The task representations are then passed through the decoder and the LLM to perform the new task, similar to training time.

Original: This is also illustrated in the right part of Figure 2.
Revised: This is also illustrated in the right part of Figure 2.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: We conduct extensive quantitative experiments to validate the performance of cross-task generalization of CTR.
Revised: We conduct extensive quantitative experiments to validate the performance of cross-task generalization of CTR.

Original: We mainly consider two settings, the zero-label setting and the few-shot setting.
Revised: We mainly consider two settings, the zero-label setting and the few-shot setting.

Original: Aside from quantitative experiments, we perform qualitative analysis to understand the cross-task generalization ability by investigating the association between the discrete latent codes and the tasks.
Revised: Aside from quantitative experiments, we perform qualitative analysis to understand the cross-task generalization ability by investigating the association between the discrete latent codes and the tasks.

Original: 4.1. D ATASETS
Revised: 4.1. D ATASETS

Original: CTR requires a large multi-task set for training, and a held-out set of tasks whose types are never seen during training for evaluation.
Revised: CTR requires a large multi-task set for training, and a held-out set of tasks whose types are never seen during training for evaluation.

Original: We follow the T0 benchmark 1 .
Revised: We follow the T0 benchmark.

Original: The training part consists of 39 tasks of 8 task types, including closed-book question answering (QA), multiple-choice QA, extractive QA, sentiment analysis, paraphrase identification, topic classification, summarization, and structure to text.
Revised: The training part consists of 39 tasks of 8 task types, including closed-book question answering (QA), multiple-choice QA, extractive QA, sentiment analysis, paraphrase identification, topic classification, summarization, and structure to text.

Original: The test part consists of 11 tasks of 4 task types, including natural language inference (RTE (Dagan et al., 2006), CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020)), sentence completion (COPA (Roemmele et al., 2011),
Revised: The test part consists of 11 tasks of 4 task types, including natural language inference (RTE (Dagan et al., 2006), CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012), Winogrande (Sakaguchi et al., 2020)), sentence completion (COPA (Roemmele et al., 2011),

Original: StoryCloze (Mostafazadeh et al., 2017), Hellaswag (Zellers et al., 2019)), and word sense disambiguation (WiC (Pilehvar & Camacho-Collados, 2019)).
Revised: StoryCloze (Mostafazadeh et al., 2017), Hellaswag (Zellers et al., 2019)), and word sense disambiguation (WiC (Pilehvar & Camacho-Collados, 2019)).

Original: Both the training and test parts are disjoint in task types, ensuring the zero-label setting.
Revised: Both the training and test parts are disjoint in task types, ensuring the zero-label setting.

Revised: We follow T0 to use the accuracy on the validation split of test tasks as the metric.

Original: 4.1. B ASELINES
Revised: 4.1. B ASELINES

Original: For the zero-label setting, we primarily compare CTR with the following baselines.
Revised: For the zero-label setting, we primarily compare CTR with the following baselines.

Revised: It is noteworthy that all baselines share a similar model size as CTR (i.e., about 770M), thus are comparable.

Revised: We provide implementation details of baselines in Appendix A.7.

Original: T0 T0 (Sanh et al., 2022) shares similar goals as CTR, which uses prompted multi-task training to improve the generalization performance.
Revised: • T0 (Sanh et al., 2022) shares similar goals as CTR, which uses prompted multi-task training to improve the generalization performance.

Original: T0 reports the average results over multiple prompts.
Revised: T0 reports the average results over multiple prompts.

Original:  Self-Training Since unlabeled data are accessed, we consider the self-training (Schick & Sch¨utze, 2020) method as one of the baselines.
Revised: • Self-Training Since unlabeled data are accessed, we consider the self-training (Schick & Sch¨utze, 2020) method as one of the baselines.

Original: Starting from T0, it uses T0 to label the unlabeled data, and further finetunes T0 with the pseudo-labeled data.
Revised: Starting from T0, it uses T0 to label the unlabeled data, and further finetunes T0 with the pseudo-labeled data.

Original: It also reports average performance over prompts.
Revised: It also reports average performance over prompts.

Original:  Manual-Code Another baseline is to use artificial features to define a task space.
Revised: • Manual-Code uses artificial feature vectors in place of CTR vector.

Original: Specifically, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning,whether it includes options into inputs, and whether it is a classification task, etc.
Revised: Specifically, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, and whether it is a classification task, etc.

Original: Each task can be represented as a artificially-defined discrete feature vector, each dimension associated with one of the aspects.
Revised: Each task can be represented as an discrete feature vector, each dimension associated with one of the aspects.

Original: “Manual-Code” uses the artificial feature vectors in place of CTR vector.
Revised: • ZPS (Liao et al., 2022a) is method for zero-label prompt selection. It first labels a set of unlabeled data through prompt ensemble, and use the pseudo-labeled data to select the best natural language prompt for the test task. We apply ZPS to multi-task T as

Original:  For the few-shot setting, we mainly consider five baseline methods.
Revised: a baseline For the few-shot setting where there are 32 test-set labeled examples, based on the multi-task T0, we experimented the following five baseline methods.

Original: Model Tuning It directly finetunes the pretrained model using  labeled data.
Revised: • ModelTuning directly finetunes the pretrained model using the test-set labeled data.

Original: Specifically, we follow the few-shot setting in Zheng et al.
Revised: Specifically, we follow the few-shot setting in Zheng et al.

Original: (2021) to use 16 labeled data to finetune and another 16 labeled samples for model selection.
Revised: (2021) to use 16 labeled data for finetuning and another 16 labeled samples for model selection.

Original: Prompt Tuning (Lester et al., 2021) introduce additional continuous prompts to LLM, and trains the continuous prompts using the  labeled data.
Revised: • PromptTuning (Lester et al., 2021) introduces additional continuous prompts to the backbone model (i e., T0) and trains the continuous prompts using the few labeled data.

Original:  GPS (Xu et al., 2022) is a genetic prompt search method that gradually mutates the prompts with a generative model and selects candidates based on performance on a labeled dataset.
Revised: • GPS (Xu et al., 2022) is a genetic prompt search method. Based on T0, GPS gradually mutates the prompts with a generative model and uses the few labeled data to selects prompt candidates.

Original:  GRIPS (Prasad et al., 2022) is asimilar work which introduced a gradient-free edit-based method for optimal prompt search
Revised: • GRIPS (Prasad et al., 2022) is a gradient-free edit-based method for optimal prompt search.

Original:  Black-Box Tuning(BBT) (Sun et al., 2022) isalso a gradient-free  method.
Revised: • Black-Box Tuning (BBT) (Sun et al., 2022) is a gradient-free few-shot prompt selection method.

Original: Unlike GPS and GRIPS, the method aims at searching for the best soft prompt embedding in the continuous space.
Revised: Unlike GPS and GRIPS, it searches for the best soft prompt embedding in a continuous space.

Original: 4.1. T RAINING D ETAILS
Revised: 4.1. T RAINING D ETAILS

Original: We instantiate our CTR with T5-Large (Raffel et al., 2019) being the LLM.
Revised: We instantiate our CTR with T5-Large (Raffel et al., 2019) being the LLM.

Original: We implement both the encoder and the decoder in our CTR model as two linear networks.
Revised: We implement both the encoder and the decoder in our CTR model as two linear networks.

Original: We have experimented with various architectures (see Appendix A.4 for more details).
Revised: We have experimented with various architectures (see Appendix A.4 for more details).

Original: For the first stage of training
Revised: For the first stage of training

Original: where the
Revised: where the

Original: LLM is frozen, we use the Adam Optimizer with a learning rate of 1e-2, a decay rate of 0.1, and a batch size of 2048.
Revised: LLM is frozen, we use the Adam optimizer with a learning rate of 1e-2, a decay rate of 0.1, and a batch size of 2048.

Original: We use an codebook embedding dimension of 1024, which is the same with hidden dimension.
Revised: We use a codebook embedding dimension of 1024, which is the same as the hidden dimension.

Original: The CTR length is set at 10 and each position can be assigned values ranging from 0 to 127; i.e.,  codebook size is 128. 
Revised: The CTR length is set at 10 and each position can be assigned values ranging from 0 to 127; i.e., the codebook size is 128. The hyperparameter β is set at 0.1.

Original: We experimented with different codebook sizes and CTR lengths, and detailed results are provided in Appendix A.5.
Revised: We experimented with different codebook sizes and CTR lengths, and detailed results are provided in Appendix A.5.

Original: For the secondstage of training  where all parameters are updated, we use  a batch size of 1024 and a learning rate of 1e-4.
Revised: For the second training phase where all parameters are updated, we use the Adam optimizer with a learning rate of 1e-4 and a batch size of 1024.

Original: We follow the training recipe as T0 (Sanh et al., 2022) for the rest of the hyperparameters. 
Revised: We follow the training recipe as T0 (Sanh et al., 2022) for the rest of the hyperparameters. [Table]

Original: 4. M AIN R ESULTS AND A NALYSIS
Revised: 4. M AIN R ESULTS AND A NALYSIS

Original: The performance of cross-task generalization, respectively under the zero-label setting and the fewshot setting, are shown in Table 1.
Revised: The performance of cross-task generalization, respectively under the zero-label setting and the fewshot setting, are shown in Table 1.

Original: Our CTR outperforms all baseline methods on average under both settings.
Revised: Our CTR outperforms all baseline methods on average under both settings.

Original: Comparing CTR with T0-Large and its variant (i.e., self-training), CTR outperforms them respectively by more than 4 points and by almost 1.5 points on average.
Revised: Comparing CTR with T0-Large and its variant (i.e., self-training), CTR outperforms them respectively by more than 4 points and by almost 1.5 points on average.

Original: The improvements potentially originate from two aspects—(a) the learned compositional task representations benefit from better generalization abilities than the discrete manual prompts used by T0/self-training; (b) our CTR is able to select a high-performing compositional task representation for the unseen task.
Revised: The improvements potentially originate from two aspects—(a) the learned compositional task representations benefit from better generalization abilities than the discrete manual prompts used by T0/self-training; (b) our CTR can select a high-performing compositional task representation for the unseen task.

Original: Comparing with Manual-Code, our CTR demonstrate significant advantages of more than 4.5 points, proving that artificially-designed features of tasks are unreliable, and CTR provides an effective way of automatically training such data-driven compositional task representations using multi-tasks.
Revised: Compared with Manual-Code, our CTR demonstrate significant advantages of more than 4.5 points, proving that artificially-designed features of tasks are unreliable, and CTR provides an effective way of automatically training such data-driven compositional task representations using multi-tasks.

Original: [Table]

Original: Compared with Model Tuning, Prompt Tuning and BBT, all of which requireextra parameter updates, our CTR shows better performance of cross-task generalization on average.
Revised: Compared with Model Tuning, Prompt Tuning, and BBT, where they require parameter updates over the test-task data, CTR shows better performance of cross-task generalization on average.

Original: Compared with baselines that do not tune parameters (i.e., GPS, GRIPS),our CTR shows even larger and more consistent advantages on most of the test tasks; i.e., on 9/11 tasks CTR shows dominating performance. 
Revised: Compared with baselines that do not tune parameters (i.e., GPS, GRIPS), CTR shows even larger and more consistent advantages on most of the test tasks; i.e., on 9/11 tasks CTR shows dominating performance. On co-reference tasks , CTR performs worse under the zero-label setting and better under the fewshot setting.

Revised: The zero-label setting uses pseudo-labeled data to select test codes while the few-shot setting uses real-labeled data to select codes.

Revised: The reason for the decreased performance lies in that the pseudo data of the co-reference task were of low quality and did not select effective task codes.

Original: Please refer to Appendix A.3 for generated zero-label cases from CTR .
Revised: Please refer to Appendix A.3 for generated zero-label cases from CTR .

Original: 4. A BLATION S TUDY 4.3. A RE M ANUAL P ROMPTS N ECESSARY ?
Revised: 4. A BLATION S TUDY 4.3. A RE M ANUAL P ROMPTS N ECESSARY ?

Original: We are interested in whether CTR will be further improved when combined with discrete manual prompts.
Revised: We are interested in whether CTR will be further improved when combined with discrete manual prompts.

Original: We conduct comparative experiments of CTR respectively with and without manual prompts.
Revised: We conduct comparative experiments of CTR respectively with and without manual prompts.

Revised: Specifically, for CTR without manual prompts, the inputs are a direct concatenation of multiple text fields, with compositional task representations appended in front of it.

Revised: For CTR with manual prompts, the inputs are constructed by leveraging T0 prompts , with CTR placed in front.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 2.

Original: We report both the CTR results as well as its “upperbound” results.
Revised: We report both the CTR results as well as its “upper-bound” results.

Original: It is noteworthy that the “upper-bound” results are post-hoc, which are obtained by the best code/prompt with observing the test task performance, and are merely given for estimating the potential of the two methods.
Revised: It is noteworthy that the “upper-bound” results are post-hoc, which are obtained by the best code/prompt with observing the test task performance, and are merely given for estimating the potential of the two methods.

Original: We shall observe that adding additional discrete manual prompts does not improve the CTR performance as well as the upper-bound.
Revised: We shall observe that adding additional discrete manual prompts does not improve the CTR performance as well as the “upper-bound”.

Original: From the results, we shall conclude that (1) our CTR does not rely on manual prompts to obtain optimal performance.
Revised: From the results, we shall conclude that (1) our CTR does not rely on manual prompts to obtain optimal performance.

Original: CTR can work as an alternative to the prompt-based methods, which has the potential of mitigating the labor work of hand-crafting discrete prompts (2) By comparing the CTR performance and the CTR upper-bound, there is still room to optimize the code-searching algorithm, and as a result to further improve the generalization performance.[Table]
Revised: CTR can work as an alternative to the prompt-based methods. By comparing the CTR performance and the CTR upper-bound, there is still room to optimize the code-searching algorithm, and as a result to further improve the generalization performance.

Original: [Table]
Revised: [Table]

Original: 4.3. T RAINING O BJECTIVES
Revised: 4.3. T RAINING O BJECTIVES

Original: Our CTR consists of three training loss items, including the LM loss, the commitment loss [Table] and the embedding loss.
Revised: Our CTR consists of three training loss items.

Original: Intuitively, the LM loss is optimized to predict the correct answer of the tasks.
Revised: Intuitively, the LM loss is optimized to predict the correct answer of the tasks.

Original: The embedding loss and the commitment loss are optimized to minimize the distance between the query embeddings and the CTR that are lookup from the codebook embeddings.
Revised: The embedding loss and the commitment loss are optimized to minimize the distance between the query embeddings and the CTR that are lookup from the codebook embeddings.

Original: To study the effectiveness of each loss item, we conduct an ablation study on them [Table] and the results are shown in Table 3.
Revised: To study the effectiveness of each loss item, we conduct an ablation study on them, and the results are shown in Table 3.

Original: Results show that removing either of the loss items will drastically hurt the zero-label performance on average, proving that either loss item is indispensable for the training of CTR 
Revised: Results show that removing either of the loss items will drastically hurt the zero-label performance , proving that either loss item is indispensable for the training of CTR .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. I N -D EPTH A NALYSIS 4.4. I NTERPRETABILITY
Revised: 4. I N -D EPTH A NALYSIS 4.4. I NTERPRETABILITY

Original: Since CTR is trained to represent tasks with compositional codes, each code associated with one of the key aspects of tasks, CTR demonstrates a certain degree of interpretability.
Revised: Since CTR is trained to represent tasks with compositional codes, each code associated with one of the key aspects of tasks, CTR demonstrates a certain degree of interpretability.

Original: Table 4 presents examples of CTR that shows how each compositional task code is possibly interpreted. 
Revised: Table 4 presents examples of CTR that shows how each compositional task code is possibly interpreted. Interestingly

Original: In our experiments, we use a codebook size of 128. We analyze which tasks each code has occurred in, and what features these common tasks share.
Revised: [Table]

Original: Interestingly, it frequently occurred that tasks that share similar features indeed have the same compositional code.
Revised: it frequently occurred that tasks that share similar features indeed have the same compositional code.

Original: For example, the code 52 occurs in tasks that require extracting information from the given contexts, including samsum * (summarization task), wiki bio * (structured data to text task), and paws labeled final paraphrase task (paraphrase generation) etc.
Revised: For example, the code 52 occurs in tasks that require extracting information from the given contexts, including samsum * (summarization task), wiki bio * (structured data to text task) [Table] and paws labeled final paraphrase task (paraphrase generation) etc.

Original: Another example is that code 111 exists in most of the tasks that require generating long setences, including multi news * and samsum * (both are summarization tasks).
Revised: Another example is that code 111 exists in most of the tasks that require generating long setences, including multi news * and samsum * (both are summarization tasks).

Original: We also validate these explanations on unseen test tasks.
Revised: We also validate these explanations on unseen test tasks.

Original: Results show that these possible explanations still hold.
Revised: Results show that these possible explanations still hold.

Original: For example, the COPA task generates long answers given two candidate choices, and its compositional code has the code 111 (indicating sentence generation).
Revised: For example, the COPA task generates long answers given two candidate choices, and its compositional code has the code 111 (indicating sentence generation).

Original: 4.4. C ONTROLLABILITY
Revised: 4.4. C ONTROLLABILITY

Original: CTR also demonstrates a certain degree of contollability.
Revised: CTR also demonstrates a certain degree of controllability.

Original: Specifically, by modifying one bit of the compositional task code, our CTR will exhibit a different task behavior.
Revised: Specifically, by modifying one bit of the compositional task code, our CTR will exhibit a different task behavior.

Original: Table 5 shows several examples of controlling CTR.
Revised: Table 5 shows several examples of controlling CTR.

Original: We shall observe that, given the same inputs, by simply changing one  bit of the compositional task code, the task behaviour of CTR turns from DIALOGUE GENERATION toTOPIC CLASSIFICATION , from REVIEW RATING to SUMMARIZATION , from SENTIMENT
Revised: We shall observe that, given the same inputs, by simply changing one [Table] bit of the compositional task code, the task behavior of CTR turns from DIALOGUE GENERATION to

Original: ANALYSIS to TOPIC CLASSIFICATION , etc.
Revised: TOPIC CLASSIFICATION , from to

Original: We shall come to the following conclusions.
Revised: , from

Original:  (1) Since the inputs are randomly sampled from all tasks, some of which are quite different in data distribution, CTR performs well on them, proving that CTR indeed learns the ability of performing different tasks, instead of simply memorizing/overfitting to a certain dataset.
Revised: We shall conclude— (1) Since the inputs are randomly sampled from all tasks, some of which are quite different in data distribution, CTR performs well on them, proving that CTR indeed learns the ability to perform different tasks, instead of simply memorizing/overfitting to a certain dataset.

Original: (2) CTR is capable of switching between different tasks by simply changing one bit of the compositional code, proving that CTR effectively encodes the “task behavior” factor into the bits of the compositional code, while disentangling other factors.
Revised: (2) CTR is capable of switching between different tasks by simply changing one bit of the compositional code, proving that CTR effectively encodes the “task behavior” factor into the bits of the compositional code while disentangling other factors.

Original:  H OW CTR G ENERALIZES TO N EW T ASKS
Revised: 4.4. H OW CTR G ENERALIZES TO N EW T ASKS

Original: To reveal how CTR essentially works, we use Figure 1 to explain how CTR generalizes from training tasks to unseen tasks such as COPA.
Revised: To reveal how CTR essentially works, we use Figure 1 to explain how CTR generalizes from training tasks to unseen tasks such as COPA.

Original: During the training phase, CTR learns the compositional task[Table] code for each task, each code associated with certain features of the task.
Revised: During the training phase, CTR learns the compositional task code for each task, each code associated with certain features of the task.

Original: For example, codeis associated with many of the multiple-choice/reasoning-based QA tasks; code 111 is associated with most of the long-sentence-generation tasks, etc.
Revised: For example, codeis associated with many of the multiple-choice/reasoning-based QA tasks; code 111 is associated with most of the long-sentence-generation tasks, etc.

Original: For a new task, its task code is a re-arranged composition of the code existing in training tasks.
Revised: For a new task, its task code is a re-arranged composition of the code existing in training tasks.

Original: If the new task shares the same code snippet as the training tasks, it inherits corresponding task features as well.
Revised: If the new task shares the same code snippet as the training tasks, it inherits corresponding task features as well.

Original: In the example of Figure 1, a real case produced by our model, the new task COPA is performed by combing the abilities of reasoning-based QA, sentence generation, and multi-choice selection from training tasks.
Revised: In the example of Figure 1, a real case produced by our model, the new task COPA is performed by combining the abilities of reasoning-based QA, sentence generation, and multi-choice selection from training tasks.

Original: These observations are consistent with our motivation of learning compositional task representations for cross-task generalization.
Revised: These observations are consistent with our motivation of learning CTR for cross-task generalization.

Original: 4.4. G ENERALIZING TO NEW UNSEEN DOMAINS / CLASSES
Revised: 4.4. G ENERALIZING TO NEW UNSEEN DOMAINS / CLASSES

Original: Another interesting finding is that, for tasks with limited categorical classes, e.g., topic classification tasks, CTR is capable of generalizing to new unseen classes/domains.
Revised: Another interesting finding is that, for tasks with limited categorical classes, e.g., topic classification tasks, CTR is capable of generalizing to new unseen classes/domains.

Original: Table 6 shows several examples.
Revised: Table 6 shows several examples.

Original: Given different inputs that are randomly sampled from a large set of tasks and the compositional task code of AGNews, CTR not only predicts topic classes that are originally definedby AGNews, but also predicts new topic classes that never occur in AGNews. Please refer to Appendix A.1 and Appendix A.2 for more cases. [Table]
Revised: Given different inputs that are randomly sampled from a large set of tasks and the compositional task code of AGNews, CTR not only predicts topic classes that are originally defined

Original: If Jim moves some particles of matter farther apart, what will happen to the rate at which they can pass vibrations on to nearby particles decrease or increase?

Original: Assuming that: When particles of matter are closer together, they can more quickly pass the energy of vibrations to nearby particles.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: DIALOGUE :

Original: W

Original: [Table]

Original: Hi, Jim.

Original: I’m Jim.

Original: I’m a physicist at the University of California, Berkeley.

Original: [Table]

Original: A

Original: [Table]

Original: Parti

Original: [Table]

Original: was marred

Original: [Table]

Original: an

Original: [Table]

Original: on a

Original: [Table]
Revised: [Table]

Original: elect Pauline Marois.

Original: A man

Original: [Table]

Original: fire during her

Original: [Table]
Revised: [Table]

Original: speech, killing one person and critically wounding another, reports CTV.

Original: ... “I have convictions and I am going to

Original: [Table]

Original: them.”

Original: Marois

Original: [Table]

Original: during her [Table] speech. “There will be a referendum when the Quebec population wants a referendum.”
Revised: by AGNews, but also predicts new topic classes that never occur in AGNews.

Original: [Table]

Original: One person was killed and another critically injured in a shooting at a victory rally for the separatist Parti Quebecois

Original: [Table]

Original: City

Original: [Table]

Original: [Table]

Original: here

Original: [Table]

Original: like

Original: [Table]

Original: where

Original: [Table] On [Table] and Wednesday
Revised: Please refer to Appendix A.1 and Appendix A.2 for more cases.

Original: they

Original: [Table]

Original: on manicures

Original: [Table]

Original: pedicures

Original: [Table]

Original: 5 cheaper

Original: [Table]

Original: original

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper [Table] we explore cross-task generalization from a new perspective—compositional generalization.
Revised: In this paper, we explore cross-task generalization from a new perspective—compositional generalization.

Original: We propose the Compositional Task Representations (CTR) method that learns a discrete compositional codebook for tasks and generalizes to new unseen tasks by forming new compositions of the task codes.
Revised: We propose the Compositional Task Representations (CTR) method that learns a discrete compositional codebook for tasks and generalizes to new unseen tasks by forming new compositions of the task codes.

Original: For the inference of CTR [Table] we propose two algorithms — Code Ensemble and
Revised: For the inference of CTR , we propose two algorithms — Code Ensemble and

Original: Bitwise Search, respectively for zero-label and few-shot settings.
Revised: Bitwise Search, respectively for zero-label and few-shot settings.

Original: Experiments demonstrate that our
Revised: Experiments demonstrate that our

Original: CTR significantly outperforms existing prompted-based methods on both zero-label and few-shot settings.
Revised: CTR significantly outperforms existing prompted-based methods on both zero-label and few-shot settings.

Original: Analysis of the learned compositional task codes proves that some of the CTR codes show certain degrees of interpretability and controllability [Table]
Revised: Analysis of the learned compositional task codes proves that some of the CTR codes show certain degrees of interpretability and controllability.

Original: A A PPENDIX A. E XAMPLES OF AGN EWS
Revised: A A PPENDIX A. E XAMPLES OF AGN EWS

Original: [Table]
Revised: [Table]

Original: Fears
Revised: Fears

Original: [Table]
Revised: [Table]

Original: T N pension after talks Unions representing workers at Turner Newall say they are ’disappointed’ after talks with stricken parent firm Federal Mogul.
Revised: T N pension after talks Unions representing workers at Turner Newall say they are ’disappointed’ after talks with stricken parent firm Federal Mogul.

Original: [Table]
Revised: [Table]

Original: Q
Revised: Q

Original: [Table]
Revised: [Table]

Original: : Prediction Unit Helps Forecast Wildfires (AP) AP - It’s barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring.
Revised: : Prediction Unit Helps Forecast Wildfires (AP) AP - It’s barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring.

Original: Lightning will strike in places he expects.
Revised: Lightning will strike in places he expects.

Original: Winds will pick up, moist places will dry and flames will roar.
Revised: Winds will pick up, moist places will dry and flames will roar.

Original: [Table]
Revised: [Table]

Original: Science and technology
Revised: Science and technology

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: How did serfdom develop in and then leave Russia?
Revised: How did serfdom develop in and then leave Russia?

Original: [Table]
Revised: [Table]

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: : There are 10 apples on an apple tree.
Revised: : There are 10 apples on an apple tree.

Original: Three fall off.
Revised: Three fall off.

Original: Now there are X apples.
Revised: Now there are X apples.

Original: [Table]
Revised: [Table]

Original: I NPUT : the rock is destined to be the 21st century’s new “conan” and that he’s going to make a splash even greater than arnold schwarzenegger, jean-claud van damme or steven segal.
Revised: I NPUT : the rock is destined to be the 21st century’s new “conan” and that he’s going to make a splash even greater than arnold schwarzenegger, jean-claud van damme or steven segal.

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: Film
Revised: Film

Original: I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.
Revised: I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.

Original: A major part of their output was under sub-contract to motor vehicle manufacturers.
Revised: A major part of their output was under sub-contract to motor vehicle manufacturers.

Original: Their business closed in 1972.
Revised: Their business closed in 1972.

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: O UTPUT ( OUT OF ORIGINAL CLASSES ): Company
Revised: O UTPUT ( OUT OF ORIGINAL CLASSES ): Company

Original: I NPUT : Stuning even for the non-gamer This sound track was beautiful!
Revised: I NPUT : Stuning even for the non-gamer This sound track was beautiful!

Original: It paints the senery in your mind so well I would recomend it even to people who hate vid.
Revised: It paints the senery in your mind so well I would recomend it even to people who hate vid.

Original: game music!
Revised: game music!

Original: I have played the game Chrono Cross but out of all of the games I have ever played it has the best music!
Revised: I have played the game Chrono Cross but out of all of the games I have ever played it has the best music!

Original: It backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras.
Revised: It backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras.

Original: It would impress anyone who cares to listen!
Revised: It would impress anyone who cares to listen!

Original: M
Revised: M

Original: ODEL O
Revised: ODEL O

Original: [Table]
Revised: [Table]

Original: OUT
Revised: OUT

Original: [Table]
Revised: [Table]

Original: I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.
Revised: I NPUT : E. D. Abbott Ltd - Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929.

Original: A major part of their output was under sub-contract to motor vehicle manufacturers.
Revised: A major part of their output was under sub-contract to motor vehicle manufacturers.

Original: Their business closed in 1972.
Revised: Their business closed in 1972.

Original: M
Revised: M

Original: ODELO
Revised: ODEL

Original:  UTPUT ( OUT
Revised: O UTPUT ( OUT

Original: OF
Revised: OF

Original: ORIGINAL CLASSES ): Company
Revised: ORIGINAL CLASSES ): Company

Original: Q UESTION : Slack (2003) compares three groups that conducted biological research at Yale during overlapping periods between 1910 and 1970.
Revised: Q UESTION : Slack (2003) compares three groups that conducted biological research at Yale during overlapping periods between 1910 and 1970.

Original: Yale proved important as a site for this research.
Revised: Yale proved important as a site for this research.

Original: The leaders of these groups were Ross Granville Harrison, Grace E. Pickford, and G. Evelyn Hutchinson, and their members included both graduate students and more experienced scientists.
Revised: The leaders of these groups were Ross Granville Harrison, Grace E. Pickford, and G. Evelyn Hutchinson, and their members included both graduate students and more experienced scientists.

Original: All produced innovative research, including the opening of new subfields in embryology, endocrinology, and ecology, respectively, over a long period of time.
Revised: All produced innovative research, including the opening of new subfields in embryology, endocrinology, and ecology, respectively, over a long period of time.

Original: Harrison’s group is shown to have been a classic research school; Pickford’s and Hutchinson’s were not.
Revised: Harrison’s group is shown to have been a classic research school; Pickford’s and Hutchinson’s were not.

Original: Pickford’s group was successful in spite of her lack of departmental or institutional position or power.
Revised: Pickford’s group was successful in spite of her lack of departmental or institutional position or power.

Original: Hutchinson and his graduate and postgraduate students were extremely productive, but in diverse areas of ecology rather than one focused area of research or the use of one set of research tools.
Revised: Hutchinson and his graduate and postgraduate students were extremely productive, but in diverse areas of ecology rather than one focused area of research or the use of one set of research tools.

Original: Hutchinson’s example shows that new models for research groups are needed, especially for those that include extensive field research.
Revised: Hutchinson’s example shows that new models for research groups are needed, especially for those that include extensive field research.

Original: M ODEL A NSWER ( OUT OF ORIGINAL CLASSES ): Ecology
Revised: M ODEL A NSWER ( OUT OF ORIGINAL CLASSES ): Ecology

Original: A. E XAMPLES OF DBPEDIA
Revised: A. E XAMPLES OF DBPEDIA

Original: Q UESTION : Federation of International Trade Associations - The Federation of International Trade
Revised: Q UESTION : Federation of International Trade Associations - The Federation of International Trade

Original: Associations (FITA) based in Reston Virginia and New York New York USA was founded in 1984.
Revised: Associations (FITA) based in Reston Virginia and New York New York USA was founded in 1984.

Original: It fosters international trade by seeking to strengthen the role of associations in the United States
Revised: It fosters international trade by seeking to strengthen the role of associations in the United States

Original: Mexico and Canada.
Revised: Mexico and Canada.

Original: FITA is the strategic partner of the United States Commercial Service for ecommerce.
Revised: FITA is the strategic partner of the United States Commercial Service for ecommerce.

Original: company, educational institution, artist, athlete, office holder, mean of transportation, building, natural place, village, animal, plant, album, film or written work
Revised: company, educational institution, artist, athlete, office holder, mean of transportation, building, natural place, village, animal, plant, album, film or written work

Original: M ODEL
Revised: M ODEL

Original: A NSWER
Revised: A NSWER

Original: WITHIN THE ORIGINAL CLASSES
Revised: WITHIN THE ORIGINAL CLASSES

Original: ): Company
Revised: ): Company

Original: Q UESTION : Furian knife
Revised: Q UESTION : Furian knife

Original: In this movie sequel, Vin Diesel returns as Riddick, an escaped convict with a price on his head.
Revised: In this movie sequel, Vin Diesel returns as Riddick, an escaped convict with a price on his head.

Original: Riddick has been hiding on a snow planet for the last five years, when a group of mercenaries try to capture him.
Revised: Riddick has been hiding on a snow planet for the last five years, when a group of mercenaries try to capture him.

Original: Riddick returns to the planet Helion, and finds out that his friend Jack, is in prison on Crematoria, a very hot planet.
Revised: Riddick returns to the planet Helion, and finds out that his friend Jack, is in prison on Crematoria, a very hot planet.

Original: While on Helion, the planet is invaded by the Necromongers, led by Lord Marshal (Colm Feore), who wants to rule the universe.
Revised: While on Helion, the planet is invaded by the Necromongers, led by Lord Marshal (Colm Feore), who wants to rule the universe.

Original: Riddick is captured by the mercs and flown to the same unsavoury and possibly illegal prison Kyra is in.
Revised: Riddick is captured by the mercs and flown to the same unsavoury and possibly illegal prison Kyra is in.

Original: Turns out Jack is now known as Kyra (Alexa Davalos) and is tough as nails.
Revised: Turns out Jack is now known as Kyra (Alexa Davalos) and is tough as nails.

Original: M ODEL
Revised: M ODEL

Original: A
Revised: A

Original: NSWER
Revised: NSWER

Original: ( WITHIN THE
Revised: ( WITHIN THE

Original: ORIGINAL CLASSES ):
Revised: ORIGINAL CLASSES ):

Original: Film
Revised: Film

Original: Q UESTION : lila abu-lughod’s: - name - alma mater - website - known for - birth date - employer - nationality - occupation Bio: lila abu-lughod -lrb- born 1952 -rrb- is an american with palestinian and jewish ancestry who is professor of anthropology and women’s and gender studies at columbia university in new york city.
Revised: Q UESTION : lila abu-lughod’s: - name - alma mater - website - known for - birth date - employer - nationality - occupation Bio: lila abu-lughod -lrb- born 1952 -rrb- is an american with palestinian and jewish ancestry who is professor of anthropology and women’s and gender studies at columbia university in new york city.

Original: a specialist of the arab world, her seven books, most based on long term ethnographic research, cover topics from sentiment and poetry to nationalism and media, from gender politics to the politics of memory.
Revised: a specialist of the arab world, her seven books, most based on long term ethnographic research, cover topics from sentiment and poetry to nationalism and media, from gender politics to the politics of memory.

Original: M ODEL A NSWER ( WITHIN THE ORIGINAL CLASSES ): Educational Institution
Revised: M ODEL A NSWER ( WITHIN THE ORIGINAL CLASSES ): Educational Institution

Original: I NPUT : when he dug his massive body into the batter’s box tuesday night at legends field during an exhibition game against the new york yankees, adam dunn not only wasn’t hitting his weight, he wasn’t hitting the weight of an UNK bag of baseballs.
Revised: I NPUT : when he dug his massive body into the batter’s box tuesday night at legends field during an exhibition game against the new york yankees, adam dunn not only wasn’t hitting his weight, he wasn’t hitting the weight of an UNK bag of baseballs.

Original: M
Revised: M

Original: ODEL 
Revised: ODEL O UTPUT ( OUT OF ORIGINAL CLASSES

Original: O UTPUT OUT OF ORIGINAL CLASSES

Original: ): Sports
Revised: ): Sports

Original: I NPUT :
Revised: I NPUT :

Original: We know that, thanks to our DNA, each of us is a little bit different.
Revised: We know that, thanks to our DNA, each of us is a little bit different.

Original: Some of those differences are obvious, like eye and hair color.
Revised: Some of those differences are obvious, like eye and hair color.

Original: Others are not so obvious, like how our bodies react to medication.
Revised: Others are not so obvious, like how our bodies react to medication.

Original: Researchers are beginning to look at how to tailor medical treatments to our genetic profiles, in a relatively new field called pharmacogenomics.
Revised: Researchers are beginning to look at how to tailor medical treatments to our genetic profiles, in a relatively new field called pharmacogenomics.

Original: Some of the biggest breakthroughs have been in cancer treatment.
Revised: Some of the biggest breakthroughs have been in cancer treatment.

Original: M ODEL O UTPUT (
Revised: M ODEL O UTPUT (

Original: OUT
Revised: OUT

Original: OF ORIGINAL CLASSES ): Science
Revised: OF ORIGINAL CLASSES ): Science

Original: I NPUT : how did athenians make money?
Revised: I NPUT : how did athenians make money?

Original: Other Greek cities set up democracies, and even though most followed an Athenian model, none were as powerful, stable, nor as well-documented as that of
Revised: Other Greek cities set up democracies, and even though most followed an Athenian model, none were as powerful, stable, nor as well-documented as that of

Original: Athens.
Revised: Athens.

Original: M
Revised: M

Original: ODEL O UTPUT ( OUT OF ORIGINAL CLASSES 
Revised: ODEL O UTPUT ( OUT OF ORIGINAL CLASSES )

Original: City
Revised: City

Original: I NPUT : Great food, portions could be smaller.
Revised: I NPUT : Great food, portions could be smaller.

Original: A little pricey for Middleton.
Revised: A little pricey for Middleton.

Original: Would try somewhere else next time but would also not hesitate to return
Revised: Would try somewhere else next time but would also not hesitate to return

Original: M ODEL O
Revised: M ODEL O

Original: UTPUT
Revised: UTPUT

Original: OUT OF ORIGINAL CLASSES
Revised: OUT OF ORIGINAL CLASSES

Original: ): Restaurant
Revised: ): Restaurant

Original: I NPUT : Facts: - name: mr. marcus - spelling: us - caption: mr. october 2007 marcus at a porn star karaoke event, - height: 5 8 - alias: mister marcus, mr. marquis - image: mr marcus, 2007.jpg - birth date: 4 september 1970 - birth name: jesse spencer - weight: 200 lb kg on - ethnicity: black - birth place: pomona, california, usa - hair color: black - number of films: 1,782 -lrb- per iafd -rrb- as a performer & 36 as a director - eye color: brown mr. marcus.
Revised: I NPUT : Facts: - name: mr. marcus - spelling: us - caption: mr. october 2007 marcus at a porn star karaoke event, - height: 5 8 - alias: mister marcus, mr. marquis - image: mr marcus, 2007.jpg - birth date: 4 september 1970 - birth name: jesse spencer - weight: 200 lb kg on - ethnicity: black - birth place: pomona, california, usa - hair color: black - number of films: 1,782 -lrb- per iafd -rrb- as a performer & 36 as a director - eye color: brown mr. marcus.

Original: M ODEL O
Revised: M ODEL O

Original: UTPUT
Revised: UTPUT

Original: ( OUT OF ORIGINAL CLASSES ): Actor
Revised: ORIGINAL CLASSES ): Actor

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C OMPONENT A RCHITECTURE
Revised: C OMPONENT A RCHITECTURE

Revised: [Table]

Revised: study on different architectures of the CTR encoder and decoder.

Revised: All experiments are conducted under the zero-label setting.

Revised: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).

Revised: Results show that the simple “Linear - Linear” combination achieves the best performance.

Original: Table 10 shows the zero-label results when using different architecture of the encoder and the decoder.
Revised: Table 10 shows the zero-label results when using different architecture of the encoder and the decoder.

Original: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).
Revised: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).

Original: Results show that simply using a linear network for both the encoder and the decoder performs best, while complicated architectures, e.g., Transformer and MLP, yield poor performance.
Revised: Results show that simply using a linear network for both the encoder and the decoder performs best, while complicated architectures, e.g., Transformer and MLP, yield poor performance.

Original: This could because that the learning of codebook embeddings as well as the code mapping does not necessarily need complicated transformation.
Revised: This could because that the learning of codebook embeddings as well as the code mapping does not necessarily need complicated transformation.

Original: When using complicated architectures, it instead increases the difficulties of the learning process.
Revised: When using complicated architectures, it instead increases the difficulties of the learning process.

Original: A. C ODEBOOK S IZE AND CTR L ENGTH
Revised: A. C ODEBOOK S IZE A ND CTR L ENGTH

Original: There are two critical hyper-parameters for CTR —the codebook size and the CTR length.
Revised: There are two critical hyper-parameters for CTR —the codebook size and the CTR length.

Original: We conduct experiments of different selection of the two hyper-parameters, and see how it influence the performance of zero-label generalization.
Revised: We conduct experiments of different selection of the two hyper-parameters [Table] and see how it influence the performance of zero-label generalization.

Original: Table 11 presents the results.
Revised: Table 11 presents the results.

Original: We shall observe that when the codebook size decreases below 64, the performance decreases to a large degree.
Revised: We shall observe that when the codebook size decreases below 64, the performance decreases to a large degree.

Original: We conjecture that it is due to the capacity of a codebook size below 64 is not sufficient for representing the multiple aspects of the tasks.
Revised: We conjecture that it is due to the capacity of a codebook size below 64 is not sufficient for representing the multiple aspects of the tasks.

Original: In addition, experimenting with a CTR length of 10 generally outperforms those with larger CTR length.
Revised: In addition, experimenting with a CTR length of 10 generally outperforms those with larger CTR length

Original: [Table]
Revised: [Table]

Original:  study on different architectures of the CTR encoder and decoder.
Revised: 11: Ablation study on different codebook size and CTR length.

Original: All experiments are conducted under the zero-label setting.
Revised: All experiments are conducted under the zero-label setting.

Original: We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None).
Revised: We experiment three of the codebook size, respectively 128,and 48, and two of the CTR length, including 10 and 20. Results show that ”128 - 10” combination achieves the best performance. A. P ERFORMANCE S ENSITIVITY TO DIFFERENT SELECTION OF CODES

Original: Results show that  the simple “Linear - Linear” combination achieves the best performance 
Revised: We want to verify whether different selections of codes will affect the performance of the model. Intuitively, if a code is merely noise that gets ignored by the model eventually, the model will not sensitive to different selections of codes and vice versa. Therefore, we randomly sampled several codes of training tasks and evaluate their performance on different test tasks. The results are shown in Table 12.

Original:  [Table] 
Revised: As shown in the table, the model is highly sensitive to a different selection of codes. For example, the code of task amazon polarity user satisfied can perform well on RTE and significantly better than the performance of code of task cnn dailymail 3.0.0 news card view [Table] A. E XPERIMENTAL D ETAILS

Revised: For the data preprocessing of all experiments, to balance the number of data for different tasks, we restrict the maximum data examples for each training task to be 50,000, which empirically yields better results.

Revised: Training details of each baseline method under the zero-label setting are illustrated as follows.

Revised: T0-Large Based on T5-Large-LM-Adapted, it performs multi-task training for 10000 steps.

Revised: We set the maximum length of input and target sequences to 384 and 32 respectively.

Revised: We use the Adam optimizer with a learning rate of 1e-4, a dropout rate of 0.1, and a batch size of 1024.

Revised: Following

Revised: T0 Sanh et al.

Revised: (2022), we use the same task prompts from PromptSource(Bach et al., 2022).

Revised: We report the average accuracy of multiple prompts for each test task.

Revised: Note that our reproduced T0Large results are much better than those reported in the original paper Sanh et al.

Revised: (2022), which sets a much stronger baseline for comparison.

Revised: We report the average accuracy of prompts for each test task.

Revised: We believe our baseline is well-optimized.

Revised: Because the performance on test tasks is comparable to the results of T0-3B reported in Sanh et al.

Revised: , even our baseline only contains 770M parameters.

Revised: Self-Training For a fair comparison, we randomly sample 32 unlabeled data for self-training, which is the same as CTR.

Revised: The Self-Training method trains from the T0-Large with these pseudo-labeled examples for 5 epochs and reports average performance over prompts.

Revised: For the training, we use a batch size of 32 and the Adam optimizer with a learning rate of 1e-4.

Revised: Manual-Code

Revised: In practice, we manually label a set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, whether it includes options into inputs, and whether it is a classification task, etc.

Revised: Each task can be represented as an artificially-defined discrete feature vector, each dimension associated with one of the aspects.

Revised: Manual-Code follows exact the same training recipe as the second training phase of our CTR.

Revised: Training details are presented in Section 4.1.3.

Revised: Zero-Label Prompt Selection (ZPS)

Revised: (Liao et al., 2022a)

Revised: For each task, we use 32 unlabeled data for producing pseudo-labeled data.

Revised: Finally, we report the accuracy of the selected prompt.

Revised: For the few-shot setting, we consider the following five baseline methods.

Revised: All few-shot baselines are based on our reproduced T0-Large.

Revised: Model Tuning We use the Adam optimizer with a batch size of 256 and a learning rate of 1e-4.

Revised: We combine all training data of the test tasks for training.

Revised: The maximum training step is 100.

Revised: We use a validation set for model selection.

Revised: Finally, we report the average accuracy of the selected best checkpoint.

Revised: Prompt Tuning (Lester et al., 2021)

Revised: We use the Adam optimizer with a batch size of 128 and a learning rate of 0.05.

Revised: We combine all training data of the test tasks for training.

Revised: The maximum training step is 100.

Revised: The length of continuous prompts for each task is 20.

Revised: We use a validation set for model selection.

Revised: Finally, we report the average accuracy of the selected best checkpoint.

Revised: GPS (Xu et al., 2022)

Revised: We follow hyper-parameters reported in the original paper.

Revised: Specifically, we run the GPS for 6 steps.

Revised: At each step, new prompts are generated by a T5-xxl-lm-adapted model.

Revised: GRIPS In our experiments, we set max patience P = 2 , candidate l = 5 , and step m = 5 .

Revised: For the rest of the hyper-parameters, we follow the original GRIPS

Revised: paper(Prasad et al., 2022).

Revised: Black-Box Tuning (BBT) (Sun et al., 2022)

Revised: In practice, we use the Adam optimizer with a learning rate of 0.05.

Revised: For each test task, we train the soft prompt for 200 steps.

Revised: We set the prompt length

Revised: L = 10 , subspace dimension d = 500 , and cma budget 1000.

Revised: For the rest of the hyper-parameters, we follow the original BBT(Sun et al., 2022).