Original: L EARNING WITHOUT F ORGETTING : T ASK A WARE
Revised: L EARNING WITHOUT F ORGETTING : T ASK A WARE

Original: M ULTI -T ASK
Revised: M ULTI -T ASK

Original: L EARNING
Revised: L EARNING

Original: FOR
Revised: FOR

Original: M ULTI -M ODALITY
Revised: M ULTI -M ODALITY

Original: T ASKS
Revised: T ASKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing Multi-Task Learning(MTL) strategies like joint or meta-learning focus more on shared learning and have little to no scope for task-speciﬁc learning.
Revised: Existing Multi-Task Learning(MTL) strategies like joint or meta-learning focus more on shared learning and have little to no scope for task-speciﬁc learning.

Original: This creates the need for a distinct shared pretraining phase and a task-speciﬁc ﬁnetuning phase.
Revised: This creates the need for a distinct shared pretraining phase and a task-speciﬁc ﬁnetuning phase.

Original: The ﬁnetuning phase creates separate models for each task, where improving the performance of a particular task necessitates forgetting some of the knowledge garnered in other tasks.
Revised: The ﬁnetuning phase creates separate models for each task, where improving the performance of a particular task necessitates forgetting some of the knowledge garnered in other tasks.

Original: Humans, on the other hand, perform task-speciﬁc learning in synergy with general domain-based learning.
Revised: Humans, on the other hand, perform task-speciﬁc learning in synergy with general domain-based learning.

Original: Inspired by these learning patterns in humans, we suggest a simple yet generic task aware framework to incorporate into existing MTL strategies.
Revised: Inspired by these learning patterns in humans, we suggest a simple yet generic task aware framework to incorporate into existing MTL strategies.

Original: The proposed framework computes task-speciﬁc representations to modulate the model parameters during MTL.
Revised: The proposed framework computes task-speciﬁc representations to modulate the model parameters during MTL.

Original: Hence, it performs both shared and task-speciﬁc learning in a single phase resulting in a single model for all the tasks.
Revised: Hence, it performs both shared and task-speciﬁc learning in a single phase resulting in a single model for all the tasks.

Original: The single model itself achieves signiﬁcant performance gains over the existing MTL strategies.
Revised: The single model itself achieves signiﬁcant performance gains over the existing MTL strategies.

Original: For example, we train a model on Speech Translation (ST), Automatic Speech Recognition (ASR), and Machine Translation (MT) tasks using the proposed task aware multitask learning approach.
Revised: For example, we train a model on Speech Translation (ST), Automatic Speech Recognition (ASR), and Machine Translation (MT) tasks using the proposed task aware multitask learning approach.

Original: This single model achieves a performance of 28.64 BLEU score on ST MuST-C English-German,
Revised: This single model achieves a performance of 28.64 BLEU score on ST MuST-C English-German,

Original: WER of 11.61 on ASR TEDLium v3, and BLEU score of 23.35 on MT WMT
Revised: WER of 11.61 on ASR TEDLium v3, and BLEU score of 23.35 on MT WMT

Original: English-German tasks.
Revised: English-German tasks.

Original: This sets a new state-of-the-art performance (SOTA) on the ST task while having close to SOTA performance on ASR and competitive performance on the MT task.
Revised: This sets a new state-of-the-art performance (SOTA) on the ST task while outperforming the existing end-to-end ASR systems with a competitive performance on the MT task.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: The process of Multi-Task Learning (MTL) on a set of related tasks is inspired by the patterns displayed by human learning.
Revised: The process of Multi-Task Learning (MTL) on a set of related tasks is inspired by the patterns displayed by human learning.

Original: It involves a pretraining phase over all the tasks, followed by a ﬁnetuning phase.
Revised: It involves a pretraining phase over all the tasks, followed by a ﬁnetuning phase.

Original: During pretraining, the model tries to grasp the shared knowledge of all the tasks involved, while in the ﬁnetuning phase, task-speciﬁc learning is performed to improve the performance.
Revised: During pretraining, the model tries to grasp the shared knowledge of all the tasks involved, while in the ﬁnetuning phase, task-speciﬁc learning is performed to improve the performance.

Original: However, as a result of the ﬁnetuning phase, the model forgets the information about the other tasks that it learnt during pretraining.
Revised: However, as a result of the ﬁnetuning phase, the model forgets the information about the other tasks that it learnt during pretraining.

Original: Humans, on the other hand, are less susceptible to forgetfulness and retain existing knowledge/skills while mastering a new task.
Revised: Humans, on the other hand, are less susceptible to forgetfulness and retain existing knowledge/skills while mastering a new task.

Original: For example, a polyglot who masters a new language learns to translate from this language without losing the ability to translate other languages.
Revised: For example, a polyglot who masters a new language learns to translate from this language without losing the ability to translate other languages.

Original: Moreover, the lack of task-based ﬂexibility and having different ﬁnetuning/pretraining phases cause gaps in the learning process due to the following reasons:
Revised: Moreover, the lack of task-based ﬂexibility and having different ﬁnetuning/pretraining phases cause gaps in the learning process due to the following reasons:

Original: Role Mismatch: Consider the MTL system being trained to perform the Speech Translation(ST),
Revised: Role Mismatch: Consider the MTL system being trained to perform the Speech Translation(ST),

Original: Automatic Speech Recognition(ASR) and Machine Translation(MT) tasks.
Revised: Automatic Speech Recognition(ASR) and Machine Translation(MT) tasks.

Original: The Encoder block has a very different role in the standalone ASR, MT and ST models and hence we cannot expect a single encoder to perform well on all the tasks without any cues to identify/use task information.
Revised: The Encoder block has a very different role in the standalone ASR, MT and ST models and hence we cannot expect a single encoder to perform well on all the tasks without any cues to identify/use task information.

Original: Moreover, there is a discrepancy between pretraining and ﬁnetuning hampering the MTL objective.
Revised: Moreover, there is a discrepancy between pretraining and ﬁnetuning hampering the MTL objective.

Original: Task Awareness:
Revised: Task Awareness:

Original: At each step in the MTL, the model tries to optimize over the task at hand.
Revised: At each step in the MTL, the model tries to optimize over the task at hand.

Original: For tasks like ST and ASR with the same source language, it is impossible for the model to identify the task and alter its parameters accordingly, hence necessitating a ﬁnetuning phase.
Revised: For tasks like ST and ASR with the same source language, it is impossible for the model to identify the task and alter its parameters accordingly, hence necessitating a ﬁnetuning phase.

Original: A few such examples have been provided in Table 1.
Revised: A few such examples have been provided in Table 1.

Original: Humans, on the other hand, grasp the task they have to perform by means of context or explicit cues.
Revised: Humans, on the other hand, grasp the task they have to perform by means of context or explicit cues.

Original: [Table]
Revised: [Table]

Original: Confounding

Original: [Table]

Original: Target

Original: En ASR
Revised: En ASR

Original: En-De ST 
Revised: En-De ST sogar ihre eigenen Eltern even their own parents

Original: Mobilit¨at opportunity to..
Revised: Mobilit¨at opportunity to..

Original: Mobility opportunity to ..
Revised: Mobility opportunity to ..

Original: En-De MT
Revised: En-De MT

Original: En-Ro MT
Revised: En-Ro MT

Original: Situat , ia este alarmant˘a
Revised: Situat , ia este alarmant˘a

Original: Die Situation ist alarmierend
Revised: Die Situation ist alarmierend

Original: En-De ST
Revised: En-De ST

Original: En ASR
Revised: En ASR

Original: I don’t understand it.
Revised: I don’t understand it.

Original: Das verstehe ich nicht 
Revised: Das verstehe ich nicht was a success or not.

Original:  Although MTL strategies help the ﬁnetuned models to perform better than the models directly trained on those tasks, their applicability is limited to ﬁnding a good initialization point for the ﬁnetuning phase.
Revised: war ein Erfolg oder nicht. Although MTL strategies help the ﬁnetuned models to perform better than the models directly trained on those tasks, their applicability is limited to ﬁnding a good initialization point for the ﬁnetuning phase.

Original: Moreover, having a separate model for each task increases the memory requirements, which is detrimental in low resource settings.
Revised: Moreover, having a separate model for each task increases the memory requirements, which is detrimental in low resource settings.

Original: In order to achieve the goal of jointly learning all the tasks, similar to humans, we need to perform shared learning in synergy with task-speciﬁc learning.
Revised: In order to achieve the goal of jointly learning all the tasks, similar to humans, we need to perform shared learning in synergy with task-speciﬁc learning.

Original: Previous approaches such as Raffel et al.
Revised: Previous approaches such as Raffel et al.

Original: (2019) trained a joint model for a set of related text-to-text tasks by providing the task information along with the inputs during the joint learning phase.
Revised: (2019) trained a joint model for a set of related text-to-text tasks by providing the task information along with the inputs during the joint learning phase.

Original: However, providingtask information is not straightforward for certain modalities such as speech and image. Also, providing explicit task information might not alwaysbe desirable, e.g.,  automatic multilingual speech recognition task. 
Revised: However, providing explicit task information is not always desirable, e.g., consider the automatic multilingual speech translation task. In order to ensure seamless user experience, it is expected that the model extracts the task information implicitly.

Original: Thus, a holistic joint learning strategy requires a generic framework which learns task-speciﬁc information without any explicit supervision.
Revised: Thus, a holistic joint learning strategy requires a generic framework which learns task-speciﬁc information without any explicit supervision.

Original: In this work, we propose a generic framework which can be easily integrated into the MTL strategies which can extract task-based characteristics.
Revised: In this work, we propose a generic framework which can be easily integrated into the MTL strategies which can extract task-based characteristics.

Original: The proposed approach helps align existing MTL approaches with human learning processes by incorporating task information into the learning process and getting rid of the issues related to forgetfulness.
Revised: The proposed approach helps align existing MTL approaches with human learning processes by incorporating task information into the learning process and getting rid of the issues related to forgetfulness.

Original: We design a modulation network for learning the task characteristics and modulating the parameters of the model during MTL.
Revised: We design a modulation network for learning the task characteristics and modulating the parameters of the model during MTL.

Original: As discussed above, the task information may or may not be explicitly available during the training.
Revised: As discussed above, the task information may or may not be explicitly available during the training.

Original: Hence, we propose two different designs of task modulation network to learn the task characteristics; one uses explicit task identities while the other uses the examples from the task as input.
Revised: Hence, we propose two different designs of task modulation network to learn the task characteristics; one uses explicit task identities while the other uses the examples from the task as input.

Original: The model, coupled with the modulation network, jointly learns on all the tasks and at the same time, performs the task-speciﬁc learning.
Revised: The model, coupled with the modulation network, jointly learns on all the tasks and at the same time, performs the task-speciﬁc learning.

Original: The proposed approach tackles issues related to forgetfulness by keeping a single model for all the tasks, and hence avoiding the expensive ﬁnetuning phase.
Revised: The proposed approach tackles issues related to forgetfulness by keeping a single model for all the tasks, and hence avoiding the expensive ﬁnetuning phase.

Original: Having a single model for all the tasks also reduces memory constraints, improving suitability for low resource devices.
Revised: Having a single model for all the tasks also reduces memory constraints, improving suitability for low resource devices.

Original: To evaluate the proposed framework, we conduct two sets of experiments.
Revised: To evaluate the proposed framework, we conduct two sets of experiments.

Original: First, we include the task information during MTL on text-to-text tasks to show the effect of task information.
Revised: First, we include the task information during MTL on text-to-text tasks to show the effect of task information.

Original: Secondly, we train a model on tasks with different modalities and end goals, where providing task information is not straightforward.
Revised: Secondly, we train a model on tasks with different modalities and end goals, with highly confounding tasks.

Original: Our proposed framework allows the model to learn the task characteristics without any explicit supervision, and hence train a single model which performs well on all the tasks.
Revised: Our proposed framework allows the model to learn the task characteristics without any explicit supervision, and hence train a single model which performs well on all the tasks.

Original: The main contributions of this work are as follows:
Revised: The main contributions of this work are as follows:

Original: • We propose an approach to tackle the issue of forgetfulness which occurs during the ﬁnetuning phase of existing MTL strategies.
Revised: • We propose an approach to tackle the issue of forgetfulness which occurs during the ﬁnetuning phase of existing MTL strategies.

Original: • Our model, without any ﬁnetuning, achieves superior performance on all the tasks which alleviates the need to keep separate task-speciﬁc models.
Revised: • Our model, without any ﬁnetuning, achieves superior performance on all the tasks which alleviates the need to keep separate task-speciﬁc models.

Original: • Our proposed framework is generic enough to be used with any MTL strategy involving tasks with multiple modalities.
Revised: • Our proposed framework is generic enough to be used with any MTL strategy involving tasks with multiple modalities.

Original:  T ASK -A WARE J OINT L EARNING
Revised:  T ASK -A WARE M ULTITASK L EARNING

Original: An overview of our proposed approach is shown in Figure 1.[Figure]
Revised: An overview of our proposed approach is shown in Figure 1.

Original:  ASE M ODEL
Revised: 2. B ASE M ODEL

Original: In general, the sequence-to-sequence architecture consists of two components: (1) an encoder which computes a set of representations X = { x 1 , · · · , x m } ∈ R m × d corresponding to x , and a decoder coupled with attention mechanism (Bahdanau et al., 2015) dynamically reads encoder’s output and predicts target language sequence Y =
Revised: In general, the sequence-to-sequence architecture consists of two components: (1) an encoder which computes a set of representations X = { x 1 , · · · , x m } ∈ R m × d corresponding to x , and a decoder coupled with attention mechanism (Bahdanau et al., 2015) dynamically reads encoder’s output and predicts target language sequence Y =

Original: { y 1 , · · · , y n } ∈ R n × d .
Revised: { y 1 , · · · , y n } ∈ R n × d .

Original: It is trained on a dataset D to maximize the p ( Y | X ; θ ) , where θ are parameters of the model.
Revised: It is trained on a dataset D to maximize the p ( Y | X ; θ ) , where θ are parameters of the model.

Original: We use the Transformer Vaswani et al.
Revised: We use the Transformer Vaswani et al.

Original: (2017) as our base model.
Revised: (2017) as our base model.

Original: Based on the task modalities, we choose the preprocessing layer in the Transformer, i.e., speech or the text (text-embedding) preprocessing layer.
Revised: Based on the task modalities, we choose the preprocessing layer in the Transformer, i.e., speech or the text (text-embedding) preprocessing layer.

Original: The speech preprocessing layer consists of a stack of k CNN layers with stridefor both time and frequency dimensions.
Revised: The speech preprocessing layer consists of a stack of k CNN layers with stridefor both time and frequency dimensions.

Original: This layer compresses the speech sequence and produces the output sequence such that input sequences corresponding to all the tasks have similar dimensions, d .
Revised: This layer compresses the speech sequence and produces the output sequence such that input sequences corresponding to all the tasks have similar dimensions, d .

Original: The overview of the base sequence-to-sequence model is shown in the rightmost part of Figure 1.
Revised: The overview of the base sequence-to-sequence model is shown in the rightmost part of Figure 1.

Original: 2. T ASK M ODULATION N ETWORK
Revised: 2. T ASK M ODULATION N ETWORK

Original: The task modulation network performs two operations.
Revised: The task modulation network performs two operations.

Original: In the ﬁrst step, it computes the task characteristics ( t e ) using the task characteristics layer.
Revised: In the ﬁrst step, it computes the task characteristics ( t e ) using the task characteristics layer.

Original: It then modulates the model parameters θ using t e in the second step.
Revised: It then modulates the model parameters θ using t e in the second step.

Original: 2.2. T ASK C HARACTERISTICS N ETWORK :
Revised: 2.2. T ASK C HARACTERISTICS N ETWORK :

Original: We propose two types of Task Characteristics Networks(TCN) to learn the task characteristics, where one uses explicit task identities while the other uses source-target sequences as input.
Revised: We propose two types of Task Characteristics Networks(TCN) to learn the task characteristics, where one uses explicit task identities while the other uses source-target sequences as input.

Original: Explicit Task Information:
Revised: Explicit Task Information:

Original: In this approach, the tasks involved are represented using different task identities and fed as input to this TCN as one hot vectors.
Revised: In this approach, the tasks involved are represented using different task identities and fed as input to this TCN as one hot vectors.

Original: This network consists of a feed-forward layer which produces the task embedding used for modulating the model parameters.
Revised: This network consists of a feed-forward layer which produces the task embedding used for modulating the model parameters.

Original: [Equation]
Revised: [Equation]

Original: where e ∈ R s is a on-hot encoding of s tasks used during joint learning.
Revised: where e ∈ R s is a one-hot encoding of s tasks used during joint learning.

Original: Implicit Task Information: Providing task information for all the tasks might not be the best option since some of the tasks might have overlapping goals/characteristics.

Original: For example, translating written text along with spoken text sentences from English-to-German, or automatic multilingual speech recognition.

Original:  The Implicit TCN computes the task embeddings using example sequences from the tasks.
Revised: Implicit Task Information: The Implicit TCN computes the task embeddings using example sequences from the tasks without any external supervision.

Original: It consists of four sub-layers: (1) Sequence Representation Layer, (2) Bi-directional Attention Layer, (3) Sequence Summary Layer, and (4) Task
Revised: It consists of four sub-layers: (1) Sequence Representation Layer, (2) Bi-directional Attention Layer, (3) Sequence Summary Layer, and (4) Task

Original: Embedding Layer.
Revised: Embedding Layer.

Original: The sequence representation sub-layer consists of uni-directional Transformer Encoder (TE) blocks
Revised: The sequence representation sub-layer consists of uni-directional Transformer Encoder (TE) blocks

Original: Vaswani et al.
Revised: Vaswani et al.

Original: It takes the source and target sequences from the tasks as input and produces self-attended source and target sequences.
Revised: It takes the source and target sequences from the tasks as input and produces self-attended source and target sequences.

Original: [Equation]
Revised: [Equation]

Revised: where X sa ∈ R M × d , Y sa ∈ R N × d .

Original: This sub-layer computes the contextual representation of the sequences.
Revised: This sub-layer computes the contextual representation of the sequences.

Original: The Bi-directional attention sub-layer takes the self-attended source and target sequences from the previous layer as input and computes the relation between them using Dot-Product Attention
Revised: The Bi-directional Attention (BiA) sub-layer takes the self-attended source and target sequences from the previous layer as input and computes the relation between them using Dot-Product Attention

Original: Luong et al. (2015).
Revised: Luong et al. (2015).

Original: As a result, we get target aware source ( x at  ) and source aware target ( y as  ) representations as outputs.
Revised: As a result, we get target aware source ( X at ∈ R M × d ) and source aware target ( Y as R N × d ) representations as outputs.

Original: [Equation]
Revised: [Equation]

Original: The sequence summary sub-layer is similar to the sequence representation sub layer and summarizes the sequences.
Revised: The sequence summary sub-layer is similar to the sequence representation sub layer and summarizes the sequences.

Original: The sequence summaries are given by:
Revised: The sequence summaries are given by:

Original: [Equation]
Revised: [Equation]

Original:  Equation 4 summarizes the sequences x at and y as which contain the contextual and attention information.
Revised: where X s ∈ R M × d , Y s ∈ R N × d . The Equation 4 summarizes the sequences X at and Y as which contain the contextual and attention information.

Original: Speciﬁcally, we take the last tokens from both  x s  and y s as they can see the whole sequence; they act as a summary of the sequence.
Revised: We take the last tokens from both the x s ∈ R d and y s ∈ R d , since the last token can see the whole sequence and acts as a summary of the sequence.

Original: The task embedding layer computes t e by taking the output of the Sequence Summary Layer( s ) and applying a feed-forward network:
Revised: The task embedding layer computes t e by taking the outputs of the sequence summary sub-layer and applying a feed-forward network:

Original: [Equation]
Revised: [Equation]

Original: where m and n are the source and target sequence lengths.

Original: 2.2. M ODULATING M ODEL P ARAMETERS
Revised: 2.2. M ODULATING M ODEL P ARAMETERS

Original: We modulate the parameters ( θ ) of the network (Section 2.1) to account for the task-speciﬁc variation during MTL over a set of tasks.
Revised: We modulate the parameters ( θ ) of the network (Section 2.1) to account for the task-speciﬁc variation during MTL over a set of tasks.

Original: We achieve this by scaling ( γ ) and shifting ( β ) the outputs of each layer (e.g., transformer block) including any preprocessing layers in the model adopted based on the
Revised: We achieve this by scaling ( γ ) and shifting ( β ) the outputs of each layer (e.g., transformer block) including any preprocessing layers in the model adopted based on the

Original: Feature-wise Linear Modulation (FiLM; Perez et al.
Revised: Feature-wise Linear Modulation (FiLM; Perez et al.

Original: The γ and β parameters are obtained from the task embedding t e either by using equation 1 or 5.
Revised: The γ and β parameters are obtained from the task embedding t e either by using equation 1 or 5.

Original: [Equation]
Revised: [Equation]

Original: where t e ∈ R 2 d , and d is the hidden dimension of the model.
Revised: where t e ∈ R 2 d , and d is the hidden dimension of the model.

Original: Once we have γ and β , we apply the feature-wise linear modulation (Perez et al., 2018) to compute the modulated output ( O l ) for each block of the model.
Revised: Once we have γ and β , we apply the feature-wise linear modulation (Perez et al., 2018) to compute the modulated output ( O l ) for each block of the model.

Original: [Equation]
Revised: [Equation]

Original: where L is the total number of blocks in the model and f l represents the l th block of the model with parameters θ l ∈ θ and inputs v l .
Revised: where L is the total number of blocks in the model and f l represents the l th block of the model with parameters θ l ∈ θ and inputs v l .

Original: 2. T RAINING
Revised: 2. T RAINING

Original: MTL has been successfully applied across different applications of machine learning such as natural language processing (Hashimoto et al., 2016; Collobert & Weston, 2008), speech recognition (Liu et al., 2019; Deng et al., 2013), computer vision (Zhang et al., 2014; Liu et al., 2015; Girshick, 2015), and drug discovery (Ramsundar et al., 2015).
Revised: MTL has been successfully applied across different applications of machine learning such as natural language processing (Hashimoto et al., 2016; Collobert & Weston, 2008), speech recognition (Liu et al., 2019; Deng et al., 2013), computer vision (Zhang et al., 2014; Liu et al., 2015; Girshick, 2015), and drug discovery (Ramsundar et al., 2015).

Original: It comes in many forms: joint learning, learning to learn, and learning with auxiliary tasks.
Revised: It comes in many forms: joint learning, learning to learn, and learning with auxiliary tasks.

Original: We consider two MTL strategies: (1) joint learning and (2) learning to learn to train on set of S tasks, T = { τ 1 , · · · , τ S } with corresponding datasets
Revised: We consider two MTL strategies: (1) joint learning and (2) learning to learn to train on set of S tasks, T = { τ 1 , · · · , τ S } with corresponding datasets

Original: D =
Revised: D =

Original: { D 1 , · · · , D S } .
Revised: { D 1 , · · · , D S } .

Original: As our ﬁrst training strategy, we use Joint Learning (JL) (Caruana, 1997), which is the most commonly used training strategy for MTL.
Revised: As our ﬁrst training strategy, we use Joint Learning (JL) (Caruana, 1997), which is the most commonly used training strategy for MTL.

Original: In JL, the model parameters, including the output layer, are shared across all the tasks involved in the training.
Revised: In JL, the model parameters, including the output layer, are shared across all the tasks involved in the training.

Original: For the second training strategy under the learning-tolearn approach, we use a variant of meta-learning, Modality Agnostic Meta Learning (MAML)(Finn [Table]
Revised: For the second training strategy under the learning-tolearn approach, we use a variant of meta-learning, Modality Agnostic Meta Learning (MAML)

Original:  et al., 2017a).
Revised: (Finn et al., 2017a).

Original: Even though MAML is mostly used in few-shot learning settings, we use it since it  allows for task-speciﬁc learning during the meta-train step and it has also been shown to provide improvements in the ﬁeld of speech translation(Indurthi et al., 2020).
Revised: Even though MAML is mostly used in few-shot learning settings, we use it since it [Table] allows for task-speciﬁc learning during the meta-train step and it has also been shown to provide improvements in the ﬁeld of speech translation(Indurthi et al., 2020).

Original: We resolve the source-target vocabulary mismatch across different tasks in MTL by using a vocabulary of subwords (Sennrich et al., 2016) computed from all the tasks.
Revised: We resolve the source-target vocabulary mismatch across different tasks in MTL by using a vocabulary of subwords (Sennrich et al., 2016) computed from all the tasks.

Original: We sample a batch of examples from
Revised: We sample a batch of examples from

Original: D s and use this as input to the TCN and the Transformer model.
Revised: D s and use this as input to the TCN and the Transformer model.

Original: To ensure that each training example uses the task embedding computed using another example, we randomly shufﬂe this batch while using them as input to the TCN.
Revised: To ensure that each training example uses the task embedding computed using another example, we randomly shufﬂe this batch while using them as input to the TCN.

Original: This random shufﬂing improves the generalization performance by forcing the network to learn task-speciﬁc characteristics ( t e ) in Equation 1 or 5.
Revised: This random shufﬂing improves the generalization performance by forcing the network to learn task-speciﬁc characteristics ( t e ) in Equation 1 or 5.

Original: We compute the task embedding in the meta-train step as well; however, the parameters of the TCN are updated only during the meta-test step.
Revised: We compute the task embedding in the meta-train step as well; however, the parameters of the TCN are updated only during the meta-test step.

Original: During inference time, we use the precomputed task embeddings using a batch of examples randomly sampled from the training set.
Revised: During inference time, we use the precomputed task embeddings using a batch of examples randomly sampled from the training set.

Original:  E XPERIMENTS 3. T ASKS AND D ATASETS
Revised:  E XPERIMENTS 3. T ASKS AND D ATASETS

Original: We conduct two sets of experiments, one with the tasks having the same input modality, i.e., text and another over tasks having different input modalities, i.e., speech and text.
Revised: We conduct two sets of experiments, one with the tasks having the same input modality, i.e., text and another over tasks having different input modalities, i.e., speech and text.

Original: The main motivation behind the text-based experiments is to establish the importance of providing task information in
Revised: The main motivation behind the text-based experiments is to establish the importance of providing task information in

Original: MTL.
Revised: MTL.

Original: The second set of experiments help us demonstrate the effectiveness of our approach in amore generic setup.
Revised: Our main experiments, containing different input modalities involve highly confusing tasks. These experiments help us demonstrate the effectiveness of our approach in a generic setup.

Original: We incorporate the proposed task modulation framework into joint and meta-learning strategies and analyze its effects.
Revised: We incorporate the proposed task modulation framework into joint and meta-learning strategies and analyze its effects.

Original: 3.1. S INGLE M ODALITY E XPERIMENTS
Revised: 3.1. S INGLE M ODALITY E XPERIMENTS

Original: We perform the small scale text-to-text machine translation task over three language pairs EnglishGerman/Romanian/Turkish (En-De/Ro/Tr).
Revised: We perform the small scale text-to-text machine translation task over three language pairs EnglishGerman/Romanian/Turkish (En-De/Ro/Tr).

Original: We keep English as the source language, which makes it crucial to use task information and produce different outputs from the same input.
Revised: We keep English as the source language, which makes it crucial to use task information and produce different outputs from the same input.

Original: Since it is easier to provide task identity through one-hot vectors in text, we provide the task information by simply prepending the task identity to the source sequence of each task, e.g., ” translate from English to
Revised: Since it is easier to provide task identity through one-hot vectors in text, we provide the task information by simply prepending the task identity to the source sequence of each task, e.g., ” translate from English to

Original: German ”, ” translate from English to Turkish ” similar to Raffel et al.
Revised: German ”, ” translate from English to Turkish ” similar to Raffel et al.

Original: We also train models using our proposed framework to learn the task information and shared knowledge jointly.
Revised: We also train models using our proposed framework to learn the task information and shared knowledge jointly.

Original: For En-De, we use 1.9M training examples from the Europarl v7 dataset.
Revised: For En-De, we use 1.9M training examples from the Europarl v7 dataset.

Original: Europarl dev2006 and News Commentary nc-dev2007 are used as the dev and Europarl devtest2006 , Europarl test2006 and News Commentary nc-devtest2007 as the test sets.
Revised: Europarl dev2006 and News Commentary nc-dev2007 are used as the dev and Europarl devtest2006 , Europarl test2006 and News Commentary nc-devtest2007 as the test sets.

Original: For En-Tr we train using 200k training examples from the setimes2 dataset.
Revised: For En-Tr we train using 200k training examples from the setimes2 dataset.

Original: We use newsdev2016 as the dev and newstest2017 as the test set.
Revised: We use newsdev2016 as the dev and newstest2017 as the test set.

Original: For En-Ro, we use 600k training examples from Europarl v8 and setimes2 datasets.
Revised: For En-Ro, we use 600k training examples from Europarl v8 and setimes2 datasets.

Original: We use newsdev2016 as dev and newstest2016 as the test set.
Revised: We use newsdev2016 as dev and newstest2016 as the test set.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  M ULTIPLE M ODALITY E XPERIMENTS
Revised:  M ULTIPLE M ODALITY E XPERIMENTS

Original: To alleviate the data scarcity issue in Speech Translation (ST), several MTL strategies have been proposed to jointly train the ST task with Automatic Speech Recognition (ASR) and Machine
Revised: To alleviate the data scarcity issue in Speech Translation (ST), several MTL strategies have been proposed to jointly train the ST task with Automatic Speech Recognition (ASR) and Machine

Original: Translation (MT) tasks.
Revised: Translation (MT) tasks.

Original: These MTL approaches lead to signiﬁcant performance gains on both ST and
Revised: These MTL approaches lead to signiﬁcant performance gains on both ST and

Original: ASR tasks after the ﬁnetuning phase.
Revised: ASR tasks after the ﬁnetuning phase.

Original: We evaluate our proposed framework based on this multimodal
Revised: We evaluate our proposed framework based on this multimodal

Original: MTL setting since passing the task information explicitly via prepending labels(like
Revised: MTL setting since passing the task information explicitly via prepending labels(like

Original: the text-to-text case) in the source sequence is not possible.
Revised: the text-to-text case) in the source sequence is not possible.

Original: We use the following datasets for ST English-German,
Revised: We use the following datasets for ST English-German,

Original: ASR English, MT English-German tasks:
Revised: ASR English, MT English-German tasks:

Original: MT En-De: We use the Open Subtitles (Lison et al., 2019) and WMT 19 corpora
Revised: MT En-De: We use the Open Subtitles (Lison et al., 2019) and WMT 19 corpora

Original: WMT 19 consists of Common Crawl, Europarl v9, and News Commentary v14 datasets(22M training examples).
Revised: WMT 19 consists of Common Crawl, Europarl v9, and News Commentary v14 datasets(22M training examples).

Original: ASR English : We used ﬁve different datasets namely LibriSpeech
Revised: ASR English : We used ﬁve different datasets namely LibriSpeech

Original: (Panayotov et al., 2015), MuST-C (Di Gangi et al., 2019), TED-LIUM (Hernandez et al., 2018), Common Voice (Ardila et al., 2020) and ﬁltered IWSLT 19 (nie, 2019) to train the English ASR task.
Revised: (Panayotov et al., 2015), MuST-C (Di Gangi et al., 2019), TED-LIUM (Hernandez et al., 2018), Common Voice (Ardila et al., 2020) and ﬁltered IWSLT 19 (IWS, 2019) to train the English ASR task.

Original: ST Task:
Revised: ST Task:

Original: We use the Europarl ST (Iranzo-S ´ anchez et al., 2019), IWSLT 2019 (nie, 2019) and
Revised: We use the Europarl ST (Iranzo-S ´ anchez et al., 2019), IWSLT 2019 (IWS, 2019) and

Original: MuST-C (Di Gangi et al., 2019) datasets.
Revised: MuST-C (Di Gangi et al., 2019) datasets.

Original: Since ST task has lesser training examples, we use data augmentation techniques (Lakumarapu et al., 2020) to increase the number of training examples.
Revised: Since ST task has lesser training examples, we use data augmentation techniques (Lakumarapu et al., 2020) to increase the number of training examples.

Original: Please refer to the appendix for more details about the data statistics and data augmentation techniques used.
Revised: Please refer to the appendix for more details about the data statistics and data augmentation techniques used.

Original: All the models reported in this work use the same data settings for training and evaluation.
Revised: All the models reported in this work use the same data settings for training and evaluation.

Original: 3. I MPLEMENTATION D ETAILS AND M ETRICS
Revised: 3. I MPLEMENTATION D ETAILS AND M ETRICS

Original: We implemented all the models using Tensorﬂow 2.2 framework.
Revised: We implemented all the models using Tensorﬂow 2.2 framework.

Original: For all our experiments, we use the
Revised: For all our experiments, we use the

Original: Transformer(Vaswani et al., 2017) as our base model.
Revised: Transformer(Vaswani et al., 2017) as our base model.

Original: The hyperparameter settings such as learning rate, scheduler, optimization algorithm, and dropout have been kept similar to the Transformer, other than the ones explicitly stated to be different.
Revised: The hyperparameter settings such as learning rate, scheduler, optimization algorithm, and dropout have been kept similar to the Transformer, other than the ones explicitly stated to be different.

Original: The ASR performance is measured using Word Error
Revised: The ASR performance is measured using Word Error

Original: Rate (WER) while ST and MT performances are calculated using the detokenized cased BLEU score (Post, 2018).
Revised: Rate (WER) while ST and MT performances are calculated using the detokenized cased BLEU score (Post, 2018).

Original: We generate word-piece based universal vocabulary(Gu et al., 2018b) of size 32k using source and target text sequences of all the tasks.
Revised: We generate word-piece based universal vocabulary (Gu et al., 2018a) of size 32k using source and target text sequences of all the tasks.

Original: For the task aware MTL strategies, we choose a single model to report the results rather than ﬁnding the best model for each task separately.
Revised: For the task aware MTL strategies, we choose a single model to report the results rather than ﬁnding the best model for each task separately.

Original: We train the text-to-text translation models using 6 Encoder and Decoder layers with a batch size of 2048 text tokens.
Revised: We train the text-to-text translation models using 6 Encoder and Decoder layers with a batch size of 2048 text tokens.

Original: The training is performed using NVIDIA P40 GPU for 400k steps.
Revised: The training is performed using NVIDIA P40 GPU for 400k steps.

Original: In multi-modality experiments, the speech signals are represented using 80-dimensional log-Mel features and use 3 CNN layers in the preprocessing layer described in Section 2.1.
Revised: In multi-modality experiments, the speech signals are represented using 80-dimensional log-Mel features and use 3 CNN layers in the preprocessing layer described in Section 2.1.

Original: We use 12 Encoder and Decoder layers and train for 600k steps using 8 NVIDIA V100 GPUs.
Revised: We use 12 Encoder and Decoder layers and train for 600k steps using 8 NVIDIA V100 GPUs.

Original: For the systems without
Revised: For the systems without

Original: TCN, we perform ﬁnetuning for 10k steps on each task.
Revised: TCN, we perform ﬁnetuning for 10k steps on each task.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 3. R ESULTS 3.3. S INGLE M ODALITY EXPERIMENTS
Revised: [Figure] S [Figure] M ODALITY E

Original:  The results for the text-to-text translation models trained with different MTL strategies have been provided in Table 2.
Revised: [Figure] The results for the text-to-text translation models trained with different MTL strategies have been provided in Table 2.

Original: The MTL models with prepended task label (Raffel et al., 2019) are referred to as OHV (One Hot Vector).
Revised: The MTL models with prepended task label (Raffel et al., 2019) are referred to as OHV (One Hot Vector).

Original: Unlike T5, we don’t initialize the models with the text embeddings from large pretrained language model (Devlin et al., 2018).
Revised: Unlike T5, we don’t initialize the models with the text embeddings from large pretrained language model (Devlin et al., 2018).

Original: Instead, we focus on establishing the importance of task information during MTL and having a single model for all the tasks.
Revised: Instead, we focus on establishing the importance of task information during MTL and having a single model for all the tasks.

Original: As we can see from the results [Table] providing the task information via text labels or implicitly using the proposed task aware MTL leads to signiﬁcant performance improvements compared to the MTL without the task information.
Revised: As we can see from the results, providing the task information via text labels or implicitly using the proposed task aware MTL leads to signiﬁcant performance improvements compared to the MTL without the task information.

Original: The models trained using OHV have better performance than those trained using implicit TCN.
Revised: The models trained using OHV have better performance than those trained using implicit TCN.

Original: However, providing OHV via text labels is not always possible for tasks involving non-text modalities such as speech and images.
Revised: However, providing OHV via text labels is not always possible for tasks involving non-text modalities such as speech and images.

Original: 3.3. M ULTI M ODALITY E XPERIMENTS
Revised: 3.3. M ULTI M ODALITY E XPERIMENTS

Original: We evaluate the proposed two TCNs and compare them with the vanilla MTL strategies.
Revised: We evaluate the proposed two TCNs and compare them with the vanilla MTL strategies.

Original: The performance of all the models is reported in Table 3.
Revised: The performance of all the models is reported in Table 3.

Revised: We also extended the T5 (Raffel et al., 2019) approach to the multi modality experiments and compare it with our approach.

Original: Effect of Task Information: The models trained using task aware MTL achieve signiﬁcant performance gains over the models trained using vanilla MTL approach.
Revised: Effect of Task Information: The models trained using task aware MTL achieve signiﬁcant performance gains over the models trained using vanilla MTL approach.

Original: Our single model achieves superior performance compared to the vanilla MTL models even after the ﬁnetuning.
Revised: Our single model achieves superior performance compared to the vanilla MTL models even after the ﬁnetuning.

Original: This shows that not only is the task information essential to identify the task, but also helps to extract the shared knowledge better.
Revised: This shows that not only is the task information essential to identify the task, but also helps to extract the shared knowledge better.

Original: Our JL and MAML models trained with task aware MTL achieve improvements of (+2.65, +2.52) for ST, (-1.34, -1.18) for ASR, and (+0.72, +1.26) for the MT task.
Revised: Our JL and MAML models trained with task aware MTL achieve improvements of (+2.65, +2.52) for ST, (-1.34, -1.18) for ASR, and (+0.72, +1.26) for the MT task.

Original: MAML has some scope for task-speciﬁc learning during its meta train step, which explains why the improvements for
Revised: MAML has some scope for task-speciﬁc learning during its meta train step, which explains why the improvements for

Original: MAML are slightly lesser than JL for ST and ASR tasks.
Revised: MAML are slightly lesser than JL for ST and ASR tasks.

Original: We also report results using Direct Learning (DL) approach, where separate models are trained for each task, to compare with MTL models.
Revised: We also report results using Direct Learning (DL) approach, where separate models are trained for each task, to compare with MTL models.

Original: All the MTL models outperform the DL models on ST and
Revised: All the MTL models outperform the DL models on ST and

Original: ASR tasks have comparable performance on MT task.
Revised: ASR tasks have comparable performance on MT task.

Original: Explicit v/s
Revised: Explicit v/s

Original: Implicit TCN: Our proposed implicit TCN learns the task characteristics directly from the examples of each task and achieves slightly better performance than the models trained using explicit TCN.
Revised: Implicit TCN: Our proposed implicit TCN learns the task characteristics directly from the examples of each task and achieves a performance comparable to the models trained using explicit TCN.

Original: This indicates that it is better to learn the task information implicitly, speciﬁcally for tasks having overlapping characteristics.
Revised: This indicates that it is better to learn the task information implicitly, speciﬁcally for tasks having overlapping characteristics.

Original: Figure 2 contains the tSNE plots for task embeddings obtained from the implicit TCN for single and multi-modality experiments.
Revised: Figure 2 contains the tSNE plots for task embeddings obtained from the implicit TCN for single and multi-modality experiments.

Original: We can observe that the implicit TCN is also able to separate all the three tasks effectively without any external supervision.
Revised: We can observe that the implicit TCN is also able to separate all the three tasks effectively without any external supervision.

Original: Single model for all tasks: We select one single model for reporting the results for our approach, since, having a single model for multiple tasks is favourable in low resource settings.
Revised: Single model for all tasks: We select one single model for reporting the results for our approach, since, having a single model for multiple tasks is favourable in low resource settings.

Original: However, we also report the best models corresponding to each task (row 8 and 11 of Table 3).
Revised: However, we also report the best models corresponding to each task (row 8 and 11 of Table 3).

Original: We observe that choosing a single model over task-speciﬁc models did not result in any signiﬁcant performance loss.
Revised: We observe that choosing a single model over task-speciﬁc models did not result in any signiﬁcant performance loss.

Revised: Feature-wise v/s Input based modulation: We also implemented the input based conditioning (Toshniwal et al., 2018; Raffel et al., 2019) where we prepend the TCN output, i.e., task information to the source and target sequences.

Revised: As compared to our approach, this approach provides a comparable performance on the ASR task.

Revised: However, the ST performance is erratic and the output is mixed between

Original: Comparison with existing works: The design of our system, i.e., the parameters and  the related tasks were ﬁxed keeping the ST task in mind.
Revised: ST and ASR tasks. This shows that the feature-wise modulation is more efﬁcient way to carry out task-based conditioning for highly confusing tasks like ST and ASR. Number of parameters added: The Explicit TCN, which is a dense layer, roughly 1500 new parameters are added. For the Implicit TCN, roughly 1 million new additional parameters are added.

Revised: However, simply increasing the number of parameters is not sufﬁcient to to improve the performance.

Revised: For e.g., we trained several models by increasing the number of layers for encoder and decoder upto 16.

Revised: However, these models gave inferior performance as compared to the reported models withencoder and decoder layers.

Revised: Scaling with large number of tasks:

Revised: The t-sne plots in the Figure 2b are drawn using the three test datasets.

Revised: However, we used multiple datasets for each of the ASR(Librispeech, Common voice,

Revised: TEDLIUM, MuSTC-ASR), ST (MuSTC, IWSLT20, Europarl), and MT (WMT19, OpenSubtitles) tasks in the multi-modality experiments.

Revised: We analyze whether or not our proposed approach is able to separate the data coming from these different distributions.

Revised: As compared to data coming from different tasks, separating the data coming from the same task(generated from different distributions) is more difﬁcult.

Revised: Earlier, in Figure 2b, we observed that the output is clustered based on the tasks.

Revised: Figure 2c shows that within these task-based clusters, there are sub-clusters based on the source dataset.

Revised: Hence, the model is able to identify each sub-task based on the source dataset.

Revised: The model also gives decent performances on all of them.

Revised: For example, the single model achieves a WER of 7.5 on the Librispeech tst-clean, 10.35 on MuSTC, 11.65 on the TEDLIUM v3 and 20.36 on the commonvoice test set.

Revised: For the ST task, the same model gives a BLEU score of 28.64 on the

Revised: MuSTC test set, 27.61 on the IWSLT tst-2010, and 27.57 on the Europarl test set.

Revised: This shows that our proposed approach scales well with the total number of tasks.

Revised: Comparison with existing works: The design of our system, i.e., the parameters and the related tasks were ﬁxed keeping the ST task in mind.

Original: We compare the results of our system with the recent works in Table 4.
Revised: We compare the results of our best systems(after checkpoint averaging) with the recent works in Table 4.

Original: We set a new state-of-the-art (SOTA) on the ST En-De MuST-C task.
Revised: We set a new state-of-the-art (SOTA) on the ST En-De MuST-C task.

Original: Although we did not achieve a SOTA on the ASR task, we  achieve a 9.03% improvement in the WER as compared to theTransformer based model(Pham et al., 2019) with the same number of Encoder and Decoder blocks.
Revised: For the ASR task, we outperform the very deep Transformer based model Pham et al. We achieve a 19.2% improvement in the WER as compared to the model with the same number of Encoder and Decoder blocks.

Original: The best transformer-based MT model achieves a BLEU score of 30.10, however, it uses  Encoder blocks.
Revised: The best transformer-based MT model achieves a BLEU score of 30.10, however, it uses 60 Encoder blocks.

Original: The performance drop on the MT task is attributed to simply training a bigger model without using any additional initialization techniques proposed in Liu et al.
Revised: The performance drop on the MT task is attributed to simply training a bigger model without using any additional initialization techniques proposed in Liu et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: However, the MT task helps the other tasks and improves the overall performance of the system.
Revised: However, the MT task helps the other tasks and improves the overall performance of the system.

Original:  [Equation]
Revised:  R ELATED W ORK

Original: Various MTL techniques have been widely used to improve the performance of end-to-end neural networks.
Revised: Various MTL techniques have been widely used to improve the performance of end-to-end neural networks.

Original: These techniques are known to solve issues like overﬁtting and data scarcity.
Revised: These techniques are known to solve issues like overﬁtting and data scarcity.

Original: Joint learning (Caruana, 1997) improves the generalization by leveraging the shared information contained in the training signals of related tasks.
Revised: Joint learning (Caruana, 1997) improves the generalization by leveraging the shared information contained in the training signals of related tasks.

Original: MAML (Finn et al., 2017b) was proposed for training a joint model on a variety of tasks, such that it can quickly adapt to new tasks.
Revised: MAML (Finn et al., 2017b) was proposed for training a joint model on a variety of tasks, such that it can quickly adapt to new tasks.

Original: Both the learning approaches require a ﬁnetuning phase resulting in different models for each task.
Revised: Both the learning approaches require a ﬁnetuning phase resulting in different models for each task.

Original: Moreover, during ﬁnetuning phase the model substantially forgets the knowledge acquired during the large-scale pretraining.
Revised: Moreover, during ﬁnetuning phase the model substantially forgets the knowledge acquired during the large-scale pretraining.

Original: One of the original solutions to this problem is pseudo-rehearsal, which involves learning the new task while rehearsing generated items representative of the previous task.
Revised: One of the original solutions to this problem is pseudo-rehearsal, which involves learning the new task while rehearsing generated items representative of the previous task.

Original: This has been investigated and addressed to a certain extent in Atkinson et al.
Revised: This has been investigated and addressed to a certain extent in Atkinson et al.

Original: and Li & Hoiem (2018).
Revised: and Li & Hoiem (2018).

Original: He et al.
Revised: He et al.

Original: (2020) address this by using a mix-review ﬁnetuning strategy, where they include the pretraining objective during the ﬁnetuning phase.
Revised: (2020) address this by using a mix-review ﬁnetuning strategy, where they include the pretraining objective during the ﬁnetuning phase.

Original: Raffel et al.
Revised: Raffel et al.

Original: (2019) take a different approach by providing the task information to the model and achieve performance improvements on different text-to-text tasks.
Revised: (2019) take a different approach by providing the task information to the model and achieve performance improvements on different text-to-text tasks.

Original: Although this alleviates the need for ﬁnetuning, it cannot be extended to the tasks involving complex modalities.
Revised: Although this alleviates the need for ﬁnetuning, it cannot be extended to the tasks involving complex modalities.

Original: In our work, we propose a generic framework on top of MTL to provide task information to the model which can be applied irrespective of the task modalities.
Revised: In our work, we propose a generic framework on top of MTL to provide task information to the model which can be applied irrespective of the task modalities.

Original: It also removes the need for ﬁnetuning, tackling the issue of forgetfulness at its root cause.
Revised: It also removes the need for ﬁnetuning, tackling the issue of forgetfulness at its root cause.

Original: In the domain of neural machine translation, several MTL approaches have been proposed (Gu et al., 2018a;c).
Revised: A few approaches have also tried to train multiple tasks with a single model, Cheung et al. (2019) project the input to orthogonal sub-spaces based on the task information. In the approach proposed by Li & Hoiem (2018), the model is trained on various image classiﬁcation tasks having the same input modality.

Revised: They preserve the output of the model on the training example such that the parameters don’t deviate much from the original tasks.

Revised: This is useful when the tasks share the same goal, e.g. classiﬁcation.

Revised: However, we train on a much more varied set of tasks, which might also have the same inputs with different end goals.

Revised: Strezoski et al.

Revised: (2019) propose to apply a ﬁxed mask based on the task identity.

Revised: Our work can be seen as a generalization of this work.

Revised: As compared to all these approaches, our model is capable of performing both task identiﬁcation and the corresponding task learning simultaneously.

Revised: It learns to control the interactions among various tasks based on the inter-task similarity without any explicit supervision.

Revised: In the domain of neural machine translation, several MTL approaches have been proposed (Gu et al., 2018a;b).

Original: Similarly, recent works have shown that jointly training ST, ASR, and MT tasks improved the overall performance (Liu et al., 2019; Indurthi et al., 2020).
Revised: Similarly, recent works have shown that jointly training ST, ASR, and MT tasks improved the overall performance (Liu et al., 2019; Indurthi et al., 2020).

Original: However, all these require a separate ﬁnetuning phase and hence suffer from the issues mentioned above.
Revised: However, all these require a separate ﬁnetuning phase.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work proposes a task-aware framework which helps to improve the learning ability of the existing multitask learning strategies.
Revised: This work proposes a task-aware framework which helps to improve the learning ability of the existing multitask learning strategies.

Original: It addresses the issues faced during vanilla multitask learning, which includes forgetfulness during ﬁnetuning and the problems associated with having separate models for each task.
Revised: It addresses the issues faced during vanilla multitask learning, which includes forgetfulness during ﬁnetuning and the problems associated with having separate models for each task.

Original: The proposed approach helps to align better the existing multitask learning strategies with human learning.
Revised: The proposed approach helps to align better the existing multitask learning strategies with human learning.

Original: It achieves signiﬁcant performance improvements with a single model on a variety of tasks which is favourable in low resource settings.  
Revised: It achieves signiﬁcant performance improvements with a single model on a variety of tasks which is favourable in low resource settings.  A PPENDIX 6. D ATASETS 6.1. D ATA A UGMENTATION FOR S PEECH T RANSLATION

Revised: Table 5 provides details about the datasets used for the multi-modality experiments.

Revised: Since En-De

Revised: ST task has relatively fewer training examples compared to ASR and MT tasks, we augment the

Revised: ST dataset with synthetic training examples.

Revised: We generate the synthetic speech sequence and pair it with the synthetic German text sequences.

Revised: obtained by using the top two beam search results of the two trained English-to-German NMT models.

Revised: For speech sequence, we use the Sox library to generate the speech signal using different values of speed, echo, and tempo parameters similar to (Potapczyk et al., 2019).

Revised: The parameter values are uniformly sampled using these ranges : tempo ∈ (0 . 85 , 1 . 3) , speed ∈ (0 . 95 , 1 . 05) , echo delay ∈ (20 , 200) , and echo decay ∈ (0 . 05 , 0 . 2) .

Revised: We also train two NMT models on EN-De language pair to generate synthetic German sequence.

Revised: The ﬁrst model is based on Edunov et al.

Revised: (2018) and the second model (Indurthi et al., 2019) is trained on

Revised: WMT’18 En-De and OpenSubtitles datsets.

Revised: We increase the size of the IWSLT 19(ﬁltered) ST dataset to ﬁve times of the original size by augmenting 4x data – four text sequences using the top two beam results from each EN-De NMT model and four speech signals using the Sox parameter ranges.

Revised: For the Europarl-ST, we augment 2x examples to triple the size.

Revised: The TED-LIUM 3 dataset does not contain speech-to-text translation examples originally; hence, we create 2x synthetic speech-to-text translations using speech-to-text transcripts.

Revised: Finally, for the MuST-C dataset, we only create synthetic speech and pair it with the original translation to increase the dataset size to 4x.

Revised: The Overall, we created the synthetic training data of size approximately equal to four times of the original data for the ST task.

Revised: 6.1. T ASK I DENTIFICATION WITHOUT T ASK I NFORMATION

Revised: Under the multi-modality setting, we conducted smaller scale experiments using only one dataset for each ST, ASR, and ST tasks.

Revised: The details of the datasets used have been provided in Table 7.

Revised: We trained on single p40 GPU for 400k steps.

Revised: The corresponding results have been reported in Table 6.

Revised: All the results have been obtained without any ﬁnetuning.

Revised: Even though our task-aware MTL model achieves signiﬁcant performance improvement over vanilla MTL models, we can observe that the vanilla MTL models are also able to give a decent performance on all tasks without any ﬁnetuning.

Revised: An explanation for this is that we used MuST-C dataset for the En-De ST task and TEDLium v3 for the ASR task, which means that the source speech is coming from 2 different sources.

Revised: However, if we use the same datasets for both the tasks(after data augmentation), the MTL models get confused and the ST, ASR outputs are mixed.

Revised: The MTL models might be able to learn the task identities simply based on the source speech sequences, since these sequence are coming from different datasets for each task type–MuST-C for ST and TED-LIUM v3 for ASR.

Revised: However, this does not mean that vanilla

Revised: MTL models perform joint learning effectively.

Revised: A human who can perform multiple tasks from the same input is aware of the task he has to perform beforehand.

Revised: Similarly, it is unreasonable to expect different outputs (translation, transcription) from a model to the same type of input (English speech) without any explicit task information.

Revised: 6.1. I MPLEMENTATION D ETAILS

Revised: The detailed hyperparameters settings used for the single modality and multi modality experiments have been provided in the Table 8.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]