Original: EM-N ETWORK : L EARNING B ETTER L ATENT V ARIABLE FOR S EQUENCE - TO -S EQUENCE M ODELS
Revised: EM-N ETWORK : L EARNING B ETTER L ATENT V ARIABLE FOR S EQUENCE - TO -S EQUENCE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In a sequence-to-sequence (seq2seq) framework, the use of an unobserved latent variable, such as latent alignment and representation, is important to address the mismatch problem between the source input and target output sequences.
Revised: In a sequence-to-sequence (seq2seq) framework, the use of an unobserved latent variable, such as latent alignment and representation, is important to address the mismatch problem between the source input and target output sequences.

Original: Existing seq2seq literature typically learns the latent space by only consuming the source input, which might produce a sub-optimal latent variable for predicting the target.
Revised: Existing seq2seq literature typically learns the latent space by only consuming the source input, which might produce a sub-optimal latent variable for predicting the target.

Original: Extending an expectation-maximization (EM)-like algorithm, we introduce

Original:  EM-Network that can yield the promising latent variable by leveraging the target sequence as the model’s additional training input.
Revised: In this paper, we introduce EM-Network that can yield the promising latent variable by leveraging the target sequence as the model’s additional training input.

Original: The target input is used as guidance to provide the target-side context and reduce the candidates of the latent variable.
Revised: The target input is used as guidance to provide the target-side context and reduce the candidates of the latent variable.

Original: The proposed framework is trained in a new self-distillation setup, allowing the original sequence model to benefit from the latent variable of the EM-Network.
Revised: The proposed framework is trained in a new self-distillation setup, allowing the original sequence model to benefit from the latent variable of the EM-Network.

Original: Specifically, the EM-Network’s prediction serves as a soft label for training the inner sequence model, which only takes the source as input.
Revised: Specifically, the EM-Network’s prediction serves as a soft label for training the inner sequence model, which only takes the source as input.

Original: We theoretically show that our training objective can be a lower bound for the log-likelihood of the sequence model and is justified from the EM perspective.

Original: We conduct comprehensive experiments on two sequence learning tasks: speech recognition and  machine translation.
Revised: We conduct comprehensive experiments on two types of seq2seq models: connectionist temporal classification (CTC) for speech recognition and attention-based encoder-decoder (AED) for machine translation.

Original: Experimental results demonstrate that the EM-Network significantly advances the current state-of-the-art self-supervised learning approaches.
Revised: Experimental results demonstrate that the EM-Network significantly advances the current stateof-the-art approaches.

Original: It improves over the best prior work on speech recognition and establishes state-ofthe-art performance on WMT’14 and IWSLT’14 datasets.
Revised: It improves over the best prior work on speech recognition and establishes state-of-the-art performance on WMT’14 and IWSLT’14 datasets.

Original: Moreover, the proposed method even achieves considerable performance improvement for fully supervised learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Throughout the literature on deep learning, sequence-to-sequence (seq2seq) learning has achieved great success in a wide range of applications, especially in speech and natural language processing.
Revised: Throughout the literature on deep learning, sequence-to-sequence (seq2seq) learning has achieved great success in a wide range of applications, especially in speech and natural language processing.

Original: Given a source-target pair ( x , y ), a task of the seq2seq learning is to learn a function for mapping a source sequence x to a target sequence y , which generally suffers from source-target mismatch problems, e.g., unequal length, different domain, and modality mismatch.
Revised: Given a source-target pair ( x , y ), a task of the seq2seq learning is to learn a function for mapping a source sequence x to a target sequence y , which generally suffers from source-target mismatch problems, e.g., unequal length, different domain, and modality mismatch.

Original: To deal with this issue, learning the latent variable z and how to improve its quality are deemed critically important in sequence modeling.
Revised: To deal with this issue, learning the latent variable z and how to improve its quality are deemed critically important in sequence modeling.

Original: For example, in automatic speech recognition (ASR), the connectionist temporal classification (CTC) (Graves et al., 2006) model defines the latent alignment z to learn the mapping between the speech feature x
Revised: For example, in automatic speech recognition (ASR), the connectionist temporal classification (CTC) (Graves et al., 2006) model defines the latent alignment z to learn the mapping between the speech feature x

Original: and the word sequence y , as shown in Figure 1.
Revised: and the word sequence y , as shown in Figure 1.

Original: For the counterpart speech synthesis task, some studies (Kim et al., 2020; 2021) have proposed learning an internal alignment z between the text x and the corresponding speech y .
Revised: For the counterpart speech synthesis task, some studies (Kim et al., 2020; 2021) have proposed learning an internal alignment z between the text x and the corresponding speech y .

Original: In natural language processing, BERT family models (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) employ masked language modeling (MLM) to learn the contextualized representations z , where the randomly masked tokens are predicted given the context of other tokens.
Revised: In natural language processing (NLP), BERT family models (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) employ masked language modeling (MLM) to learn the contextualized representations z , where the randomly masked tokens are predicted given the context of other tokens.

Original: From the perspective of z , the self-supervised learning (SSL)-based pre-training enables the model to obtain the robust latent representation z from the input x , which offers the desired performance for predicting the target y on the downstream task. 
Revised: From the perspective of z , the self-supervised learning (SSL)-based pre-training enables the model to obtain the robust latent representation z from the input x , which offers the desired performance for predicting the target y on the downstream task. However, it is difficult to learn the optimal latent variable for the learning task.

Revised: For example, in the case of ASR, CTC models often converge to sub-optimal alignment distributions and produce over-confident predictions (Liu et al., 2018; Yu et al., 2021; Miao et al., 2015).

Original: Despite the promise, their shortcoming is that the latent variable captures only the source-side information, which might yield a sub-optimal solution for predicting the target.
Revised: Since there is an exponential number of possible alignment paths, and the alignment information between source and target sequences is rarely available during training, settling on the optimal alignment is quite challenging. From the feature perspective, powerful representation is important to achieve the desired performance. For the recent NLP studies, including machine translation (MT), a large pre-trained LM is highly required to obtain the contextualized representation (Zhu et al.,

Original:  In this paper, we propose  EM-Network that can effectively estimate thepromising latent variable z for seq2seq learning.
Revised: ; Wu et al. Xu et al., 2021). In this paper, we propose a novel framework termed EM-Network that can effectively improve the quality of the latent variable z and thus the overall quality of the seq2seq model.

Original: In particular, EM-Network encapsulates two key components.
Revised: In particular, EM-Network encapsulates two key components.

Original: First, the proposed framework leverages the target sequence as the model’s additional training input, where the target input y is used as guidance to capture the target-side context and reduce the candidates of z .
Revised: First, the proposed framework leverages the target sequence as the model’s additional training input, where the target input y is used as guidance to capture the target-side context and reduce the candidates of z .

Original: Second, based on the usage of the target input, we present a new self-distillation strategy as collaborative training, where the original sequence model can benefit from the EM-Network’slatent variable knowledge in a one-stage manner.
Revised: Second, based on the usage of the target input, we present a new self-distillation strategy as collaborative training, where the original sequence model can benefit from the EM-Network’s knowledge in a one-stage manner.

Original: The prediction of the EM-Network serves as a soft label for training the inner sequence model, which consumes only the source input.
Revised: The prediction of the EM-Network serves as a soft label for training the inner sequence model, which consumes only the source input.

Original: It is important to note that the parameters of  the original sequence model are included in the EM-Network, and only the sequence model is used to generate the prediction during the inference. We theoretically show that the proposed  trainingscheme is justified from the expectation-maximization (EM)-like algorithm perspective, and the objective function is a lower bound to  the log-likelihood of the supervised sequence model.
Revised: The proposed self-distillation acts as a sort of regularization for the seq2seq model, and its performance gain comes from a deep mutual learning scheme (Zhang et al., 2018), where the students learn collaboratively and teach each other. However, the main difference from the previous mutual learning approaches (Zhang et al., 2018; Liang et al., 2021) is that the proposed method utilizes the target input when training the EM-Network instead of merely considering the ground truth as the sole target in training. Since the target input is used as guidance to provide the target-side information, the prediction of the EM-Network (teacher mode) is more accurate than that of the inner sequence model (student mode).

Original: Although the proposed method could find applications in many fields as a general approach to seq2seq learning, here our paper focuses on CTC and cross entropy (CE) objective functions, which are widely used in the seq2seq field.
Revised: Therefore, the sequence model can effectively benefit from the soft labels of the EM-Network, which will be additionally discussed in Section 6. In addition, we attempt to apply the proposed self-distillation to the CTC framework, an unexplored area in mutual learning research.

Original: As shown in Figure 1, we develop two types of EM-Network approaches:
Revised: Modeling the conditional probability of the EM-Network (teacher mode) is determined by whether the latent variable is explicitly defined, as shown in Figure 1.

Original: CTC-based EM-Network and CE-based EM-Network. • CTC-based EM-Network  aims at predicting a better CTC alignment z by leveraging the source and target inputs, which can be formulated as P ( z | x, y ) .
Revised: The CTC computation adopts the alignment z , and it is difficult to settle on the optimal alignment with the conventional framework. The proposed EM-Network computes the posterior P ( z | x, y ) for the loss, which aims at predicting a better CTC alignment z by leveraging the source and target inputs.

Original: We then treat its prediction as the soft labels for training the CTC-based sequence model.
Revised: Therefore, the CTC model distilled from the EM-Network does not have to consider the exponential number of possible CTC alignments.

Original: Since the relationship between z and y is many-to-one, EM-Network can use the target information y as the model’s input while preventing a trivial solution, e.g., directly copying the target input to the output.
Revised: We theoretically show that the proposed objective function can serves as the proposed Q-function and is justified from the EM-like algorithm perspective.

Original: • In CE-based EM-Network, the  explicit latent alignment is not defined, and thus simply taking the target y as the additional input may cause an obvious but trivial solution, where the model converges with the conditional probability P ( y | x, y )
Revised: For the attention-based encoder-decoder (AED), where there is no explicit latent alignment, it is challenging to directly apply the same training scheme as the EM-Network for CTC. Simply taking the target y as the additional input may cause an obvious but trivial solution, where the model converges with the conditional probability P ( y | x, y )

Original: =
Revised: =

Original: δ ( y ) .
Revised: δ ( y ) .

Original: Inspired by the MLM, we present an alternative that employs the masked version of target ˜
Revised: Inspired by the MLM, we present an alternative that employs the masked version of target ˜

Original: y as the additional input instead of using the whole target y .
Revised: y as the additional input instead of using the whole target y .

Original: Different from the CTC-based EM-Network that focuses on improving the latent CTC alignment, the CE-based EM-Network provides more robust contextualized representations that can benefit the learning task.
Revised: The EM-Network for AED computes the posterior P (˜ y | x, y ) for loss and provides more robust contextualized representations that can benefit the learning task.

Original: We conduct comprehensive experiments on multiple benchmarks, including ASR and machine translation (MT) tasks.
Revised: We conduct comprehensive experiments on multiple benchmarks, including ASR and MT tasks.

Original: Since the current state-of-the-art (SOTA) approaches are based on self-supervised learning (SSL), the EM-Network is applied to the supervised fine-tuning stage of the SSL-based models  for ASR and MT tasks.
Revised: The CTC and AED models are considered for ASR and MT tasks, respectively.

Original: Experimental results demonstrate that the EM-Network improves over the best prior work on ASR and establishes SOTA performance on WMT’14 and IWSLT’ datasets.
Revised: Experimental results demonstrate that the EM-Network improves over the best prior work on ASR and establishes SOTA performance on WMT’14 and IWSLT’14 datasets.

Original: Also, the proposed method can significantly improve the performance of the sequence model with the fully supervised learning-based model.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we introduce how to design the EM-Network for learning better latent variable z
Revised: In this section, we introduce how to design the EM-Network for learning better latent variable z

Original: (Section 2.1 and Section 2.2), and its applications with CTC (Section 2.3) and CE objective functions (Section 2.4).
Revised: (Section 2.1 and Section 2.2), and its applications with CTC (Section 2.3) and AED (Section 2.4).

Original: We further provide a theoretical interpretation of the proposed framework from an EM-like algorithm perspective in Section 3.
Revised: We further provide a theoretical interpretation of the proposed framework from an EM-like algorithm perspective in Section 3.

Original: 2. T HE A RCHITECTURE OF EM-N ETWORK
Revised: 2. T HE A RCHITECTURE OF EM-N ETWORK

Original: EM-Network.
Revised: EM-Network.

Original: As shown in Figure 2, EM-Network mainly consists of two parts: (1) a sequence model that performs the original learning task given the source input x , and (2) an auxiliary network to learn a meaningful representation of both source x and target y .
Revised: As shown in Figure 2, EM-Network mainly consists of two parts: (1) a sequence model that performs the original learning task given the source input x , and (2) an auxiliary network to learn a meaningful representation of both source x and target y .

Original: We add the auxiliary network on top of the conventional sequence model.
Revised: We add the auxiliary network on top of the conventional sequence model.

Original: For the convenience of notation, we let ϕ , θ , and ρ denote the parameters of the EM-Network, the sequence model, and the auxiliary network, respectively.
Revised: For the convenience of notation, we let ϕ , θ , and ρ denote the parameters of the EM-Network, the sequence model, and the auxiliary network, respectively.

Original: Note that the EM-Network includes the parameters of the sequence model, where θ ⊂ ϕ .
Revised: Note that the EM-Network includes the parameters of the sequence model, where θ ⊂ ϕ .

Original: Sequence Model.
Revised: Sequence Model.

Original: Since the sequence model in the EM-Network performs the same task as the conventional seq2seq models, we can flexibly select the architecture of the sequence model depending on the task demands.
Revised: Since the sequence model in the EM-Network performs the same task as the conventional seq2seq models, we can flexibly select the architecture of the sequence model depending on the task demands.

Original: In our ASR experiments, the conventional SSL-based architecture, consisting of 12 transformer (Vaswani et al., 2017) blocks, is applied as the sequence model.
Revised: For example, in our ASR experiments, the conventional CTC model consisting of 12 transformer (Vaswani et al., 2017) blocks, is applied as the sequence model.

Original: For the MT task, the architecture of the sequence model follows that of the BiBERT (Xu et al., 2021).
Revised: For the MT task, the architecture of the sequence model follows that of the BiBERT (Xu et al., 2021).

Original: Auxiliary Network.
Revised: Auxiliary Network.

Original: The auxiliary network performs a fusion of the source x and target y representations when generating the prediction, and its architecture is based on the self and cross-attention modules.
Revised: The auxiliary network performs a fusion of the source x and target y representations when generating the prediction, and its architecture is based on the self and cross-attention modules.

Original: Specifically, the cross-attention takes the representation of x as the query and the representation of y as the key-value pairs.
Revised: Specifically, the cross-attention takes the representation of x as the query and the representation of y as the key-value pairs.

Original: 2. T RAINING AND I NFERENCE P ROCEDURES
Revised: 2. T RAINING AND I NFERENCE P ROCEDURES

Original: Self-distillation.
Revised: Self-distillation.

Original: EM-Network first yields the predictions, which serve as soft labels in distillation, by encoding both the source x and target y inputs (teacher mode).
Revised: EM-Network first yields the predictions, which serve as soft labels in distillation, by encoding both the source x and target y inputs (teacher mode).

Original: Then, the sequence model learns the parameter θ to predict the soft labels given the source input x (student mode).
Revised: Then, the sequence model learns the parameter θ to predict the soft labels given the source input x (student mode).

Original: Training.
Revised: Training.

Original: In the proposed framework, there are three kinds of losses during the training phase: (1) the original sequence learning loss L org ( θ ) , such as CTC  and CE, with the source input x to train the sequence model, (2) the proposed sequence learning loss L em ( ϕ ) given both source and target inputs for training the EM-Network, and (3) self-distillation loss L kd ( ϕ, θ ) that trains the sequence model to predict the soft labels generated from the EM-Network.
Revised: In the proposed framework, there are three kinds of losses during the training phase: (1) the original sequence learning loss L org ( θ ) , such as CTC loss and frame-wise cross entropy (CE), with the source input x to train the sequence model, (2) the proposed sequence learning loss L em ( ϕ ) given both source and target inputs for training the EM-Network, and (3) self-distillation loss L kd ( ϕ, θ ) that trains the sequence model to predict the soft labels generated from the EMNetwork.

Original: We optimize the following loss to train the proposed framework:
Revised: We optimize the following loss to train the proposed framework:

Original: [Equation]
Revised: [Equation]

Original: where α is a tunable parameter.
Revised: where α is a tunable parameter.

Original: In our self-distillation manner, the soft labels from the EM-Network are updated in each iteration since ϕ is affected by the two losses L em ( ϕ ) and L kd ( ϕ, θ ) .
Revised: In our self-distillation manner, the soft labels from the EM-Network are updated in each iteration since ϕ is affected by the two losses L em ( ϕ ) and L kd ( ϕ, θ ) .

Original: Inference.
Revised: Inference.

Original: During the inference, only the sequence model is used to generate the prediction .
Revised: During the inference, only the sequence model is used to generate the prediction .

Original: Since the auxiliary network is removed during the inference, the additional computational load of ρ is only required for the training procedure.
Revised: Since the auxiliary network is removed during the inference, the additional computational load of ρ is only required for the training procedure.

Original: In our SSL-based ASR experiments, about 10 M parameters are used as the auxiliary parameters to train the EM-Network, where the number of parameters of θ is about 93 M. For the fully-supervised ASR setting, the number of parameters of θ is about 13 M, and only 1 M parameters are required for the auxiliary network.
Revised: In our SSL-based ASR experiments, about 10 M parameters are used as the auxiliary parameters to train the EM-Network, where the number of parameters of θ is about 93 M. For the fully-supervised ASR setting, the number of parameters of θ is about 13 M, and only 1 M parameters are required for the auxiliary network.

Original: As aforementioned in Section 1, EM-Network can cover two popular loss functions for sequence learning: CTC and CE objective.
Revised: As aforementioned in Section 1, EM-Network can cover two popular loss functions for sequence learning: CTC and CE objective.

Original: We begin in the CTC objective setting (Secion 2.3) and then turn to the CE-based EM-Network (Section 2.4).
Revised: We begin in the CTC objective setting (Secion 2.3) and then turn to the CE-based EM-Network (Section 2.4).

Original: 2. EM-N ETWORK WITH C ONNECTIONIST T EMPORAL C LASSIFICATION (CTC)
Revised: 2. EM-N ETWORK FOR CTC

Original: Standard CTC Objective.
Revised: Standard CTC Objective.

Original: Before describing the proposed method in the CTC framework (Graves et al., 2006), it might be beneficial to briefly discuss the CTC objective function.
Revised: Before describing the proposed method in the CTC framework (Graves et al., 2006), it might be beneficial to briefly discuss the CTC objective function.

Original: Given an input sequence x , the core idea of CTC is leveraging intermediate alignment z by allowing label repetitions possibly interleaved with a special blank token ( ϵ ).
Revised: Given an input sequence x , the core idea of CTC is leveraging intermediate alignment z by allowing label repetitions possibly interleaved with a special blank token ( ϵ ).

Original: CTC trains the sequence model θ to minimize the following loss function:
Revised: CTC trains the sequence model θ to minimize the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where B is a many-to-one mapping in the CTC algorithm with y
Revised: where B is a many-to-one mapping in the CTC algorithm with y

Original: = B ( z ) .
Revised: = B ( z ) .

Original: B − 1 ( y ) is the set of possible alignments z compatible to y .
Revised: B − 1 ( y ) is the set of possible alignments z compatible to y .

Original: The mapping is done by merging repeated labels from the paths and then removing all blank tokens ( ϵ ), e.g., B ( { ϵ ,a,a, ϵ , ϵ ,a,b,b } ) = { aab } ) where { aab } denotes y and the sequence { ϵ ,a,a, ϵ , ϵ ,a,b,b } represents z .
Revised: The mapping is done by merging repeated labels from the paths and then removing all blank tokens ( ϵ ), e.g., B ( { ϵ ,a,a, ϵ , ϵ ,a,b,b } ) = { aab } ) where { aab } denotes y and the sequence { ϵ ,a,a, ϵ , ϵ ,a,b,b } represents z .

Original: Modeling Conditional Probability with Target Input.
Revised: Modeling Conditional Probability with Target Input.

Original: As mentioned earlier, the crucial challenge is how to use the target input while preventing the trivial solution.
Revised: As mentioned earlier, the crucial challenge is how to use the target input while preventing the trivial solution.

Original: Since the CTC algorithm explicitly adopts the latent alignment z , we can easily apply the target y as the EM-Network’s input.
Revised: Since the CTC algorithm explicitly adopts the latent alignment z , we can easily apply the target y as the EM-Network’s input.

Original: The relationship between z and y is many-to-one, which can be formulated as B ( z )
Revised: The relationship between z and y is many-to-one, which can be formulated as B ( z )

Original: =
Revised: =

Original: y .
Revised: y .

Original: Intuitively, it is challenging to predict z (many) given the target y
Revised: Intuitively, it is challenging to predict z (many) given the target y

Original: (one).
Revised: (one).

Original: Therefore, the training objective function with the target input can be given as
Revised: Therefore, the training objective function with the target input can be given as

Original: [Equation]
Revised: [Equation]

Original: The target input is used as guidance to provide the target-side information and reduce the candidates of the latent variable.
Revised: The target input is used as guidance to provide the target-side information and reduce the candidates of the latent variable.

Original: Distillation Loss.
Revised: Distillation Loss.

Original: EM-Network is trained in a self-distillation setup where the prediction given source and target inputs serves as a soft label for training the original sequence model.
Revised: EM-Network is trained in a self-distillation setup where the prediction given source and target inputs serves as a soft label for training the original sequence model.

Original: The distillation loss in the proposed method is as follows:
Revised: The distillation loss in the proposed method is as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˆ z em and ˆ z org are the predicted latent alignments of the EM-Network and the sequence model, respectively. 
Revised: where ˆ z em and ˆ z org are frame-wise softmax outputs of the EM-Network and the sequence model, respectively. Note that the argmax value of ˆ z is the predicted CTC alignment.

Original: Since the conventional distillation loss using CE or Kullback-Leibler (KL)-divergence generally fails to converge in the CTC framework, as reported in previous studies (Senior et al., 2015; Takashima et al., 2018; 2019; Yoon et al., 2021b), we follow the distillation loss of Yoon et al.
Revised: Since the conventional distillation loss using CE or Kullback-Leibler (KL)-divergence generally fails to converge in the CTC framework, as reported in previous studies (Senior et al., 2015; Takashima et al., 2018; 2019; Yoon et al., 2021b), we follow the distillation loss of Yoon et al.

Original: (2021b) that adopts l 2 loss function for transferring the latent alignment.
Revised: (2021b) that adopts l 2 loss function for transferring the latent alignment.

Original: 2. EM-N ETWORK WITH C ROSS E NTROPY (CE)
Revised: 2. EM-N ETWORK FOR AED

Original: Standard CE Objective.
Revised: Standard CE Objective.

Original: Different from the CTC objective function, the CE algorithm does not explicitly define the latent alignment z and the many-to-one mapping B .
Revised: Different from the CTC, the AED model does not explicitly define the latent alignment z and the many-to-one mapping B .

Original: The CE loss for sequence learning can be formulated as
Revised: The CE loss for AED can be formulated as

Original: [Equation]
Revised: [Equation]

Original: Since simply taking the target y as the CE-based model’s input may cause an obvious but trivial solution where the model converges with the conditional probability P ( y | x, y ; θ ) = δ ( y ) , it is difficult to learn a meaningful representation of the target input y .
Revised: Since simply taking the target y as the AED model’s input may cause an obvious but trivial solution where the model converges with the conditional probability P ( y | x, y ; θ ) = δ ( y ) , it is difficult to learn a meaningful representation of the target input y .

Original: Modeling Conditional Probability with Target Input.
Revised: Modeling Conditional Probability with Target Input.

Original: To address this issue, we employ a masking strategy in the MLM task.
Revised: To address this issue, we employ a masking strategy in the MLM task.

Original: The masked version of the target ˜
Revised: The masked version of the target ˜

Original: y is applied as the additional input instead of directly using the target y .
Revised: y is applied as the additional input instead of directly using the target y .

Original: Given the target sequence y , we use a masking function M , which randomly masks the tokens of y with the probability of λ .
Revised: Given the target sequence y , we use a masking function M , which randomly masks the tokens of y with the probability of λ .

Original: Then, the masked target
Revised: Then, the masked target

Original: M ( y, λ ) =
Revised: M ( y, λ ) =

Original: ˜
Revised: ˜

Original: y is fed to the EM-Network as the auxiliary input.
Revised: y is fed to the EM-Network as the auxiliary input.

Original: The training objective function with the masked target ˜
Revised: The training objective function with the masked target ˜

Original: y can be calculated as follows:
Revised: y can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: Distillation Loss.
Revised: Distillation Loss.

Original: In the case of the CE objective function, we utilize the conventional distillation loss, where the KL divergence is computed using softmax outputs between the student and the teacher.
Revised: In the case of the CE objective function, we utilize the conventional distillation loss, where the KL divergence is computed using softmax outputs between the student and the teacher.

Original: The distillation loss for CE-based EM-Network is given by
Revised: The distillation loss for AED-based EM-Network is given by

Original: [Equation]
Revised: [Equation]

Original: where ˆ y em , ˆ y org denote the softmax outputs of the EM-Network and the sequence model.
Revised: where ˆ y em , ˆ y org denote the softmax outputs of the EM-Network and the sequence model.

Original: D KL represents the KL-divergence.
Revised: D KL represents the KL-divergence.

Original:  W HY EM-N ETWORK IMPROVES SEQUENCE LEARNING ?
Revised:  C ONNECTION TO EM A LGORITHM

Original: In this section, we give theoretical justification on why the proposed framework helps the training of the sequence model.
Revised: In this section, we explore the connection with the EM algorithm.

Original: We begin in the CTC objective setting, and the theoretical analysis regarding the CE objective function is available in Appendix B.2.
Revised: The detailed derivations are provided in Appendix B.

Original: Standard EM Algorithm.
Revised: Standard EM Algorithm.

Original: The traditional EM algorithm finds maximum likelihood parameters of the model that depends on unobserved latent variables z .
Revised: The traditional EM algorithm finds maximum likelihood parameters of the model that depends on unobserved latent variables z .

Original: It iteratively performs an expectation (E) step and a Maximization step (M step).
Revised: It iteratively performs an expectation (E) step and a Maximization step (M step).

Original: The E step defines a Q-function Q ( θ | θ ( t ) ) as the expected value of the log-likelihood of θ , which can be formulated as follows:
Revised: The E step defines a Q-function Q ( θ | θ ( t ) ) as the expected value of the log-likelihood of θ , which can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, the M step computes the parameters that maximize the expected log-likelihood found on the E step.
Revised: Then, the M step computes the parameters that maximize the Q-function found on the E step.

Original: EM- like Perspective.
Revised: EM-like Perspective for CTC-based EM-Network

Original: The proposed framework is motivated by the EM algorithm.
Revised: The proposed framework is motivated by the EM algorithm.

Original: In the case of the EM-Network, the Q-function is calculated as follows:
Revised: In the case of the EM-Network for CTC, the Q-function is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where P ( z | x, y ; θ ( t ) , ρ ( t ) ) represents the conditional probability of the CTC-based EM-Network in
Revised: where P ( z | x, y ; θ ( t ) , ρ ( t ) ) represents the conditional probability of the CTC-based EM-Network in

Original: Eq.
Revised: Eq.

Original: (3), and log P ( z | x ; θ ) denotes the CTC-based sequence model in Eq.
Revised: (3), and log P ( z | x ; θ ) denotes the CTC-based sequence model in Eq.

Original: Here, we ignore the constant factor − H ( P ( z | x, y ; θ ( t ) , ρ ( t ) )) in the fourth equality and the first approximation.
Revised: Here, we ignore the constant factor − H ( P ( z | x, y ; θ ( t ) , ρ ( t ) )) in the fourth equality and the first approximation.

Original: As shown in Eq.
Revised: As shown in Eq.

Original: (8), a negative value of KL-divergence between the EM-Network and the sequence model serves as the Q-function of the EM-Network.
Revised: (8), a negative value of KL-divergence between the EM-Network and the sequence model serves as the Q-function of the EM-Network.

Original: However, as aforementioned in Section 2.3,
Revised: However, as aforementioned in Section 2.3,

Original: CTC models often fail to converge with the distillation loss using the KL-divergence  (Senior et al., 2015; Takashima et al., 2018; 2019; Yoon et al., 2021b).
Revised: CTC model often fails to converge with the distillation loss using the KL-divergence due to its alignment-free property (Senior et al., 2015; Takashima et al., 2018; 2019; Yoon et al., 2021b).

Revised: To perform the distillation in the CTC-framework, the alternative is required instead of KL-divergence.

Original: To sidestep this convergence problem, we use l 2 loss instead of the KL-divergence, corresponding to the distillation loss L kd − ctc .
Revised: To sidestep this convergence problem, we use l 2 loss instead of the KL-divergence, corresponding to the distillation loss L kd − ctc .

Original: Eq.
Revised: Eq.

Original: (9) shows that L kd − ctc can be considered as the approximation of the EM-Network’s Q-function.
Revised: (9) shows that L kd − ctc can be considered as the approximation of the EM-Network’s Q-function.

Original: Note that B ( z ) =
Revised: Note that B ( z ) =

Original: y and ϕ
Revised: y and ϕ

Original: = θ + ρ in the CTC-based EM-Network.
Revised: = θ + ρ in the CTC-based EM-Network.

Original: Based on the above perspective, we can actually show our proposed training objective in Eq.
Revised: Based on the above perspective, we can actually show our proposed training objective in Eq.

Original: (4) is a lower bound for the log-likelihood of the original sequence model.
Revised: (4) is a lower bound for the log-likelihood of the original sequence model.

Original: Our goal is to maximize the log-likelihood below,
Revised: Our goal is to maximize the log-likelihood below,

Original: [Equation]
Revised: [Equation]

Original: where the first inequality follows from Jensen’s inequality.
Revised: where the first inequality follows from Jensen’s inequality.

Original: We provide the detailed derivation in

Original: Appendix B.1.

Original: From Eq.
Revised: From Eq.

Original: (10), we can confirm that a negative value of L kd − ctc serves as the lower bound for the log-likelihood of the original CTC-based sequence model log P ( y | x ; θ ) .
Revised: (10), we can confirm that a negative value of L kd − ctc serves as the lower bound for the log-likelihood of the original CTC-based sequence model log P ( y | x ; θ ) .

Original: Therefore, both Eq.
Revised: Therefore, both Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10) indicate that maximizing the EM-Network’s Q-function is equivalent to maximizing the lower bound of the sequence model’s likelihood.
Revised: (10) indicate that maximizing the EM-Network’s Q-function is equivalent to maximizing the lower bound of the sequence model’s likelihood.

Revised: The tight lower bound will be additionally discussed in Appendix E.

Revised: Maximizing the lower bound (minimizing L kd − ctc ) is also closely related to the upper bound for the sequence model’s log-likelihood.

Revised: The conventional KD generally assumes that the teacher’s performance determines the upper bound of the student (Zhang et al., 2020;

Revised: Mishra & Marr, 2018; Clark et al., 2019).

Original: In Section 5, we empirically show that maximizing the lower bound  in Eq.
Revised: Under the assumption, the log-likelihood of the EM-Network (teacher mode) can be regarded as the upper bound for the log-likelihood of the sequence model. Considering that the EMNetwork uses the ground truth as the additional input to perform the learning task, this assumption seems more reasonable. Different from the previous offline KD methods where the teacher model is generally fixed, we update the EM-Network and sequence model simultaneously. The training loss curves are given in Figure 4.

Revised: The proposed distillation loss L kd − ctc regularizes the EM-Network (teacher mode) to learn the knowledge of the sequence model (student mode), providing the tight upper bound for the sequence model.

Revised: For the counterpart, minimizing L kd − ctc also corresponds to making the sequence model mimic the behavior of the EM-Network, indicating that the loglikelihood of the sequence model is close to its upper bound.

Revised: Therefore, we can derive the tight upper bound by maximizing the lower bound.

Revised: On top of that, the upper bound can be maximized by minimizing the loss L em − ctc in Eq.

Original: (10) leads to better performance of the sequence model. 
Revised: (3), indicating that the proposed objective function partially maximizes the likelihood of the sequence model. EM-like Perspective for AED-based EM-Network When considering the AED, the Q-function can be computed with a similar perspective in Eq. (8), which is given by

Revised: [Equation]

Revised: Then, we start from the log-likelihood of the AED model, which is formulated in Eq.

Original: E XPERIMENTAL S ETUP
Revised: [Equation] Eq. (12) indicates that the KL-divergence between the EM-Network ( P ( y | x, ˜ y ; θ, phi ) ) and the AED model ( P ( y | x ; θ )) )

Revised: accords with the distillation loss in Eq.

Revised: By Eq.

Revised: (11), we know that a negative value of the KL-divergence between the EM-Network and the sequence model serves as the Qfunction of the EM-Network.

Revised: Therefore, maximizing the AED-based EM-Network’s Q-function is equivalent to maximizing the lower bound of the sequence model’s likelihood.

Revised: As mentioned above, the log-likelihood of the EM-Network serves as the tight upper bound for the sequence model.

Revised:  E XPERIMENTAL S ETUP

Original: Speech Recognition.
Revised: Speech Recognition.

Original: Our experiments were conducted using the fairseq (Ott et al., 2019) toolkit.
Revised: Our experiments were conducted using the fairseq (Ott et al., 2019) toolkit.

Original: We utilized a widely used LibriSpeech (Panayotov et al., 2015) dataset, an English speech corpus derived from audiobooks and a standard benchmark in the speech community.
Revised: We utilized a widely used LibriSpeech (Panayotov et al., 2015) dataset.

Original: In the proposed framework, we first pre-trained the sequence model ( θ ) on the full 960 hours of LibriSpeech (LS960) by using the objective of the data2vec since it is the current SOTA model in the SSL literature.
Revised: In the proposed framework, we first pre-trained the sequence model ( θ ) on the full 960 hours of LibriSpeech (LS-960) by using the objective of the data2vec since it is the current SOTA model in the SSL literature.

Original: Then, the whole EM-Network ( ϕ ) was fine-tuned with LS-960, where the dev-other subset was used as the validation.
Revised: Then, the whole EM-Network ( ϕ ) was fine-tuned with LS-960, where the dev-other subset was used as the validation.

Original: The architecture of the sequence model followed the Base setup of the conventional
Revised: The architecture of the sequence model followed the Base setup of the conventional

Original: SSL-based model, consisting of 12 transformer blocks.
Revised: SSL-based model, consisting of 12 transformer blocks.

Original: The tunable parameter α was experimentally set to 2.
Revised: The tunable parameter α was experimentally set to 2.

Original: When applying the language model (LM), we used the official 4-gram KenLM (Heafield, 2011).
Revised: When applying the language model (LM), we used the official 4-gram KenLM (Heafield, 2011).

Original: The detailed implementation is additionally presented in Appendix C.
Revised: The detailed implementation is additionally presented in Appendix C.

Original: Machine Translation.
Revised: Machine Translation.

Original: We evaluated the EM-Network on IWSLT’14 and WMT’14 datasets for
Revised: We evaluated the EM-Network on IWSLT’14 and WMT’14 datasets for

Original: English-to-German (En-De) and German-to-English (De-En) translation tasks. 
Revised: English-to-German (En-De) and German-to-English (De-En) translation tasks. The implementation

Original: The implementation of the EM-Network was based on the official source code provided by the previous work (Xu et al., 2021).
Revised: [Table] [Table] of the EM-Network was based on the official source code provided by the previous work (Xu et al., 2021).

Original: Firstly, we pre-trained the sequence model with SSL, and then the whole EM-Network was fine-tuned on the MT task.
Revised: Firstly, we pre-trained the sequence model with SSL, and then the whole EM-Network was fine-tuned on the MT task.

Original: In the case of the pre-training, the sequence model was trained on the same English and German texts as BiBERT (Xu et al., 2021), derived from the OSCAR (Su´arez et al., 2020) corpus.
Revised: In the case of the pre-training, the sequence model was trained on the same English and German texts as BiBERT (Xu et al., 2021), derived from the OSCAR (Su ´ arez et al., 2020) corpus.

Original: The architecture sequence model was based on the BiBERT, a recent SOTA method on the MT task, and we followed its pre-training scheme.
Revised: The architecture sequence model was based on the BiBERT, a recent SOTA method on the MT task, and we followed its pre-training scheme.

Original: During the fine-tuning, the parameter α and the masking ratio λ were set to 2 and 50 %, respectively.
Revised: During the fine-tuning, the parameter α and the masking ratio λ were set to 2 and 50 %, respectively.

Original:  M AIN E MPIRICAL RESULTS
Revised:  M AIN E MPIRICAL RESULTS

Revised: Since the current SOTA models were typically based on the SSL, we used the pre-trained SSL model in our experiments.

Revised: Due to the space limit, the experimental results with the fully supervised learning-based models were described in Appendix D. From the results, we observed that the proposed method achieved considerable performance improvement for fully supervised learning.

Original: Speech Recognition.
Revised: Speech Recognition.

Original: We compared the EM-Network with previous SOTA works from the literature, including wav2vec 2.0 (Baevski et al., 2020), HuBERT (Hsu et al., 2021), ILS-SSL (Wang et al., 2022), and data2vec (Baevski et al., 2022).
Revised: We compared the EM-Network with previous SOTA works from the literature, including wav2vec 2.0 (Baevski et al., 2020), HuBERT (Hsu et al., 2021), ILS-SSL (Wang et al., 2022), and data2vec (Baevski et al., 2022).

Original: As shown in Table 1, EM-Network outperformed the recent approaches on the LibriSpeech test clean/other datasets.
Revised: As shown in Table 1, EM-Network outperformed the recent approaches on the LibriSpeech.

Original: It is important to note that our approach followed the pre-training scheme of the data2vec.

Original: The difference between the EM-Network and the data2vec relied on the fine-tuning stage.

Original: In the proposed framework, we utilized the auxiliary network ( ρ ) and the additional losses (Eq.

Original: (3) and Eq.

Original: (4)) to effectively use the target input [Table] [Table] for sequence learning, while the data2vec only used the standard CTC objective (Eq. (2))
Revised: It is important to note that our approach followed the pretraining scheme of the data2vec, and we applied the proposed scheme to the data2vec’s fine-tuning stage.

Original: given the source input during the fine-tuning.

Original: On the LibriSpeech, we improved upon the best data2vec by 0.12 %/0.

Revised: On the LibriSpeech, we improved upon the best data2vec by 0.12 %/0.

Original: % on the clean/other test datasets with greedy decoding, yielding a relative error rate reduction (RERR) of 4.32 [-%/4.
-]
Revised: % on the clean/other test datasets with greedy decoding, yielding a relative error rate reduction (RERR) of 4.32 [-%/4.
-]%/4.

Original: % compared to the data2vec baseline.
Revised: % compared to the data2vec baseline.

Original: Also, the proposed approach consistently achieved superior WER results when applying beam-search decoding with the LM.
Revised: Also, the proposed approach consistently achieved superior WER results when applying beam-search decoding with the LM.

Original: Specifically, compared to the beam-search decoding case, the performance improvement of the EM-Network was much more significant with the greedy decoding.
Revised: Specifically, compared to the beam-search decoding case, the performance improvement of the EM-Network was much more significant with the greedy decoding.

Original: Since the EM-Network considered linguistic (text) information by using the additional target input, the sequence model could produce strong performance even without the LM.
Revised: Since the EM-Network considered linguistic (text) information by using the additional target input, the sequence model could produce strong performance even without the LM.

Original: Due to the space limit, the experimental results with the fully supervised learning-based models were described in Appendix D. From the results, we observed that the proposed method achieved considerable performance improvement for fully supervised learning.

Original: Machine Translation.
Revised: Machine Translation.

Original: Table 2a shows a comparison of the proposed method with the recent approaches on IWSLT’14 En-De and De-En translations.
Revised: Table 2a shows a comparison of the proposed method with the recent approaches on IWSLT’14 En-De and De-En translations.

Original: We chose the eight best-performed MT algorithms to date: 1) Adversarial MLE (Wang et al., 2019), 2) DynamicConv (Wu et al., 2019) 3)
Revised: We chose the eight best-performed MT algorithms to date: 1) Adversarial MLE (Wang et al., 2019), 2) DynamicConv (Wu et al., 2019) 3)

Original: Macaron Net (Lu et al., 2020), 4) BERT-Fuse (Zhu et al., 2020), 5) multi-branch encoders (MAT) (Fan et al., 2020), 6) mixed representations from different tokenizers (Mixed Representation) (Wu et al., 2020), 7) uniting different dropout techniques (UniDrop) (Wu et al., 2021), and 8) BiBERT (Xu et al., 2021).
Revised: Macaron Net (Lu et al., 2020), 4) BERT-Fuse (Zhu et al., 2020), 5) multi-branch encoders (MAT) (Fan et al., 2020), 6) mixed representations from different tokenizers (Mixed Representation) (Wu et al., 2020), 7) uniting different dropout techniques (UniDrop) (Wu et al., 2021), and 8) BiBERT (Xu et al., 2021).

Original: From the results, it is verified that EM-Network outperformed all of them and yielded around 1.4 (En-De) and 0.9 (De-En) BLEU point gains over the previous SOTA results.
Revised: From the results, it is verified that EM-Network outperformed all of them and yielded around 1.4 (En-De) and 0.9 (De-En) BLEU point gains over the previous SOTA results.

Original: Recall that EM-Network used the same settings as BiBERT — the main difference was applying the auxiliary network with the additional input ˜ y and the proposed self-distillation framework during the fine-tuning stage — yet the EM-Network yielded a significant improvement over BiBERT.
Revised: Recall that EM-Network used the same settings as BiBERT — the main difference was applying the auxiliary network with the additional input ˜ y and the proposed self-distillation framework during the fine-tuning stage — yet the EM-Network yielded a significant improvement over BiBERT.

Original: For the high-resource scenario, we conducted experiments on WMT’14 En-De and De-En translations and compared the EM-Network with prior existing works that achieved high BLEU scores on
Revised: For the high-resource scenario, we conducted experiments on WMT’14 En-De and De-En translations and compared the EM-Network with prior existing works that achieved high BLEU scores on

Original: WMT’14 dataset, including large batch training (Ott et al., 2018),
Revised: WMT’14 dataset, including large batch training (Ott et al., 2018),

Original: Evolved Transformer (So et al., 2019), initializing the BERT by leveraging pre-trained checkpoints (Rothe et al., 2020), BERT-Fuse[Table] (Zhu et al., 2020), and BiBERT (Xu et al., 2021).
Revised: Evolved Transformer (So et al., 2019), initializing the BERT by leveraging pre-trained checkpoints (Rothe et al., 2020), BERT-Fuse (Zhu et al., 2020), and BiBERT (Xu et al., 2021).

Original: From Table 2b, we verified that our model also gave the best BLEU scores on the high-resource WMT’14 En-De and De-En translations.
Revised: From Table 2b, we verified that our model also gave the best BLEU scores on the high-resource WMT’14 En-De and De-En translations.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: Latent CTC Alignment.
Revised: CTC Alignment.

Original: As visualized in Figure 3, we contrasted the latent alignments of the best baseline data2vec
Revised: As visualized in Figure 3, we contrasted the frame-wise softmax outputs of the best baseline data2vec

Original: and our sequence model θ in the EM-Network.
Revised: and our sequence model θ in the EM-Network.

Revised: The argmax value of the frame-wise label probability corresponds to the predicted CTC alignment.

Original: In Figure 3, the conventional data2vec converted a given speech into “NOT ON SO ON BOARD THE JOHN WAS HART” and made erroneous predictions with “ON SO” and “HART”.
Revised: In Figure 3, the conventional data2vec converted a given speech into “NOT ON SO ON BOARD THE JOHN WAS HART” and made erroneous predictions with “ON SO” and “HART”.

Original: When considering only the acoustic feature (speech voice), it is challenging to distinguish “ON SO”/“A SOUL” and “HART”/“HURT”.
Revised: When considering only the acoustic feature (speech voice), it is challenging to distinguish “ON SO”/“A SOUL” and “HART”/“HURT”.

Original: However, the proposed sequence model provided a more accurate prediction than the data2vec.
Revised: However, the proposed sequence model provided a more accurate prediction than the data2vec.

Original: Different from the previous seq2seq approaches, the EM-Network performed a fusion of the acoustic (speech source input) and linguistic (text target input) representations.
Revised: Different from the previous seq2seq approaches, the EM-Network performed a fusion of the acoustic (speech source input) and linguistic (text target input) representations.

Original: Since the EM-Network could consider not only the acoustic information but also the linguistic one, the sequence model learned better alignment and obtained satisfactory performance.
Revised: Since the EM-Network could consider not only the acoustic information but also the linguistic one, the sequence model learned better alignment and obtained satisfactory performance.

Original: Even though there was a single erroneous word “HART”, we can discover the possible correct alignment regarding ”U” (green circle in Figure 3) in the prediction of the sequence model.
Revised: Even though there was a single erroneous word “HART”, we can discover the probability regarding ”U” (green circle in Figure 3) in the prediction of the sequence model.

Original: Also, unlike the data2vec that included many irrelevant alignments (red boxes in Figure 3), our sequence model had relatively fewer redundant ones, implying that the use of the target input effectively reduced the candidates of the latent CTC alignment z .
Revised: Also, unlike the data2vec that included many irrelevant label probabilities (red boxes in Figure 3), our sequence model had relatively fewer redundant ones, implying that the use of the target input effectively reduced the candidates of the latent CTC alignment z .

Original: From the results, it is verified that the proposed framework could learn the promising CTC alignment by using the target input and the self-distillation setup.
Revised: From the results, it is verified that the proposed framework could learn the promising CTC alignment by using the target input and the self-distillation setup.

Original: Translation Consistency.
Revised: Translation Consistency.

Original: The token repetition ratio represents the degree of the inconsistency problem in MT (Song et al., 2021;
Revised: The token repetition ratio represents the degree of the inconsistency problem in MT (Song et al., 2021;

Original: Ghazvininejad et al., 2020).
Revised: Ghazvininejad et al., 2020).

Original: We computed the token repetition ratio by dividing all consecutively repeating tokens by the total number of tokens.
Revised: We computed the token repetition ratio by dividing all consecutively repeating tokens by the total number of tokens.

Original: The teacher forcing and the greedy decoding were applied to fairly compare the predictions.
Revised: The teacher forcing and the greedy decoding were applied to fairly compare the predictions.

Original: From the results  in Table 3, we can confirm that the EM-Network effectively addressed the token repetition issue compared to the prior SOTA approach.
Revised: From the results [Table] in Table 3, we can confirm that the EM-Network effectively addressed the token repetition issue compared to the prior SOTA approach.

Original: In the case of the conventional BiBERT, it only considered the past target information during the training via the teacher forcing technique.
Revised: In the case of the conventional BiBERT, it only considered the past target information during the training via the teacher forcing technique.

Original: However, the EM- Network leveraged a more global target context by using themasked version of the target  ˜ y .
Revised: However, the EMNetwork leveraged a more global target context by using the target input ˜ y .

Original: Since our approach used both past and future target information in generating the prediction, it alleviated translation inconsistency and produced bettertranslation quality.
Revised: Since our approach used both past and future target information in generating the prediction, it alleviated translation inconsistency and produced better quality.

Original: In other words, the sequence model in the proposed framework could benefit from a more optimal and consistent representation of the EM-Network.
Revised: In other words, the sequence model in the proposed framework could benefit from a more optimal and consistent representation of the EM-Network.

Original: Training Loss Curves.
Revised: Training Loss Curves.

Original: Figure 4 shows the training loss curves of the EM-Network for the  task.
Revised: Figure 4 shows the training loss curves of the EM-Network for the ASR task.

Original: We observed that the loss L em of the whole EMNetwork ( ϕ ) consistently decreased, implying that soft labels from the EM-Network were updated in each iteration.
Revised: We observed that the loss L em of the whole EM-Network ( ϕ ) consistently decreased, implying that soft labels from the EM-Network were updated in each iteration.

Original: In this way, unlike the conventional distillation that the knowledge is generally fixed, the EM-Network transferred the knowledge more adaptively.
Revised: In this way, unlike the conventional distillation that the knowledge is generally fixed, the EM-Network transferred the knowledge more adaptively.

Original: Compared to the previous self-distillation methods (Pei et al., 2021; Kong et al., 2022; Baevski et al., 2022), the proposed distillation framework did not require the additional update strategy, such as a moving-average of parameters.
Revised: Compared to the previous self-distillation methods (Pei et al., 2021; Kong et al., 2022; Baevski et al., 2022), the proposed distillation framework did not require the additional update strategy, such as a moving-average of parameters.

Original: It could be implemented by simply applying the additional losses L em and L kd .
Revised: It could be implemented by simply applying the additional losses L em and L kd .

Original: Also, since L em was lower than L org during the training, the prediction of the EMNetwork (teacher mode) was more accurate than that of the inner sequence model (student mode).
Revised: Also, since L em was lower than L org during the training, the prediction of the EM-Network (teacher mode) was more accurate than that of the inner sequence model (student mode).

Original: Thus, the sequence model could effectively benefit from the soft labels of the EM-Network.
Revised: Thus, the sequence model could effectively benefit from the soft labels of the EM-Network.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Training with Target Input.
Revised: Training with Target Input.

Original: There are many techniques in the seq2seq learning literature that exploit the target-related information.
Revised: There are many techniques in the seq2seq learning literature that exploit the target-related information.

Original: Autoregressive approaches (Vaswani et al., 2017; Chorowski et al., 2015; Graves, 2012) have adopted a teacher forcing technique that supplies a ground truth value as conditional input during the training.
Revised: Autoregressive approaches (Vaswani et al., 2017; Chorowski et al., 2015; Graves, 2012) have adopted a teacher forcing technique that supplies a ground truth value as conditional input during the training.

Original: The teacher forcing allows the model to predict the current target by utilizing the context representation of the source input and the history of labels.
Revised: The teacher forcing allows the model to predict the current target by utilizing the context representation of the source input and the history of labels.

Original: To make good global planning, Feng et al.
Revised: To make good global planning, Feng et al.

Original: (2021) proposed the additional seer decoder into the encoder-decoder framework, which embodies future ground truth to guide the behaviors of the conventional decoder.
Revised: (2021) proposed the additional seer decoder into the encoder-decoder framework, which embodies future ground truth to guide the behaviors of the conventional decoder.

Original: In the case of the non-autoregressive framework, some studies (Higuchi et al., 2020; Chan et al., 2020) attempted to improve the CTC model’s prediction by conditioning on previously generated tokens.
Revised: In the case of the non-autoregressive framework, some studies (Higuchi et al., 2020; Chan et al., 2020) attempted to improve the CTC model’s prediction by conditioning on previously generated tokens.

Original: Also, Yoon et al.
Revised: Also, Yoon et al.

Original: (2021a) introduced the Oracle Teacher, which uses the target information to improve the teacher model’s performance in the offline distillation setup.
Revised: (2021a) introduced the Oracle Teacher, which uses the target information to improve the teacher model’s performance in the offline distillation setup.

Original: Our motivation was quite different from the conventional methods.

Original: Inspired by the EM algorithm that finds maximum likelihood parameters of the model that depends on latent variables, the proposed method aimed at learning the promising latent variable by leveraging the additional target input.

Original: Compared to the others, we went much further regarding the experiments’ scale and analysis depth.

Original: Firstly, We studied a wide range of tasks and confirmed that the EM-Network could achieve impressive results even for the SSL-based models.

Original: To date, using the target input to improve the

Original: SSL-based models is still an unexplored area in previous literature.

Original: We also investigated how the proposed training objective can improve the sequence model from the EM-like perspective.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we introduced EM-Network that can effectively yield a promising latent variable from both source and target inputs.
Revised: In this paper, we introduced EM-Network that can effectively yield a promising latent variable from both source and target inputs.

Original: Instead of merely considering the ground truth as the sole target in training, the target was utilized as the model’s additional training input.
Revised: Instead of merely considering the ground truth as the sole target in training, the target was utilized as the model’s additional training input.

Original: The proposed selfdistillation framework enabled the sequence model to benefit from the EM-Network’s latent variable in a one-stage manner.
Revised: The proposed selfdistillation framework enabled the sequence model to benefit from the EM-Network’s latent variable in a one-stage manner.

Original: We theoretically showed that our training objective, which was motivated by the EM-like algorithm, could be a lower bound for the log-likelihood of the sequence model.
Revised: We theoretically showed that our training objective, which was motivated by the EM-like algorithm, could be a lower bound for the log-likelihood of the sequence model.

Original: Empirically, EM-Network significantly advanced the current state-of-the-art approaches on ASR and MT tasks.
Revised: Empirically, EM-Network significantly advanced the current state-of-the-art approaches on ASR and MT tasks.

Original: We hope our study will draw more attention from the community toward a richer view of the latent variable for seq2seq learning.
Revised: We hope our study will draw more attention from the community toward a richer view of the latent variable for seq2seq learning.

Original: A S OURCE C ODE
Revised: A S OURCE C ODE

Original: Source code can be found at https://github.com/em-net/em-network-anonymous.

Original: git .
Revised: [Table]

Original: B THEORETICAL ANALYSISB. D ERIVATION Derivation of Eq. (10)
Revised: B THEORETICAL ANALYSIS Derivation of Eq. (8)

Revised: [Equation] Derivation of Eq. (10)

Original: [Equation]
Revised: [Equation]

Revised: θ

Original: [Equation]
Revised: [Equation]

Original: B. T HEORETICAL A NALYSIS WITH CE OBJECTIVE
Revised: t

Original: When considering the CE objective function, we can also interpret our framework from a similar perspective in Eq.
Revised: [Equation]

Original: We start from the log-likelihood ofthe CE-based sequence model, which is formulated in Eq. 
Revised: Derivation of Eq. (12)

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Revised: x

Revised: [Equation]

Revised: ˜ y

Revised: [Equation]

Revised: θ ( t ) , ρ ( t ) )

Original: Eq.
Revised: P ( [Equation] θ ( t ) [Equation]

Revised: ρ ( t ) )

Revised: [Equation]

Original: (12) indicates that the KL-divergence between the EM-Network (  P ( y | x, ˜ y ; ϕ ) ) and the CE-based sequence model (  P ( y | x ; θ )) )
Revised: ˜ y [Equation] (˜ y − M ( y, λ )) P ( y | x ; θ ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) )

Original: accords with the distillation loss in Eq.
Revised: = log (cid:88) ˜ y =

Revised: M ( y,λ )

Original: By Eq.
Revised: P ( y | x ; θ ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) = log (cid:88) ˜

Revised: y

Revised: =

Original: (8), we already know that a negative value of the KL-divergence between the EM-Network and the sequence model serves as the Q-function of the EM-Network.
Revised: M ( y,λ ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) P ( y | x ; θ ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) ≥ (cid:88) ˜

Revised: y

Revised: =

Original: Therefore, maximizing the CE-based EM-Network’s
Revised: M ( y,λ ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) log P ( y | x ; θ ) P ( y | x, ˜ y ; θ ( t ) , ρ ( t ) )

Revised: [Equation]

Original: Q-function is equivalent to maximizing the lower bound of the sequence model’s likelihood.
Revised: [Equation] ( y | x, ˜ y ; θ ( t ) , ρ ( t ) ) ∥ P ( y | x ; θ )) [Equation]

Original: C I MPLEMENTATION D ETAILS
Revised: C I MPLEMENTATION D ETAILS

Revised: [Table]

Revised: BiBERT 206 M

Revised: EM

Revised: [Equation]

Revised: Network

Original: Speech Recognition.
Revised: [Equation] 206 M [Equation] 19 M)

Revised: MT

Revised: [Equation]

Revised: WMT’14)

Revised: BiBERT 324 M

Revised: EM-Network 374 M ( θ

Revised: [Equation]

Revised: 324 M

Original:  For the pre-training stage, we followed the pre-training regime of data2vec (Baevski et al., 2022).
Revised: [Equation] 50 M) Speech Recognition. For the pre-training stage, we followed the pre-training regime of data2vec (Baevski et al., 2022).

Original: After the pre-training, pre-trained models were fine-tuned for the ASR task by applying a linear projection layer.
Revised: After the pre-training, pre-trained models were fine-tuned for the ASR task by applying a linear projection layer.

Original: We fine-tuned the model with Adam (Kingma & Ba, 2015) optimizer with an initial learning rate of 1e-4.
Revised: We fine-tuned the model with Adam (Kingma & Ba, 2015) optimizer with an initial learning rate of 1e-4.

Original: For the LibriSpeech, the character set had a total of 29 labels plus a word boundary token.
Revised: For the LibriSpeech, the character set had a total of 29 labels plus a word boundary token.

Original: The BASE setup contained 12 transformer blocks with model dimension 768 and 8 attention heads.
Revised: The BASE setup contained 12 transformer blocks with model dimension 768 and 8 attention heads.

Original: Also, BASE models utilized a batch size of 3.2m samples per GPU and were fine-tuned on eight Quadro RTX 8000 GPUs (each with 48GB of memory), where training updates were set to 320k.
Revised: Also, BASE models utilized a batch size of 3.2m samples per GPU and were fine-tuned on eight Quadro RTX 8000 GPUs (each with 48GB of memory), where training updates were set to 320k.

Original: Following the recommendation of the language model settings (Baevski et al., 2020), test performance was measured with beam 1,500 for the 4-gram LM.
Revised: Following the recommendation of the language model settings (Baevski et al., 2020), test performance was measured with beam 1,500 for the 4-gram LM.

Original: In the case of EM-Network, the architecture of the auxiliary network was based on the self and cross-attention modules, and we additionally employed the feed-forward layers, layer normalization layers, and residual connections for the attention layer.
Revised: In the case of EM-Network, the architecture of the auxiliary network was based on the self and cross-attention modules, and we additionally employed the feed-forward layers, layer normalization layers, and residual connections for the attention layer.

Original: Machine Translation.
Revised: Machine Translation.

Original: For the IWSLT’14 dataset, we used four Titan V GPUs (each with 12GB of memory) with 2048 tokens per GPU and accumulated the gradient 4 times.
Revised: For the IWSLT’14 dataset, we used four Titan V GPUs (each with 12GB of memory) with 2048 tokens per GPU and accumulated the gradient 4 times.

Original: The optimizer was
Revised: The optimizer was

Original: Adam (Kingma & Ba, 2015), with a learning rate of 0.0004.
Revised: Adam (Kingma & Ba, 2015), with a learning rate of 0.0004.

Original: We followed the same byte-pair encoding (BPE) settings of the previous study (Xu et al., 2021).
Revised: We followed the same byte-pair encoding (BPE) settings of the previous study (Xu et al., 2021).

Original: At inference time, we applied beam search with width 4 and a length penalty of 0.6.
Revised: At inference time, we applied beam search with width 4 and a length penalty of 0.6.

Original: In the case of the WMT’14 dataset, newstest and newstest2013 were combined as the validation set, and we used newstest2014 as the test set.
Revised: In the case of the WMT’14 dataset, newstest and newstest2013 were combined as the validation set, and we used newstest2014 as the test set.

Original: Followed by the BiBERT(Xu et al., 2021), we used a unified 52K vocabulary for the decoder.
Revised: Followed by the BiBERT(Xu et al., 2021), we used a unified 52K vocabulary for the decoder.

Original: We utilized four Quadro RTX 8000 GPUs (each with 48GB of memory) with a batch size of 4096 tokens per GPU.
Revised: We utilized four Quadro RTX 8000 GPUs (each with 48GB of memory) with a batch size of 4096 tokens per GPU.

Original: The gradient was accumulated 32 times.
Revised: The gradient was accumulated 32 times.

Original: The initial learning rate was set to 0.001.
Revised: The initial learning rate was set to 0.001.

Revised: Number of Parameter.

Revised: Table 4 compares the number of parameters between the best seq2seq baseline and the EM-Network.

Revised: We used the best baseline as the inner sequence model of the EMNetwork, and thus the parameter for θ corresponded to that of the best baseline, such as data2vec and BiBERT.

Revised: The size of the auxiliary network ρ was mainly determined by the vocabulary size.

Revised: Since the EM-Network for WMT’14 dataset used about 52K vocabulary for the decoder, it required the relatively large parameters for ρ .

Original:  D F URTHER E XTENSION : E XPERIMENTAL RESULTS WITH FULLY SUPERVISED LEARNING - BASED MODEL D. S ETUP
Revised: However, during the inference, only the sequence model was used to generate the prediction . Since the auxiliary network was removed during the inference, the additional computational load of ρ was only required for the training procedure. Considering the performance improvements of the proposed approach, this computational load for training seems reasonable. D F URTHER E XTENSION : E XPERIMENTAL RESULTS WITH FULLY SUPERVISED LEARNING - BASED MODEL D. S ETUP

Original: For the fully supervised learning-based models,  models were implemented in the NeMo (Kuchaiev et al., 2019) toolkit.
Revised: For the fully supervised learning-based models, ASR models were implemented in the NeMo (Kuchaiev et al., 2019) toolkit.

Original: We trained the EM-Network with LS-960, and the sequence model ( θ ) was based on a conformer-CTC architecture consisting of 16 conformer (Gulati et al., 2020) blocks with 176 dimensions.
Revised: We trained the EM-Network with LS-960, and the sequence model ( θ ) was based on a conformer-CTC architecture consisting of 16 conformer (Gulati et al., 2020) blocks with 176 dimensions.

Original: We used four Quadro RTX 8000 GPUs (each with 48GB of memory), andepochs were spent for training the models.
Revised: We used four Quadro RTX 8000 GPUs (each with 48GB of memory), and 100 epochs were spent for training the models.

Original: AdamW algorithm (Loshchilov & Hutter, 2019) was employed as an optimizer with an initial learning rate of 5.0.
Revised: AdamW algorithm (Loshchilov & Hutter, 2019) was employed as an optimizer with an initial learning rate of 5.0.

Original: In the case of Conformer-CTC large, the current SOTA ASR model, we used the pre-trained checkpoint provided by the NeMo (Kuchaiev et al., 2019) toolkit.
Revised: In the case of Conformer-CTC large, the current SOTA ASR model, we used the pre-trained checkpoint provided by the NeMo (Kuchaiev et al., 2019) toolkit.

Revised: For MT task, we evalutated the models with IWSLT’14 datasets for En-De and De-En translations.

Revised: The sequence model ( θ ) was based on Transformer (Vaswani et al., 2017).

Revised: We used four Titan V GPUs (each with 12GB of memory), and 75 epochs were spent for training.

Revised: BiBERT (Xu et al., 2021), which is the current SOTA MT model, was adopted as the teacher model for conducting the conventional distillation.

Original: D.  R ESULTS
Revised: D. ASR R ESULTS

Original:  In addition to the previous experiments, we checked whether the EM-Network could improve the performance of the sequence models with fully supervised learning.
Revised: [Table] In addition to the previous experiments, we checked whether the EM-Network could improve the performance of the sequence models with fully supervised learning.

Original: Table 4 shows the results on the LibriSpeech test set.
Revised: Table 5 shows the results on the LibriSpeech test set.

Original: The SKD (Yoon et al., 2021b) is the recent KD method in the ASR task, and we used its distillation loss for transferring the soft labels of the EM-Network (Eq.
Revised: We applied Guided CTC training (Kurata & Audhkhasi, 2019) and SKD (Yoon et al., 2021b) as the competing KD methods for ASR. The Conformer-CTC Large was adopted as the teacher for Guided CTC training and SKD. The Guided CTC training is the effective KD method for CTC-based ASR model.

Revised: The student can be guided to align with the frame-level alignment of the teacher by using the guided mask.

Revised: The SKD is the recent KD method in the ASR task, and we used its distillation loss for transferring the soft labels of the EM-Network (Eq.

Original: The main difference between the SKD and the EM-Network was that the EM-Network was trained with the self-distillation setup with the additional target input, while the conventional SKD was based on offline knowledge distillation.
Revised: The main difference between the SKD and the EM-Network was that the EM-Network was trained with the self-distillation setup with the additional target input, while the conventional SKD was based on offline knowledge distillation.

Original: Even though the teacher’s knowledge in  SKD was fixed during the student model training, the soft label of the EM-Network was updated in each iteration so that our approach could transfer the knowledge more adaptively.
Revised: Even though the teacher’s knowledge in Guided CTC training and SKD was fixed during the student model training, the soft label of the EM-Network was updated in each iteration so that our approach could transfer the knowledge more adaptively.

Original: As shown in Table 4, SKD required the additional teacher model (122 M parameters) to perform the distillation.
Revised: As shown in Table 5, both Guided CTC training and SKD required the additional teacher model (122 M parameters) to perform the distillation.

Original: However, only 1 M[Table] parameters, which corresponded to the auxiliary network in the EM-Network, were required to distill the knowledge in the proposed framework.
Revised: However, only 1 M parameters, which corresponded to the auxiliary network in the EM-Network, were required to distill the knowledge in the proposed framework.

Original: From the results, it is verified that the EM-Network performed well with the fully-supervised learning-based model.
Revised: From the results, it is verified that the EM-Network performed well with the fully-supervised learning-based model.

Original: Compared to the conventional SKD using the SOTA ASR model as the teacher, EM-Network achieved better performance on the LibriSpeech test dataset.
Revised: Compared to the conventional KD methods using the SOTA ASR model as the teacher, EM-Network achieved better performance on the LibriSpeech test dataset.

Original: We confirmed that the proposed method can significantly improve the performance of the sequence model with the fully supervised learning-based model, yielding 4.29 % (RERR: 11.91 %) and 10.81 % (RERR: 10.29 %) on test-clean and test-other, respectively. 
Revised: We confirmed that the proposed method can significantly improve the performance of the sequence model with the fully supervised learning-based model, yielding 4.29 % (RERR: 11.91 %) and 10.81 % (RERR: 10.29 %) on test-clean and test-other, respectively. D. MT R ESULTS [Table]

Revised: For MT task, we compared the EM-Network with other previous methods, including sequence-level

Revised: KD (Kim & Rush, 2016) and R-Drop (Liang et al., 2021), as shown in Table 6.

Revised: The sequencelevel KD is a widely-used KD method in sequence generation tasks, and the R-Drop is a recent and effective regularization method built upon dropout.

Original: E L ATENT CTC A LIGNMENT V ISUALIZATION
Revised: As the teacher model for the sequence-level KD, we adopted BiBERT Xu et al. (2021), which is a SOTA approach for MT task. The results in Table 6 report that the proposed EM-Network yielded the best BLEU performance on IWSLT’14 dataset, yielding 29.84 and 36.47 on En-De and De-En translation tasks, respectively. The main difference from the R-Drop was the training loss L em of the EM-Network.

Revised: While the

Revised: R-Drop only consumed the source input, the proposed method utilized the target input when training the EM-Network so that the prediction of the EM-Network (teacher mode) was more accurate than that of the inner sequence model (student mode).

Revised: From the results, we can verify that the target input was effectively used as guidance to provide the target-side information while producing a considerable performance gain.

Revised: Compared to the R-Drop that did not require the additional parameters, 19 M parameters were used as the auxiliary network ρ to train the EM-Network.

Revised: However, since the auxiliary network was removed during the inference, the additional computational load of ρ was only required for the training procedure.

Original: F E FFECT OF α AND λ
Revised: Considering the performance improvements, this additional load during the training seemed reasonable. Also, when we reduced the vocabulary size, the additional parameters would be much smaller. E A DDITIONAL V ISUALIZATION We additionally compared the frame-wise softmax outputs of the best baseline data2vec, our sequence model θ (student mode) in the EM-Network, and the EM-Network ϕ (teacher mode), as visualized in Figure 5.

Revised: The argmax value of the frame-wise label probability corresponds to the predicted CTC alignment.

Revised: The conventional data2vec made erroneous predictions with “ON SO” and “HART”.

Revised: When considering only the acoustic feature (speech voice), it is challenging to distinguish “ON SO”/“A SOUL” and “HART”/“HURT”.

Revised: However, the EM-Network gave the correct prediction result with few irrelevant alignments, indicating that the proposed method effectively leveraged the target information to find the optimal CTC alignment.

Revised: By benefiting from the knowledge of the

Revised: EM-Network, the proposed sequence model provided a more accurate prediction than

Revised: By Eq.

Revised: 10, we confirmed that minimizing Eq.

Revised: 4 maximizes a lower bound of the CTC model’s loglikelihood.

Revised: The lower bound is tight when achieving a low distillation loss between EM-Network and the sequence model, meaning a relatively large overlap between the predictions.

Revised: Due to the alignment-free property of the CTC framework, the conventional CTC models trained with the same settings, such as model architecture, training data, etc., can have different frame-level alignments, and this difference often leads to the convergence issue of the distillation (Yoon et al., 2021b).

Revised: However, as shown in Figure 5, both EM-Network (teacher mode) and the sequence model (student mode) could produce similar frame-wise predictions (green boxes in Figure 5), indicating that the proposed method was indeed able to perform the frame-wise distillation effectively and give a considerable alignment overlap, making a tight lower bound.

Original: When training the CE-based EM-Network, we considered two tunable parameters α in Eq.
Revised: When training the CE-based EM-Network, we considered two tunable parameters α in Eq.

Original: (1) and λ in Eq.
Revised: (1) and λ in Eq.

Original: The parameter α was used to balance the distillation loss L kd , and λ was applied as the masking probability in the masking function M .
Revised: The parameter α was used to balance the distillation loss L kd , and λ was applied as the masking probability in the masking function M .

Original: We explored the effect of α and λ on EM-Network performance, as shown in Figure 6.
Revised: We explored the effect of α and λ on EM-Network performance, as shown in Figure 6.

Original: Firstly, we evaluated the EM-Network on IWSLT’14 En-De translation while varying α .
Revised: Firstly, we evaluated the EM-Network on IWSLT’14 En-De translation while varying α .

Original: α was selected from { 0, 1, 2, 3, 4 } , and λ was set to 50 %.
Revised: α was selected from { 0, 1, 2, 3, 4 } , and λ was set to 50 %.

Original: We observed that on IWSLT’14 En-De, the performance was stably better than the prior SOTA approach, and the best performance was obtained at α = 2 .
Revised: We observed that on IWSLT’14 En-De, the performance was stably better than the prior SOTA approach, and the best performance was obtained at α = 2 .

Original: Interestingly, the EM-Network with α = 0 was slightly better than the performance of the BiBERT, but the difference was negligible.
Revised: Interestingly, the EM-Network with α = 0 was slightly better than the performance of the BiBERT, but the difference was negligible.

Original: Secondly, we showed the effect of λ on EM-Network performance.
Revised: Secondly, we showed the effect of λ on EM-Network performance.

Original: In this case, α was set to 2, and we selected λ from { 10%, 30%, 50%, 70% } .
Revised: In this case, α was set to 2, and we selected λ from { 10%, 30%, 50%, 70% } .

Original: From the results, it is verified that performance on IWLST’14 En-De achieved the best result when λ = 50% .
Revised: From the results, it is verified that performance on IWLST’14 En-De achieved the best result when λ = 50% .