CTC Alignments Improve Autoregressive Translation (2023)

Abstract: Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment. However for translation, CTC exhibits clear limitations due to the contextual and non-monotonic nature of the task and thus lags behind attentional decoder approaches in terms of translation quality. In this work, we argue that CTC does in fact make sense for translation if applied in a joint CTC/attention framework wherein CTC’s core properties can counteract several key weaknesses of pure-attention models during training and decoding. To validate this conjecture, we modify the Hybrid CTC/Attention model originally proposed for ASR to support text-to-text translation (MT) and speech-to-text translation (ST). Our proposed joint CTC/attention models outperform pure-attention baselines across six benchmark translation tasks.

## introduction automatic speech recognition (asr), machine translation (mt), and speech translation (st) have conspicuous differences but are all closely related sequence-to-sequence problems. researchers from these respective fields have long recognized the opportunity for cross-pollinating ideas @xcite , starting from the coupling of statistical asr @xcite and @xcite which gave rise to early approaches for st @xcite . notably in the end-toend era, attentional encoder-decoder approaches emerged in both mt @xcite and asr @xcite , rising to great prominence in both fields. during this same period, there has been another prominent end-to-end approach in asr: connectionist temporal classification (ctc) @xcite . unlike the highly flexible attention mechanism which can handle asr, mt, and st alike, ctc models sequence transduction as a monotonic alignment of inputs to outputs and thus fits more naturally with asr than it does with translation. still, many interested in non-autoregressive translation have applied ctc to mt @xcite and st @xcite and promising techniques have emerged, shrinking the gap between autoregressive approaches @xcite @xcite . these recent developments suggest that the latent alignment ability of ctc is a promising direction for translation -this leads us to question: can ctc alignments improve autoregressive translation? in particular, we are interested in frameworks that leverage the strength of ctc while minimizing its several harmful incompatibilities (see §3) with translation tasks. inspired by the success of hybrid ctc/attention in asr @xcite , we investigate jointly modeling ctc with an autoregressive attentional encoder-decoder for translation. our conjecture is that the monotonic alignment and conditional independence of ctc, which weaken purely ctc-based translation, counteract particular weaknesses of attentional models in joint ctc/attention frameworks. in this work, we seek to investigate how each ctc property interacts with corresponding properties of the attentional counterpart during joint training and decoding. we design a joint ctc/attention architecture for translation ( §4) and then examine the positive interactions which ultimately result in improved translation quality compared to pure-attention baselines, as demonstrated on the iwslt @xcite , @xcite , and mtedx @xcite mt/st corpora ( §6). what are the critical differences between the ctc and attention frameworks? as shown in the first two columns of secondly, the attentional decoder models each output unit y 1 with conditional dependence on not only the input x, but also the previous output units y 1:l-1 . in contrast, ctc makes a conditional independence assumption that each z t does not depend on z 1:t-1 if already conditioned on x (as denoted by the strike-through in finally, the attentional decoder is an autoregressive generator that decodes the output until a stop token, <eos>. comparing likelihoods for sequences of different lengths requires a heuristic brevity penalty. furthermore label bias with respect to the stop token manifests as a length problem where likelihoods degenerate for unexpectedly long outputs @xcite . in comparison, ctc is an input-synchronous emitter that consumes an input unit in order to produce an output unit. therefore, ctc cannot produce an output longer than the input representation which feeds the final posterior output layer -but this also means that ctc does not require end detection. as previously shown by @xcite , jointly modeling ctc and an attentional decoder is highly effective in asr. the foundation of this architecture is a shared encoder, enc, which feeds into both ctc, p ctc (•), and attentional decoder, p attn (•), posteriors: joint decoding is typically performed with a onepass beam search where ctc plays a secondary role as a joint scorer while attention leads the major hypothesis expansion and end detection functions in the algorithm @xcite . however, ctc is capable of taking over the lead role if called upon (e.g. for streaming applications) @xcite . ## potential ctc limitations in mt/st why exactly does this joint ctc/attention framework perform so well in asr? as summarized in column 3 of so can ctc and attention also complement each other when applied jointly to translation? foot_1 asr, mt, and st can all be generalized as sequence transduction tasks following the bayesian formulation. attentional decoders have been a predominant technical solution to each of these tasks. however, the ctc framework appears to have several limitations specific to mt/st that are not present in asr; this seemingly diminishes the promise of the joint ctc/attention framework for translation. in this work, we seek to address the following three concerns about mt/st ctc which appear to inhibit the ctc/attention framework (please refer back to l1 can ctc encoders perform sophisticated input-to-output mappings required for translation? unlike asr, translation entails non-monotonic mappings due to variable word-ordering across languages. additionally, inputs may be shorter than outputs as mappings are not necessarily one-to-one. furthermore, the mapping task for st is compositional where logically a speech signal first maps to a source language transcription before being mapped to the ultimate translation. all of these complications appear to directly contradict the hard alignment of ctc. if ctc cannot produce stable encoder representations for mt/st, then during joint training attention does not receive the optimization benefit as in asr (per row 2 of l2 does ctc-based translation quality lag too far behind attention-based to be useful? ctc-based asr has recently shown competitive performance due in large part to improved neural architectures @xcite and selfsupervised learning @xcite , but the gap between ctc and attention for translation appears to be greater @xcite . perhaps the conditional independence of ctc inhibits the quality to such a degree in mt/st where these likelihoods cannot ease the label/exposure biases of the attentional decoder as they do in asr (per row 3 of ## results and analyses in this section, we first present our main results on 6 benchmark mt and st tasks. we then present evidence that hierarchical encoding ( §4.1) produces stable encoder representations that simplify the decoder's source attention (addressing l1 in §3). next we present evidence that joint decoding is beneficial despite the fact that ctc-only performance lags behind that of attention-only (addressing l2 in §3). finally, we present evidence that ctc's alignment information resolves attention's end-detection problem in both input and output synchronous joint decoding ( §4.2) (addressing l3 in §3). ## experimental setup data we examine the efficacy of our proposed approaches on two language pairs for each of the mt and st tasks. for mt, we use german-to-english (de-en) and spanish-to-english (es-en) from iwslt14 @xcite . for st, we use english-to-german (en-de) and english-to-japanese (en-ja) from must-c-v2, reporting tst-common results @xcite . we also examine the multilingual setting of 6 european languages to english (all-en) from mtedx @xcite for both tasks. full dataset descriptions for reproducibility are in §b. ## results and analyses in this section, we first present our main results on 6 benchmark mt and st tasks. we then present evidence that hierarchical encoding ( §4.1) produces stable encoder representations that simplify the decoder's source attention (addressing l1 in §3). next we present evidence that joint decoding is beneficial despite the fact that ctc-only performance lags behind that of attention-only (addressing l2 in §3). finally, we present evidence that ctc's alignment information alleviates attention's end-detection problem in both input and output synchronous joint decoding ( §4.2) (addressing l3 in §3). ## related works the idea of using latent alignments to improve autoregressive translation has been explored previously by @xcite who concluded that ctc alignments are not compatible with teacher forcing. the key difference is that we train ctc and autoregressive models jointly while @xcite sought to apply ctc to train autoregressive models, replacing cross-entropy entirely. more recently in a concurrent work, @xcite have also shown the effectiveness of jointly training ctc and attention in the context of st for unwritten languages where no asr transcriptions are available. we believe that our contribution showing the effectiveness of also jointly decoding ctc and attention demonstrates an additional technique which can further improve their direction. our work also differs in that we seek to incorporate the asr objective into st via hierarchical encoding. other concurrent works integrated ctc and attention within blockwise streaming @xcite and compositional multi-decoder @xcite 2022) architectures for st in particular. our work supports their findings by addressing why ctc is helping, and we provide a unified approach that generalizes to both mt and st. prior works have also used the non-autoregressive property of ctc as means for speeding up autoregressive models during inference @xcite , but these works focus on latency and do not apply ctc to improve translation quality. ## conclusion we propose to jointly train and decode ctc/attention models for mt and st using 1) hierarchical encoding to resolve incompatibilities between ctc and the non-monotonic mappings in translation and 2) synchronous decoding to ease the exposure/label biases of autoregressive decoders with ctc's conditionally independent alignment information. our analyses reveal several reasons why even weak ctc models benefit autoregressive translation via joint modeling, suggesting that future explorations into jointly modeling attentional decoders with other latent alignment models @xcite @xcite may uncover similar benefits.