Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models (2023)

Abstract: One challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. To address this mismatch, we adapt large language models (LLMs) to split long ASR transcripts into segments that can be independently translated so as to maximize the overall translation quality. We overcome the tendency of hallucination in LLMs by incorporating finite-state constraints during decoding; these eliminate invalid outputs without requiring additional training. We discover that LLMs are adaptable to transcripts containing ASR errors through prompt-tuning or fine-tuning. Relative to a state-of-the-art automatic punctuation baseline, our best LLM improves the average BLEU by 2.9 points for English–German, English–Spanish, and English–Arabic TED talk translation in 9 test sets, just by improving segmentation.

## introduction with the proliferation of long-form audiovisual content online, translation and captioning become paramount for accessibility. cascade models remain the dominant approach for speech translation @xcite , decomposing the problem into automatic speech recognition (asr), post-processing of the transcript, and machine translation (mt). the cascade's mt component typically operates on sentence-like units, with each sentence translated independently of the others. when asked to translate long passages, models regularly fail or degenerate @xcite @xcite . this differs considerably from the expectations for automatic speech recognition models (e.g. @xcite that can process inputs of unbounded lengths. mt models must either be able to cope with potentially long, multi-sentence inputs or, alternatively, they must be able to determine cutpoints at which the transcript can be segmented into compact, independently translatable units. this work introduces a new, effective approach for the latter. while numerous text segmentation techniques have been proposed to improve spoken language translation ( §6), the problem remains hard and unsolved. indeed, li et al. @xcite demonstrate that poor sentence segmentation degrades performance almost twice as much as transcript lexical errors. we cast sentence segmentation as a sequenceto-sequence task, rather than a traditional structured prediction task that tags sentence-final tokens. while this lets us leverage large language models, such models' outputs can be ill-formed. even by using additional data for fine-tuning, residual adapters @xcite , or future discriminators @xcite , simple syntactic constraints can be difficult to enforce. moreover, all three require modifying the model or storing additional learned parameters. in light of these concerns, we introduce a simple, flexible, and modular approach to generating well-formed task-specific strings at inference time without any additional training. we compactly express constraints on the output format as finitestate machines, then efficiently enforce these via composition. while the approach is simple, it remains unexplored for large language models, and it yields automatic gains on downstream performance, advancing the state of the art for speech translation and thereby applicable to existing systems. moreover, the approach is sufficiently general that it can be applied to other domains in a plug-and-play manner. we benchmark our approach as a component in a speech translation cascade. experiments in three language pairs indicate that our approach outperforms both a baseline cascade system that predicts punctuation marks before inferring sentence boundaries and a strong neural structured prediction model. overall, we improve the bleu score on the iwslt test sets by 2.9 points, closing 3 /4 of the gap between the previous best and the oracle system. our contributions are three-fold: ## windowing approach one major challenge in modeling and inference of long-form transcript segmentation is that the input sequences can be very long. for example, a ted talk can contain more than one thousand words @xcite . we take a divide-and-conquer approach that operationalizes two straightforward principles in modeling. first, words on the left and right are both useful for deciding if a sentence delimiter should be present at the current word position. second, distant words are less useful than nearby words. from these two principles, we design a top-level sliding window algorithm to balance the need for bidirectional modeling and efficiency of computation. we divide the passage into windows at both training and test time, with a small context window on each side to inform decisions at window edges (figure 1 ). with this top-level inference algorithm, the sequence-to-sequence machine learning problem is now reduced to the window-level. the problem is now to predict a sequence of segmentation decisions @xmath0, . . . , y w for each text window of size at most w tokens: @xmath1, . . . , x w . ## modeling approaches a classic approach to discriminative sequence modeling is the conditional random field (crf) @xcite . this conditional graphical model allows incorporating arbitrary features of the transcript, including linguistic variables and word embeddings. ## experiments we evaluate our proposed method for using large language models for long-form speech translation with three sets of experiments: (1) analysis of hyperparameters, (2) comparison with competing methods, and (3) robustness to speech recognition errors. in each case, we are concerned with translation quality as measured by bleu. we also assess the llm output directly by qualitative analysis, well-formedness percentage, and (for diagnostic purposes, following @xcite segmentation f 1 score against the sentence-segmented reference. our experiments are carried out on the iwslt speech translation data sets, subjected to the same pre-processing as described in @xcite . we use the 2014 data for dev and 2015 and 2018 for test. the fourteen reference transcripts in our dev set range from 861 to 1234 words; by contrast, the median length of a sentence in written english is close to 17 words @xcite . we use the publicly available speech-to-text google api 1 to generate asr transcripts. we remove the automatically predicted punctuation and lowercase the asr transcripts and use english-{german,spanish,arabic} mt models trained with the same preprocessing on the source side as @xcite . the mt model is a transformer with a model dimension of 1024, hidden size of 8192, 16 attention heads, 6 encoder layers, and 8 decoder layers. we decode with a beam size of 4. in our experiments, the three mt model instances and the asr model (and thereby its transcripts) are fixed while we vary the sentence segmentation policies. ## conclusion we have presented new methods for long-form speech translation by coupling source-side large language models with finite-state decoding constraints, allowing large language models to be used for structured prediction with a guarantee for wellformedness in the output space. finite-state constraints are especially effective when the model is decoder-only, relatively small, or has not been completely fine-tuned (only prompt-tuned, or fewshot-learned) for the structured prediction task. we also observe that even though complete fine-tuning and enlarging model size can reduced the rate of invalid output, models alone are not capable of completely eliminate invalid output. fine-tuning on in-domain asr transcripts containing recognition errors and disfluency improves segmentation accuracy over training on clean transcripts. our qualitative analysis shows the largest category of remaining errors is asr errors which make transcripts difficult to parse and segment. the fact that llms are capable of adapting to asr errors points to future research directions of contextualized asr error recovery.