Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLM s for Simultaneous Translation (2024)

Abstract: Large language models (LLMs) have achieved state-of-the-art performance in various language processing tasks, motivating their adoption in simultaneous translation. Current fine-tuning methods to adapt LLMs for simultaneous translation focus on prompting optimization strategies using either data augmentation or prompt structure modifications. However, these methods suffer from several issues, such as unnecessarily expanded training sets, computational inefficiency from dumping the key and value cache, increased prompt sizes, or restriction to a single decision policy. To eliminate these issues, in this work, we propose SimulMask, a new paradigm for fine-tuning LLMs for simultaneous translation. It utilizes a novel attention mask approach that models simultaneous translation during fine-tuning by masking attention for a desired decision policy. Applying the proposed SimulMask on a Falcon LLM for the IWSLT 2017 dataset, we have observed a significant translation quality improvement compared to state-of-the-art prompting optimization strategies on five language pairs while reducing the computational cost.

## introduction simultaneous translation refers to the process of producing a target output translation concurrently with an oncoming source input. in our increasingly interconnected world, where communication across languages in real-time is desired, simultaneous translation is becoming a requirement. as such, there is a need for machine learning models to fill this role. current literature has primarily focused on adapting end-to-end transformer models @xcite to overcome the difficulties of simultaneous machine translation (simulmt) due to their reduced parameter counts and greater inference speed @xcite . however, the recent successes of large language models (llms) @xcite @xcite has prompted preliminary research applying them to simulmt through fine-tuning and inference techniques @xcite @xcite @xcite . unfortunately, most modern works have neglected the computational increases created by dumping the target sequence's key and value (kv) cache @xcite . furthermore, there has yet to be a universal approach to finetuning llms for simulmt that is not unnecessarily computationally expensive by either expanding the dataset through data augmentation, a process referred to as prefix fine-tuning @xcite @xcite or increasing the prompt length through prompt restructuring @xcite . the lack of an efficient fine-tuning strategy of llms for simulmt has led us to propose a new paradigm, referred to as simulmask. simulmask is a novel attention mask to model simulmt during fine-tuning by redistributing the attention under a decision policy. by design, simulmask is broadly applicable to both flexible and fixed decision policies, creating a path forward for future work to build upon it. furthermore, if we avoid injecting positional information into the keys and values through a modified alibi @xcite , simulmask allows for kv caching during simulmt without accuracy degradation. to validate the efficacy of simulmask, we finetuned and evaluated 1.3 billion parameter falcon models pre-trained on the refinedweb dataset using simulmask @xcite and compared them against identical falcon models that adopt existing prefix fine-tuning or prompt restructuring methods on the iwslt 2017 dataset @xcite . from the results, we demonstrate models fine-tuned with simulmask outperform prefix fine-tuning and prompt restructuring models at simulmt for a given latency regime with a reduced computational cost. the main contributions of the paper include: 1. providing insights on the shortcomings of current methods in adapting llms to simulmt. 2. proposing a novel attention masking approach to fine-tune simulmt llms that enables efficient training and inference. 3. demonstrating the efficacy of our approach in terms of translation quality and computational costs by evaluating them on multiple language pairs across varied latencies. ## prompting optimization methods current methods of fine-tuning llms for simulmt fall under prompting optimization. we define prompting optimization as either employing data augmentation to help with prompting @xcite @xcite or redefining the prompt structure @xcite to somewhat simulate simulmt. ## analysis and shortcomings of prompting optimization methods prompting optimization, while functional to a certain degree, is inherently deficient, possessing a host of fine-tuning and inference issues. these issues include a persistent fine-tuning-inference mismatch, consistent positional confusion in the target sequence, and high computational costs. ## simulmask: a paradigm shift in this work, we propose simulmask, which we believe could be a paradigm shift in fine-tuning llms for simulmt that eschews current methods of prompting optimization. by restricting attention during fine-tuning, simulmask efficiently solves the fine-tuning-inference mismatch and positional confusion problem. we demonstrate simulmask through its application for the wait-k decision policy, but it should be noted that simulmask broadly applies to various decision policies. ## conclusion in this work, we first examine current llm finetuning approaches for simulmt and identify their shortcomings. we then propose a new paradigm for fine-tuning llms for simulmt that we call simulmask, which avoids the shortcomings of previous methods. when employing simulmask, the target sequence is prevented from attending to a portion of the source sequence according to an arbitrary decision policy modeling simulmt. through the application of simulmask, we can efficiently fine-tune a llm for simulmt and reduce the computational costs of inference by eliminating the recomputation of the kv cache for the target sequence, unlike prior works. furthermore, we can exceed or match the translation quality of prior works at all wait-k values across multiple language pairs.