Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection (2023)

Abstract: When translating words referring to the speaker, speech translation (ST) systems should not resort to default masculine generics nor rely on potentially misleading vocal traits. Rather, they should assign gender according to the speakers’ preference. The existing solutions to do so, though effective, are hardly feasible in practice as they involve dedicated model re-training on gender-labeled ST data. To overcome these limitations, we propose the first inference-time solution to control speaker-related gender inflections in ST. Our approach partially replaces the (biased) internal language model (LM) implicitly learned by the ST decoder with gender-specific external LMs. Experiments on en \rightarrow es/fr/it show that our solution outperforms the base models and the best training-time mitigation strategy by up to 31.0 and 1.6 points in gender accuracy, respectively, for feminine forms. The gains are even larger (up to 32.0 and 3.4) in the challenging condition where speakers’ vocal traits conflict with their gender.

## introduction the problem of gender bias in automatic translation particularly emerges when translating from genderless or notional gender languages (e.g., english) -which feature limited gender-specific markinginto grammatical gender languages (e.g., spanish) -which exhibit a rich lexical and morpho-syntactic system of gender @xcite . in this scenario, when gender-neutral words are translated into gender-marked words (e.g. en: the nurse -es: el/la enfermero/a), both machine translation (mt) and speech translation (st) systems are often biased towards masculine or stereotypical predictions @xcite @xcite , especially in absence of explicit cues (en: the nurse and his dog). a common instance is represented by words that refer to the first-person subject (henceforth referred to as speaker-dependent words, such as i'm a young nurse). in these cases, direct st systems @xcite have been shown to rely on vocal traits to determine gender inflections @xcite . this, however, does not eliminate the bias toward masculine forms and is not inclusive for those individuals whose vocal properties do not align with their gender, such as people with vocal impairments, children, and transgenders @xcite . therefore, whenever the speaker's gender foot_0 is known (e.g. in talks or lectures), such information should be exploited to control gender translation and avoid relying on potentially misleading physical cues. so far, this topic has been investigated only by @xcite . their best solution consists in creating two gender-specific specialized models by fine-tuning a generic direct st system on sentences uttered by female/male speakers. though effective, this method has inherent limitations. first, it requires parallel audio-text data labeled with speakers' gender, which are scarcely available and costly to collect. second, the fine-tunings are computationally demanding as they involve processing audio data, which are much longer (∼8×) than their textual equivalents @xcite . to overcome these limitations, we propose the first inference-time solution in direct st to control gender translation for speaker-dependent words when the speaker's gender is known. 2 our approach guides gender translation by partially substituting the biased internal language model implicitly learned by the st decoder of a base model with a gender-specific external language model learned on monolingual textual data. through experiments on three language pairs (@xmath0, we demon-strate that, in terms of gender accuracy, our solution outperforms the base system by up to 31.0 points (for feminine forms) and is on par with the best training-time approach (with up to 1.6 of gain for feminine forms). its effectiveness is also confirmed when speakers' vocal traits conflict with their gender, with gains up to 32.0 and 3.4 over the base system and the best training-time solution. ## ilm/elm for gender translation the autoregressive decoder of an encoder-decoder architecture is trained to predict the next target token given the previous ones and the encoder output. thereby, it implicitly learns to model the target language from the training data, thus developing an internal language model (ilm) @xcite . we assume that, in a direct st model trained on unbalanced data where female speakers (and consequently feminine speaker-dependent words) are under-represented @xcite , the ilm is biased toward masculine forms. therefore, we propose to guide the generation of the st model with respect to speakerdependent words by substituting the biased ilm with a gender-specific external language model (elm). to this aim, we train two elms on monolingual text corpora (easy to collect, unlike labelled audio data) containing either feminine or masculine speaker-dependent words (see §3). at inference time, when we have prior knowledge of the speaker's gender from the metadata, we i) integrate the elm specialized in either masculine or feminine forms (depending on the speaker's gender) into the st model, and ii) (partially) remove the ilm contribution. the integration of end-to-end models with elms is a widespread solution to leverage text data in speech recognition @xcite @xcite . successful applications span from recognizing rare words @xcite to coping with out-of-vocabulary terms @xcite , domain adaptation @xcite and under-resourced conditions @xcite . however, to the best of our knowledge, elm integration has not been explored in the field of direct st, nor in the context of gender translation, as we do here. among the various methods proposed for the elm integration @xcite @xcite @xcite , we avoid those that require training-time interventions, and we resort to shallow fusion @xcite , an effective technique @xcite that consists in the log-linear combination of the posterior of the base model (p m b ) and the prior of the elm (p elm ). as regards the ilm removal, which previous studies already shown to amplify the performance gains yielded by elm integration @xcite @xcite , the most critical aspect is its estimation. in fact, since the ilm is implicitly modeled in the decoder, disentangling its contribution from the rest of the network is a challenging task @xcite @xcite . among the estimation methods demonstrated by @xcite to yield the best results, we select the global encoder average, as it does not require training-time interventions. this method computes the ilm prior (p ilm ) as: p ilm (y) = p m b decoder (y|c) namely, by feeding the st decoder with the average c of the encoder outputs h n,t over all the t n timesteps of the n training samples, where c is: therefore, given an audio input x, the output y of our solution is the translation y that maximizes the log-linear combination of p m b , p elm and p ilm : the three components (p m b , p elm , and p ilm ) convey different information: i) p m b embeds both the acoustic and the linguistic information learned from the st data; ii) p ilm represents the estimated linguistic knowledge learned by m b ; iii) p elm embeds linguistic information (in our case genderspecific forms) learned from external textual resources. therefore, β ilm and β elm must be set to values that effectively integrate the internal and external linguistic knowledge, so that the gender bias affecting the st decoder is mitigated by the elm. at the same time, the linguistic contribution supplied by the elm must not override the acoustic modelling capabilities of p m b , so as to avoid translation quality drops. accordingly, we estimate β ilm and β elm by optimizing the harmonic mean of the two metrics (gender accuracy and bleusee §3) used to measure gender bias and overall translation quality, so as to equally weigh our two objectives. in appendix a, we discuss the computation of β ilm and β elm values, also showing that their precise estimation is not critical since final results are rather robust to small weight variations. ## data and metrics our @xmath1. this resource includes a manual annotation of the speakers' gender @xcite , which is used to determine the gender translation of speakerdependent words. to train the elms, we collected gendercrawl, foot_2 a set of monolingual corpora for each target language and gender. each corpus is made of sentences with speaker-dependent words that clarify the speaker's gender (e.g., es: soy @xmath2: i am new to this area], es: debía ser fiel a mi @xmath3: i had to be true to myself ]). these sentences were automatically selected from paracrawl @xcite through regular expressions representative of morpho-syntactic patterns matching references to the first-person singular. additionally, we have also collected a validation set by applying the same regular expressions to the must-c training sets. the statistics of all these datasets are presented in we evaluate our systems on the ted-derived and gender-sensitive must-she benchmark @xcite . in particular, we focus on its "category 1", which contains from 560 to 607 sentences (depending on the target language) with speaker-dependent words annotated in the refer-ence. to assess gender translation, we use the official must-she evaluation script foot_4 , which produces two measures: i) term coverage, i.e. the percentage of annotated words that are generated by the system (disregarding their gender marking), and on which gender translation is hence automatically measurable, and ii) gender accuracy, i.e. the percentage of words generated in the correct gender among the measurable ones. lastly, overall translation quality is calculated with sacrebleu @xcite . 5 ## results for each language pair, we evaluate our approach by training: i) an st baseline model (m b ) that is not aware of the speaker's gender; ii) the specialized models (m sp ) presented in @xcite , re-implemented as upper bound to compare our inference-time solution with the best trainingtime approach; iii) the combination of m b with the gender-specific elms and the ilm removal (m b-ilm+elm ); iv) a variant of the approach, where the ilm is not removed (m b+elm ), serving as an ablation study to disentangle the ilm and elm contributions. detailed experimental settings and model description are provided in appendix b. ## conclusions we proposed the first inference-time solution to control gender translation of speaker-dependent words in direct st. our approach partially replaces the biased ilm of the st decoder with a genderspecific elm. as such, it can be applied to existing models without the need for labeled st data or computationally expensive re-trainings, overcoming the limitations of existing training-time methods. experiments on three language pairs proved the effectiveness of our technique in controlling gender inflections of words referring to the firstperson subject, regardless of whether the speakers' vocal traits are aligned with their gender or not. in addition to significantly increasing the gender accuracy of base st models, it achieves substantial parity with the best training-time method while consistently increasing the correct generation of feminine forms.