First the Worst: Finding Better Gender Translations During Beam Search (2022)

Abstract: Generating machine translations via beam search seeks the most likely output under a model. However, beam search has been shown to amplify demographic biases exhibited by a model. We aim to address this, focusing on gender bias resulting from systematic errors in grammatical gender translation. Almost all prior work on this problem adjusts the training data or the model itself. By contrast, our approach changes only the inference procedure. We constrain beam search to improve gender diversity in n-best lists, and rerank n-best lists using gender features obtained from the source sentence. Combining these strongly improves WinoMT gender translation accuracy for three language pairs without additional bilingual data or retraining. We also demonstrate our approach’s utility for consistently gendering named entities, and its flexibility to handle new gendered language beyond the binary.

## introduction neural language generation models optimized by likelihood have a tendency towards 'safe' word choice. this lack of output diversity has been noted in nmt @xcite and throughout nlp @xcite . model-generated language may be repetitive or stilted. more insidiously, generating the most likely output based only on corpus statistics can amplify any existing biases in the corpus @xcite . potential harms arise when biases around word choice or grammatical gender inflections reflect demographic or social biases @xcite . the resulting gender mistranslations could involve implicit misgendering of a user or other referent, or perpetuation of social stereotypes about the 'typical' gender of a referent in a given context. past approaches to the problem almost exclusively involve retraining (vanmassenhove et al., * now at rws language weaver 2018; escudé @xcite or tuning @xcite on gender-adjusted data. such approaches are often computationally expensive and risk introducing new biases @xcite . instead, we seek to improve translations from existing models. @xcite highlight beam search's tendency to amplify gender bias and @xcite show that very shallow beams degrade gender translation accuracy; we instead guide beam search towards better gender translations further down the n-best list. our contributions are as follows: we rerank nmt n-best lists, demonstrating that we can extract better gender translations from the original model's beam. we also generate new n-best lists subject to gendered inflection constraints, and show this makes correctly gendered entities more common in n-best lists. we make no changes to the nmt model or training data, and require only monolingual resources for the source and target languages. ## finding consistent gender in the beam there are two elements to our proposed approach. first, we produce an n-best list of translations using our single model per language pair. we use either standard beam search or a two-pass approach where the second pass searches for differently-gendered versions of the highest likelihood initial translation. we then select a translation from the list, either by log likelihood or by how far the target language gender features correspond to the source sentence. the second reinflection pass uses a target language gender inflection transducer which defines grammatically gendered reinflections. for example, spanish definite article el could be unchanged or reinflected to la, and profession noun médico could be reinflected to médica (and vice versa). composing the reinflections with the original hypothesis generates a constrained hypothesis lattice. we can now perform constrained beam search, which can encourage nmt to output specific vocabulary @xcite . the only difference from standard beam search is that gender-constrained search only expands translations forming paths in the constrained hypothesis lattice. in the figure 1 example, beamn search would produce the n most likely translations, while the gender-constrained pass would only produce the 4 translations in the lattice. importantly, for each language pair we use just one nmt model to produce gendered variations of its own hypotheses. unlike saunders and byrne (2020) we do not reinflect translations with a separate gender-sensitive model. this removes the complexity, potential bias amplification and computational load of developing the gender-translationspecific models central to their approach. while we perform two full inference passes to simplify implementation, further efficiency improvements are possible. for example, the source sentence encoding could be reused for the reinflection pass. in principle, some beam search constraints could be applied in the first inference pass, negating the need for two passes. these potential efficiency gains would not be possible if using a separate nmt model to reinflect the translations. ## experimental setup we translate english into german, spanish and hebrew using transformers @xcite . we train the en-de model on wmt19 newstask data including filtered paracrawl @xcite , en-es on uncorpus data @xcite , and en-he on the iwslt corpus @xcite . ## conclusions this paper attempts to improve gender translation without a single change to the nmt model. we demonstrate that gender-constraining the target language during inference can encourage models to produce n-best lists with correct hypotheses. moreover, we show that simple reranking heuristics can extract more accurate gender translations from the n-best lists using oracle or inferred information. unlike other approaches to this problem we do not attempt to counter unidentified and potentially intractable sources of bias in the training data, or produce new models. however, our approach does significantly boost the accuracy of a prior datacentric bias mitigation technique. in general we view our scheme as orthogonal to such approaches: if a model ranks diverse gender translations higher in the beam initially, finding better gender translations during beam search becomes simpler.