Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation (2024)

Abstract: End-to-end speech translation (ST) presents notable disambiguation challenges as it necessitates simultaneous cross-modal and cross-lingual transformations. While word sense disambiguation is an extensively investigated topic in textual machine translation, the exploration of disambiguation strategies for ST models remains limited. Addressing this gap, this paper introduces the concept of speech sense disambiguation (SSD), specifically emphasizing homophones - words pronounced identically but with different meanings. To facilitate this, we first create a comprehensive homophone dictionary and an annotated dataset rich with homophone information established based on speech-text alignment. Building on this unique dictionary, we introduce AmbigST, an innovative homophone-aware contrastive learning approach that integrates a homophone-aware masking strategy. Our experiments on different MuST-C and CoVoST ST benchmarks demonstrate that AmbigST sets new performance standards. Specifically, it achieves SOTA results on BLEU scores for English to German, Spanish, and French ST tasks, underlining its effectiveness in reducing speech sense ambiguity. Data, code and scripts are freely available at https://github.com/ytf-philp/AmbigST.

## introduction speech translation (st) translates acoustic speech from one language to another and sees a steady increase in its application @xcite @xcite @xcite . particularly, recent research involves the development of unified end-to-end st models, leveraging the joint pre-training of speech and text @xcite @xcite @xcite . ## speech sense disambiguation typically, a speech translation corpus includes triples of speech-transcription-translation, denoted as @xmath0, x , y). we leverage transcription x as the basis to explain the notion of speech sense ambiguity. consequently, we curate a dedicated dataset that embodies homophone information, thereby providing valuable resources to tackle the issue of ambiguity in st tasks. problem definition homophones, for example, "i" versus "eye" and "would" versus "wood", are prevalent linguistic phenomena @xcite . ssd's process entails discerning the specific meaning of a homophone, which shares its pronunciation with other words but carries a distinct meaning. given the auditory context a, the goal of ssd is to identify the correct speech sense ê from a defined set of potential senses e for a given speech sequence s. this process can be mathematically formulated as: annotating the dataset figure 1 provides the process of constructing a homophone dictionary. we use transcriptions x to construct a homophone dictionary by identifying sets of words that possess identical phonemes, facilitated by the utilization of the montreal forced aligner @xcite , an open-source tool engineered for the accurate alignment of speech with its corresponding orthographic transcription. the homophone dictionary consists of sets of words that share the same pronunciation. for instance, the phonemes of "p eh r" would include terms such as pear, pair. similarly, the phonemes of "p l ey n" would encompass terms like plain, plane. the annotated dataset, organized in quintuples, highlights ambiguous words and their positions within sentences. a detailed example is presented in ## analysis 6.1 effect of homophone dictionary size to evaluate the robustness of our approach against varying sizes of homophone dictionaries, we conducted experiments using the english homophone dictionary segmented into distinct scales {0.3, 0.5, 0.7}. we reconstituted the annotated en-de dataset to determine the impact of homophone dictionary size on ambigst performance. as depicted in figure 3 , our method consistently enhances performance across different dataset sizes, demonstrating its efficacy and stability. as the size of the homophone dictionary increases, there is a correspond-ing improvement in overall performance, affirming the critical influence of dictionary size. by employing our fully constructed homophone dictionary, we achieved peak performance across all metrics, thereby confirming the precision and effectiveness of our homophone dictionary construction process. ## conclusion to mitigate speech sense ambiguity in st models, this paper represents a pioneering effort that involves the construction of comprehensive homophone dictionaries for en, fr, de, es and six language pair-annotated datasets that include pertinent homophone information. in the wake of these advancements, we introduce ambigst, a novel homophone-aware contrastive learning approach that incorporates a homophone-aware masking strategy operating at the token, sentence, and model levels. experimental results on the must-c and covost benchmark demonstrate that our proposed ambigst approach can leverage speech sense disambiguation efficiently.