Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment (2022)

Abstract: Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods. Recent findings have shown the improvement in accuracy and robustness of unsupervised word translation (UWT) by utilizing visual observations, which are universal representations across languages.Our work investigates the potential of using not only visual observations but also pretrained language-image models for enabling a more efficient and robust UWT. We develop a novel UWT method dubbed Word Alignment using Language-Image Pretraining (WALIP), leveraging visual observations via the shared image-text embedding space of CLIPs (Radford et al., 2021). WALIP has a two-step procedure. First, we retrieve word pairs with high confidences of similarity, computed using our proposed image-based fingerprints, which define the initial pivot for the alignment.Second, we apply our robust Procrustes algorithm to estimate the linear mapping between two embedding spaces, which iteratively corrects and refines the estimated alignment.Our extensive experiments show that WALIP improves upon the state-of-the-art performance of bilingual word alignment for a few language pairs across different word embeddings and displays great robustness to the dissimilarity of language pairs or training corpora for two word embeddings.

## introduction translating words across different languages is one of the long-standing research tasks and a standard building block for general machine translation. word translation is helpful for various downstream applications, such as sentence translation @xcite or cross-lingual transfer learning in language models (de vries and nissim, 2020). unsupervised word translation (uwt) has recently drawn a great deal of attention (artetxe we can connect english and french words in an unsupervised fashion through the shared images. clip models @xcite can be used as human simulators to associate words with images. @xcite @xcite , reducing the need for bilingual supervision. without any prior knowledge of the languages' connection, aligning their words is non-trivial. most works on uwt exploit the structural similarity between continuous word embedding spaces across languages @xcite to learn a linear mapping. early works @xcite @xcite @xcite focus on using only the text data to establish the bilingual alignment and solve the procrustes problem @xcite . these methods rely on the similarity between pairs of languages and training corpora, thus not working well when the languages or corpora are dissimilar @xcite . they may also need a large amount of data to achieve good alignments @xcite . words can also be connected via the visual world. visual similarity provides additional prior knowledge for easing language translation @xcite . recent works @xcite demonstrate the promise of using visual information to improve uwt. however, they mostly require intense joint training for the embedding shared between images or videos with texts of multiple languages. more-over, these embeddings are used for translating all words, whereas not every word can be described by images or videos. thus, it is unclear how they are helpful for non-visual words and whether the methods properly utilize topological similarity between word vector spaces @xcite . ## problem setup and preliminaries we formally describe the target problem of unsupervised word alignment and provide two preliminaries to our method: procrustes and csls. unsupervised word alignment. we focus on the word alignment (translation) problem: finding the mapping from a dict to b dict , where here, [n] = {1, 2, • • • , n} is defined as the set of positive integers up to a positive number n. note that we focus on unsupervised word alignment in which no ground-truth word pairs (a i , b π(i) ) are given to the algorithm. to solve this problem, we assume the access to three ingredients: (1) a large-scale image dataset with d images denoted by @xmath0, • • • , g d }, (2) a pre-trained monolingual clip model for each language, and (3) static word embeddings @xcite for all words in dictionaries. procrustes problem. let x, @xmath1. the procrustes problem aims to find @xmath2. regularizing w with the orthogonality is found to improve the translation @xcite , where the optimal w is w * = argmin ## walip we first provide the high-level idea and then specify each stage of walip. algo. 2 in appendix presents the pseudocode for our algorithm. ## experiments we evaluate walip on bilingual alignment tasks. sec. 5.2 compares walip and baselines in multiple language pairs. the following sections provide additional experimental results that either highlight the benefits of walip or help understand the component that enables the high performance of walip. our code is available at https: //github.com/uw-madison-lee-lab/walip . evaluation. we evaluate methods on the dictionary datasets (sigurdsson et al., 2020), which are test sets used in the muse benchmark @xcite . each dictionary is a set of translation pairs where each word in the source language may have multiple translations in the target language. we report recall@n used in @xcite , which presents the fraction of source words correctly translated. a retrieval is correct for a given query if at least one of n retrieved words is the correct translation. by default, we report recall@1, which is equivalent to precision@1, and the accuracy used in @xcite . baselines. our baselines include the videogrounding method @xcite and the image-grounding method globetrotter @xcite . we also compare our method with two versions of the text-only method muse @xcite : the default one trained on the dictionary dataset (with 1.5k-3k words per dictionary), and the other one trained on the muse training data (with 200k words per dictionary); we call the latter one as muse (extravocabulary). we also consider a simple baseline using clip, denoted by clip-nn, which performs 1-nearest neighbor (1-nn) based estimation on the embedding spaces of two clip models: we first find the image nearest to the source word, and then find the target word nearest to the image found in the first step. for measuring recall@n of this baseline, we replace 1-nn with ⌈ √ n⌉-nn. ## conclusion we propose walip, a novel unsupervised bilingual word alignment method using pretrained clip models. walip first leverages the visual similarity between words as the auxiliary for matching initial and simple word pairs via the image-based fingerprint representation computed by language-image pretraining models. then walip uses these initial pairs as pivots to learn the linear transformation between two static word embeddings. we introduce a robust procrustes algorithm based on errorweighting to estimate the linear mapping. compared with existing baselines, walip needs less computation for aligning two embeddings, thanks to the aid of visual information and pretrained clip models. walip achieves the sota alignment performances on several language pairs across word embedding types, especially for pairs in which two languages are highly dissimilar. walip also displays the robustness against the dissimilarity of static word embeddings' training corpora.