Cultural and Geographical Influences on Image Translatability of Words across Languages (2021)

Abstract: Neural Machine Translation (NMT) models have been observed to produce poor translations when there are few/no parallel sentences to train the models. In the absence of parallel data, several approaches have turned to the use of images to learn translations. Since images of words, e.g., horse may be unchanged across languages, translations can be identified via images associated with words in different languages that have a high degree of visual similarity. However, translating via images has been shown to improve upon text-only models only marginally. To better understand when images are useful for translation, we study image translatability of words, which we define as the translatability of words via images, by measuring intra- and inter-cluster similarities of image representations of words that are translations of each other. We find that images of words are not always invariant across languages, and that language pairs with shared culture, meaning having either a common language family, ethnicity or religion, have improved image translatability (i.e., have more similar images for similar words) compared to its converse, regardless of their geographic proximity. In addition, in line with previous works that show images help more in translating concrete words, we found that concrete words have improved image translatability compared to abstract ones.

## introduction neural machine translation (nmt) for lowresource languages has drawn a lot of attention due to the increasing awareness of the lack of linguistic and geographic diversity in nlp research @xcite . since parallel data for these languages is scarce, it necessitates the use of other data to help translation e.g., monolingual texts in unsupervised mt @xcite or images in multimodal mt @xcite . previous works on using images for translation typically accept that images are useful due to their language invariance @xcite . since everyday words such as chair denote concepts that exist independently of any language, images that ground their meanings should also be invariant to the language. however, to the best of our knowledge, this conjecture on image-language invariance has never been tested. as images' usefulness for translation has only been shown to be marginal @xcite @xcite , it is important to study this conjecture in relation to the characteristics of languages to understand when and to what extent images can aid translation. an alternative view would be that images may be different to some extent in different languages since they reflect the ways different people interact with these concepts; this may depend on where they live and the communities they live in @xcite . for example, images of the word breakfast in different languages may reflect the different cuisines of the communities that speak the languages. while most multimodal mt datasets are limited to a small set of european languages that come from the same language family, and are spoken by communities that are culturally and geographically close, the massively multilingual image dataset (mmid) @xcite is constructed specifically to facilitate large-scale multilingual research in translating words via images. mmid consists of up to 10k words and 100 images per word in 98 languages. this dataset provides an opportunity for us to examine how geographical and cultural relatedness between languages affect translation of words via images. as the use of parallel data from related languages have been found to improve mt for low resource languages @xcite @xcite , we want to study if the same extends to translation via images. specifi-cally, we want to explore if translatability of words between two languages via images is influenced by the cultural similarity and geographical proximity of their communities. a recent study, @xcite , has observed such correlations of culture and geography to semantic alignment of word meanings between languages that are measured through similarities in the word embeddings. we hypothesize that the same is true for images, and that the alignment of meanings conveyed via images coincides with culture and geography. in this work, we primarily define culture as the set of "language, norms and beliefs" of a community @xcite . these elements form our interpretation of cultural closeness between languages, which consist of their common linguistic, ethnic, and religious properties. our goal is to intrinsically evaluate to what extent images can aid in word translation, for words in languages close in a variety of these characteristics. assuming that each word is associated with a number of images that convey its meaning, we measure the degree to which images of words that are translations of each other in different languages have similar representations (thus will help in translation). we call this measure image translatability or the capacity of word meaning to be transferred from one language to another via images. if images are indeed language invariant, we should observe similar image translatability across different language pairs. we identify how close word translations are in terms of their image representations (embeddings). our findings suggest that languages with cultural similarity (defined as a combination of linguistic, ethnicity, or religious similarity of the communities at the cultural centres of the languages by glottolog @xcite ) coincides with their translatability via images, and that the translatability of languages with cultural similarity outperforms that in those with geographical proximity. our paper is structured as follows: in section 2 we discuss previous research on image-aided word translation, and how roots, geography and cultural characteristics of languages correlate with semantic alignment of words. in section 3 we describe our dataset and text-image corpora. we also introduce the language pairs we examine and estimate their closeness in culture and geography. in section 4, we present our approach for measuring translatability of words in terms of the similarity of their image representations. section 5 shows an analysis of our results and how translatability of words via images, which affects the images' fitness for translation, correlates with language properties. in section 6, we discuss noteworthy examples that illustrate our findings, before concluding in section 7. ## methodology given images that are associated with the word w s in the source language s, and w t in the target language t, we define two measures that determine how well a word can be translated by its images. the first measures whether w s and w t have overlapping or disjoint image embeddings. the second measures whether the spread of the image embedding for w s , and, similarly, for w t , is tight or loose: such a measure of image dispersion has been found to help predict the usefulness of image representations for translation @xcite . specifically, when images of w s and w t are tight and overlapping in the embedding space, it shows that the images have little diversity (low dispersion) and are similar between w s and w t , indicating potentially good translation between them. conversely, if the images are either spread out or disjoint, it means that the images have greater diversity (high dispersion) or differ between w s and w t , indicating potentially poor translation between them. we refer to the degree of overlap between two clusters of images associated with w s and w t respectively as their inter-cluster similarity, and to the degree of tightness or looseness of the images in each cluster as their intra-cluster similarity. our conjecture is that this is equivalent to representing image embeddings as samples from some generator distribution g. we can call the generator distribution for a given source word g s and a generator distribution for a given target word g t . two words are translations of each other when g @xmath0, and conversely two words are poor translations when g @xmath1. thus, inter-cluster similarity checks to see if an image embedding from g s could have been produced by g t . note that this is a necessary condition but it is not sufficient to say g @xmath2, because an image embedding from g s can also be produced by some random image embedding gen-erator g r with g @xmath3. intra-cluster similarity is a measure of how similar samples from a single generator are to each other. this will ensure that g t and g s are not random generators and are accurate representations of the word they are generating image embeddings for. in other words, having a high intra-cluster similarity implies that g @xmath4. thus, we have sufficient conditions when we have high inter-and intra-cluster similarities to say g @xmath5. ## results in order to compare the image translatability of two language pairs with different characteristics, we compare the ratio of the number of word pairs that are good translations divided by the total number of word pairs in each language pair: @xmath6,wt is lower than the homogeneity threshold h thld s,t and its median max cosine similarities i.e., medmax ws and medmax wt are higher than the thresholds medmax thld s and medmax thld t , respectively. the higher this ratio, the more translatable the language pair is via images. when we compare two language pairs that have the same source language but different target languages (with different shared characteristics with the source), we can distinguish how different characteristics such as cultural similarity or geographical proximity affect image translatability. in to understand the role of concreteness in translation via images, we also compute how many concrete words have good translations according to our image translatability measures. we consider a word pair, w s and w t , to be concrete if w s has a concreteness score greater than 3. source word concreteness is taken to as the pair concreteness, considering translation directionality. the ratio of how many concrete words in each language pair have good translations is also shown in to test whether the difference in image translatability between language pairs that share the same source language (e.g., finnish to norwegian vs. finnish to hungarian) is statistically significant, we conduct a simple t-test between their normalized score distributions. the resulting p-values signal the difference between their distributions. low p-values (< 0.05) indicate statistical significance and high variation between the distributions, while higher values suggest low variation and large similarities between the language pairs ( from the t-test, we find that the difference in distributions of pairs that share the same source language is almost in all statistically significant (p-@xmath7.05, of other pairs whose normalized score differences are statistically significant and whose differences in translatability ratios are high (boldfaced, language pair i language pair ii p-value az tr az ru 2.26 × 10 -25 ko zh ko ja 2.2 × 10 -4 zh ja zh ko 10.6 × 10 -5 ja zh ja ko 0.43 ar ur ar fa 9.44 × 10 -29 ar fa ar he 2.39 × 10 -13 ar he ar ur 16.35 × 10 -5 es fr es pt 4.3 × 10 -47 fi hu fi no 2.66 × 10 -104 af nl af sw 10 -100 on the contrary, korean to japanese and korean to chinese pairs, and chinese to korean and chinese to japanese pairs, have at least one difference in their attributes, accounting for the pairs' results statistical significance. in addition, we observe that concreteness of words largely affects the quality of translations due to the low diversity in its image representations, which facilitate translation between words. on average, across language pairs, 62.4% of words with normalized scores above the threshold are concrete, while only 37.6% are abstract. at the same time, in ## discussion our work has identified that language relatedness affect word translations via images. we observe that languages with cultural relatedness have better image translatability; suggesting that cultural relatedness should be taken into account when using images to aid translation. the image translatability measures we have defined can be used to identify a potentially good or poor translation or discover a cultural similarity or disconnect between words in two languages. for example, a word pair that has a high intra-cluster similarity and a high inter-cluster similarity in their image representations indicates that the image clusters are tight and overlapping, signaling a good translation between them. for example, the word heelal in afrikaans and the word universum in dutch have tight and overlapping image clusters (i.e., low homogeneity score) as can be seen in the pca plot of their image embeddings and in their images (figure 2 ). on the other hand, when a word pair has tight but disjoint image clusters, it can mean that their images express different meanings of the word. for example, the word dance: dans in afrikaans and the word kucheza in swahili have tight but disjoint image clusters (i.e., high homogeneity score) as can be seen in the pca plot of their image embeddings and in the images (figure 3 ), as kucheza means both to play and to dance in swahili. we observe that afrikaans, for example, has a higher image translatability to dutch due to their cultural (ethnolinguistic) similarity, than to swahili, despite the relative distances of their cultural centers-south africa is more distant geographically to the netherlands than to tanzania, defined in glottolog as being the cultural center of the swahili language. we observe higher visual similarities between words that are translations of each other in afrikaans and dutch than in afrikaans and swahili. for example, images of the word park in afrikaans are more visually similar to images of the word park in dutch than to images of the word park in swahili (hifadhi) (figures 4 , 5 ). the images of park in afrikaans and in dutch refer to a western style park, while its images in swahili refer more to a wildlife reservation, a culturally different representation of the word park that is potentially influenced by how speakers of the different languages interact with the concept of the word. interestingly, such connotation that is apparent in images may not be apparent in word embeddings, since hifadhi is used similarly with park in the texts of the language. ## conclusions in this paper, we study when images may be useful for translating words between two languages from the perspective of their cultural and geographical relatedness. we observe that translatability of words via images vary in different language pairs, with language pairs sharing cultural similarities having better image translatability. in the future, it will be interesting to study image translatability of more language pairs and their characteristics, including those outside mmid, as well as extend our work to sentence-level image aided translation. it will also be of great value to study if adding considerations of cultural relatedness to image-aided mt can further improve its performance. additionally, using a different metric for intra-cluster similarity that does not calculate similarity with respect to the origin may be more accurate depending on the application. as many similarity functions, aside from cosine similarity, have been used in the computer vision literature, improving this function could be fruitful future work.