Assessing Multilingual Fairness in Pre-trained Multimodal Representations (2022)

Abstract: Recently pre-trained multimodal models, such as CLIP, have shown exceptional capabilities towards connecting images and natural language. The textual representations in English can be desirably transferred to multilingualism and support downstream multimodal tasks for different languages. Nevertheless, the principle of multilingual fairness is rarely scrutinized: do multilingual multimodal models treat languages equally? Are their performances biased towards particular languages? To answer these questions, we view language as the fairness recipient and introduce two new fairness notions, multilingual individual fairness and multilingual group fairness, for pre-trained multimodal models. Multilingual individual fairness requires that text snippets expressing similar semantics in different languages connect similarly to images, while multilingual group fairness requires equalized predictive performance across languages. We characterize the extent to which pre-trained multilingual vision-and-language representations are individually fair across languages. However, extensive experiments demonstrate that multilingual representations do not satisfy group fairness: (1) there is a severe multilingual accuracy disparity issue; (2) the errors exhibit biases across languages conditioning the group of people in the images, including race, gender and age.

## introduction recently pre-trained vision-and-language representations @xcite @xcite @xcite @xcite @xcite @xcite have received a surge of attention. such pre-trained multimodal representations have shown great capabilities of bridging images and natural language on the downstream tasks, including image captioning @xcite , im-age retrieval @xcite , visual qa @xcite ), text-to-image generation @xcite , etc. while it is commonly recognized that the multimodal representations trained on english corpora can be generalized to multilingualism by cross-lingual alignment @xcite , recent studies criticize that the multilingual textual representations do not learn equally high-quality representations for all the languages @xcite , especially for low-resource languages. @xcite emphasize the need for general-purpose representations to seek equal performance across all languages. however, there is still a lack of a nuanced understanding of how multilingual representations fare on vision-and-language benchmarks. this paper provides a novel perspective for analyzing the principles of multilingual fairness in multimodal representations from two aspects. first, existing frameworks for measuring multilingual biases usually emulate text sources in different languages, which may have ambiguous meanings in varied contexts @xcite ). in contrast, we leverage visual grounding as the anchor to bridge text in different languages-text snippets in different languages but with similar semantics should be equitably relevant to the same images. second, we equate a language as an aggregated group of individuals (e.g., french as a group of french sentences) in the terminology of fairness. as @xcite has pointed out, "each language has a distinct identity, defined by its vocabulary, syntactic structure, its typological features, amount of available resources, and so on." the notions of fairness, such as individual fairness @xcite and group fairness @xcite @xcite , can be naturally adapted by comparing the multimodal model's treatment across languages. therefore, we introduce two fairness notions: multilingual individual fairness presumes similar outcomes between similar language expressions grounding on the same images; multilingual group fairness postulates that multimodal models should induce similar predictive performance across different languages. these fairness notions are formalized to compare the multimodal model's treatment of one language versus another for either the individual target or the aggregated group. our contributions are as follows: ## background notation. throughout the paper, we use the uppercase letter i to denote images and t to denote text. we use the superscript (l) in t (l) to represent the text is in language l. when we are jointly using t (l) and t (l ) for two languages l and l , we often assume that they share the same semantic meanings. lowercase letters v and t are used to denote the visual and textual representation vectors encoded by model m , respectively. to simplify the presentation, we use s(•, •) to generally represent the similarity between images and text. specifically, s(i, t ) refers to the similarity scores predicted by the model m between the image i and text t , while s(v, t) refers to the cosine similarity between vectors v and t. ## multilingual individual fairness for an ideal multilingual vision-and-language model, text descriptions in different languages referring to similar semantic meanings should be equally similar or dissimilar to the same grounding images. we note that there are no language expressions that are perfectly identical to each other in real-world scenarios due to linguistic features. nevertheless, at least in a normal vision-and-language task, multilingual models are desired to impose equal treatment to different languages. for instance, "this is a cat" (in english) and "das ist eine katze" (in german) should be similarly related to an image of a cat in image-text retrieval. this intuition aligns with individual fairness in a multilingual manner. in this section, we investigate to what degree multilingual representations are individually fair. individual fairness requires that similar people should be treated similarly @xcite . in our multilingual setting, we require that the text snippets expressing similar semantics in different languages should be similarly related to the same images. taking the euclidean distance function to measure the distance between text features, we can define α-multilingual individual fairness by: given a set of image-text pairs {(i, t )}, a multimodal model m satisfies α-multilingual individual fairness if for all (i, t ), for languages l and l : here, α is a parameter to control the ratio of similarity gap to the text feature vectors' distance, and smaller α indicates the model is individually fairer. note that the similarity gap is at most 2, because the range of cosine similarity is [0, 1]. in general settings, s(i, t ) is measured by the cosine similarity between the encoded visual vector v and textual vector t. lemma 1. denote o ρ (t) = {x | @xmath0. then for any visual representation vector v, sup we defer the proof to appendix a.1. lemma 1 implies that when the distance between multilingual textual representation vectors is bounded, the similarity with images can be bounded in terms of their distance. it is worth noting that the bounds are independent of the visual representation vectors. nevertheless, the form of upper bound in theorem 1 is a bit sophisticated, and can be simplified when ρ l) . theorem 2 is a direct application of lemma 1 when the distance between multilingual vectors is small enough, and extends in many natural cases to approximate the multilingual individual fairness with α ≈ 1 t (l) . the proof can be found in appendix a.2. theorem 2 implicates to what degree the multimodal model satisfies individual fairness when text snippets are well aligned between different languages. ## multilingual group fairness distinct from individual fairness, multilingual group fairness appeals to the idea that multimodal models should achieve equivalent predictive performance across different languages. from the perspective of representations, it is hard to carry out this demand without well-defined tasks and metrics. hence it is natural to ask how to define group fairness in this scenario properly? in this section, we shall answer this question by equating language as a unique dimension of group membership relating to the text modality. we formulate the criteria by equalizing the accuracy rates over different languages. we also observe that images are often connected to people in protected or unprotected groups. given the image-text pairs, we consider the accuracy disparity across different languages conditioning the subgroup of images. ## evaluations in this section, we work with the pre-trained multilingual clip @xcite ) model to study multilingual fairness. we validate the extent to which the model is individually fair across different languages in section 5.1. we characterize the prevalence of multilingual group unfairness on human faces in section 5.2 and section 5.3. these empirical evaluations shed light on potential directions for mitigating unfairness in multilingual multimodal representations. ## conclusion and limitation our work extends a growing body of fairness discourse in multilingual and multimodal learning to explore how the multilingual fairness notions, characterized by individual fairness and group fairness, are formulated on the multimodal representations. we stress that multimodal representations are individually fair, but do not prevent accuracy disparity across groups. our extensive experimental results reveal the negative impacts caused by carelessly applying pre-trained general-purpose multimodal representations. just one example of this, as discussed in section 5.3, is the significant disparities between cross-lingual gender gaps occurred in age classification. we believe the findings and insights gained through this work can encourage future work to investigate how to mitigate multidimensional biases in representation learning and prevent disparities in the downstream decision-making process. our work also has limitations. this work does not provide a thorough explanation on whether the biases and disparities result from the multilingual model itself, or from the datasets it is pre-trained on. however, to give a convincing explanation, it requires either access to large amounts of training data with privacy concerns (the complete datasets for training clip are not released yet), or ample computational resources for reproducing the training process. this research question itself is important and worth investigating further.