Paper: Cultural and Linguistic Diversity Improves Visual Representations (ICLR-2024)

Reviewer comment (Experiments):
The use of machine translation systems to have a working set of data that is only in English is a major weakness of the work. There is no explanation of possibly failed attempts to use non-English parsers or text embedding. Given the lack of meaningful discussion on this part, it is unclear whether this could in fact work using the non-English languages directly.

Author response:
1. “What efforts did you make to conduct your analysis in the original languages?”

The goal of our paper is to evaluate differences in caption content. We use a range of metrics to measure qualities of content. Some of these are implemented only in English. For instance, the LIWC metrics [1] are hand-crafted by linguists and therefore difficult to reproduce in other languages. In these cases, it is not possible to use non-English data. Other metrics have multilingual implementations, but they are not language agnostic. For instance, even advanced multilingual embeddings have been shown to encode language-specific information, especially typology [2, 3, 4, 5]. Implementations of linguistic parsers across languages vary widely in implementation and performance because their design depends on features of that language [6, 7]. Neural-based approaches for parsing multiple languages with one model usually rely on translation anyway [8]. In sum, we found analysis in original languages to be technically infeasible and introducing too many confounders, compared to translating to English.

Reviewer comment (Evaluation):
The main claims of the paper are about how different languages can be shown to express things differently. However, the only example of the differences between English and non-English text, shown in Table 16 in the Appendix, has two dubious examples of the type of diversity found in translated versions of other languages. From these examples and the comments, it looks like the non-English captions include entities that are synonymous with other English entities.

Author response:
2. “What are the main types of differences driving the differences in different non-English translated text?”

We agree that the examples in Table 16 on ground truth coverage in Visual Genome are not so strong. We have since updated that figure with several more representative examples. We do suggest looking at Figure 2, which gives some examples of English and non-English captions and demonstrates that captions in different languages are mentioning different objects.

It was hard for us to empirically validate clear, human-interpretable patterns between languages, although we do have many hypotheses we are testing for future work. For instance, some languages seem to mention objects in the background with greater frequency or mention certain types of attributes more often, which appears to align with existing work in cross-cultural psychology. See additional examples we have added in Tables 16, 19, 20 for more samples. 

The main contribution of our work is to show substantive differences between the content of captions in different languages across many metrics, and we hope future work can investigate explanations.

Reviewer comment (Experiments):
What efforts did you make to conduct your analysis in the original languages? In Section 3.1, you write "we require the consistent application of linguistic tools, such as parsers, which are more mature for English than other languages".

Author response:
1. “What efforts did you make to conduct your analysis in the original languages?”

The goal of our paper is to evaluate differences in caption content. We use a range of metrics to measure qualities of content. Some of these are implemented only in English. For instance, the LIWC metrics [1] are hand-crafted by linguists and therefore difficult to reproduce in other languages. In these cases, it is not possible to use non-English data. Other metrics have multilingual implementations, but they are not language agnostic. For instance, even advanced multilingual embeddings have been shown to encode language-specific information, especially typology [2, 3, 4, 5]. Implementations of linguistic parsers across languages vary widely in implementation and performance because their design depends on features of that language [6, 7]. Neural-based approaches for parsing multiple languages with one model usually rely on translation anyway [8]. In sum, we found analysis in original languages to be technically infeasible and introducing too many confounders, compared to translating to English.

Reviewer comment (Evaluation):
What are the main types of differences driving the differences in different non-English translated text? What types of objects or entities are being created? The examples in Table 16 are not so convincing: "alleyway" can be synonymous with "street", while "screen" can be synonymous with "monitor".

Author response:
2. “What are the main types of differences driving the differences in different non-English translated text?”

We agree that the examples in Table 16 on ground truth coverage in Visual Genome are not so strong. We have since updated that figure with several more representative examples. We do suggest looking at Figure 2, which gives some examples of English and non-English captions and demonstrates that captions in different languages are mentioning different objects.

It was hard for us to empirically validate clear, human-interpretable patterns between languages, although we do have many hypotheses we are testing for future work. For instance, some languages seem to mention objects in the background with greater frequency or mention certain types of attributes more often, which appears to align with existing work in cross-cultural psychology. See additional examples we have added in Tables 16, 19, 20 for more samples. 

The main contribution of our work is to show substantive differences between the content of captions in different languages across many metrics, and we hope future work can investigate explanations.

Reviewer comment (Experiments):
Why are the final conclusions reliable when you have used a machine translation tool to convert the non-English text into English text?

Author response:
4. “Why are the final conclusions reliable when you have used a machine translation tool to convert the non-English text into English text?”

We run a user study and show that native speakers find that the content of languages is preserved across translation (see Appendix A.2). We ensure that objects, their attributes, and relations between them are preserved. Consider the example of a bird in the background. A caption which mentions a bird will result in a translation which has a bird, and a caption which does not will result in a translation which does not have a bird. Therefore, we believe the signal we are measuring (the presence of objects, attributes, relations) is not likely to be sensitive to artifacts introduced during translation.

Reviewer comment (Writing):
What are the actual benefits of the Google Vertex API? The paper describes that the "outputs reflect established standards and have gone through quality checks" but I don't understand what that is supposed to mean.

Author response:
5. “What are the actual benefits of the Google Vertex API?”

Google Vertex API represents one popular commercial solution to multilingual image captioning. By "outputs reflect established standards and have gone through quality checks", we simply mean that Google has done extensive testing to ensure the model’s behavior [13] and that it is a worthy benchmark to analyze.

While we were running the experiments, we struggled to find comparable, high-quality, and open-source multilingual image captioning models. Since then, we have gained access to GPT-4V and added several examples in the appendix – see Table 19 in Appendix D.9.

Reviewer comment (Evaluation):
Certain findings presented in the paper may be contextually constrained by the scale of the experiments conducted. For instance, in the case of the fine-tuning experiment, it is plausible that the results observed may not extrapolate seamlessly to scenarios of much larger scale, such as foundation model pretraining. Given this potential limitation, a more cautious and nuanced articulation of these conclusions might be more suitable.

Author response:
We appreciate this note. We have modified our language and claims to be more nuanced throughout the paper. We are currently working in our future work on applying these results at a large scale over LAION.

Reviewer comment (Experiments):
Some segments of the research are founded on datasets of relatively modest size. For instance, the dataset analysis is based on 30 descriptions across two languages (English & Japanese), contributed by 10 individuals per language. This sample size may not fully encapsulate the breadth of diversity and nuance inherent to linguistic expression, potentially limiting the representativeness of the findings.

Author response:
Indeed, our user study recruiting American and Japanese participants to caption images was fairly limited in size. However, it should be noted that this user study was not a main result but rather an attempt to reproduce some of the behavior observed in cross-cultural psychology experiments (e.g. [1]). We base our claims off of the fundamental results of the paper, which are conducted over fairly large datasets. We analyze [3600 images × 7 languages × 3 captions each × 3 sources (original dataset, Vertex API, LLaVA) = 226,800] captions in CrossModal and [2000 images × 7 languages × 3 captions each × 3 sources (original dataset, Vertex API, LLaVA) = 126,000] captions in Visual Genome. However, we are pursuing future work to determine if this still holds in even more general contexts, such as web-scraped datasets like LAION.

We hope this helps clarify! Please let us know if there are any other comments or questions which may arise. If we addressed your concerns, we would appreciate an increase in your score.

– the authors




References

[1] Masuda, T., & Nisbett, R. E. (2001). Attending holistically versus analytically: comparing the context sensitivity of Japanese and Americans. Journal of personality and social psychology, 81(5), 922–934. https://doi.org/10.1037//0022-3514.81.5.922

Reviewer comment (Evaluation):
Regarding the diversity of the scene graph, I'm curious if issues in normalization could give the impression that multilingual descriptions contain more concepts. Even though you've included a step for canonicalization, can you confirm whether this process accurately aligns 100% of identical concepts? Is it possible that some concepts from multilingual sources, once translated to English, are interpreted as slightly different, leading to inflated diversity scores?

Author response:
This is certainly a possibility, and we appreciate the reviewer for bringing it up. Concept alignment will always be a messy process. As such, we designed our canonicalization process to err on the side of grouping two concepts together as opposed to apart. In our canonicalization process, we first evaluate WordNet similarity (the commonly used procedure). If it fails, we evaluate similarity of embeddings. If even this fails, we look towards other associated information (e.g. attributes and relations from objects) to make a match if possible, with even lower thresholds for similarity. Therefore, we believe that our results stand despite, rather than in part because of, irreducible error in concept alignment.

Reviewer comment (Evaluation):
On a related note, I tend to think that translation inherently introduces linguistic diversity, as it's rarely perfectly accurate. For instance, back translation is a popular method for data augmentation, despite the expectation that the core semantic content should remain consistent. Could you address the possibility that the observed differences in your study are simply artifacts of the translation process? I understand that human evaluation was conducted for GPT-4 translations, but subtle nuances may still have been overlooked.

Author response:
It is true that translation introduces some minimal amount of error or diversity into text. However, we are principally concerned with the content (objects, attributes, relations) of captions, which we confirmed in our human studies are preserved across translation. Consider the example of a bird in the background. A caption which mentions a bird will result in a translation which has a bird, and a caption which does not will result in a translation which does not have a bird. Therefore, we believe the signal we are measuring (the presence of objects, attributes, relations) is not likely to be sensitive to artifacts introduced during translation.

It is possible that translation artifacts may have a larger effect on the representational and linguistic diversity measures, which evaluate over raw text rather than scene graphs, which are parsed for content. However, observe from Tables 4 and 5 that the diversity metrics across all 7 languages increases substantially over the diversity metrics from groups of 3 languages. If the text was more or less the same and translation was introducing most of the diversity, we would not expect as much of an increase. Likewise, observe that the diversity metrics for groups of 3 non-English (i.e. all translated) languages are generally similar to the diversity metrics for groups of 1 English and 2 non-English languages. All of this is to suggest that although there may be an irreducible translation error, we have good reason to believe it does not significantly affect our diversity metrics.