CAT M any N ames: A New Dataset for Object Naming in C atalan (2022)

Abstract: Object Naming is an important task within the field of Language and Vision that consists of generating a correct and appropriate name for an object given an image. The ManyNames dataset uses real-world human annotated images with multiple labels, instead of just one. In this work, we describe the adaptation of this dataset (originally in English) to Catalan, by (i) machine-translating the English labels and (ii) collecting human annotations for a subset of the original corpus and comparing both resources. Analyses reveal divergences in the lexical variation of the two sets showing potential problems of directly translated resources, particularly when there is no resource to a proper context, which in this case is conveyed by the image. The analysis also points to the impact of cultural factors in the naming task, which should be accounted for in future cross-lingual naming tasks.

## introduction most nlp resources are only available for a small percentage of languages @xcite , being the rest of the languages spoken in the world left behind. this affects also catalan, which can be considered a moderately under-resourced language @xcite . in the language and vision area, although significantly large datasets of annotated images have been created for a variety of tasks for english, to date no resources of this kind exist for catalan. in this work we present cat manynames foot_0 , the catalan version of the manynames dataset, which is the first available resource for the task of object naming in catalan. the dataset has been translated from the english version and its test set has been human annotated to assess the quality of the translation. we also provide analyses of the sources of variation between the human annotated dataset and its translated counterpart. while the task of object naming has been studied in both language and vision and psycholinguistics, and it is related to object recognition tasks in computer vision, each field has a different approach to the task: within the field of language and vision, datasets typically collect free and natural referential utterances 3 produced by annotators for a given realworld image. some relevant datasets are refcoco (and its newer variant refcoco+) @xcite , and vi-sualgenome @xcite . although naming occurs within those datasets, it is not normally marked up and linked to its corresponding image regions. the task of picture naming constitutes an important experimental paradigm on research in cognitive science and psycholinguistics, and has been traditionally used to assess language impairments and difficulties recalling general knowledge from semantic memory @xcite . subjects reach a high agreement in this task, but it must be taken into account that participants are normally shown line drawing pictures that depict a prototypical category rather than real-world images that show objects in a context. in computer vision, the task of object recognition identifies and classifies objects into several different categories @xcite . nevertheless, current recognition benchmarks use labels and images from imagenet @xcite that assume a single ground-truth label, ignoring linguistic variation. as we can see, the task of object naming is addressed differently in cognitive science, language and vision and computer vision, but it would highly benefit from bringing together the particularities of each field so as to generate and provide quality resources. ## a new dataset for object naming in catalan the main motivations for using the manynames dataset as source data are (i) its consideration of linguistic variation in object naming, which is widely ignored up to now in computer vision, and (ii) the better accuracy that has shown to perform against single-label datasets in language and vision modelling. in order to obtain a catalan version of manynames, we decided to automatically translate all the annotations in the original english dataset to catalan using a state-of-the-art machine translation (mt) tool. to assess the quality of the resulting resource, we collected real human annotations for a subset of the dataset, consisting of around 1k images. although the size of the manually annotated subset may seem small, it can be considered standard for a test set with the purpose of evaluating the quality of automatic annotations. ## analysis and discussion the purpose of the analysis was, on the one hand, to assess the quality of the automatic translation in the subset that had been human annotated by comparing both the translation and the human annotations, and on the other, to explore possible differences in lexical choices based on cultural biases. to this end, the accuracy of the top name, the degree of variation per image, the average number of different responses per image and the agreement on the top name were computed for both test sets. the most immediate measure to evaluate the quality of the translated test set was to compute the accuracy of the translation of the most frequent response per image (aka the top name) by comparing it with the corresponding top name in the human annotated set. this accuracy only reached 67,91%, which is a clear indication of how different both resources are. another interesting metric to be computed was the degree of lexical variation in the two sets. despite the difference in the number of annotations (36 for the translated vs 10 for the human annotated), the average number of types in the translated test set was 2.1 responses per image, whereas in the human annotated test set, it was 3.1 16 , showing greater lexical variation in the human annotated test set. to account for this clear divergence, we could hypothesize that often two different names get conflated into one in the translation process. however, the ratio of the translated dataset (2.1) is very close to the 2.2 names per object on average in the original manynames dataset. a related metric that was also applied is agreement on the top name per image, which is computed by dividing the number of responses for the top name by the number of total responses. since more variation is observed in the human annotated set, we expect a higher agreement in the translated set. indeed, the median is higher in the translated data (0.93) than in the human annotated data (0.7). a qualitative analysis was performed by sorting both test sets by domain and top name and manually inspecting them to spot divergent cases of translation between english and catalan. several findings account for the observed richer lexical variation in catalan: it was found that catalan speakers tended to choose a subordinate name (portaveu, esportista, tennista, etc.) rather than a taxonomic name (dona, noi, noia, etc.) in the people domain, the exception being images that involved specific terminology of an activity or a sport not specific to the catalan culture, i.e. baseball or skateboarding. in those cases, catalan speakers tended to choose the basic level (jugador, noi) rather than the subordinate level (batedor, patinador). certain domains, such as clothing showed catalan to be more specific than english (which had repercussions in the translation). for example, jacket can be translated as americana or jaqueta, depending on the formality of the event. in addition, catalan speakers may opt for the use of a diminutive (trenet vs tren), but this is a lexical option that english speakers do not have. our analyses show major divergences between the automatically translated dataset and the manually annotated subset, both in terms of degree of internal lexical variation and accuracy of the translated top names. manual inspection of the results further confirms that these divergences can be attributed to linguistic and even cultural differences. automatic translation of language resources from well-resourced languages to less-resourced ones is a common practice in nlp and related fields. our results show that linguistic and cultural differences may affect the quality of automatically translated resources, such as the one presented here. ## conclusions in this paper, we have presented a new dataset for the task of object naming in catalan, namely cat manynames. the new resource is the result of the machine translation of the english manynames dataset, with some pre-and post-processing steps. it also includes a subset of 1,072 images which has been entirely human annotated with 10 annotations per image. the comparison between the translated and the human annotated subsets reveals culturalbased divergences in lexical choices that can affect the quality of the machine-translated resource. our results shows potential weaknesses in resources built up by translating annotations, particularly in the language and vision field, where context is provided by the image and thus is not available to the machine translation system. since current literature on object naming within the language and vision field is scarce, these findings could serve as a starting point for research on cross-lingual object naming, and on the impact of automatic translation in the annotatation of multilingual resources.