How Do Image Description Systems Describe People? A Targeted Assessment of System Competence in the PEOPLE -domain (2020)

Abstract: Evaluations of image description systems are typically domain-general: generated descriptions for the held-out test images are either compared to a set of reference descriptions (using automated metrics), or rated by human judges on one or more Likert scales (for fluency, overall quality, and other quality criteria). While useful, these evaluations do not tell us anything about the kinds of image descriptions that systems are able to produce. Or, phrased differently, these evaluations do not tell us anything about the cognitive capabilities of image description systems. This paper proposes a different kind of assessment, that is able to quantify the extent to which these systems are able to describe humans. This assessment is based on a manual characterisation (a context-free grammar) of English entity labels in the PEOPLE domain, to determine the range of possible outputs. We examined 9 systems to see what kinds of labels they actually use. We found that these systems only use a small subset of at most 13 different kinds of modifiers (e.g. tall and short modify HEIGHT, sad and happy modify MOOD), but 27 kinds of modifiers are never used. Future research could study these semantic dimensions in more detail.

## introduction image description systems are typically trained and evaluated using datasets of described images, such as flickr30k and ms coco @xcite . automated metrics such as bleu, meteor, and cider compare the generated descriptions to a set of reference descriptions, and produce a textual similarity score @xcite @xcite . the overall score is said to convey the performance of the system. these metrics (and especially bleu) are often criticized because of their low correlation to human ratings @xcite @xcite . human ratings, in turn, are typically collected using likert scales, where participants rate descriptions for their fluency, correctness, overall quality, and other quality criteria @xcite . while these measures do afford us with some insight into system performance, they are very general, and not tied to any specific cognitive ability (e.g. to produce descriptions containing negations, to reason about past and future events, or to successfully refer to properties or entities in a particular domain). this is a problem because it means we do not know what systems are good or bad at. 1 perhaps the most informative metric in common use is spice @xcite , which enables researchers to assess the ability of image description systems to produce color terms, for example, by computing precision/recall of these terms with regard to a scene graph representing the relevant image. instead of general evaluation metrics, we propose to develop specific metrics to assess 2 whether image description systems are able to produce specific kinds of descriptions. this paper presents a proof-ofconcept to assess the extent to which systems are able to produce different kinds of person-labels. our main idea is that, to assess model performance, we should first characterize the domain of interest, such that we have a clear sense of the range of possible descriptions. following this characterisation, we can see whether the model outputs cover the full range (which is unlikely), or whether it produces a small subset of the possible descriptions. our starting point is the taxonomy of person-labels developed by @xcite , who manually categorised all person-labels in the flickr30k @xcite and visual genome @xcite datasets (both us english), with the head nouns male, female, males, females, man, woman, men, women, boy, girl, boys and girls. they implemented their categorisation scheme in a context-free grammar (cfg; using nltk @xcite ), so that all combinations of modifiers and head nouns would be covered. to illustrate: with only two modifiers (tall and happy) and 12 head nouns, there are already 48 possible combinations (24 with single modifiers, and another 24 with both modifiers in two different orders). all variations can be captured by the following cfg rules (where commas are used to indicate a choice between different terminals): the original cfg is limited to the eight head nouns specified above. we extended the grammar to obtain broader coverage of system output data (both in terms of head nouns and in terms of modifiers). we then used the grammar to analyse this data, and to see what kinds of labels are typically produced by image description systems. we found that systems are limited in their coverage: at most 13 out of 40 modifier categories were used. this means that 26 different kinds of semantic properties that are attested in human image description data, cannot be found in the output data. 3 2 method corpus data. @xcite developed two taxonomies using english corpus data from flickr30k and the visual genome dataset (the latter dataset contains 108,077 images from ms coco). for this paper, we merged the two taxonomies to have broader coverage. after combining the different sets of words for each category, we manually inspected and revised the category files. 4 system data. we use output data from nine different image description systems, collected by @xcite , who obtained this data by contacting the authors of all image description papers that appeared in conferences and journals in 2016-2017. all systems were trained o the ms coco training set, and all descriptions were generated for the ms coco validation set (40,504 images). although the systems are slightly older, many are still competitive with the state-of-the-art. furthermore, we do not want to make any claims about current model performance. rather, our aim is to provide a proof-of-concept of a new assessment procedure for image description systems. preparation. we updated the grammar in two stages. in the first stage, we added more head nouns. people are not just referred to using gendered nouns. some are referred to using their occupation (police officer) or their (social) role (friend, neighbour, mother). thus we added more categories (with corresponding lexical items) for different kinds of head nouns. this required us to determine which nouns refer to people. because this experiment is a proof-of-concept, we focus only on the nouns that actually occur in the output data for the systems mentioned above. we used spacy @xcite to identify noun chunks in the generated descriptions, and identified 5385 unique sequences of one or more nouns (assuming that multiple nouns together form compound nouns). we then manually selected 170 heads that refer to people and developed category labels for all different kinds of nouns, adding new labels until all nouns were categorised. a single head may be part of multiple different categories, and thus receive multiple labels. some examples are provided in in the second stage, we updated the set of modifiers. since different head nouns may be modified in different ways, we extended coverage of the modifiers to include the ones that are used for nouns other than the twelve original nouns. this was done by identifying all noun chunks that refer to people (i.e. end with one of the heads identified in the previous step), analysing them with our updated grammar, and manually categorising all modifiers that weren't already covered by our grammar. we found that the activity baseball player, commuter, kite surfer, pedestrian, runner, skier, swimmer, spectators age adult, baby, babies, child, children, kid, toddler, girl, lady, woman, women, man, men gendered boy, girl, man, females, woman, ladies, bride, groom, daughter, cowboy, camera man plural or mass nouns crowd, passengers, persons, business people, construction workers, students, spectators relation father, friend, patient, bride, daughter, couple, family, friends, owner, customer, spectators status or occupation clown, coach, police officer, farmer, king, camera man, students, emergency personnel ## results genderednoun/agenoun). this might be due to their gan-based approach, which aims to make descriptions more human-like. some observations. below are some initial observations about the results: lack of diversity. the descriptions are not very diverse @xcite , and this is reflected in our data: there is little variation within each of the categories. for example, in the build category, only three adjectives (out of 48 possible modifiers) are used: big, small and tiny. overgeneralisation. our parser overgeneralises. for example, when we take a closer look at the labels containing modifiers relating to the build of a person (e.g., big, small, tiny), we find multiple instances where these modifiers are not used in the relevant sense, namely: a big crowd, a small family, a small team. this shows that manual analysis is currently still necessary (and the system would clearly benefit from word sense disambiguation), but our parser serves as a good filter: without it, it would not be possible to find different kinds of modifiers at all. with the parser and some manual effort, we have a better picture of the kinds of person-labels a system is able to produce. biases. the generated labels seem to be biased with respect to gender and ethnicity (cf. van miltenburg (2016)). for example: attractiveness modifiers are used almost exclusively with women and children. the only counter-example is ambiguous: 'a pretty young man' (pretty young or pretty man). haircolor is only specified for women. ethnicity is only marked for asian people. annotations. although we took care to double-check all the labels, the taxonomy has been developed by a single annotator. future work should investigate the replicability of our taxonomy. (although there are as many taxonomies as taxonomists, and some amount of disagreement is inevitable, there should be a clear overlap between our data and future replications.) coverage. our taxonomy is driven by existing sets of descriptions. should image description systems become more creative in the future, or rely on different datasets, then this will affect the coverage of our taxonomy, and updates will be necessary. we do believe that this process could be automated, and that it is feasible to update the taxonomy in a matter of days (which we believe is acceptable, and proportionate to the development time of a new image description system). ambiguity. as mentioned in the results section, a context-free grammar is not powerful enough to disambiguate different word senses. so although our parser can act as a filter to select relevant noun phrases from system outputs, this will result in false positives for different modifiers. to reduce this issue, one could either refine our taxonomy and tie it to the output of word sense disambiguation systems, or manually correct the outputs of our parser to create a training set, turning description categorisation into a sequence labeling problem. the latter solution would hopefully also generalise to unseen nouns and modifiers, reducing coverage issues caused by our grounded-in-data approach. ## conclusion our work shows that it is feasible to assess how image description systems label people, using a contextfree grammar. this idea could be applied more broadly, to assess systems' coverage of the original training data (i.e. which patterns the system managed to capture). the development of a taxonomy of expressions in a particular domain does represent a serious time investment, but once this is done, we can more closely assess system performance in a particular domain, for any number of systems. (and perhaps this is also just the cost of wanting to develop end-to-end systems using crowdsourced image description corpora.) as noted by @xcite , developing a taxonomy also forces us to think about the desirability of different kinds of expressions that may be used in the generated descriptions. this will hopefully lead to improved image description guidelines, which may later be formalised in an assessment procedure, similar to the one reported in this paper.