Gradations of Error Severity in Automatic Image Descriptions (2020)

Abstract: Earlier research has shown that evaluation metrics based on textual similarity (e.g., BLEU, CIDEr, Meteor) do not correlate well with human evaluation scores for automatically generated text. We carried out an experiment with Chinese speakers, where we systematically manipulated image descriptions to contain different kinds of errors. Because our manipulated descriptions form minimal pairs with the reference descriptions, we are able to assess the impact of different kinds of errors on the perceived quality of the descriptions. Our results show that different kinds of errors elicit significantly different evaluation scores, even though all erroneous descriptions differ in only one character from the reference descriptions. Evaluation metrics based solely on textual similarity are unable to capture these differences, which (at least partially) explains their poor correlation with human judgments. Our work provides the foundations for future work, where we aim to understand why different errors are seen as more or less severe.

## introduction recent years have seen a growing discomfort with the use of automatic metrics like bleu @xcite for the evaluation of natural language generation (nlg) systems (e.g., @xcite @xcite . much of the criticism centers around the fact that these metrics show poor agreement with human judgments. while many researchers have tried to develop new metrics that are better suited to evaluate nlg systems (e.g. tailored to the domain like spice @xcite or with intensive pre-training like bleurt; @xcite , we are not aware of any studies attempting to explain why we see such a poor correlation between human judges and automatic metrics. this paper aims to explore this hypothesis, focusing on the evaluation of automatic image description systems. we focus on image descriptions because it is relatively easy for humans to judge whether a given description correctly describes an image. compare this to the webnlg data @xcite , where participants would have to judge whether a given sentence verbalizes a set of triples containing information about properties of different entities, and how different entities relate to each other. the format of the input data, as well as the numerous ways to verbalise each triple separately, and express them jointly, perhaps through some process of aggregation, is bound to make the judgment more challenging. ## hypotheses in line with the error analysis from @xcite , our experiment explores the perceived severity of four common kinds of errors found in automatic image description systems, relating to 1. age, 2. gender, 3. clothing-color, and 4. clothing-type. this section discusses our expectations regarding the quality scores for sentences containing these types of errors. earlier studies in linguistics have shown that not every word in a sentence is equally prominent (see lockwood and macaulay 2012; himmelmann and primus 2015 for an overview). for example, the subject of a sentence is more prominent than the direct object, which in turn is more prominent than the indirect object. by the same token, certain expressions may achieve prominence due to the type of entity they denote. for example, people may be more prominent than inanimate objects (like clothes); an observation also borne out by studies on how humans process visual inputs (cf. @xcite . animacy also plays an important role in referring expression generation @xcite . we believe that animacy might also play a (yet to be determined) role in quality judgments, and our intuition is: 2 hypothesis 1: the perceived quality of descriptions with people-related errors is lower than the perceived quality of descriptions with clothing-related errors. next our hunch is that, in most situations, gender errors are worse than age errors, and errors regarding clothing type are worse than errors regarding clothing color. two perspectives come to mind: while it can be argued that all categories exhibit some degree of vagueness, age-denoting expressions tend to be more vague than genderdenoting ones (e.g. whether someone is old is more often debatable than whether someone is male); likewise, color-denoting expressions tend to be more vague than clothing-type-denoting ones. we therefore expect that participants are more forgiving when judging the truthfulness of age-denoting and clothing-color denoting expressions than when judging the truthfulness of gender-denoting or clothing-type denoting ones. this difference in severity may also become entrenched, so that one type of error may generally be perceived as worse than another. our intuitions are as follows: hypothesis 2: the perceived quality of descriptions with an age error is higher than the perceived quality of descriptions with a gender error. hypothesis 3: the perceived quality of description with clothing color error is higher than the perceived quality of descriptions with clothing type error. ## method we provide a detailed description of our method below. all data and stimuli are provided in the supplementary materials. ## conclusion we carried out a tightly controlled study, comparing minimal pairs of image descriptions with different types of errors. our results reveal big differences in perceived quality between these descriptions. moreover, we even found preliminary evidence that there are also differences within error categories. our results show that we need to take a closer look at the determinants of description quality, and take seriously the idea of different levels of importance for different aspects of an image. on a broader level, gradations in error severity are probably not limited to image descriptions alone. we encourage researchers in nlg to take a closer look at common output errors in their domain, and to consider the different impact that each of those errors may have.