Grounding Plural Phrases: Countering Evaluation Biases by Individuation (2021)

Abstract: Phrase grounding (PG) is a multimodal task that grounds language in images. PG systems are evaluated on well-known benchmarks, using Intersection over Union (IoU) as evaluation metric. This work highlights a disconcerting bias in the evaluation of grounded plural phrases, which arises from representing sets of objects as a union box covering all component bounding boxes, in conjunction with the IoU metric. We detect, analyze and quantify an evaluation bias in the grounding of plural phrases and define a novel metric, c-IoU, based on a union box’s component boxes. We experimentally show that our new metric greatly alleviates this bias and recommend using it for fairer evaluation of plural phrases in PG tasks.

## introduction phrase grounding (pg) describes the multimodal task of identifying objects in images and connecting them to free-form phrases in a textual description (caption). a phrase usually describes one, or sometimes several, specific objects. grounding phrases in image regions provides an essential link between texts and images and serves as a foundation for multimodal understanding tasks, including sentence-to-image alignment, visual qa, visual common-sense reasoning (vcr), etc. benchmarks for training and evaluating pg systems @xcite @xcite @xcite generally provide rectangular bounding boxes as ground truth (gt). therefore a pg ground truth is represented as a phrase linking to a (gold) bounding box enclosing the image patch referred to by the phrase. some datasets provide pixel segmentation masks @xcite , which enable more precise evaluations but are more difficult and costly to produce. thus, the trend towards annotating bounding boxes persists in recent datasets @xcite . plural phrases describe multiple entities in an image, either through a collective term (e.g. crowd) or a plural form (e.g. two children). depending on the annotation, the gold box consists either of a single box enclosing all entities or several component boxes representing the individual entities. by convention foot_0 , component boxes are merged into one union box spanning all individual boxes, functioning as a single gold box. figure (2.a) gives an example of a union box for a plural phrase with two components. this reduction of multiple boxes to a single union box is widely established in pg evaluation, both for ground truths and predictions. although plural phrases are underrepresented in pg benchmarks, they constitute substantial proportions, and appropriate annotation and evaluation of component boxes is essential to achieve highquality mappings for all phrase types. however, the annotation of plural phrases is challenging, as shown in @xcite ; @xcite who investigate how phrases can refer to groups of objects or several entities within a group. while iou is a simple and effective metric for evaluating 1:1 mappings, we claim that it is unsuitable for evaluating plural phrases. we show that the union box is in fact not an ideal gold representation for plural phrases: it can make the gold box overly large, especially when including areas that do not represent any components, and thus introduces an evaluation bias favoring large prediction boxes. our contributions are as follows: i) we detect, describe and quantify an evaluation bias in the grounding of plural phrases when applying standard practice of measuring iou over union boxes, using an unsupervised pg system on the pg dataset flickr30k. ii) we propose a novel evaluation metric based on component boxes rather than union boxes. iii) we show that the new metric alleviates this bias and reduces the evaluation failures. ## evaluation bias iou is the standard evaluation metric used in pg and rewards predictions that highly overlap with their gold boxes. for a plural phrase that links to multiple ground truth boxes, a union box enclosing all components is generated, so the same evaluation metric can be used for singular and plural phrase types. however, we argue that this method introduces a considerable bias, which may result in unfair evaluations. when evaluating on union boxes, we ignore all information about the components' sizes and positions, and only consider the union box outline. if components are spread across the image, a union box can become much larger than the combined size of its component boxes, which makes them imprecise and ambiguous. figures (2.a) vs. (2.b) show an example of twocomponent union boxes that are highly overlapping -one for pins, the other for hands. hence, a system that returns the prediction (2.b) for juggling pins will be unduly considered as correct. similarly, for a prediction with too few or too many components, iou often fails to detect such mistakes, as in (2.c) vs. (2.d). the ground truth (2.c) for two boys includes only two components, yet a system predicting four components (including the men on the boat) will still be correct according to iou. this type of 'false positive' arises from the generation of union boxes, in conjunction with the relatively forgiving nature of the iou metric for large predicted boxes. given such undesirable failure cases, we conjecture that iou can lead to unwanted evaluation biases and we investigate whether it is a sound metric for evaluating plural phrases. ## our new evaluation metric c-iou we aim at a metric that is not based on the union box, but its components. however -any metric is only as good as the quality of its underlying ground truth. when studying the annotation of plural phrases in flickr30k, we found that many of them are imprecise or incomplete. nearly one third of plural phrases are annotated with a single bounding box without components and for 9% the number of components does not match the cardinality of the referring phrase (e.g. two component boxes for three women), leaving 37% of the plural phrases without proper representation of their components. this high level of noise precludes any metric that relies on matching the number of component boxes of ground truth and predictions. in §2.1 we identified the filler space of union boxes -jointly with iou evaluation -as the source of the detected evaluation bias. to combat this, we define an adapted iou that is not computed over the union box, but its aggregated components, by taking the intersection of all gold and predicted component boxes and dividing by the area of the union of all (gold and predicted) component boxes. we call this metric component iou (c-iou). c-iou is analogous to standard iou for single-object boxes, and only affects the evaluation of plural phrases, as seen in fig. (1) . by considering the area covered by all component boxes, it gains robustness against annotation noise. ## evaluation of component iou (c-iou) we evaluate c-iou in the same way as we did in §2.2 to quantify biases under iou, and give results in table (1.b). we expect c-iou to avoid biases for plurals and ensuing false predictions. the experiments confirm our expectation: large prediction boxes yield lower scores with c-iou. max box predictions, computed on all objects found in the image, yields 9.31 pp. lower accuracy on plural phrases. pg system performance with enlarged predictions measured on singular objects for plural phrases increases by 2.69 pp. to 33.84 for iou, yet decreases by 1.16 pp. to 29.84 for c-iou. therefore, c-iou better detects wrong predictions. but c-iou does not catch all incorrect predictions: with pruning, accuracy measured with c-iou increases less (+12.72 pp. to 50.09) compared to iou (+19.86 pp. to 56.46). data inspection shows that without pruning, c-iou allows plural objects for singular phrases (and vice versa) -typically the plural object includes the targeted object plus another small object in the background (see fig. as a final test, we evaluate both metrics on artificially generated false plural box predictions. for each phrase, we assemble a random prediction consisting of 2-5 components with different labels. ideally, most predictions should be labeled as incorrect, thus a lower accuracy indicates a more sensitive metric. c-iou indeed returns much lower scores than iou: 9.17 and 7.62 (c-iou) vs. 19.97 and 24.09 (iou) on all phrases and plural phrases, showing that c-iou effectively counters the bias. ## conclusion we have detected, described and quantified an evaluation bias for plural phrases in the pg literature. our alternative c-iou metric, acting on components rather than union boxes, alleviates this bias, as we show in experiments with an unsupervised pg system. future work could test more systems to assess by how much state-of-the-art performance is lower than currently estimated. evaluation of plural phrases is further impeded by the low quality of the gold boxes. therefore, future benchmarks need to annotate plural phrases with all their components if we wish to enable pg systems to better learn the intricacies of language (including plural expressions) in relation to the visual modality. figure 3 : gt and prediction cases with union boxes (dashed) and components (blue); the check marks show whether iou (red) and c-iou (blue) correctly evaluate the prediction (for further explanation see figure 2 ).