To Find Waldo You Need Contextual Cues: Debiasing Who’s Waldo (2022)

Abstract: We present a debiased dataset for the Person-centric Visual Grounding (PCVG) task first proposed by Cui et al. (2021) in the Who’s Waldo dataset. Given an image and a caption, PCVG requires pairing up a person’s name mentioned in a caption with a bounding box that points to the person in the image. We find that the original Who’s Waldo dataset compiled for this task contains a large number of biased samples that are solvable simply by heuristic methods; for instance, in many cases the first name in the sentence corresponds to the largest bounding box, or the sequence of names in the sentence corresponds to an exact left-to-right order in the image. Naturally, models trained on these biased data lead to over-estimation of performance on the benchmark. To enforce models being correct for the correct reasons, we design automated tools to filter and debias the original dataset by ruling out all examples of insufficient context, such as those with no verb or with a long chain of conjunct names in their captions. Our experiments show that our new sub-sampled dataset contains less bias with much lowered heuristic performances and widened gaps between heuristic and supervised methods. We also demonstrate the same benchmark model trained on our debiased training set outperforms that trained on the original biased (and larger) training set on our debiased test set. We argue our debiased dataset offers the PCVG task a more practical baseline for reliable benchmarking and future improvements.

## introduction a newly released task called person-centric visual grounding @xcite poses an interesting angle into contextual reasoning in vision-language. the task is motivated by humans' reasoning ability. humans viewing an image with a caption as shown in figure 1 can reason (and if needed, speculate) which name refers to which person in the image. this reasoning task involves multiple abilities, such as perceiving characteristics and behaviors of people, understanding their actions in context, speculating about their intentions and effects human of actions @xcite , and connecting visually perceived characteristics with grounded descriptions in natural language @xcite @xcite . in many cases, this task can be performed without knowing the names of the people; for instance in the example on the right, one person is signing and the other is not, as such it is possible to predict which person refers to president and secretary of state respectively. however, in cases such as the example on the left, if all persons are performing the same action (run-ning on a track), then it is hard to match names with these runners without any additional information. progress in the pcvg task can thus help better capture what exact contextual cues are needed to learn about a person's characteristics in a scenario, and can aid improvements in visual understanding about human interactions and behaviors. to support this task, @xcite offer a large-scale dataset called who's waldo which consists of 272k annotated real life images. ideally, the dataset should consist of input-output pairs (such as the example on the right in figure 1 ) which are 'solvable' as opposed to the one on the left which is ambiguous. however, as we explore the original who's waldo dataset, we encounter a great portion of cases that resemble the left example in figure 1 , unsolvable data with insufficient contextual cues. given such context, if we do not recognize who exactly is in the picture, even we human beings cannot tell which name is who. we can then only make predictions with biased assumptions, such as the first named person would always be on the leftmost, or the main subject would always make up the largest area. such biases in the original dataset may explain why the heuristic methods perform very strongly, outperforming random guessing by a big 27% increase in test accuracy and trailing the top benchmark only by 6%. we believe a fair dataset should not encourage approaches to adopt biases to such an extent, and thus the original baseline model overestimates its performance. inspired by dataset debiasing works such as vqa-cp @xcite and gqa-ood @xcite , we create a debiased collection of 84k annotated image-captions out of the who's waldo dataset by filtering out all biased data with insufficient context. we evaluate the quality of our new dataset by applying the original heuristic methods as well as who's waldo's benchmark model. results show that our debiased dataset greatly reduces the heuristic biases from the original dataset and provides the pcvg task a more practical baseline for future developments. we also learn from various techniques to amend priors, biases, or shortcuts in datasets. repair @xcite uses resampling to fix representation biases in image datasets. @xcite incorporate compositional information into sentence embeddings for natural language inference. dqi @xcite offers quantitative metrics to assess biases in automated dataset creation in natural language processing. le bras et al. (2020) introduce adversarial measures to mitigate biases in various natural language processing and computer vision tasks. visual grounding. the pcvg task adapts previous supervised visual grounding models as its original baselines. the visual grounding task is defined as locating specific objects in an image from a textual description. first established by @xcite , following researches have evolved into extracting attention information such as works by @xcite and @xcite . a huge variation of datasets for visual grounding have also been created, including flicker30k @xcite , visual genome @xcite , and refcoco @xcite . ## method in this section, we introduce the person-centric visual grounding task, discuss the original who's waldo dataset, and provide our analysis of shortcuts, biases, and other issues that we discovered in the dataset. we describe the process via which we curate, debias, and filter the dataset. ## conclusion we present a refined dataset for the pcvg task with samples that contain contextual information required for the task. we address prominent biases that we identified in the original task dataset by filtering out a large number of unsolvable cases, and report new baseline performances on the new benchmark. our refined dataset can serve as a more reliable benchmark to enable fair comparisons for new modeling techniques and training protocols.