Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA (2023)

Abstract: Natural language is ambiguous. Resolving ambiguous questions is key to successfully answering them. Focusing on questions about images, we create a dataset of ambiguous examples. We annotate these, grouping answers by the underlying question they address and rephrasing the question for each group to reduce ambiguity. Our analysis reveals a linguistically-aligned ontology of reasons for ambiguity in visual questions. We then develop an English question-generation model which we demonstrate via automatic and human evaluation produces less ambiguous questions. We further show that the question generation objective we use allows the model to integrate answer group information without any direct supervision.

## introduction the ability to ask questions allows people to efficiently fill knowledge gaps and convey requests; this makes questions a natural interface for interacting with digital agents. visual question answering (vqa) models more specifically seek to answer questions about images, which can be useful in a variety of settings, such as assistive tech @xcite . a number of datasets have been proposed for training vqa models, including vqav2 @xcite , vizwiz @xcite , and gqa @xcite . such datasets are not only useful for training -they represent the aggregate judgements of speakers on a variety of factors, including ambiguity. ambiguity is a core feature of natural language, and can exist at all levels of linguistic analysis @xcite . in the context of data annotation, ambiguity often leads to disagreement between annotators. given that the data resulting from crowdsourced annotation projects is typically used in a categorical fashion to train and evaluate models, annotator disagreements are problematic. past work has often looked at detecting and resolving disagreements from the perspective of trust @xcite , where some annotators are assumed to be more or less trustworthy. however, in the case of ambiguity, an annotator's honest effort might still lead to disagreement; in such cases, collecting more annotations can fail to establish a consensus. this differs from disagreements arising as a result of mistakes and cheating, where gathering more annotations would effectively outvote low-quality annotations. ambiguity in the context of questions presents a particularly rich problem: firstly, from a formal point of view, question semantics are an area of active development and debate @xcite ; this makes empirical accounts of questions particularly useful. secondly, questions are increasingly relevant to natural language processing (nlp) research. many nlp tasks are cast as question-answering (qa), including a growing number of tasks which can be cast as few-shot qa. against this backdrop, we aim to document and describe ambiguities in vqa as well as to introduce a model for resolving them. our main contributions are: 1. we examine how ambiguity appears in the vqav2 data by constructing a dataset of 1,820 annotated visual image-question-answer triples. for each question, we ask annotators to re-group answers according to the underlying question they answer, and to rewrite questions to unambiguously correspond to that group. an example from our dataset can be seen in fig. 1 . for the ambiguous vqa question given at the top, annotators group existing answers into two topical groups (species and color). then, for each group, annotators rewrite the original question such that it could be answered by answers from the corresponding group, but not from other groups. 2. we create an ontology of causes for linguistic ambiguity based on the propbank ontology (kingsbury and @xcite @xcite , and annotate our data with these causes. 3. we develop a visual question generation model which learns to rewrite questions; we validate this model with the re-grouped answers and re-written questions from our dataset. our model can be used to cluster answers into their groups without any supervision for answer groups. in the vqav2 annotations, each image has multiple questions, with each question being redundantly answered by up to 10 annotators. this redundancy is crucial for our annotations, as it provides us with multiple judgments per question, some of which may indicate ambiguity. we define ambiguous examples as ones where annotators are responding to different underlying questions. note that this definition is not exhaustive, as it relies on the annotations; an example could be ambiguous but have few annotations, resulting in complete agreement between annotators. we contrast this definition with visual underspecification and uncertainty, which are categorized by a lack of visual information needed to answer a question, rather than ambiguity about what the question is. these can appear simultaneously, e.g. in fig. 3 where there is both ambiguity and underspecification. fig. 2 gives an example of underspecification, as the information being queried is absent in the image and must be inferred. past efforts examining reasons for annotator disagreement in vqa have addressed this distinction: @xcite introduce a dataset of 45,000 vqa examanswers harley davidson honda vqa q: what make of motorcycle is that? harley dirt bike dirt suzuki blue and white ples annotated with reasons for disagreement, including ambiguity and lack of visual evidence as two separate categories. in practice, however, many examples labeled as ambiguous (such as fig. 2) are cases of underspecification or unambiguous questions paired with visually ambiguous images. we use the ambiguous examples from bhattacharya et al. (2019) as a starting point for our dataset. ## data to properly study linguistic ambiguity in vqa, we collect a dataset of ambiguous examples, which represents a resource for categorizing and analyzing ambiguous questions and contains 1,820 answers to 241 image-question pairs. the data contains answers grouped by their underlying questions; there are 629 rewritten underlying questions. our dataset is intended for evaluating models and for performing analysis, not for training. the size of the ambiguous subset of vqa from @xcite prohibits our reannotating the whole dataset, so we create a subset of data that is likely to be linguistically ambiguous. first, we sort the annotations into a priority queue using several heuristics. to merge synonymous answers (e.g. "cat", "the cat", "feline") we embed each answer into continuous space using glove embeddings @xcite , meanpooling across words for multi-word answers and apply k-means @xcite to the resulting embeddings, iteratively increasing the number of clusters k. examples are scored by combining the k-means inertia score with a penalty for each additional cluster, trading off cluster coherence and having as few clusters as possible. these are subsequently sorted by how balanced their clusters are -balanced clusters are more likely to be ambiguous, as unbalanced clusters are often a result of a single bad annotation. we remove yes-no questions with only "yes" and "no" answers, as they answer the same question. note that we do not include questions from gqa (hudson and manning, 2019) in our dataset. because gqa questions were generated from a grammar rather by annotators, we do not expect there to be as much ambiguity. furthermore, gqa questions are not part of the labeled data from @xcite . ## model the data collected in section 3 consists of questions rewritten according to their answer clusters. we develop a visual question generation (vqg) model which takes in answers and images and produces questions. after confirming the performance of the vqg model for generation generally, we evaluate the performance of a vqg model with respect to the answer clusters in our dataset. specifically, we examine how the model can be used for clustering answers within an answer group together. given that the answer clusters are based on the underlying question the answer is answering, we hypothesize that a good vqg model should not only learn to generate questions with a high similarity to the reference questions, but learn input representations that contain answer group information. note that this information would have to emerge in an unsupervised fashion, as we do not provide any answer group information during training. we present a simple model for vqg consisting of a pre-trained vision-language encoder followed by a pretrained text-to-text encoder-decoder model. the encoder embeds an image and an answer into a shared representation space; the decoder produces a text question conditioned on this shared representation. we use vilt @xcite as our vision-language encoder. vilt is a pre-trained fully transformer-based 87.4m-parameter model. the available vilt model fine-tuned for vqa was trained on the entirety of the vqav2 training data; since the annotations for @xcite come from the training set, our annotations also are sourced from the vqav2 training set. to avoid test-set leakage, we fine-tune our own version of vilt on a modified training set that excludes our annotations. our input to vilt is the image i i and a text answer a i from the set of answers for instance i, a i . to generate text, we feed the output of vilt to a pre-trained t5-base encoder-decoder model @xcite with ∼ 220m parameters, accessed via huggingface transformers @xcite . we replace the t5 embedding layer with the output of our vilt encoder, and train the model using all answers in the dataset with "yes" or "maybe" confidence ratings. we use categorical cross-entropy loss computed against the original question q i as our loss function. note that the question q i is taken directly from the vqav2 data, which we refer to as "original data" -we do not train on the annotations collected in section 3. ## visual question generation before analyzing performance on our dataset, we verify that the question-generation model we proposed is able to generate reasonable questions for the dataset more broadly. here, we follow past work in reporting several string-based metrics: bleu @xcite , cider @xcite , rouge-l @xcite scores. we also report bertscore @xcite . ## human evaluation the metrics in section 5 suggest that our model holds promise as a method for rephrasing ambiguous questions; ## discussion while there are many competing accounts of question semantics, a general consensus maintains that the analysis of questions requires an analysis of the dialogic context in which they exist @xcite . this differs substantially from the vqa domain, where questions are being posed and answered outside of a dialogue. specifically, annotators are asked to answer questions in a single attempt, with no recourse to follow-up questions or interactions. one umbrella reason for disagreement and ambiguity in this setting that we have not explored in our analysis is guided by the question-under-discussion (qud) framework @xcite , which frames a dialogue as a series of moves around asking and resolving quds. in the setting we explore, much of the disagreement seems to be driven by annotators disagreeing on which qud to resolve with their response to the question; it is possible that having additional context would reduce the disagreement between annotators by providing additional information. indeed, the presence of contextual information is a key reason given by @xcite for the existence of ambiguity in communication systems like natural language: if a listener can reasonably be expected to reconstruct a speaker's intended meaning from the speaker's utterance combined with the prior context, then the speaker does not have to specify the (redundant) contextual information in their utterance, lowering their effort. given the artificial nature of the vqa annotation setting, with one set of annotators producing questions and another answering them in a single turn, it is unsurprising that ambiguities arise. similarly, the rewritten questions from our dataset and our rewriting model can be framed as questions specifying which qud is being resolved by a given answer. this is especially true of property-based questions, where the qud influences which property is used to partition the space of possible answers (cf. appendix b.1). ## conclusion we have presented a dataset of ambiguous vqa questions, annotated with reasons why they are ambiguous, as well as answers grouped by the underlying disambiguated question they are answering. we then introduced a model for rephrasing ambiguous questions according to their answers, finding that the model, which is trained purely on visual question generation, is able to recover information about the underlying question. we validate both our dataset and model using automatic and human evaluations, where we find that both reduce question ambiguity.