Detecting Temporal Ambiguity in Questions (2024)

Abstract: Detecting and answering ambiguous questions has been a challenging task in open-domain question answering. Ambiguous questions have different answers depending on their interpretation and can take diverse forms. Temporally ambiguous questions are one of the most common types of such questions. In this paper, we introduce TEMPAMBIQA, a manually annotated temporally ambiguous QA dataset consisting of 8,162 open-domain questions derived from existing datasets. Our annotations focus on capturing temporal ambiguity to study the task of detecting temporally ambiguous questions. We propose a novel approach by using diverse search strategies based on disambiguate versions of the questions. We also introduce and test non-search, competitive baselines for detecting temporal ambiguity using zero-shot and few-shot approaches.

## introduction in the field of open-domain question answering (odqa) detecting and avoiding ambiguous questions is quite important @xcite . @xcite found that over 50% of the questions in google search queries are ambiguous. current odqa systems, however, usually operate under an implicit assumption that there is a single correct answer for every question. temporal ambiguity, in particular, occurs when a question involves unclear or unspecified time frames, leading to different answers depending on the assumed temporal context @xcite . temporal ambiguity is then a specific type of ambiguity where the interpretation of a question depends on the time frame being referred to. for example, the question "who was the president of nbc universal?" is temporally ambiguous as the answer depends on the specific time frame. temporal ambiguity poses unique challenges for qa systems, as these need to understand the temporal context of a question to provide the correct answer @xcite . however, its detection should be useful for improving temporal ir and qa systems @xcite @xcite . the objective of our work is to stimulate the design of odqa systems that are able to distinguish between temporally ambiguous and nonambiguous questions. this involves understanding the temporal context of a question, a challenge that is particularly prevalent in open-domain questionanswering systems. our work, therefore, extends the existing research in the field of ambiguity in open-domain questions, with a specific focus on temporal aspects. to foster the research in temporal ambiguity detection, we construct a dataset called tempam-biqa having 8,162 questions (3,879 ambiguous questions and 4,283 unambiguous questions) using an open-domain version of situatedqa @xcite , archivalqa @xcite and ambigqa @xcite datasets. for each question, we manually identify its temporal context, and label the question as ambiguous or unambiguous. if a question has multiple answers due to temporal ambiguity, we annotate the question as ambiguous. for example, consider the question "who won the world cup when it was held in south america?". this is a temporally ambiguous question because the answer depends on the specific time frame. a person reading this question would note that it could refer to different time frames @xcite , each with a different winner. the question could have multiple answers depending on temporal context. we establish initial performance benchmarks on tempambiqa by introducing a comprehensive set of strong baseline methods. (1) zero-shot ques- q: who was the commissioner for long beach california? dq 1 : who was the commissioner for long beach california as of 1987? dq k : who was the commissioner for long beach california as of 2003? dq 1 dq k dq 1 : who ... as of 1987? dqk: who ... 1988? is the answer for q1 and q2 same? write only one word between 'yes' and 'no'. a k disambiguation component answer equivalence testing component search compoent linear k k'=k+1 skip-list (s) k k'=k+s random k k' dac k k'=(k+1)/2 no a k ambiguous false k' is valid unambiguous yes k' true figure 1 : overview of different search strategies for detecting temporally ambiguous questions. the disambiguation component generates questions dq 1 and dq k , referred to as q1 and q2 in the prompts, respectively. the answer equivalence testing component compares them, classifying q as temporally ambiguous if the answer equivalence (a k ) is "no". if "yes", the search proceeds to find the next valid year k' within the defined time range, generating the next disambiguation question dq k' to continue the classification process. if no valid k' is found, the question q is classified as temporally unambiguous. a valid year k' is the one that falls within the specified time range (e.g., 2000-2024). our contributions can be summarised as follows: ## related work the development of datasets that explicitly consider temporal and ambiguous aspects remains limited. while datasets like ambigqa @xcite and others @xcite incorporate certain elements of ambiguity, they do not focus on the temporal dimension. ambigqa highlights the prevalence of ambiguities in natural questions and studies the disambiguation of questions to address them. however, it does not focus specifically on temporal ambiguities. archivalqa @xcite is a temporal odqa dataset featuring questions derived from the new york times news collection @xcite @xcite propose a qa dataset that focuses on temporal and geographical context-dependent questions. other works @xcite @xcite use clarification questions to handle ambiguities, but these approaches primarily refine user intents rather than directly resolving temporal confusion. self-calibrating models @xcite have been proposed to estimate confidence and handle ambiguities, but these methods have not been specifically tailored to address the temporal aspect in qa, indicating a potential area for future research. to the best of our knowledge, we are the first to introduce a dataset specifically focused on temporally ambiguous questions and propose approaches for detecting temporal ambiguity in questions. ## data collection tempambiqa is constructed by incorporating questions from different qa datasets, archivalqa, situatedqa, and ambigqa. to create tempam-biqa, we combined a subset of archivalqa that were designated by its authors as containing temporal ambiguous questions, a test set of timedependent questions from situatedqa, and the development set of ambigqa. we then manually labeled all the questions as temporally ambiguous or unambiguous by carefully checking if answers vary over time or the questions can be reformulated into multiple unambiguous variants. this process resulted in a dataset comprising in total 8,162 questions, with 3,879 labeled as ambiguous and 4,283 as unambiguous. further details about the statistics of tempambiqa dataset and its few examples can be found in appendix a. ## search methods answers to temporally ambiguous questions change based on the time period they refer to. for example, "who was the first female governor of india?" has a single answer, making it unambiguous. but, "who was a governor of india?" is temporally ambiguous, as the answer varies over time. to detect such ambiguity, we employ various search strategies by explicitly specifying the relevant year in the question. to identify a question as ambiguous, we need to find at least two different answers for the same question, each from a different time frame. we do this by disambiguating the question by adding a specific year such as "as of 2001?" at the end of the question. for the tempambiqa, we use two time frames based on the original datasets. for questions from archivalqa, the time frame spans from 1987-2007 to match the news collection period. for questions from situatedqa and ambigqa, which are not tied to a specific time range, we set a time frame to span 2000 -2024. figure 1 provides an overview of the framework based on a search method used for the classification. the framework consists of three main components described below: disambiguation component, answer equivalence testing component, and search component. ## experimental setup we analyze different search strategies against zero-shot, few-shot approaches using diverse models, such as t5 @xcite , llama3 @xcite , qwen @xcite , mixtral (jiang et al., 2024), gpt-3.5 (brown et al., 2020), and gpt-4 @xcite , with different parameter values. prompts used for the analyzed approaches are given in appendix c. additionally, we also analyze search strategies against a fine-tuned bert @xcite model. we utilize standard evaluation metrics, such as accuracy (acc), precision (pr), recall (rc), and f1 score (f1). ## results in figure 2 we illustrate the number of comparisons for all search strategies. the best-case scenario occurs when temporal ambiguity is detected after the first comparison, while the worst-case scenario happens when it is identified only after the final comparison. the average case lies between the extremes but tends to be much closer to the best case regarding efficiency. the efficiency in the average case scenario is more similar to the bestcase than worst-case scenarios. comparing figure 2 and ## conclusion in this paper, we address the novel task of detecting temporally ambiguous questions in odqa by introducing tempambiqa, a manually annotated dataset of 8,162 temporally ambiguous and unambiguous questions. we perform several experiments using different search strategies to classify the question as temporally ambiguous. our experiments demonstrate the effectiveness of searchbased methods in detecting temporally ambiguous questions. for future work, we plan to explore more dynamic strategies for determining time ranges, such as identifying the timing of recurring events mentioned in the questions and generating disambiguation questions based on those timelines. additionally, we aim to refine the granularity of the timelines, moving beyond yearly intervals to include finer distinctions, such as months.