Question Answering as Programming for Solving Time-Sensitive Questions (2023)

Abstract: Question answering plays a pivotal role in human daily life because it involves our acquisition of knowledge about the world. However, due to the dynamic and ever-changing nature of real-world facts, the answer can be completely different when the time constraint in the question changes. Recently, Large Language Models (LLMs) have shown remarkable intelligence in question answering, while our experiments reveal that the aforementioned problems still pose a significant challenge to existing LLMs. This can be attributed to the LLMs’ inability to perform rigorous reasoning based on surface-level text semantics. To overcome this limitation, rather than requiring LLMs to directly answer the question, we propose a novel approach where we reframe the \textbf{Q} uestion \textbf{A} nswering task \textbf{a} s \textbf{P} rogramming ( \textbf{QAaP} ). Concretely, by leveraging modern LLMs’ superior capability in understanding both natural language and programming language, we endeavor to harness LLMs to represent diversely expressed text as well-structured code and select the best matching answer from multiple candidates through programming. We evaluate our QAaP framework on several time-sensitive question answering datasets and achieve decent improvement, up to 14.5% over strong baselines.

## introduction in our pursuit of knowledge and understanding, we often rely on factual questions to uncover the truth about the world around us. however, a critical aspect that is often overlooked is the presence of underlying temporal constraints within these questions. time, an important dimension in the physical world, emerges as a ubiquitous and formidable constraint that can significantly impact the accuracy and relevance of the answer. consider, for instance, the seemingly straightforward question: "who is the current president of the united states?". while the answer may appear apparent, its validity depends entirely on the specific point in time when the question is posed. this realization underscores the importance of recognizing and accounting for these temporal constraints in our quest for reliable and meaningful information. recently, llms have exhibited excellent intelligence in question answering and challenged the dominance of traditional search engines. nevertheless, despite their impressive performance, relying solely on llms for question answering still faces many shortcomings and limitations. as illustrated in fig. 1 (a), current approaches to utilizing llms for question answering can be primarily categorized into two types: (1) relying on the internal knowledge of llms to generate direct answers, such as chain-of-thought (cot) prompt @xcite and (2) presenting llms with retrieved documents for reading and subsequent answering, such as react @xcite . however, there are several challenges in answering time-sensitive factual questions with llms: 1) llms are insensitive to numbers @xcite and struggle to comprehend the sequential, overlapping and inclusive relationships between dates, which can be attributed to their nature as probabilistic models and their inability to perform rigorous reasoning as symbolic systems; 2) it is difficult for llms to directly locate the relevant information and provide the correct answer, as relevant facts may often be scattered across a long document and expressed in diverse ways, as a result, simply searching for keywords in the question or relying on retrievers may fail to identify the relevant paragraphs; 3) it is very likely that the answer provided by llms fails to satisfy the constraint and it is hard to verify the correctness of the answer since they are given in natural language. in this work, we endeavor to bridge the gap between existing approaches and the specific challenge posed by time-sensitive factual questions that require both rich world knowledge and intricate reasoning. to tackle the aforementioned challenges, we seek to harness llms' strong ability in natural language and programming language understanding to reframe the question answering task as programming (qaap), depicted in fig. 1 (b) . specifically, our method consists of two phases: 1.represent all as codes, since llms cannot rigorously reason by themselves, we endeavor to harness their strong coding ability to transform the question and context into well-structured codes. we first parse the given question into a python dict, then extract relevant information from the provided context and store these items in a python list. this allows a comprehensive gathering and organizing of relevant information dispersed throughout the documents. besides, the collected code format information helps to facilitate the subsequent processing and avoid reasoning based on surface-level text semantics, 2.choose answer through programming, due to the notorious hallucination inherent in llms, it is necessary to check if the extracted contents are faithful to the corresponding context. furthermore, as there may be multiple potential answers, we need to reason out the best-matching answer to the question. since all the obtained in-formation is represented as codes, we can easily construct two functions check and match to reduce hallucination and ensure accuracy. with our approach, we move beyond the traditional paradigm of using llms to directly generate an answer or read a given context and then provide an answer, avoiding the inability to verify whether the answer satisfies the constraints set out in the question. furthermore, by storing intermediate information in code, we overcome the length limit of the model input, thus empowering llms to read through the passages and uncover the eligible answer concealed within the lengthy documents. experimental evaluation on multiple time-sensitive question-answering datasets shows that our approach consistently outperforms strong baselines and approximates the performance of supervised methods. notably, we achieve up to 14.5%, 10.5% and 8.6% absolute improvements on timeqa, tempquestions and timequestions over state-of-the-art few-shot methods. in summary, our contributions are as follows. ## conclusion in this work we propose a novel approach, qaap (question answering as programming), to tackle the challenges posed by time-sensitive factual questions. by leveraging llms' exceptional abilities in natural language understanding and programming, qaap can transform diversely expressed text into well-structured codes, enabling llms to capture both the desired knowledge and the underlying constraints, particularly the temporal aspects. experiments demonstrate that existing llms face significant difficulty in effectively comprehending the temporal constraint stated in the question. while our approach consistently demonstrates superior performance over strong baselines with llms. we hope this work can shed light on the future research direction on enhancing llms' reasoning ability to tackle real-world questions with various constraints and developing more efficient methods to reduce the hallucinations llms frequently encounter.