Beyond Boundaries: A Human-like Approach for Question Answering over Structured and Unstructured Information Sources (2024)

Abstract: Answering factual questions from heterogenous sources, such as graphs and text, is a key capacity of intelligent systems. Current approaches either (i) perform question answering over text and structured sources as separate pipelines followed by a merge step or (ii) provide an early integration, giving up the strengths of particular information sources. To solve this problem, we present “HumanIQ”, a method that teaches language models to dynamically combine retrieved information by imitating how humans use retrieval tools. Our approach couples a generic method for gathering human demonstrations of tool use with adaptive few-shot learning for tool augmented models. We show that HumanIQ confers significant benefits, including i) reducing the error rate of our strongest baseline (GPT-4) by over 50% across 3 benchmarks, (ii) improving human preference over responses from vanilla GPT-4 (45.3% wins, 46.7% ties, 8.0% loss), and (iii) outperforming numerous task-specific baselines.

## introduction the ability to answer factual questions reliably is one of the hardest problems in ai. large language models (llms) have shown promise on factual question answering (qa) tasks. however, llms that rely only on internal parameters can become stale and do not perform well on tail knowledge @xcite . moreover, it is difficult to curate llm's parametric knowledge or attribute responses to information sources. this has led to research in which llms are equipped with external tools to retrieve information from external sources @xcite to reduce hallucinations and establish a connection between sources of evidence and the generated response to a question. such external information sources can be in structured form, e.g., in the shape of a knowledge graph, or in unstructured form, e.g., text corpora. textual sources usually provide higher recall whereas structured sources offer potentially higher precision and support executing complex queries. the field of hybrid question answering, which pre-dates the rise of llms, has investigated approaches to answer questions by retrieving information from structured and unstructured information sources. a main approach to hybrid qa (a field sometimes also referred to as heterogeneous qa) is to use late fusion: in this line of research, the input question is processed by separate systems optimised for specific types of information sources and the responses are merged or selected afterwards. while this has the advantage of being easy to parallelize, a drawback is that the different information sources do not interact with each other, i.e., evidence found in one source does not influence pipelines for other sources. to remedy this, early fusion qa approaches integrate information sources into a unified format before performing inference steps. this allows to run a single pipeline over all sources. however, the unified format represents a least common denominator of the input sources and does not allow to use the strength of different input sources: (i) the flexibility and generality of unstructured representations and (ii) the ability to perform compositional queries to aggregate, combine, sort, and order structured data. to overcome these restrictions, we propose to follow a new direction in this field: rather than performing early or late fusion, we integrate inference steps over heterogeneous information sources by mimicking how humans find responses to question. we find that human reasoning tends to transcend the boundaries of different information sources and can benefit from mutually complementary information. specifically, we propose figure 1 : overview of aggregated evaluation results for humaniq applied to hybrid question answering (hqa): on the left, the percentage of correctly answered questions over 3 qa benchmarks is plotted. on the right, the results of a human evaluation using questions from the same benchmarks is shown. for each question, annotators could indicate preference for the output of a system. detailed results can be found in section 5. the humaniq (''human-like intelligent queries'') methodology for collecting human solution processes and using those to prompt llms. the methodology can be applied to tasks that use tools and benefit from human-like thought processes. 1 the methodology is divided into a preparation stage, performed by humans, and an inference stage. during the preparation stage, the method requires to identify suitable tools and solution processes using those tools. for hybrid qa, this includes search and retrieval over unstructured information, entity linking, and querying over structured information. solution processes describe how those tools can be combined to find an answer for a particular question, in particular, they also contain thoughts in natural language humans have when using those tools. at inference stage, we aim to replicate those solution processes. we use an llm-based approach in which we provide tools to an llm and the reasoning is done by generating thoughts about the usage of those tools and the interpretation of their results (also called observations). we achieve human-like reasoning by injecting complete solution processes as few-shot examples in the llm prompt. we adapt the prompt to the input by selecting few-shot examples based on their similarity to the input as well as maximizing the diversity of their tool usage. the method does not require fine-tuning the llm. we tested our approach on three different qa benchmarks and observed performance improvements over the used gpt-3.5 and gpt-4 base models as illustrated in figure 1 (left ). an additional benefit over vanilla llms is that we also obtain a reasoning trace including used sources and thoughts about observations. we observed large improvements over task-specific baselines (see section 5) and conducted a human evaluation study showing that users prefer our system over gpt-4, gpt-3.5, and mpt (figure 1 right). ## humaniq methodology the humaniq methodology is illustrated in figure 2 . we first cover its task preparation stage. the input to this stage is a set of instances of a task sampled from the training set. humans proficient at the task then need to decide which tools (also called services or apis) are most relevant for solving the task. those tools are then registered at one of the available llm-tool-augmentation frameworks, e.g., langchain foot_1 or openai function calling, foot_2 and connected to the humaniq user interface (see figure 3 ). the user interface follows the react @xcite approach dividing steps into actions (= commands for invoking tools), observations (= tool output), and thoughts (= natural language reasoning) until the llm outputs a final response. we call the combination of all of those steps a solution process. the user interface can be applied on concrete instances of a task to construct a human-like solution process. action sequences can differ substantially for different task instances. for example, some factual questions require more work around entity disambiguation, e.g., distinguishing between the ''the hunger games'' movie, movie series, or book. in some cases multiple tries are needed to construct a query returning the desired information or retrieving documents useful for solving a question. while applying the tool-augmented llm on particular task instances, humans can identify whether particular tools are missing or should be implemented differently to follow a human-like solution process. for example, in this initial phase we added a linker to map from individual documents to entity ids enabling a bridge between unstructured and structured information sources. this improvement cycle is depicted in the left in figure 2 . once the tool set is established, a set of solution processes demonstrating desired behavior is constructed resulting in few-shot library. to be sample efficient, this can be done in the spirit of active learning using a verify & fix approach: tool-augmented llms can be automatically applied on task instances in the training set and, when those fail, humans can manually build successful solution processes for those instances. during this process, the tool outputs (observations) can be rated (e.g., on a 1-5 likert scale). this can be used as supervised feedback signal for individual tools. the second part of the humaniq methodology is the inference stage aiming to replicate human solution processes: we use a base llm that we instruct to follow the general pattern for solution processes as well as announcing the tools that are available along with instructions for solving the task (e.g., ''answer the question below by accessing the information sources with the given tools!'') and few-shot examples. each few-shot example contains the entire solution process and can typically span several hundred tokens, i.e., we primarily instruct the model to follow the process, which is similar in spirit to @xcite , rather than focusing only on output. given llm context length limitations, we can usually not include all elements on the few-shot library when applying humaniq in an in-context learning setting. to select a set of few-shot examples, we therefore use a determinantal point process (dpp) @xcite to obtain solution processes that are relevant and diverse. relevance in our case is measured by the similarity to the input question using sentencebert. intuitively, similar questions should be more relevant demonstrations of the desired behavior. however, using this as sole criterion can lead to selecting examples which all exhibit very similar solution processes. therefore, we include example diversity as objective. the diversity of two given solution processes is measured via the levenshtein distance of their actions with each action encoded by a different single symbol. the dpp based on using the above similarity and diversity metrics is then conditioned to a fixed number k of desired few-shot examples. we search the configuration of examples with the highest probability density using a greedy optimiser @xcite and include those in the llm prompt. executing the inference stage successfully is a demanding task for llms: the model needs to learn human thought patterns by example, understand the intricate relationship between different tools, observations, and needs to support long context as we inject full solution processes with multiple thoughts and tool invocations. ## related work the main areas related to our work are early/late fusion approaches for hybrid qa, and information integration and human-like thoughts in llms. ## conclusions and future work we introduced the humaniq methodology and showed that applying it to hybrid question-answering allows reasoning over heterogeneous information sources in the llm prompt leading to state-of-the-art results on three benchmarks. users preferred the system over vanilla llms in a human evaluation study. a natural next step is to apply humaniq in other settings than factual qa. furthermore, we aim to investigate prompt compression and memory techniques. another interesting direction are multilingual nlp tasks, in particular retrieving from sources in different languages and using this to build evidence in the required target language.