Handling Ontology Gaps in Semantic Parsing (2024)

Abstract: The majority of Neural Semantic Parsing (NSP) models are developed with the assumption that there are no concepts outside the ones such models can represent with their target symbols (closed-world assumption). This assumption leads to generate hallucinated outputs rather than admitting their lack of knowledge. Hallucinations can lead to wrong or potentially offensive responses to users. Hence, a mechanism to prevent this behavior is crucial to build trusted NSP-based Question Answering agents. To that end, we propose the Hallucination Simulation Framework (HSF), a general setting for stimulating and analyzing NSP model hallucinations. The framework can be applied to any NSP task with a closed-ontology. Using the proposed framework and KQA Pro as the benchmark dataset, we assess state-of-the-art techniques for hallucination detection. We then present a novel hallucination detection strategy that exploits the computational graph of the NSP model to detect the NSP hallucinations in the presence of ontology gaps, out-of-domain utterances, and to recognize NSP errors, improving the F1-Score respectively by ~21%, ~24% and ~1%. This is the first work in closed-ontology NSP that addresses the problem of recognizing ontology gaps. We release our code and checkpoints at https://github.com/amazon-science/handling-ontology-gaps-in-semantic-parsing.

## introduction semantic parsing (sp) is one of the long-standing tasks in natural language understanding, aiming at mapping complex natural language to machinereadable languages (e.g., sql, sparql, kopl @xcite , and so on). these languages, which we will refer to as meaning representation languages (mrls), are designed to be precise representations of the natural language's intent, enabling efficient querying of a knowledge base (kb) to retrieve pertinent answers in a question answering (qa) agent. despite the advent of the transformer architecture @xcite , which has enabled semantic parsers to achieve extraordinary performance @xcite @xcite , semantic parsing's crux remains the handling of out-of-ontology queries; in other words, since sp models and tasks (such as kqa-pro @xcite , lc-quad 2.0 @xcite , and qald-9 @xcite ) hold a closed-world assumption, they will always try to map an utterance to a mrl even if there is no valid representation for that utterance in the target ontology, leading to wrong answers to be delivered to the model's users, called hallucinations. in fact, the closed-ontology task formulation enforces nsp models to always produce interpretations without an option to admit their lack of knowledge, inducing the models to hallucinate. therefore, the resulting models produce hallucinated outputs when they receive an utterance that requires symbols outside of their ontology, resulting in a wrong and potentially offensive answer. it is then of paramount importance to develop a system able to detect and prevent these hallucinations, so that users are not exposed to such mistakes. hallucinations in nsp differs with the notion of hallucinations in natural language generation, we report the differences in appendix a. to better understand different types of hallucinations in nsp, we classify errors into four macro categories. given a semantic q&a parsing task t , a dataset d, and an ontology o, hallucinations of a model trained over d are classified as: we show actual closed ontology semantic parsing hallucination examples in figure 2 and we report more in appendix h. high performance in detecting ood utterances in nsp can be achieved @xcite , and non-executable outputs are trivially detectable as they fail to parse; but identifying both in-ontology and out-of-ontology errors can be hard even for ex-perienced annotators, since the sheer size of most popular ontologies makes it impractical for a human to have a complete view of all the ontology symbols foot_0 . moreover, to the best of our knowledge, there are no works addressing this specific nsp issue. the research question that we want to address is: what is the most effective strategy to prevent a nspbased qa agent to deliver wrong, and potentially offensive, answers to its users? to this end, we develop the hallucination simulation framework depicted in figure 1 ; in detail, our main contributions are: • we propose a framework to stimulate, analyze and detect hallucinations in closed ontology nsp; • we propose the hallucination detection model (hdm), an architecture that analyzes an nsp model to determine whether it is hallucinating or not using several hallucination detection signals; • we introduce a model's activations as hallucination detection signals; when combined with other signals, they improve the macro f1-score by up to 21% in ontology gaps, 1% in nsp error, and 24% in ood detection. to the best of our knowledge, this is the first work that addresses the ontology gaps problem in a closed-ontology nsp task. ## related work when we do not allow models to admit their lack of knowledge, forcing them to produce an output even when they do not have the instruments to do it, they will inevitably hallucinate. in other words, in generative nlp, when the generated output displays a misunderstanding of the input utterance by the model, we say that the model is "hallucinating". typically, models hallucinate in two ways: (1) inventing additional information not included in or related to the input utterance, or (2) confusing a symbol/word with another one. one of the biggest assumptions in existing semantic parsing tasks is that every input always has a valid target logical form. in such a setup, models are always forced to generate a mrl or, in other words, to hallucinate a wrong understanding, instead of admitting a lack of knowledge. however, recently the nlp community has begun to investigate this closed-world assumption for other tasks. for example, the extractive question answering dataset squad v1 @xcite was built with the assumption that, given each question-paragraph pair, it is always possible to find an answer to the question in the paragraph. this assumption was removed in the second version of the dataset @xcite , which includes questions without an answer. another field in which this problem was addressed is entity linking, where models can produce a nil entity when they cannot find a suitable entity for a certain mention @xcite . on the other hand, most of the hallucination detection in nsp works rely on two confidence estimation techniques: (1) the sequence log-probability (also called confidence score) @xcite , or (2) monte carlo dropout (or dropout perturbation) @xcite @xcite . ## closed world assumption in nsp: a logical theory perspective the closed world assumption (cwa) originates from logic theory, and it is the assumption that only the known facts are correct, and what is not known is false @xcite . in other words, the cwa assumes total knowledge over a domain, implying that all the possible symbols (e.g., entities and predicates) are known, and that only the known facts represented using the known symbols are true. on the other hand, the open world assumption (owa) makes no assumption over what is not known; in other words, the owa allows "gaps" in the knowledge, e.g. the existence of unknown symbols or of unknown, but true, facts. for some tasks, using the cwa is safe. for exgiven a question that requires this symbol, the model produces a wrong but executable mrl leading to a wrong answer served to its user. even if this mrl is syntactically correct, it misrepresents the input due to the limitations of the training set. since the mrl is executable, it will lead to the generation of an incorrect answer. ## hallucination detection strategies in this section, we introduce the hallucination detection strategies that we use in our experiments. autodetect hallucinations a baseline approach to detect hallucinations is to enable the nsp model itself to decide whether to reject the mrl or not, in a similar fashion to the nil entity in @xcite . therefore, we add a new ontology symbol @xmath0, as a label for all the out-of-ontology sentences, i.e. moving from a cwa approach to a owa one. instead of using the nsp and hdd datasets, as we don't rely on external hallucination detection strategies, we train the nsp model using the full o dataset , marking mrls containing o unknown_symbols samples as utterances to reject. in preliminary experiments, this approach resulted in zero true positives. this happens because the model memorized the utterances marked as out-of-ontology, hence failing to generalize on the "unseen" unknown symbols in the development and test set (see section 4.2 for how the disjoint train, validation and test sets are constructed). confidence score confidence score (cs) is a standard method to detect hallucinations @xcite that measures the confidence level of a statistical model about the output it generates. however, this method relies on the strong assumption that the model will not be confident when generating hallucinations, and vice versa. this is not always guaranteed in practice: as we can see in the cs distribution in figure 4 , the confidence distributions of correct and wrong model predictions overlap. for this reason, rejecting model predictions below a certain threshold would not be sufficient to remove all the wrong mrls. to compute the cs, we calculate the posterior probability (pp) of a generated mrl w n , ..., w 1 from the beam search tree, and then we normalize it by the length n of the generated output, by applying the nth-root. @xmath1, w n-1 , ..., w 1 ) we test cs in two ways: (1) by setting a threshold to the best cs value found in a sample from the hdd train set that maximizes the hallucination detection in the hdd dev set; (2) and by using it as a feature in the hallucination detection model (hdm) that we will define in section 6. ## experimental setup dataset while the hsf is dataset-agnostic, in our experiments, we use the kqa-pro dataset @xcite , based on the kopl (knowledgeoriented programming language) mrl; this dataset is built on top of a large ontology, which is a subset of wikidata. we instead sourced ood sentences from the top v2 dataset @xcite , that contains task oriented utterances, such as "turn on the lights!". to create a test set, we merged the train and the validation set, and we split the data as follows: 60%, 20%, and 20%, respectively, for the train, validation and test set. the statistics of the hsf framework applied to the kqa-pro dataset are reported in appendix b. nsp model following the kqa-pro paper, we train the bart-base model @xcite , using the nsp training dataset. we report the hyperparameters that we use to train the nsp model in appendix m. note that as the original kqa-pro test set is not publicly available, we cannot compare our results with the original dataset paper. evaluation to measure the hallucination detection capabilities, we use the macro f1-score due to the imbalance of the dataset, as shown in appendix d and, e. we compute the individual f1-score for each type of hallucination defined in section 1: in-ontology nsp errors caused by the model hallucinating wrong symbols from o known_symbols , out-of-ontology errors caused by the need of symbols in o unknown_symbols to correctly represent the input, and zero-shot ood detection. as mentioned in section 7, we excluded non-executable mrls from our evaluation protocol because they are trivially detected by simply trying, and failing, to ex- @xcite , and on top of it we apply our hallucination detection strategies. we compute metrics only on the executable outputs that lead to an answer to be delivered to a user; for more details, see appendix f. ## discussion we report the performance of our nsp model using execution accuracy and the mrl exact match metric in from a baseline where only non-executable mrls are not delivered to the user, the hdm with activations + cs is our best-performing model, improving answer accuracy by 4% and mrl exact match by 10%, effectively reducing a user's exposition to wrong answers. the hdm with activations + cs's performance is achieved by increasing the macro f1-score by approximately 21% and 24% for out-of-ontology and out-of-domain detection w.r.t. baseline, respectively. on the other hand, the nsp errors detection performance is comparable to that of threshold cs, with only the hdm with the activations + cs + mcd combination showing a 1% improvement over the baseline in nsp error detection. this marginal gain can be attributed to the limited number of errors produced by our nsp model over utterances with known symbols only, which constitutes about 11% of the in-ontology utterances (see statistics in appendix j). however, we can notice that both cs and mcd, if optimized through the hdm, obtain large gains in terms of macro f1-score. in fact, cs improves by 17% and 10% in out-of-ontology and out-ofdomain detection, and mcd by 4%, 3%, and 10% in out-of-ontology, nsp errors, and out-of-domain detection. in addition, the hdm can combine multiple hallucination signals to obtain higher performance, as in the case of our most-performing system. for further insight, we report the precision and recall over each error category in appendix n. ## conclusions current studies of neural semantic parsing (nsp) models revolve around improving performance on academic benchmarks, but they do not take into account the trustworthiness of the model in a real world scenario where the model is used to serve answers to users of a qa system. in such scenario, nsp models can hallucinate syntactically correct, but semantically wrong mrls, that can be used to serve incorrect answers to users. this is particularly true when users ask questions that require knowledge beyond the one used by the model's target ontology, as in these cases the model simply cannot generate a correct mrl. to test nsp models under this more realistic scenario, we propose the hallucination simulation framework (hsf), where we programmatically induce nsp models to hallucinate, and then, using the hallucination detection model, we detect model errors at inference time using several different signals, including the model's activations or confidence score, or by using monte carlo dropout. we find that the best way to prevent detect hallucinations is using the hdm model with activations and cs as features, which leads to an average improvement of more than 20% w.r.t. a baseline where the only non-served mrls are just the syntactically incorrect ones.