The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources (2023)

Abstract: Many state-of-the-art natural language understanding (NLU) models are based on pretrained neural language models. These models often make inferences using information from multiple sources. An important class of such inferences are those that require both background knowledge, presumably contained in a model’s pretrained parameters, and instance-specific information that is supplied at inference time. However, the integration and reasoning abilities of NLU models in the presence of multiple knowledge sources have been largely understudied. In this work, we propose a test suite of coreference resolution subtasks that require reasoning over multiple facts. These subtasks differ in terms of which knowledge sources contain the relevant facts. We also introduce subtasks where knowledge is present only at inference time using fictional knowledge. We evaluate state-of-the-art coreference resolution models on our dataset. Our results indicate that several models struggle to reason on-the-fly over knowledge observed both at pretrain time and at inference time. However, with task-specific training, a subset of models demonstrates the ability to integrate certain knowledge types from multiple sources. Still, even the best performing models seem to have difficulties with reliably integrating knowledge presented only at inference time.

## introduction progress on natural language understanding (nlu) benchmarks has recently been driven by pretrained large language models (llms), which can be adapted to specific tasks via finetuning @xcite @xcite . these models draw on a variety of knowledge sources, such as knowledge given in inputs at inference time and knowledge contained in their parameters, usually acquired via pretraining. to effectively use these two knowledge sources, models must (1) retrieve relevant information from each knowledge source, (2) adjudicate between potentially conflicting information, and (3) integrate multiple units of information from both knowledge sources and reason over them on the fly. for example, pretrained parameters might contain the knowledge that donald trump is the president of the united states, but inference-time inputs might state that joe biden is the president. based on the contextual information available in a task, models must infer the correct president. studying whether current models can use multiple knowledge sources effectively can help identify and debug errors that models make when relying on outdated sources. to this end, we introduce a coreference resolution task designed to probe models' ability to draw on knowledge available in different sources. recent work by @xcite examined the effects of knowledge conflicts across different knowledge sources. in our work, we aim to more broadly examine the behaviour of nlu models in the presence of multiple knowledge sources. while @xcite study how models handle conflicting facts, our goal is to evaluate whether models can combine complementary knowledge drawn from multiple sources rather than choose between sources. in our task, the resolution of a given pronoun requires two types of knowledge (see figure 1 ): 1) entity-specific knowledge, e.g., "servin is a judge" and 2) background knowledge, e.g., "judges decide cases in law courts." generally, background knowledge is learned during the pretraining of llms i.e., at pretrain-time, while entity-specific knowledge is typically observed at inference time. we vary the availability of the required information such that it may either be found in a single source or in multiple sources. we evaluate a model's ability to integrate and reason over the two knowledge types (entity-specific and background knowledge), given in two knowledge sources @xcite . we propose kitmus, foot_0 a diagnostic test suite. the kitmus tests evaluate knowledge integration from multiple sources. kitmus's distinguishing feature is that it contains texts in which we methodically vary the mapping of knowledge types to knowledge sources, which allows us to pinpoint the specific strengths and limitations of models. we also analyze the behaviour of models when knowledge is available only at inference-time by introducing variants where a model needs to reason over fictional knowledge, which is presumably not contained in the parameters. unlike previous reasoning datasets, where inference-time knowledge is retrieved @xcite , we provide the knowledge necessary to solve the task in each instance of kitmus. this allows for a more controlled setting where we can focus on knowledge integration, rather than on retrieval, which we hold out as a separate problem. in a study with human participants, we empirically validated that both entity-specific and background knowledge are required to perform well on kitmus, and that the automatically generated labels are consistent with human annotations. we evaluate state-of-the-art coreference resolution models on the kitmus. in our experiments, many established models appear unable to integrate knowledge from two different knowledge sources and reason over them. with task-specific training, two models-bert4coref @xcite and c2f @xcite )-demonstrate the ability to reason over both knowledge observed at pretrain time and at inference time. however, we find that the ability to integrate knowledge from different sources seems to depend on the knowledge type in that source. while knowledge integration through concatenation at inference time seems to be effective for entity-specific knowledge, experiments with fictional knowledge indicate that even the best performing models cannot reliably integrate all types of background knowledge when provided only at inference time. ## the kitmus test suite we evaluate the knowledge integration capability of coreference resolution models from different knowledge sources: 1) pretrain-time: knowledge accumulated in the parameters during (pre-)training and 2) inference-time: knowledge observed in an input text. to design kitmus, we formulate a coreference resolution task which requires access to two facts. we systematically vary the presence of these facts across the knowledge sources to evaluate the models. as an instantiation of the idea of presenting two facts, we experiment with the following two knowledge types: -entity-specific: occupation of an entity e.g., "rosenow is an architect." -background: situation typical for an occupation e.g., "architects design building and houses." for example, consider the following task to predict whether mujica or rosenow is the correct antecedent of the pronoun "he." here, the occupations are model and architect, and the situational cue is designing building and houses. both knowledge types are required in order to resolve this coreference. an illustration of this knowledge schema can be found in figure 2 . we explore three main variants of the dataset as shown in figure 3 . with entity-specific knowledge always provided in the instance, the variants differ based on when and where background knowledge is available: -background-pretrain: background knowledge is available only in the model parameters -background-both: background knowledge is available in the model parameters and explicitly provided in the instance -background-inference: background knowledge is available only in the instance each instance of the kitmus task consists of two fragments of text that are concatenated: 1) a knowledge text-containing the inference-time knowledge that models are given access to-and 2) a task text-consisting of the coreference task that models solve. ## dataset creation to construct kitmus, we manipulate which entities are mentioned in each instance, what occupations those entities have, what situations those occupations pertain to, what contexts they are mentioned in, and whether noise is present. each entry is structured to first (1) introduce the entities, (2) then place them in the same location, and (3) finally, place one of them in a situation related to their occupation. in the background-both and background-inference variants, this is preceded by a knowledge text mapping entities to their respective occupations using the phrase "is a." the dataset entries are generated using handcrafted english-language templates and sampling from a variety of resource pools to fill the template slots. the use of templates facilitates control over the source a certain type of knowledge is stored in, which may not be possible to do with a natural dataset. we aim to minimize the likelihood of models learning to exploit any spurious correlations in the templates or resources and promote data diversity using the following methods: -we use multiple templates for each sentence. examples are shown in with these measures, we ensure that all entity names, occupations, situations, locations, templates, and noise statements that occur in the test instances do not occur in the train instances. ## discussion models trained on "general" coreference datasets fail on kitmus: the poor performance of ontonotes-trained models suggests that when trained on general coreference resolution datasets, models learn to exploit surface cues, which does not help when testing on kitmus where such cues are removed. another factor might be the structure of the texts in kitmus, which are designed to place knowledge in specific knowledge sources. this might affect models' abilities to form useful representations resulting in poor performance of ontonotes-trained models. these failures suggest that training on (what are meant to be) "general" datasets is not enough to induce knowledge integration from multiple sources and task-specific training is required. effect of dataset format and size: we observe that the models that accept input in the conll format @xcite perform better than those models that accept the gap format @xcite . this indicates that mention annotations in the knowledge text-which only the conll format provides-might be significant. to evaluate whether the failure cases are due to the small train set size of 2000, we repeat experiments with a train set size of 5000. while we do see some improvements, the general trends persist and our observations remain consistent with the previous results (see the limitations section for additional discussion). this suggests that further scaling of the train set size might not be sufficient to improve performance on cases where existing models are currently failing. performance of current pretrained llms: bert4coref seems to consistently outperform c2f. this might be due to the difference in pretrained llms: bert4coref uses the transformer architecture @xcite , which has been shown to be effective at reasoning tasks presented in natural language form @xcite and utilizing information presented in inference-time contexts @xcite , while c2f uses elmo @xcite . to verify that bert and elmo contain background knowledge mapping occupations to situations, we ran a lama probe @xcite . we find that bert is more likely to contain the background knowledge compared to elmo (see section 8 for details). this corroborates the better performance of bert on knowledge intensive tasks such as kitmus. integration of fictional knowledge: as shown in the work of a remaller is socring clatodemnly. nims is a mamser. formica is a remaller. the work of a mamser is slimbing murstly. at the birthday party, nims and formica ran into each other. the party is filled with local and national celebrities and entertainers. she shared experiences from a career of socring clatodemnly. one possible reason could be bert's tokenization strategy, which involves pooling subword representations @xcite . in character-level fictional words, the subwords are meaningless, rendering their representations unhelpful. this is consistent with previous work showing that representations of llms for character-level fictional "jabberwocky" words are less useful @xcite and that the presence of out-of-vocabulary (oov) words decreases performance of neural models for nlu tasks @xcite @xcite . despite the character-fictional occupations and situations, we expect the models to resolve the coreferences successfully in this setting. in the given example, the pronoun "she" can be resolved by matching the situation "socring clatodemnly" to the occupation "remaller" (using the word overlap between the situations and the occupation descriptions) and identifying the correct entity associated with the occupation i.e, formica. humans can successfully make these inferences by matching fictional occupations and situations. however, the current models do not perform better than a random baseline in this setting. our hope is that eventually, models should be able to handle even knowledge presented in previously unknown terms. given that languages are forever growing, robustness to neologisms is crucial, considering that oov words e.g., new occupations like "tik-toker" develop constantly. ## conclusion we investigated the ability of models to integrate knowledge from multiple knowledge sources to resolve linguistic ambiguities in a coreference resolution task. we formulated a task that requires access to two knowledge types, entity-specific and background knowledge, and controlled for two knowledge sources that knowledge is available in, pretrain-time and inference-time. our results show that with task-and datasetspecific training, some models have the ability to reason over both knowledge observed at pretrain time and at inference time. for these models, knowledge can be integrated by concatenating textual knowledge to the model inputs. furthermore, our findings imply that supplying additional information (e.g., from a retriever) at inference time to models can be successful even if the knowledge required for the task has not been observed before. however, in our task this ability seems to require task-specific training and depend on the type of knowledge being supplied. future work could explore finetuning models on kitmus to encourage knowledge integration across different sources. one might also consider extending the kitmus test suite to other languages or to create a multilingual test suite. instructions for using our code and adapting the templates and resources to other languages can be found in appendix a.1.