Self-Supervised Rule Learning to Link Text Segments to Relational Elements of Structured Knowledge (2023)

Abstract: We present a neuro-symbolic approach to self-learn rules that serve as interpretable knowledge to perform relation linking in knowledge base question answering systems. These rules define natural language text predicates as a weighted mixture of knowledge base paths. The weights learned during training effectively serve the mapping needed to perform relation linking. We use popular masked training strategy to self-learn the rules. A key distinguishing aspect of our work is that the masked training operate over logical forms of the sentence instead of their natural language text form. This offers opportunity to extract extended context information from the structured knowledge source and use that to build robust and human readable rules. We evaluate accuracy and usefulness of such learned rules by utilizing them for prediction of missing kinship relation in CLUTRR dataset and relation linking in a KBQA system using SWQ-WD dataset. Results demonstrate the effectiveness of our approach - its generalizability, interpretability and ability to achieve an average performance gain of 17% on CLUTRR dataset.

## introduction relation linking is a key step in many natural language processing (nlp) tasks including semantic parsing, triple extraction, and knowledge base question answering (kbqa). its goal is to accurately map predicate components of a natural language (nl) text segment onto their corresponding predicate (i.e., relational) elements within a knowledge base (kb). such a mapping would enable question answering systems to exploit the deep † ibm research, bangalore/gurugram, india. ‡ ibm research, t. j. watson research center, yorktown heights, new york, united states. various approaches have been tried in the past work, including graph-based approaches (pan et al., 1 provided entity mentions in the nl text are also appropriately mapped to their corresponding entity elements in the kb, through entity linking @xcite . for example, in figure 1 , entity mention kate winslet in nl text is mapped to corresponding kb (wikidata) entity wd:q202765. figure 2 : illustration of human readable rule for the example shown in figure 1 and the corresponding extended knowledge base context shown in figure 3 . note that inverse knowledge base predicate (i.e., reverse knowledge base relation) in the rule is denoted by superscript -1 , as described in section 3.2. 2019; @xcite , linguistic features based approaches @xcite , and recently neural approaches that achieve stateof-the-art performance @xcite @xcite . however, these approaches have the following drawbacks: they are (a) data hungry requiring huge amount of supervised data for training, (b) non-interpretable by design, and (c) non-trivial to adapt and generalize to new domains or knowledge graphs. in this paper, we propose a neuro-symbolic approach that formulates relation linking as a rule learning problem in the space of structured knowledge sources. our approach tries to learn human readable rules of the form illustrated in figure 2 and use them to perform relation linking, thus resulting in explainable relation linking model. in contrast to the existing approaches that build linking models in a supervised fashion, we adopt a self-supervised learning approach that offer benefits: does not require annotating large amounts of training data, and offers generalization across domains. for self-supervision we use masked training, a popularly used self-supervised training strategy for large language models. however, in contrast to the use of nl text as in the large language models (llms), masked training in our approach operates on logical statements of the sentences. in this work, we use triples foot_0 extracted from semantically parsed output of the sentences as the logical statements. an example sentence in its nl form and its corresponding triple form (logical statement) are shown in figure 1 . the masking procedure when applied on a triple may end up masking any of the three elements, i.e., subject, object, or predicate. figure 1 shows an illustration of the predicate component getting masked. given that the three elements of a triple play different semantic roles, the masked training need to be adapted to suit the role played by the element being masked. as will be seen later in the paper, such differences reflect mainly on the loss computation during training. the main contributions of our work are: relation linking has been important for various nlp tasks such as semantic parsing, knowledge graph induction @xcite @xcite and knowledge base question answering @xcite @xcite . prior to the surge of generative models, relation linking was addressed either by graph traversal based @xcite or by linguistic-features based methodologies @xcite . several learning based approaches to relation linking have been proposed @xcite @xcite . most recent approaches to relation linking have focused on generative models @xcite . these approaches are data hungry and non-interpretable. in contrast, our work is a self-supervised rule learning based approach for relation linking. the learnt rules are human readable and they can be learnt for different knowledge bases as context for different domains. rule learning, specifically in terms of detecting relations has also been popular for knowledge base completion (kbc) task. kbc problems are addressed in two different ways in the literature. rule based kbc learns explainable rules that can be used to predict the missing links in the knowledge base @xcite @xcite @xcite . these rules are learned for predicates in the knowledge base in terms of other predicates and paths in the knowledge base. our work uses the rule model and some of the ideas from the rule based kbc to build self supervised explainable rules for linking text elements to knowledge base relations. there have also been several learning based methods that use vector embeddings of the entities and relationships and then use them to predict any missing relationships @xcite @xcite @xcite @xcite ; they have the same drawbacks as neural relation linking approaches. there are many recent works that use semi-supervised/unsupervised learning for rules and knowledge induction @xcite @xcite . transformers @xcite have given rise to a wide range of masked pre-training models that try to learn latent representation of words. masked language modeling has gained popularity in building large pre-trained models that learn word representations that can easily be adapted to a variety of tasks. masked language training has resulted in building various model architectures like encoder only models which learn a vector representation of tokens using masked training @xcite @xcite . several encode-decoder llms are built using similar training strategy @xcite @xcite . recently decoder only models built using similar masked language training are gaining popularity @xcite @xcite @xcite . in this paper we take inspiration from masked language training on text and use the ideas to perform similar masking on logic statements to learn rules. ## our approach our approach to relation linking is a rule-learning framework where rules learned in a self-supervised fashion are used to map from 'nl text segments' to the 'elements of structured knowledge'. for self-supervised rule learning, we use masked training. a distinctive aspect of the masked training in our approach is that it is applied over the logical form of the input text. when logical form is not readily available, we first convert nl text into its logical form using a semantic parser, before applying masked training. in this section, first we briefly describe the conversion of input text to logical statements, and then describe the proposed rule model and its self-supervised training. ## conclusions in this paper, we presented a novel masked training based self rule learning approach for relation linking. the proposed approach takes inspiration from the masked training of large language models and rule learning in knowledge base completion to create a rule learning model that results in a relation linking model that is interpretable and generalizable to any knowledge base and domain. our evaluation on two datasets clutrr (rule learning for kinship relations) and swq (relation linking for wikidata) shows that our model is able to produce high quality rules for relation linking that generalize across domains. our next steps are to scale to learn from large amount of data, extend the framework beyond triples. we see a lot of potential for our rule learning framework beyond relation linking, like general knowledge acquisition.