D omi K now S : A Library for Integration of Symbolic Domain Knowledge in Deep Learning (2021)

Abstract: We demonstrate a library for the integration of domain knowledge in deep learning architectures. Using this library, the structure of the data is expressed symbolically via graph declarations and the logical constraints over outputs or latent variables can be seamlessly added to the deep models. The domain knowledge can be defined explicitly, which improves the explainability of the models in addition to their performance and generalizability in the low-data regime. Several approaches for such integration of symbolic and sub-symbolic models have been introduced; however, there is no library to facilitate the programming for such integration in a generic way while various underlying algorithms can be used. Our library aims to simplify programming for such integration in both training and inference phases while separating the knowledge representation from learning algorithms. We showcase various NLP benchmark tasks and beyond. The framework is publicly available at Github( https://github.com/HLR/DomiKnowS ).

## introduction current deep learning architectures are known to be data-hungry with issues mainly in generalizability and explainability @xcite . while these issues are hot research topics, one approach to address them is to inject external knowledge directly into the models when possible. while learning from examples revolutionized the way that intelligent systems are designed to gain knowledge, many tasks lack adequate data resources. generating examples to capture knowledge is an expensive and lengthy process and especially not efficient when such a knowledge is available explicitly. therefore, one main motivation of our proposed framework (domiknows) is to facilitate the integration of domain knowledge in deep learning architectures, in particular when this knowledge is represented symbolically. in this demonstration paper, we highlight the components of this framework that help to combine learning from data and exploiting knowledge in learning, including: 1) learning problem specification 2) knowledge representation 3) algorithms for integration of knowledge and learning. currently, domiknows implementation relies on py-torch and off-the-shelf optimization solvers such as gurobi. 2 however, it can be extended by developing hooks to other solvers and deep learning libraries since the interface is generic and independent from the underlying computational modules. in general, the integration of domain knowledge can be done 1) using pretrained models and transferring knowledge @xcite , 2) designing architectures that integrate knowledge expressed in knowledge bases (kb) and knowledge graphs (kg) in a way that the kb/kg context influences the learned representations @xcite , or 3) using the knowledge explicitly and logically as a set of constrains or preferences over the inputs or outputs @xcite @xcite . our current library aims at facilitating the third approach. it is still an open problem to know which method of integrating prior knowledge with neural modules is the best and the performance heavily relies on each task specifications and experimental settings. however, there are many ongoing research on the integration of soft/hard constraints @xcite @xcite on the output variables of neural modules which shows the effectiveness of such approaches in gaining better performance, especially on low-resource or even semi-supervised tasks @xcite . while applying the constraints on input is technically trivial and could be done in a data pre-processing step, applying constraints over outputs and considering those structural constraints during training is a research challenge @xcite @xcite . this requires encoding the knowledge at the algorithmic level. however, given that the constraints can be expressed logically and symbolically, having a language to express such a knowledge in a principled way is lacking in the current machine learning libraries. using our developed domiknows library, the domain knowledge will be provided symbolically and by the user utilizing a logical language that we have defined. this knowledge is used in various ways: a) as soft constraints by considering the violations as a part of loss function, this is done using a prim-dual formulation @xcite and can be expanded to probabilistic and sampling-based approaches @xcite or by mapping the constraint to differentiable operations @xcite ) b) mapping the constrains to an integer linear program setting and performing inference-based training by masking the loss @xcite . independent form the training paradigm the constraints can be always used as hard constraints during inference or not used at all. an interactive online demo of domiknows is available at google colab foot_2 and the framework is accessible on github foot_3 . ## related research integration of domain knowledge in learning relates to tools that try to express the prior or posterior information about variables beyond what is in the data. this relates to probabilistic programming languages such as @xcite , venture @xcite , stan @xcite , and infernet @xcite . the logical expression of domain knowledge is used in probabilistic logical programming languages such as problog @xcite , prism @xcite , the recent version of problog,i.e., deep problog @xcite , statistical relational learning tools, such as markov logic networks @xcite , probabilistic soft logic @xcite , bayesian logic (blog) @xcite , and slightly related to learning over graph structures @xcite . considering the structure of the output without its explicit declaration is considered in structured output prediction tools @xcite . our library is mostly related to the previous efforts for learning based programming and the integration of logical constraints in learning with classical machine learning approaches @xcite @xcite . our framework makes this connection to deep neural network libraries and arbitrarily designed architectures. the unique feature of our library is that, the graph structure is defined symbolically based on the concepts in the domain. unlike torch-struct @xcite , our library is independent from the underlying algorithms, and arbitrary structures can be expressed and used based on various underlying algorithms. in contrast to deepproblog, we are not limited to probabilistic inference and any solver can be used for inference depending on the training paradigm that is used for exploiting the logical constraints. probabilistic soft logic is another framework that considers logical constraints in learning by mapping the constraint declarations to a hing loss markov random field @xcite . drail is another declarative framework that is using logical constraints on top of deep learning and converts them to an integer linear program at the inference time @xcite . none of the above mentioned frameworks accommodate working with raw sensory data nor help in putting that in an operational structure that can form the domain predicates and be used by learning modules while our framework tries to address that challenge. we support training paradigms that make use of the inference as a black box and in those cases any constraint optimization, logical inference engine or probabilistic inference tools can be integrated and used based on our abstraction and the provided modularity. ## declarative learning-based programming we use the entity-mention-relation (emr) extraction task to describe the framework. we discuss more showcases in section 5. given an input text such as "washington is employed by associated press.", the task is to extract the entities and classify their types (e.g., people, organizations, and locations) as well as relations between them (e.g., works for, lives in). for example, for the above sentence [washington] is a person [associated press] is an organization and the relationship between these two entities is work-f or. we choose this task as it includes the prediction of multiple outputs at the sentence level, while there are global constraints over the outputs. 5 in domiknows, first, using our python-based specification language, the user describes the problem and its logical constraints declarativly and independent from the solutions. second, it defines the necessary computational units (here, pytorchbased architectures) and connect the solution to the problem specification. third, a program instance is created to execute the model using a background knowledge integration method with respect to the problem description. ## learning & evaluation in domiknows to execute the defined model considering the specified conceptual graph, domiknows uses program instances. a program instance is responsible to run the model, apply loss functions, optimize the parameters, connect the output decisions to the inference algorithms, and generate the final results and metrics. executing the program instance relies on the problem graph, model declaration, dataloaders and a backbone data structure called datanode. dataloader provides an iterable object to loop over the data. datanode is an instance of the conceptual graph to keep track of the data instances and store the computational results of the sensors and learners. for the emr task, the program definition is as follows: here, the concepts passed to the poi field specifies the training points of the program. this enables the user to train the task based on any subsets of the concepts defined in the model. for each program instance, the user should specify the domain knowledge integration method. the available methods for integration is discussed in the next sections. after initializing the program, the user can call train, test, and prediction functionalities to train and evaluate the designed model. the below snippet is to run training and evaluation on the emr task: here, the user will specify the dataloaders for different sets of the data and the hyper-parameters required to train the model. programs can be composed to address different training paradigms such as end-to-end or pipeline training by defining different training points for each program. more details are available in the appendix. following is an alternative program definition for pre-training the phrases first and then learning based on the pairs: ## showcases the effectiveness of ilp @xcite , iml @xcite , and primal-dual (nandwani et al., 2019a) methods have been already shown in their respective papers. here, we provide different tasks and settings to showcase our framework's abilities and flexibility to model various problems. the results, models implementation and details of experiments are (partially) available in the supplementary part of this paper and (fully) in the github repository of domiknows. foot_5 ## conclusions and future work domiknows makes it effortless to integrate domain knowledge into deep neural network models using a unified framework. it allows users to switch between different algorithms and benefit from a rich source of abstracted functionalities and computational modules developed for multiple tasks. it allows naming concepts, defining their relationships symbolically and combining symbolic and sub-symbolic reasoning over the named concepts. domiknows helps in interpretability of neural architectures by providing named layers and access to the neural computations at each stage of the training and evaluation process. as a future direction, we are looking to enrich our library with predefined functionalities and neural models and further extend its ability to support more techniques on integration of domain knowledge with deep neural models as well as seamless model composition. more information about technical details and the documentation of domiknows is publicly available at our website 9 and on github. 10 fusion of knowledge bases and text. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4231-4242. niket tandon, bhavana dalvi, keisuke sakaguchi, peter clark, and antoine bosselut. 2019. wiqa: a dataset for "what if..." reasoning over procedural text. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6076-6085, hong kong, china. association for computational linguistics. jingyi xu, zilu zhang, tal friedman, yitao liang, and guy van den broeck. 2018. a semantic loss function for deep learning with symbolic knowledge. in proceedings of the 35th international conference on machine learning, volume 80 of proceedings of machine learning research, pages 5502-5511. pmlr. bishan yang and tom mitchell. 2017. leveraging knowledge bases in lstms for improving machine reading. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1436-1446. xiao zhang, maria leonor pacheco, chang li, and dan goldwasser. 2016. introducing drail -a step towards declarative deep relational learning. in proceedings of the workshop on structured prediction for nlp, pages 54-62, austin, tx. association for computational linguistics. da zheng, minjie wang, quan gan, zheng zhang, and george karypis. 2020. learning graph neural networks with deep graph library. www '20, new york, ny, usa. association for computing machinery.