Modeling Content and Context with Deep Relational Learning (2021)

Abstract: Building models for realistic natural language tasks requires dealing with long texts and accounting for complicated structural dependencies. Neural-symbolic representations have emerged as a way to combine the reasoning capabilities of symbolic methods, with the expressiveness of neural networks. However, most of the existing frameworks for combining neural and symbolic representations have been designed for classic relational learning tasks that work over a universe of symbolic entities and relations. In this paper, we present DRaiL, an open-source declarative framework for specifying deep relational models, designed to support a variety of NLP scenarios. Our framework supports easy integration with expressive language encoders, and provides an interface to study the interactions between representation, inference and learning.

## introduction understanding natural language interactions in realistic settings requires models that can deal with noisy textual inputs, reason about the dependencies between different textual elements, and leverage the dependencies between textual content and the context from which it emerges. work in linguistics and anthropology has defined context as a frame that surrounds a focal communicative event and provides resources for its interpretation @xcite ). as a motivating example, consider the interactions in the debate network described in figure 1 . given a debate claim (t 1 ), and two consecutive posts debating it (p 1 , p 2 ), we define a textual inference task, determining whether a pair of text elements hold the same stance in the debate (denoted using the relation agree(x, y)). this task is similar to other textual inference tasks @xcite that have been successfully approached using complex neural representations @xcite . in addition, we can leverage the dependencies between these decisions. for example, assuming that one post agrees with the debate claim (agree(t 1 , p 2 )), and the other one does not (¬agree(t 1 , p 1 )), the disagreement between the two posts can be inferred: ¬agree(t 1 , p 1 ) ∧ agree(t 1 , p 2 ) → ¬ agree(p 1 , p 2 ). finally, we consider the social context of the text. the disagreement between the posts can reflect a difference in the perspectives their authors hold on the issue. this information might not be directly observed, but it can be inferred using the authors' social interactions and behavior, given the principle of social homophily @xcite , stating that people with strong social ties are likely to hold similar views and authors' perspectives can be captured by representing their social interactions. exploiting this information requires models that can align the social representation with the linguistic one. motivated by these challenges, we introduce drail 1 , a deep relational learning framework, which uses a combined neuro-symbolic representation for modeling the interaction between multiple decisions in relational domains. similar to other neuro-symbolic approaches @xcite , our goal is to exploit the complementary strengths of the two modeling paradigms. symbolic representations, used by logic-based systems and by probabilistic graphical models @xcite , are interpretable, and allow domain experts to directly inject knowledge and constrain the learning problem. neural models capture dependencies using the network architecture and are better equipped to deal with noisy data, such as text. however, they are often difficult to interpret and constrain according to domain knowledge. our main design goal in drail is to provide a generalized tool, specifically designed for nlp tasks. existing approaches designed for classic relational learning tasks @xcite , such as knowledge graph completion, are not equipped to deal with the complex linguistic input, whereas others are designed for very specific nlp settings such as word-based quantitative reasoning problems @xcite or aligning images with text @xcite . we discuss the differences between drail and these approaches in section 2. the examples in this paper focus on modelings various argumentation mining tasks and their social and political context, but the same principles can be applied to wide array of nlp tasks with different contextualizing information, such as images that appear next to the text, or prosody when analyzing transcribed speech, to name a few examples. drail uses a declarative language for defining deep relational models. similar to other declarative languages @xcite , it allows users to inject their knowledge by specifying dependencies between decisions using first-order logic rules, which are later compiled into a factor graph with neural potentials. in addition to probabilistic inference, drail also models dependencies using a distributed knowledge representation, denoted relnets, which provides a shared representation space for entities and their relations, trained using a relational multi-task learning approach. this provides a mechanism for explaining symbols, and aligning representations from different modalities. following our running example, ideological standpoints, such as liberal or conservative, are discrete entities embedded in the same space as textual entities and social entities. these entities are initially associated with users, however using relnets this information will propagate to texts reflecting these ideologies, by exploiting the relations that bridge social and linguistic information (see figure 1 ). to demonstrate drail's modeling approach, we introduce the task of open-domain stance prediction with social context, which combines social network analysis and textual inference over complex opinionated texts, as shown in figure 1 . we complement our evaluation of drail with two additional tasks, issue-specific stance prediction, where we identify the views expressed in debate forums with respect to a set of fixed issues @xcite , and argumentation mining @xcite , a document-level discourse analysis task. ## related work in this section, we survey several lines of work dealing with symbolic, neural, and hybrid representations for relational learning. ## learning the scoring function used for comparing output assignments can be learned locally for each rule separately, or globally, by considering the dependencies between rules. ## experimental evaluation we compare drail to representative models from each category covered in section 2. our goal is to examine how different types of approaches capture dependencies and what are their limitations when dealing with language interactions. these baselines are described in section 5.1. we also evaluate different strategies using drail in section 5.2. we focus on three tasks: open debate stance prediction (sec. 5.3), issue-specific stance prediction (sec. 5.4) and argumentation mining (sec. 5.5), details regarding the hyper-parameters used for all tasks can be found in appendix b. ## conclusions in this paper, we motivate the need for a declarative neural-symbolic approach that can be applied to nlp tasks involving long texts and contextualizing information. we introduce a general framework to support this, and demonstrate its flexibility by modeling problems with diverse relations and rich representations, and obtain models that are easy to interpret and expand. the code for drail and the application examples in this paper have been released to the community, to help promote this modeling approach for other applications.