D eep W eighted M ax SAT for A spect-based O pinion E xtraction (2020)

Abstract: Though deep learning has achieved significant success in various NLP tasks, most deep learning models lack the capability of encoding explicit domain knowledge to model complex causal relationships among different types of variables. On the other hand, logic rules offer a compact expression to represent the causal relationships to guide the training process. Logic programs can be cast as a satisfiability problem which aims to find truth assignments to logic variables by maximizing the number of satisfiable clauses (MaxSAT). We adopt the MaxSAT semantics to model logic inference process and smoothly incorporate a weighted version of MaxSAT that connects deep neural networks and a graphical model in a joint framework. The joint model feeds deep learning outputs to a weighted MaxSAT layer to rectify the erroneous predictions and can be trained via end-to-end gradient descent. Our proposed model associates the benefits of high-level feature learning, knowledge reasoning, and structured learning with observable performance gain for the task of aspect-based opinion extraction.

## introduction aspect-based opinion extraction aims to identify opinion targets (or aspects) of a review corpus that indicate specific product features, as well as the opinion terms expressed towards the aspects. for example, in the sentence "the wine list is excellent", the aspect term is wine list, whereas the opinion term is excellent. many deep learning models have been proposed for this task via enumerating high-level features @xcite @xcite @xcite . however, these methods fail to explicitly encode prior knowledge on the relationships among aspect terms and opinion terms which are crucial for the task at hand, as shown in earlier rule-based models @xcite . as in the previous example, if wine list is extracted as an aspect term and it has dependency relation "nsubj" with excellent which is an objective, then we can deduce that excellent is an opinion term. though in @xcite , rules are incorporated as constraints into a deep neural network, the constraints cannot be backpropagated to the feature learning process. recently, @xcite proposed a joint model to combine deep learning with logic rules via minimizing the discrepancy between them. their approach, however, only indirectly guides deep learning in training without the ability to rectify the predictions according to logic rules in inference. to address the aforementioned limitations for aspect-based opinion extraction, we propose a novel joint model deepwmaxsat to integrate logic knowledge via a weighted maxsat layer into a deep learning architecture. specifically, deepw-maxsat consists of 1) a dnn layer that transforms an input embedding to a high-level feature representation; 2) a weighted maxsat layer that takes dnn outputs as the initial probabilistic evaluations on the logic variables and produces the values for the output logic variables corresponding to the head atoms of selected logic rules; 3) a conditional random field (crf) @xcite layer that generates structured outputs (label sequences) considering linear context interactions among the tokens in a sequence. moreover, to fully inherit the advantages of both dnns and logic programs, we adopt a form of residual connection that combines both dnn predictions and the outputs from the weighted maxsat layer with a learnable weight, which is then fed into the crf layer. it is worth noting that the weighted maxsat layer contains all the prior knowledge about the cor-relations among aspect and opinion terms encoded in conjunctive normal form (cnf) for all the logic rules. for example, the association between the aspect term wine list and the opinion term excellent in the previous example can be expressed using cnf as ¬aspect(list) ∨ ¬nsubj(list, excellent)∨ ¬obj(excellent) ∨ opinion(excellent), which is converted from the first-order-logic (fol) rule obj(excellent) ∧ nsubj(list, excellent) ∧ aspect(list) ⇒ opinion(excellent). a learnable weight is associated to each disjunctive clause in the cnf formula to indicate its confidence. the weighted maxsat layer is able to rectify dnn predictions according to preset rules, at the same time, the loss signal for the final predictions can be back-propagated smoothly through the weighted maxsat layer to dnn parameters to guide the training of the deep learning model. though @xcite proposed a differentiable satisfiability solver that integrates maxsat into deep learning, they only assumed a fixed set of rules that are true in nature, making it less flexible for general nlp problems where data can be noisy. with this consideration, we adopt the attention mechanism to adaptively select useful rules in the weighted maxsat layer for each data instance and treat the learnable attention scores as rule weights. the intuition is that different data instances may fit to different rules with varying probabilities. to summarize, our contributions include: deep learning with logic reasoning recent years have witnessed an increasing focus on neural symbolic learning that combines deep learning systems with discrete symbolic rules @xcite @xcite @xcite by constructing a logic network or connecting the distributed systems with logic rules for reasoning and inference in the logic domains. @xcite treated logic knowledge as semantic regularization in the loss function. for nlp applications, the neural-symbolic systems were recently proposed in @xcite for relation and knowledge graph learning that embed logic into the same space as distributed features in a single system. logical knowledge has also been incorporated as a form of posterior regulariza-tion in @xcite to enhance deep learning predictions. moreover, logic rules can be used as evidences to construct adversarial sets @xcite , or as a form of indirect supervision @xcite to improve model training. @xcite further augmented deep learning models with logic neurons that can be trained together with the neural networks. ## methodology in this section, we present our proposed model in detail. to make the logic knowledge more effective that is able to directly rectify the erroneous predictions made by deep learning models, and at the same time adapt its rules selectively according to different data instances, we propose a neuralsymbolic integration by incorporating an attentionbased weighted maxsat layer. the attention mechanism is used to automatically select relevant logic rules according to each specific data instance and to weigh the importance of each rule that could affect the final objective. furthermore, we also integrate a crf layer to generate structured predictions. as a result, the joint framework inherit the advantage of high-level feature learning, knowledge reasoning and structured learning. figure 1 provides an overview of the proposed model. it consists of 3 layers: 1) a deep learning module that takes input embeddings x 1 , ..., x n as deep neural networks ## conclusion we propose a novel joint model that inherits the advantage of high-level feature learning, logic reasoning and structured learning which can be trained smoothly in an end-to-end manner. to adapt logic knowledge with noisy real applications, we introduce an attention mechanism to generate an adaptive weight corresponding to each data instance for each logic rule. the attention weights control the information flow between deep neural networks and the maxsat layer which automatically weigh the relevance of each rule towards the data given. extensive experiments are conducted to verify both quantitatively and qualitatively the effectiveness of the proposed model.