Structured Mean-Field Variational Inference for Higher-Order Span-Based Semantic Role Labeling (2023)

Abstract: In this work, we enhance higher-order graph-based approaches for span-based semantic role labeling (SRL) by means of structured modeling. To decrease the complexity of higher-order modeling, we decompose the edge from predicate word to argument span into three different edges, predicate-to-head (P2H), predicate-to-tail (P2T), and head-to-tail (H2T), where head/tail means the first/last word of the semantic argument span. As such, we use a CRF-based higher-order dependency parser and leverage Mean-Field Variational Inference (MFVI) for higher-order inference. Moreover, since semantic arguments of predicates are often constituents within a constituency parse tree, we can leverage such nice structural property by defining a TreeCRF distribution over all H2T edges, using the idea of partial marginalization to define structural training loss. We further leverage structured MFVI to enhance inference. We experiment on span-based SRL benchmarks, showing the effectiveness of both higher-order and structured modeling and the combination thereof. In addition, we show superior performance of structured MFVI against vanilla MFVI.

## introduction semantic role labeling (srl) aims to recognize the predicate-argument structures for a given sentence. srl structures have found various applications in downstream natural language understanding tasks, e.g., machine translation @xcite , question answering @xcite , machine reading comprehension @xcite . there are two types of formalisms in srl, namely dependency-based and span-based srl, where the argument is a word in the former case * corresponding author and a contiguous sequence of words (i.e., a span) in the latter case. span-based srl is more difficult as it needs to identify two boundaries of a span instead of an argument word, resulting in a much larger search space. we focus on span-based srl in this work. span-based srl is traditionally tackled by biobased sequence labeling approaches @xcite . later, researchers turn to graph-based methods @xcite @xcite wherein graph nodes are argument spans and predicate words. recently, researchers show that higher-order graph-based methods achieve state-ofthe-art performance @xcite @xcite . for higher-order graphbased methods, the main difficulty is that there are in total o(n 3 ) predicate-argument pairs and thereby o(n 5 ) second-order parts @xcite , making them computationally infeasible to model. to resolve this issue, @xcite prune the number of candidate argument spans from o(n 2 ) to o(n), and consequently, reduce the number of second-order parts from o(n 5 ) to o(n 3 ). on the other hand, @xcite decompose the original edge (between the predicate word and the argument span) into two word-to-word edges, namely predicate-to-head (predicate word to the first word of argument span, p2h) and predicateto-tail (predicate word to the last word of argument span, p2t), so the total number of second-order parts reduces from o(n 5 ) to o(n 3 ) as well. both of these two works use conditional random fields (crf) for probabilistic modeling and mean-field variational inference (mfvi) for higher-order statistical inference in cubic time. without mfvi, exact higher-order inference with crf is np-hard. moreover, mfvi is fully differentiable and thus can be incorporated into neural networks as an rnn layer @xcite for end-to-end training. hence, mfvi becomes increasingly popular in solving nlp structured prediction tasks together with higher-order crf-based modeling @xcite @xcite . besides higher-order modeling, structured modeling has also been shown to be useful in spanbased srl @xcite . span-based srl has a nice structural property that argument spans would not cross to each other in general foot_0 , since gold annotations of argument spans are mostly extracted from existing constituency parse trees. as such, we can build a partiallyobserved constituency parse tree @xcite wherein observed nodes correspond to gold argument spans. notably, this is also the case for nested named entity recognition @xcite and coreference resolution @xcite . to leverage such structural information (for free) while eliminating the need of obtaining full constituency parse trees (which could be expensive), prior works perform latent-variable probabilistic modeling with partial marginalization based on dynamic programming (i.e., the inside or cky algorithm for full constituency parsing). concretely, they train a span-based treecrf model @xcite , either maximizing the probabilities of all compatible trees (to the set of observed arguments or entity spans) via the masked inside algorithm @xcite or defining training loss based on span marginal probabilities @xcite . these works show that structured modeling indeed improves performance for aforementioned tasks. our desiderata in this work is to combine the best of two worlds, performing joint higher-order and structured modeling in a probabilistically principled manner under the crf framework. to decrease the high complexity of higher-order inference, we use a strategy similar to @xcite and introduce an additional type of edges for modeling argument spans, namely head-to-tail (the first word to the last word of the argument span, h2t). without h2t edges, there could be potential ambiguities in the decoding process. more importantly, h2t edges are the bridge for structured modeling, on which we define a span-based treecrf distribution. to combine higher-order and structured modeling, inspired by @xcite , we perform mfvi for several steps to obtain approximated marginals, on which we de-fine structured loss for the argument span parts. however, (vanilla) mfvi uses fully-factorized distributions to approximate the otherwise complex true posterior, damaging the quality of higher-order inference. to solve this issue, we further adopt structured mfvi @xcite to enhance inference, leveraging the underlying tree structures of argument spans for more delicate structured modeling. we experiment on two benchmarks of spanbased srl: conll05 and conll12, obtaining state-of-the-art performances on five out of six evaluation metrics. ablation studies confirm the effectiveness of both higher-order and structured modeling, their combination thereof, and the use of structured mfvi. ## model architecture we depict our model architecture in fig. 3 . ## experiments settings. following previous works, we conduct experiments on two benchmarks: conll05 @xcite ## related work in recent years, graph-based (or span-based) models become popular in span-based srl thanks to their ability in encoding rich span features. @xcite exhaustively search predicateargument pairs. @xcite use a pruning strategy to reduce the search complexity. they then use a neural network to predicate the relationship between candidate predicates and candidate argument spans. @xcite extend their work by using deep biaffine attention @xcite for scoring, and tackling both span-based and dependency-based srl under a single unified framework. @xcite prune argument spans via syntactic rules for multilingual srl. @xcite point out that the way to extract spans has a huge impact on the final performance. instead of taking top-k candidate spans (i.e., beam pruning) as in @xcite , they use a two-stage strategy where the first stage finds all headwords, and the second stage predicates span boundaries based on predicted headwords. they use either gold heads from dependency-srl annotations or automatically-learned heads by using the "bag loss" proposed in @xcite . they show their two-stage strategy is better than beam pruning in different settings. thanks to the advance in second-order semantic dependency parsing @xcite where they unroll several mean-field inference steps for end-to-end training, researchers adopt this technique to improvement the performance of spanbased srl. direct second-order modeling leads to a o(n 5 ) search space, which is computationally prohibitive. @xcite thus use a beam pruning strategy to select o(n) candidate spans to decrease the complexity of second-order inference. @xcite decompose predicate-argument pairs into dependency edges. by doing so, they cast span-based srl to a dependency graph parsing technique, and thus can directly use the method of @xcite without much adaptation. since there are total o(n 2 ) edges, there is no need for pruning as exhaustive search is relative cheap. semantic arguments are often constituents. this is very similar to the case in nested named entity recognition (ner) where named entities are mainly extracted from constituency trees; and in coreference resolution where mentions are often constituents. this means that, one can embed these named entities or semantic arguments or mentions into constituency trees for structured modeling. finkel and @xcite use a constituency parser to jointly model constituents and named enti-ties, however their approach needs tree annotations, which are difficult to obtain. to resolve this problem, @xcite ; @xcite view named entities as partially-observed constituency trees, and design masked inside algorithms for partial marginalization to train their treecrf models. @xcite propose structured span selectors for span-based srl and coreference resolution, training weighted context-free grammars (or essentially, treecrfs) by partial marginalization akin to @xcite ; @xcite . they leverage the cyk algorithm to produce o(n) structure-aware candidate spans, outperforming the beam pruning strategy. structured mean-field variational inference is well-studied in the literature of graphical models @xcite ), but we only find few applications in the nlp community, e.g. in @xcite . we believe structured meanfield variational inference can be used more frequently and in this work we demonstrate its usage in span-based srl. ## conclusion in this work, we tackled span-based srl using a graph-based approach, combining the advantage of higher-order and structured modeling. in addition, we leveraged structured mfvi to respect the constituency tree constraint of argument spans during inference. we showed the effectiveness of these components experimentally.