O N C OMPOSITIONAL U NCERTAINTY Q UANTIFICATION FOR S EQ 2 SEQ G RAPH P ARSING
A BSTRACT
Recent years have witnessed the success of applying seq2seq models to graph parsing tasks, where the outputs are compositionally structured (e.g., a graph or a tree).
However, these seq2seq approaches pose a challenge in quantifying the model’s compositional uncertainty on graph structures due to the gap between seq2seq output probability and structural probability on the graph.
This work is the first to quantify and evaluate compositional uncertainty for seq2seq graph parsing tasks.
First, we proposed a generic, probabilistically interpretable framework that allows correspondences between seq2seq output probability to structural probability on the graph.
This framework serves as a powerful medium for quantifying a seq2seq model’s compositional uncertainty on graph elements (i.e., nodes or edges).
Second, to evaluate uncertainty quality in terms of calibration, we propose a novel metric called Compositional Expected Calibration Error (CECE) which can measure a model’s calibration behavior in predicting graph structures.
By a thorough evaluation for compositional uncertainty on three different tasks across ten domains, we demonstrate that CECE is a better reflection for distributional shift compared to vanilla sequence ECE.
Finally, we validate the effectiveness of compositional uncertainty considering the task of collaborative semantic parsing, where the model is allowed to send limited subgraphs for human review.
The results show that the collaborative performance based on uncertain subgraph selection consistently outperforms random subgraph selection (30% average error reduction rate) and performs comparably to oracle subgraph selection (only 0.
difference in average prediction error), indicating that compositional uncertainty is an ideal signal for model errors and can benefit various downstream tasks.
I NTRODUCTION
Parsing a natural language sentence into a compositional graph structure, i.e., graph parsing , is an important task of natural language understanding beyond simple classification or text generation tasks.
It has been broadly applied in applications like semantic parsing, code generation and knowledge graph generation.
Recently, a line of research has successfully applied sequence-to-sequence ( seq2seq ) approaches to these graph parsing tasks (Vinyals et al., 2015; Xu et al., 2020; Orhan, 2021;
Cui et al., 2022; Lin et al., 2022b).
Despite achieving impressive results, these approaches pose a challenge in quantifying the model’s predictive uncertainty on graph structures, making it hard to ensure a trustworthy and reliable deployment of NLP systems such as voice assistants (see an example in Figure 1).
Meanwhile, most existing work on uncertainty estimation for seq2seq models focused on classification or language generation tasks (Kumar & Sarawagi, 2019; Vasudevan et al., 2019; Malinin & Gales, 2020; Jiang et al., 2021; Shelmanov et al., 2021;
Wang et al., 2022; Pei et al., 2022).
However, how to quantify and evaluate compositional uncertainty , the predictive uncertainty over compositional graph elements (i.e., nodes or edges), remains unresolved for seq2seq graph parsing (see related work in Appendix A.2).
In this paper, we aim to answer these questions by proposing a simple probabilistic framework and rigorous evaluation metrics.
[Figure]
on Tuesday is my planning
[Figure] [Figure]
Just to make sure.
We are looking for a meeting that’s on Tuesday right?
[Figure]
Quantifying compositional uncertainty for seq2seq graph parsing is inherently more difficult than other seq2seq tasks like machine translation or speech recognition, since there is a gap between seq2seq output probabilities and conditional probabilities on the graph.
Specifically, our interest is to express the conditional probability of a graph node v concerning its parent pa( v ) , i.e., p ( v | pa( v ) , x ) , rather than the likelihood of v conditioning on the previous tokens in the linearized string.
For example, in the graph structure of Figure 1, the subgraph rooted with node TimeSpec (in the dotted square) depends on its parent node EventSpec , while in the linearized graph, the parent node is not necessarily the previous token to the subgraph (the shaded spans).
Consequently, we cannot directly quantify compositional uncertainty without bridging this gap between different probabilistic representations.
To address this challenge, we propose a generic, probabilistic framework called
Graph Autoregressive Process (GAP) (Section 2.1) that allows the correspondence between seq2seq output probability to graphical probability, i.e., assigning model probability for a node or edge on the graph.
Thus, GAP can be used as a powerful medium for quantifying a seq2seq model’s compositional uncertainty.
Furthermore, to evaluate uncertainty quality, we propose a novel metric called
Compositional Expected Calibration Error (CECE) to measure the model’s behavior in predicting compositional graph structures (Section 2.2).
Taking semantic parsing as a canonical application, in Section 3, we build a large benchmark consisting of 3 semantic parsing tasks across 10 different domains, based on which we comprehensively evaluate compositional uncertainty under distributional shift and validate its effectiveness on a practical downstream task (collaborative semantic parsing).
First, in Section 3.1, we report different calibration metrics for a state-of-the-art seq2seq parser (Lin et al., 2022b) based on T5 (Raffel et al., 2020) as well as its different advanced uncertainty variants on the benchmark.
We demonstrate that compared to vanilla ECE based on sequence accuracy,
CECE is a better metric for reflecting distributional shift, i.e., task difficulty and domain generalization.
We also notice that, despite the strong performance brought by those advanced uncertainty baselines on classification tasks, in the settings of graph parsing, the absolute advantage of these methods no longer holds when predicting graph edges.
This suggests that developing uncertainty methods focused on compositional uncertainty can be a fruitful avenue for future research.
Second, in Section 3.2, we validate the practical effectiveness of compositional uncertainty considering the problem of collaborative semantic parsing.
In this setting, the model is allowed to send a limited number of uncertain subgraphs for human review (see Figure 1 for an example).
We test the collaborative performance on the benchmark, and find that using uncertain subgraph selection consistently outperforms random subgraph selection (selecting random subgraphs on the predicted graph) with an average error reduction rate of 30%, and performs fairly close to oracle subgraph selection (selecting incorrect subgraphs on the predicted graph) with an small difference in prediction error as 0.33.
This indicates that compositional uncertainty is an ideal signal for the likelihood on model error over graph elements, and can benefit various downstream tasks, e.g., human-AI collaborative parsing and neural-symbolic parsing (Lin et al., 2022a).
In summary, our work makes the following contributions:
• New Framework for Compositional Uncertainty Quantification.
We are the first to propose a simple and general probabilistic framework (GAP) that can quantify compositional uncertainty for seq2seq graph parsing.
GAP allows us to go beyond the conventional autoregressive sequence
probability and express parent-child conditional probability on the graph, which is compatible with any graph parsing problem and autoresgressive model.
• Rigorous Metrics for Compositional Uncertainty Calibration.
We introduce a novel quality measurement for compositional uncertainty (CECE) which can evaluate the model’s calibration on graph elements and provide a better interpretation of the model’s behavior in predicting graph structures under distributional shift.
• Practical Effectiveness.
We comprehensively evaluate the calibration behavior of modern pretrained large language models,
i
e., T5 (Raffel et al., 2020), in compositional uncertainty quantification on a broad range of semantic parsing tasks of varying complexity (Redwoods, SNIPS, and SMCalFlow).
We further evaluate the benefit of compositional uncertainty quantification in enabling new capacity (e.g, fine-grained collaborative prediction for complex semantic parsing) in downstream applications.
Specifically, our results show that compositional uncertainty can significantly benefit collaborative parsing performance, with only a 0.33 difference in prediction error compared to the headroom.
Q UANTIFYING AND E VALUATING C OMPOSITIONAL U NCERTAINTY
Problem Formulation.
In this work, we interpret the term graph parsing as mapping from surface strings (usually natural language sentences) to target representations that are explicitly or implicitly graph-structured.
Formally, the input is a natural language utterance x , and the output is a DAG
G = ⟨ N , E ⟩ , where N is the set of nodes and E ∈ N × N is the set of edges.
In the case of seq2seq parsing, G is represented as a linearized graph string g = s
1 s 2 · · · s L consists of symbols { s l } Ll =1 .
In the experiment, we use PENMAN notation (Kasper, 1989) to linearize all the formalism, which is a serialization format for the directed, rooted graphs used to encode semantic dependencies (more details are available in Appendix B).
To this end, our goal is to quantify the graph-level uncertainty p ( G | x ) from the sequence-level probability p ( g | x ) generated by a seq2seq model, which we term the compositional uncertainty .
For example, our interest to express the conditional probability of a graph node v with respect to its parent pa( v ) , i.e., p ( v | pa( v ) , x ) , rather than the likelihood of v conditioning on the previous tokens in the linearized string.
In the following sections, we will introduce how to quantify (Section 2.1) and evaluate (Section 2.2) this compositional uncertainty.
2. Q UANTIFYING C OMPOSITIONAL U NCERTAINTY VIA G RAPH A UTOREGRESSIVE P ROCESS (GAP)
To properly model the uncertainty p ( G | x ) from a seq2seq model, we need an intermediate probabilistic representation to translate the raw token-level probability to the distribution over graph elements (i.e., nodes and edges).
To this end, we introduce a simple probabilistic formalism termed
Graph Autoregressive Process (GAP), which is a probability distribution assigning seq2seq learned probability to the graph elements v ∈ G .
Specifically, as the seq2seq-predicted graph adopts both a sequence-based representation g = s 1 , ..., s L and a graph representation G = ⟨ N , E ⟩ , the GAP model adopts both an autoregressive representation p ( g | x )
= (cid:81)
i p ( s i | s <
i , x ) analogous to that of the seq2seq model (Section 2.1.1), and also a probabilistic graphical model representation p ( G | x ) =
(cid:81) v ∈ G p ( v | pa( v ) , x ) for proper quantification of model uncertainty on the graph (Section 2.1.2).
Both representations share the same set of underlying probability measures (i.e., the graphical-model likelihood p ( G | x ) can be derived from the autoregressive probabilities p ( s i | s <i , x ) ).
As we will show, GAP serves as a powerful medium for quantifying compositional uncertainty for seq2seq graph parsing.
2.1. A UTOREGRESSIVE R EPRESENTATION FOR L INEARIZED S EQUENCE g
Given an input sequence x and output sequence y =
y 1 y 2 · · · y N , the token-level autoregressive distribution from a seq2seq model is p ( y | x ) =
(cid:81)
Ni =1 p ( y i |
y <
i , x ) .
In the context of graph parsing, the output sequence describes a linearized graph g = s
1 s 2 · · · s L , where each symbol s i = { y i 1 y i 2 · · · y i Ni } represents either a node n ∈ N or an edge e ∈ E of the graph and corresponds to a collection of beam-decoded tokens { y i 1 y i 2 · · · y i Ni } .
This process is illustrated as follows:
[Equation]
s L g: y: y
y
yy 
 yy
y 7 yy N-1 y N … …
To this end, GAP assigns probability to each linearized graph g
= s
1 s 2 · · · s L autoregressively as p ( g | x ) = (cid:81)
Li =1 p ( s i | s <i , x ) , and the conditional probability p ( s i | s <i , x ) is computed by aggregating the token probability:
[Equation]
Marginal and Conditional Probability.
Importantly, GAP allows us to compute the marginal and (non-local) conditional probabilities for graph elements s i .
Given the input x , the marginal probability of s i is computed as
[Equation]
by integrating over the space of all possible subsequences s <i before the symbol s i .
Then, the (non-local) conditional probability between two graph elements ( s i , s j ) with i < j is computed as
[Equation]
by integrating over the space of subsequences s i
→ j between ( s i , s j ) and the subsequence s <i before s i .
Higher order conditional (e.g., p ( s j | ( s i , s l ) , x ) ) can be computed analogously.
Notice this gives us the ability to reason about long-range dependencies between non-adjacent symbols on the sequence.
Furthermore, the conditional probability on the reverse direction can also be computed using the Bayes’ rule: p ( s i | s j , x ) =
p ( s j |
s i ,x ) p ( s i | x ) p ( s j | x ) .
Efficient Estimation Using Beam Outputs.
In practice, we can estimate p ( s i | x ) and p ( s j | s i , x ) efficiently via importance sampling using the output from the beam decoding { g k } Kk =1 , where K is the beam size (Malinin & Gales, 2020).
The marginal probability can be computed as
[Equation]
here π k is the importance weight proportional to the beam candidate g k ’s log likelihoods, and t >is the temperature parameter fixed to a small constant (e.g., t = 0 . 1 ) (Malinin & Gales, 2020).
If the symbol s i does not appear in the k th beam, we set p ( s i | s k,<i , x ) = 0 .
As shown, the marginalized probability ˆ p ( s i | x ) provides a way to reason about the global importance of s i by integrating the probabilistic evidence p ( s i | s k,<i , x ) over the whole beam-sampled posterior space.
It is able to capture the cases of spurious graph elements s i with high local probability p
( s i | s k,<i , x ) but low global likelihood (i.e., only appear in a few low-probability beam candidates).
Therefore, it is a useful quantity for structure induction (e.g., edge and node pruning) in graphical model inference (Dianati, 2016).
Then, for two symbols ( s i , s j ) with i < j , we can estimate the conditional probability as ˆ p ( s
[Equation]
here π ik is the importance weight among beam candidates that contains s i .
Notice this is different from Equation 4 where π k is computed over all beam candidates regardless of whether it contains s i .
2.1. P ROBABILISTIC G RAPHICAL M ODEL R EPRESENTATION FOR G
So far, we have focused on probability computation based on the graph’s linearized representation p ( g | x ) =
(cid:81)
i p ( s i | s <i , x ) .
In this section, we further consider GAP’s graphical model representation p ( G | x ) = (cid:81) v ∈ G p ( v | pa( v ) , x ) .
enen 7 nene 4 nene 5 nen 8 nen 7 n… … … … … gggnneeneeenneenn
Autoregressive Sequential Representation (i.e., Beam Candidates)
Graphical Model Representation (i.e., Meta Graph) v
Specifically, given a beam sample of linearized graphs { g k } Kk =1 , well-established algorithms exist to synthesize different graph predictions into a meta graph G .
Briefly, these methods first convert each string g b into their original graph representation G k = ⟨ N k , E k ⟩ , then merge multiple graphs { G k } Kk =1 using a graph matching algorithm (Cai & Knight, 2013; Hoang et al., 2021).
A visual illustration of the resulting graph G is shown in Figure 2, where n i and e j are the candidates for the node and edge predictions collected from beam sequences.
As shown, compared to the sequence-based representation g , the meta graph G (1) explicitly enumerates different candidates for each node and edge prediction (e.g., n 1 v.s. n 2 for predicting the first element), and (2) provides an explicit account of the parent-child relationships between variables on the graph (e.g., e 7 is a child node of n 3 in the predicted graph, which is not reflected in the autoregressive representation).
From the probabilistic learning perspective, the meta graph G describes the space of possible graphs (i.e., the support ) for a graph distribution p ( G | x ) : G → [0 , 1] .
It describes the possible node and edge variables and their dependencies on the graph G (i.e., the shaded squares in the Figure 2), and also different possible values for each node and edge variable (i.e., the solid squares within each shaded square in Figure 2).
To this end, GAP assigns proper graph-level probability p ( G | x ) to graphs G sampled from the meta graph G via the graphical model likelihood:
[Equation]
where p ( v | pa( v ) , x ) is the conditional probability for v with respect to their parents pa( v ) in G .
Given the candidates graphs { G k } Kk =1 , we can express the likelihood for p ( v | pa( v ) , x ) by writing down a multinomial likelihood enumerating over different values of pa( v ) (Murphy, 2012).
This in fact leads to a simple expression for the model likelihood as a simple averaging of the beamsequence log likelihoods:
[Equation]
where c k is the value of pa( n ) in k th beam sequence, and the conditional probabilities are computed using Equation (5).
See Appendix C for a detailed derivation.
In summary, for each graph element variable v ∈ G , GAP allows us to compute the graphical-model conditional likelihood p ( v | pa ( v ) , x ) via its graphical model representation, and also to compute the marginal probability p ( v | x ) via its autoregressive presentation.
Algorithm 1 summarizes the full
GAP computation.
2. E VALUATING C OMPOSITIONAL U NCERTAINTY
In this section, we present how to evaluate compositional uncertainty.
A common approach to evaluate a model’s uncertainty quality is to measure its calibration performance, i.e., whether the
Inputs:
Beam candidates with probabilities { p ( g k | x ) } Kk =1 , Meta graph G Output:
Marginal probabilities { p ( s | x ) } , Graph model likelihood log p ( G | x ) for v ∈ G do
Compute marginal likelihood: p ( v = s | x ) (Equation 4)
Compute graphical model likelihood: log p ( v = s | pa( v ) , x ) (Equation 7)
Return
Marginal Probabilities: { p ( v | x ) }
Graphical Model Likelihood: log p ( G | x ))
= (cid:80) v ∈ G log p ( v | pa( v ) , x ) model’s predictive uncertainty is indicative of the predictive error, e.g., expected calibration error (ECE; Naeini et al., 2015).
In this work, we propose a compositional calibration metric based on
ECE, which measures the difference in expectation between the model’s predictive confidence (e.g., the maximum probability score) on graph elements (nodes or edges) and their actual match to the gold graph.
Formally, during inference time, given a input x and a target graph G , we first partition the confidence interval into B equal bins I 1 , . .
, I B .
Then in each measure the absolute difference between the node/edge accuracy and confidence of predictions in that bin.
This gives the compositional expected calibration error (CECE G ) for graph G as:
[Equation]
(8) where | G | is the number of graph elements in the target graph G , ˆ v t is the t th element (node/edge) in the predicted graph ˆ G , C
(ˆ v it , G i ) denotes if ˆ v it matches in the graph G i using a graph matching algorithm, and p (ˆ v t | x ) can be obtained by GAP in Section 2.1.
Specifically, we use the matching algorithm adopted in S MATCH (Cai & Knight, 2013), which is the same graph matching algorithm for constructing meta graph G (see Appendix E for details).
Alternatively, we can compute CECE only for node/edge predictions, namely CECE N and CECE E .
E XPERIMENTAL E VALUATION
Datasets.
In this paper, we take semantic parsing as a canonical application, and build a large benchmark consisting of three semantic parsing tasks and covering ten different domains, ranging from graph-based grammar parsing to dialogue-oriented semantic parsing:
• Redwoods: The LinGO Redwoods Treebank is a collection of hand-annotated corpora for an English grammar consisting of more than 20 datasets.
The underlying grammar is called English Resource Grammar (ERG; Flickinger et al., 2014; Bender et al., 2015), which is an open-source, domain-independent, linguistically precise, and broad-coverage grammar.
ERG can be presented into different types of annotation formalism.
This work focuses on the Elementary Dependency Structure (EDS; Oepen & Lønning, 2006)
which is a compact representation that can be expressed as a DAG.
Following previous works
for in-domain test
we train and evaluate models on the subset treebank corresponding to the 25 Wall Street Journal (WSJ) sections with standard data splits (Flickinger et al., 2012).
For out-of-domain (OOD) evaluations, we select 7 diverse datasets from Redwoods: Wikipedia (Wiki), the Brown Corpus (Brown), the Eric Raymond Essay (Essay), customer emails (E-commerce), meeting/hotel scheduling (Verbmobil), Norwegian tourism (LOGON) and the Tanaka Corpus (Tanaka) (See Appendix D for more details).
• SMCalFlow: SMCalFlow (Andreas et al., 2020) is a large corpus of semantically detailed annotations of task-oriented natural dialogues.
The annotation uses dataflow computational graphs, composed of a rich set of both general and application specific functions, to represent user requests as rich compositional expressions.
We use the standard data split in the original paper and evaluate inference results on development set.
• SNIPS: SNIPS (Coucke et al., 2018) is a slot filling dataset containing 39 slot names fromdifferent domains collecting from the SNIPS voice assistant.
It is usually formulated as a sequence
[Table] [Table]
labeling problem.
Following previous work (Yu et al., 2021), we train models on five source domains, use a sixth one for development, and test on the remaining domian.
We transfer all the data into linearized graphs (Appendix B).
To reduce the sequence length, since the number of node/edge names are limited, we set them untokenizable to the tokenizer.
Evaluation Metrics.
Consistent with previous work, the performance metric used for Redwoods is
S MATCH score (Cai & Knight, 2013), which computes the degree of overlap between two semantic graphs (see Appendix E for details).
For SMCalFlow, we use sequence accuracy (exact match).
For
SNIPS, we use slot F1 score, which is equal to node F1 score when we transfer the SNIPS data into
PENMAN notation.
For model calibration, we report naive ECE based on sequence accuracy and compositional uncertainty CECE introduced in Section 2.2.
Seq2seq Model.
We adopt T5 (Raffel et al., 2020) as the baseline model, which is a pre-trained seq2seq Transformer model that has been widely used in many NLP applications.
We use the open-sourced T5X, which is a new and improved implementation of T5 codebase in JAX and Flax.
Specifically, we use the official pretrained T5-large (770 million parameters) and finetuned it on three datasets respectively.
By evaluating the performance metrics for each task, we find that the T model is capable of achieving the state-of-the-art results on all the tasks compared to previous work (see Table 3 in Appendix F for full comparison).
3. E VALUATING C OMPOSITIONAL U NCERTAINTY UNDER D ISTRIBUTIONAL S HIFT Comparing
Compositional ECE to vanilla ECE.
Table 1 reports the evaluation results on the benchmark.
First, we find that sequence accuracy (A CC seq ) does not necessarily correlate to the
S MATCH score, which makes the vanilla ECE based on sequence accuracy, i.e., ECE seq , less informative in reflecting model’s calibration on predicting graph structures.
Second, comparing to Redwoods (in-domain) which requires parsing a natural language sentence into a pre-defined grammar representation, and SNIPS which requires labeling intent slots for a natural language sentence, SMCalFlow is more ambiguous and difficult as it involves complex dialogue histories and fine-grained intent slots (Andreas et al., 2020).
We notice that the CECE is larger for SMCalFlow on in-domain test, indicating that CECE is a better metrics to reflect this task ambiguity/difficulty.
Finally, as the model generalization degrades across different domains for Redwoods, CECE also increases accordingly, indicating that CECE can reflect model’s generalization under domain shift.
Comparing Advanced Uncertainty Methods.
In recent year, a variety of methods have been developed to improve the DNNs uncertainty quality for classification problems.
Here, we are interested to understand if the benefit brought by those advanced methods to classification setting translates also to the graph parsing setting.
In this section, we evaluate the performance of 6 uncertainty baselines on Redwoods across 8 different domains.
We consider T5-Large (Raffel et al., 2020) as the base model, and select six methods based on their practical applicability for the base model.
Specifically, we consider (1) Deterministic model which is the base T5 model; (2) Monte Carlo
Dropout ( MC Dropout ) which estimates uncertainty using the Monte Carlo average of 5 dropout samples (Gal & Ghahramani, 2016); (3) Deep Ensemble ( DE ) which trains 4 deterministic models individually and averages all predictions (Lakshminarayanan et al., 2017); (4) Batch Ensemble ( BE ),
[Figure]
an ensemble method which has much lower computational and memory costs comparing to MC
Dropout and Deep Ensemble (Wen et al., 2019); (5) Spectral-normalized Neural Gaussian Process ( SNGP ), a recent state-of-the-art approach which improves uncertainty quality by transforming a neural network into an approximate Gaussian process model (Liu et al., 2020); (6) SNGP+DE which is the deep ensemble for 4 individual SNGP models; (7) SNGP+BE which uses a combination of
Batch Ensemble and SNGP layers.
The results are shown in Figure 3.
The full evaluation results can be found in Appendix G (Table 4).
Results.
As shown in the figure, these uncertainty baselines generally follows the same pattern on domain shift, i.e., decrease in S MATCH corresponds to increase in CECE G , while we cannot infer distributional shift via ECE seq .
Some uncertainty baselines (e.g., MC Dropout and SNGP+DE) can achieve better in both ECE seq and CECE G compared to deterministic model across different domains, where MC Dropout achieves the best results in ECE seq and SNGP+DE achieves the best results in CECE G .
By further evaluating CECE N and CECE E , we notice that the improvement in CECE G mainly comes from node predictions (the difference in CECE N is more obvious than
CECE E ), while for edge predictions, only little improvement is observed.
This suggests that uncertainty estimation is structurally different for seq2seq graph parsing tasks compared to classification tasks, and further research is needed for designing better calibrated model with more focus on compositional uncertainty calibration.
3. P RACTICAL E FFECTIVENESS : U NCERTAINTY -G UIDED C OLLABORATIVE S EMANTIC P ARSING
Motivation.
To further explore the correlation between model uncertainty and performance, we plot the histogram for the T5 model’s probabilities verses the node/edge accuracies in Appendix H (Figure 5), where we find that low model probability generally corresponds to low model performance.
This serves as a motivation for collaborative semantic parsing using composotional uncertainty, where the model is allowed to send a limited number of uncertain subgraphs for human review.
This is a practical setting in lots of realistic scenarios, for example, in Figure 1, the model can ask for clarification regarding the uncertain subgraph (dotted squared) via modeling the uncertainty score for each element in the parsed semantic graph.
This process allows the system to collaborate with the users to avoid triggering unwanted actions, which cannot be achieved without properly quantifying compositional uncertainty over graph elements.
Uncertainty-based Subgraph Selection.
For a well-trained seq2seq parser in Section 3.1, we find uncertain subgraphs by (1) finding e uncertain nodes as root nodes based on ranked compositional uncertainty scores over the predicted graph; (2) tracing descendants from root nodes up to depth d 2 .
[Figure]
Training and Inference.
The training examples for the collaborative model are generated by attaching human edit results of random subgraphs to input sentences 3 .
During the inference, the test examples are generated by attaching corresponding human edit results for each subgraph selection strategy to input sentences.
Results.
The results are shown in Figure 4.
Due to space limitation, we only report results on d = 2 for Redwoods and SMCalflow, and d = 1 for SNIPS.
Full results for different combinations of example numbers and depths are reported in Table 5 and Table 6 (Appendix I).
We see that for all three tasks, uncertainty-based subgraph selection consistently outperforms random subgraph selection with average error reduction rates 13.64%, 24.45% and 52.11% respectively, and performs fairly close to Oracle with an average difference in prediction error as small as 0.33.
This shows that compositional uncertainty is effective in detecting potential incorrect subgraph predicted by the model.
Meanwhile, we notice that Oracle* performs better than Oracle , indicating that incorrect predicted subgraphs with high uncertainty are more informative to the collaborative model.
Analysis.
Theoretically, the performance of collaborative parsing is determined by how many incorrectly predicted subgraphs can be selected for human edits, where Oracle is the headroom given limited budgets.
In Table 7 (Appendix I), We further conduct an analysis to subgraphs selected by different strategies by calculating the coverage rate of incorrect nodes in subgraphs to incorrect nodes in the entire graph (i.e., error node coverage rate).
The results indicates that compositional uncertainty is effective in detecting incorrectly predicted nodes.
C ONCLUSION AND F UTURE W ORK
Over the past few years, lots of efforts have been made to apply seq2seq models to graph parsing, which is an important area in NLP.
Despite achieving the state-of-the-art in various graph parsing tasks, these seq2seq approaches pose a challenge on how to interpret model’s predictive uncertainty on predicting graph structures.
This work is the first to provide a general method to properly quantify and evaluate compositional uncertainty for seq2seq graph parsing, which is achieved by a simple probabilistic framework (GAP) and a rigorous metric (CECE).
The experimental evaluation demonstrates that CECE is an effective metric to reflect distributional shift and compositional uncertainty is a useful tool for downstream tasks such as collaborative semantic parsing.
For future work, several directions are worth investigations, including: (1) evaluating the practical effectiveness of advanced uncertainty methods in collaborative semantic parsing; (2) learning theoretic experiments to interrogate the seq2seq model’s ability in capturing the true distribution of the probabilistic graphical model; (3) extending the benefit of compositional uncertainty to more real-world NLP tasks, e.g., retrieval-augmented semantic parsing (Hashimoto et al., 2018) and active learning. A CKNOWLEDGMENTS We appreciate the insightful comments from the anonymous reviewers. We would like to thank Jie
Ren for discussion and proofreading
A R ELATED W ORK A. S EQ 2 SEQ G RAPH P ARSING
Seq2seq graph parsing is inspired by the success of recent seq2seq models (particularly pretrained models), which are the heart of modern neural machine translation.
This type of parser encodes and views a target graph as a string from another language (Vinyals et al., 2015).
However, simply applying seq2seq models to graph parsing is not always successful when the target graph is complecated, e g., for Abstract Meaning Representation (AMR; Banarescu et al., 2013) or
English Resource Grammar (ERG; Flickinger et al., 2014).
This is because effective linearization (encoding graphs as linear sequences) and data sparsity were thought to pose significant challenges (Konstas et al., 2017).
Alternatively, some specifically designed preprocessing procedures for vocabulary and entities can help to address these issues (Konstas et al., 2017; Peng et al., 2017).
These preprocessing procedures are very specific to a certain type of meaning representation and are difficult to transfer to others.
To address this, (Bevilacqua et al., 2021) propose to use special tokens to represent variables in the linearized graph and to handle co-referring nodes.
Lin et al.
(2022b) propose a variable-free top-down linearization and a compositionality-aware tokenization for ERG graph preprocessing, and successfully transfer the ERG parsing into a translation problem that can be solved by a state-of-the-art seq2seq model T5 (Raffel et al., 2020).
Their parser achieves the best known results on the in-domain test set for ERG parsing.
A. U NCERTAINTY Q UANTIFICATION FOR G RAPH P ARSING
Compared to seq2seq graph parsing, uncertainty quantification is straightforward if the parser explicitly models the target graph structures, e.g., chart parsers (Magerrnan & Marcus, 1991), factorization-based parsers (McDonald, 2006; Cao et al., 2021) or composition-based parsers (Chen et al., 2018; 2019), given that the model’s score function is naturally aligned with the graph structure.
As for transition-based parser (Fernandez Astudillo et al., 2020; Zhou et al., 2021), where the target graph is generated via a series of actions, in a process that is very similar to dependency tree parsing (Yamada & Matsumoto, 2003; Nivre, 2008), some previous work has used important sampling to estimate probabilities (Dyer et al., 2016), and model uncertainty for alignments between graph nodes and input text tokens (Drozdov et al., 2022).
These works are very specific to the formalism of the target graph, and it is difficult to transfer to other graph parsing problems.
Some previous uncertainty quantification methods have focused on sequential or token-level uncertainty for seq2seq model.
For example, Dong et al.
(2018) model uncertainty for neural semantic parsers by outlining three major causes of uncertainty including model uncertainty, data uncertainty and input uncertainty, and design various metrics to quantify these factors.
Lin et al.
(2022b) uses predictive probability generated by the T5 model as a signal for neural-symbolic parsing.
However, these uncertainty quantification methods cannot model compositional uncertainty over the graph structures.
B PENMAN N OTATION
PENMAN notation, originally called Sentence Plan Notation in the PENMAN project (Kasper, 1989), is a serialization format for the directed, rooted graphs used to encode semantic dependencies,
N 1 E
N
N N
N
N 2 E 2 E
E
E 5 E( N1:E1 ( N :E4 ( N5 ) :E5 ( N6* ) ) :E2 ( N4 :E3-of ( N2 ) :E6 ( N6* ) ) )
G r a p h
P E N M A N G r a p h most notably in the Abstract Meaning Representation (AMR) framework (Banarescu et al., 2013). It looks similar to Lisp’s S-Expression in using parentheses to indicate nested structures. To make
PENMAN notation compatible with the seq2seq learning, we adopted a variable-free version of PENMAN which was first proposed in Lin et al. (2022b). The general template is illustrated as follows:
The linearized form can only describe projective structures such as trees, so in order to capture nonprojective graphs, this notation (1) reverse some of the edges to make it can be written in top-down tree order, e.g., :E3-of here (2) use star markers to indicate a node referred later to establish a reentrancy, e.g., E6* here.
Table 2 shows some variable-free PENMAN linearized examples for different semantic parsing tasks. [Figure]
[Table]
graph (template can be found in Appendix
B) in three different semantic parsing tasks (task details can be found in Section 3).
Here :carg means corresponding spans in the sentence.
C S IMPLIFIED E XPRESSION FOR G RAPHICAL M ODEL L IKELIHOOD
Given the candidates graphs { G k } Kk =1 , we can express the likelihood for p ( v | pa( v ) , x ) by writing down a multinomial likelihood enumerating over different values of pa( v ) (Murphy, 2012).
For example, say pa( n ) =
( e 1 , e 2 ) which represents a subgraph of two edges ( e 1 , e 2 ) pointing into a node n .
Then the conditional probability p ( n | pa( n ) , x ) can be computed by enumerating over the observed values of ( e 1 , e 2 ) pair:
[Equation]
where Candidate ( e ) is the collection of possible symbols s the variable e can take, and K c is the number of times ( e 1 , e 2 ) takes a particular value c ∈ Candidate ( e 1 , e 2 ) =
Candidate ( e 1 ) ×
Candidate ( e 2 ) .
Then, the log likelihood becomes:
[Equation]
To simplify this above expression, we notice that log p ( n | pa( n ) , x ) can be divided by the constant beam size K without impacting the inference.
As a result, the log probability can be computed by simplify averaging the values of log p ( v | pa( v ) =
c k ) across the beam candidates:
[Equation]
where c k is the value of ( e 1 , e 2 ) in k th beam candidate.
D OOD D ATASETS FOR ERG P ARSING
Wikipedia (Wiki)
The DeepBank team constructed a treebank for 100 Wikipedia articles on Computational Linguistics and closely related topics.
The treebank of 11,558 sentences comprisessets of articles.
The corpus contains mostly declarative, relatively long sentences, along with some fragments.
The Brown Corpus (Brown)
The Brown Corpus was a carefully compiled selection of current
American English, totalling about a million words drawn from a wide variety of sources.
The Eric Raymond Essay (Essay)
The treebank is based on translations of the essay “The Cathedral and the Bazaar” by Eric Raymond.
The average length and the linguistic complexity of these sentences is markedly higher than the other treebanked corpora.
E-commerce While the ERG was being used in a commercial software product developed by the
YY Software Corporation for automated response to customer emails, a corpus of training and test data was constructed and made freely available, consisting of email messages composed by people pretending to be customers of a fictional consumer products online store.
The messages in the corpus fall into four roughly equal-sized categories: Product Availability, Order Status, Order Cancellation, and Product Return.
Meeting/hotel scheduling (Verbmobil)
This dataset is a collection of transcriptions of spoken dialogues, each of which reflected a negotiation either to schedule a meeting, or to plan a hotel stay.
One dialogue usually consists of 20-30 turns, with most of the utterances relatively short, including greetings and closings, and not surprisingly with a high frequency of time and date expressions as well as questions and sentence fragments.
Norwegian tourism (LOGON)
The Norwegian/English machine translation research project LOGON acquired for its development and evaluation corpus a set of tourism brochures originally written in Norwegian
and then professionally translated into English.
The corpus consists almost entirely of declarative sentences and many sentence fragments, where the average number of tokens per item is higher than in the Verbmobil and E-commerce data.
The Tanaka Corpus (Tanaka)
This treebank is based on parallel Japanese-English sentences, which was adopted to be used with in the WWWJDIC dictionary server as a set of example sentences associated within words in the dictionary.
E G RAPH M ATCHING A LGORITHM IN S MATCH
In general, finding the largest common subgraph is a well-known computationally intractable problem in graph theory.
However, for graph parsing problems where graphs have labels and a simple tree-like structure, some efficient heuristics are proposed to approximate the best match by a hillclimbing algorithm (Cai & Knight, 2013).
The initial match is modified iteratively to optimize the total number of matches with a predefined number of iterations (default value set to 5 ).
This algorithm is very efficient and effective, it was also used to calculate the S MATCH score in Cai & Knight (2013).
F C OMPARING T5 TO P REVIOUS W ORK
Table 3 shows the in-domain performance of the T5 model compare to previous work on Redwoods,
SMCalflow and SNIPS.
The results indicate that the T5 model is capable of achieving the state-ofthe-art results on all the tasks compared to previous work.
Interestingly, we notice that by rewriting
SMCalFlow into PENMAN notation, the sequence accuracy increase from 72.9 to 82.8 on development set, indicating that proper linearization and tokenization are important for graph parsing tasks.
[Table]
G F ULL R ESULTS FOR C OMPARING U NCERTAINTY B ASELINES
In Table [Table] we reported the full results for comparing different uncertainty baselines on the benchmark introduced in Section 3.
H H ISTOGRAM OF C ALIBRATIONS
The correlations between the subgraph’s probability and performance on ERG parsing are shown in
Figure 5, where we can see that low model probability generally corresponds to low model performance, i.e., the model is relatively calibrated in predicting graph structures.
I F ULL R ESULTS FOR C OLLABORATIVE S EMANTIC P ARSING
In Table 5 and Table 6, we report the full results for collaborative performance under different settings of budgets of subgraphs by different combinations of number of subgraphs ( e ) and max depth of each subgraphs ( d ).
Note that for SNIPS, since the performance is almost close to 100% when selecting number of subgraphs greater than 3, i.e., the total graph has been covered, we will not evaluate cases where e > 3 .
We can see from the table that uncertainty-based subgraph selection consistently outperforms random subgraph selection, and performs close to oracle subgraph selection.
In Table 7, We further conduct an analysis to subgraphs selected by different strategies by calculating the coverage rate of incorrect nodes in subgraphs to incorrect nodes in the entire graph (i.e., error node coverage rate).
The results indicates that compositional uncertainty is effective in detecting incorrectly predicted nodes.
J L IMITATIONS
Here we discuss some potential limitations of the current study:
Breadth The GAP model in this work is a general uncertainty quantification framework for graph parsing problems using seq2seq model, which theoretically has no restriction to formalism and languages adopted for the output graph.
In this work, we have tested GAP on Redwoods, SMCalflow and SNIPS, which are all English based, but it is worth to see how the approach [Table] can scale up (with the number of languages covered) and down (with diminished resource availability in low-resource languages).
For example, many efforts have been made to build a practical and cross-linguistically valid graph formalism to overcome language barriers, e [Table] g., Uniform Meaning Representation (UMR; Van Gysel et al., 2021) and BabelNet Meaning Representation (BMR;
Navigli et al., 2022) [Table] and it is interesting to explore the model behavior on these graph formalism in terms of compositional uncertainty [Table]
Graphical Model Specification The GAP model presented in this work considers a classical graphical model likelihood p ( G | x ) =
(cid:81) v ∈ G p ( v | pa( v ) , x ) , which leads to a clean factorization between graph elements v and fast probability computation.
However, it also assumes a local Markov property that v is conditional independent to its ancestors given the parent pa( v ) . In theory, the
[Figure]
u c i
[Figure] probability learned by a seq2seq model is capable of modeling higher order conditionals between arbitrary elements on the graph.
Therefore it is interesting to ask if a more sophisticated graphical model with higher-order dependency structure can lead to better performance in practice while maintaining reasonable computational complexity [Figure] [Table] [Table]
Essay
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
LOGON c
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
92.83 c=3,d= 91.
92.96 c=3,d= 92.
94.38 c=3,d= 92.
95.10 c=5,d= 91.
93.28 c=5,d= 93.
95.22 c=5,d= 93.
Verbmobil c=1,d= 90.
90.76 c=1,d= 90. 91.76 c=1,d= 90.
91.68 c=3,d= 89.
91.14 c=3,d= 92.
94.18 c=3,d= 93.
95.40 c=5,d= 90.
92.37 c=5,d= 94.
95.43 c=5,d= 94.
Wiki c=1,d= 91.
91.84 c=1,d= 91.
92.70 c=1,d= 91.
92.53 c=3,d= 91.
92.38 c=3,d= 92.
93.49 c=3,d= 92.
NaN 93.
93.84 c=5,d= 91.
92.76 c=5,d= 92.
94.67 c=5,d= 93.
[Table] [Table]
LOGON
Random 6.
46.74 Uncertain 14.
64.29 Oracle 25.
Verbmobil
Random 12.
62.55 Uncertain 23.
80.65 Oracle 34.
Wiki
Random 4.
43.48 Uncertain 14.
57.82 Oracle 22.
SMCalflow
Random 10.
56.20 Uncertain 14.
59.67 Oracle 21.
SNIPS
Random 39.
[Table]