S UBS : Subtree Substitution for Compositional Semantic Parsing
Abstract
Although sequence-to-sequence models often achieve good performance in semantic parsing for i.i.d.
data, their performance is still inferior in compositional generalization.
Several data augmentation methods have been proposed to alleviate this problem.
However, prior work only leveraged superficial grammar or rules for data augmentation, which resulted in limited improvement.
We propose to use subtree substitution for compositional data augmentation, where we consider subtrees with similar semantic functions as exchangeable.
Our experiments showed that such augmented data led to significantly better performance on S CAN and G EO Q UERY , and reached new SOTA on compositional split of G EO Q UERY .
We have publicly released our code at https: //github.com/GT-SALT/SUBS .
Introduction
Semantic parsing transforms natural language utterances to formal language.
Because meaning representations or programs are essentially compositional, semantic parsing is an ideal testbed for compositional generalization.
Although neural seq2seq models could achieve state-of-the-art performance in semantic parsing for i.i.d.
data, they failed at compositional generalization due to lack of reasoning ability.
That is, they do not generalize well to formal language structures that were not seen at training time.
For example, a model that observes at training time the questions “ What is the population of the largest state? ”
and “ What is the largest city in USA? ”
may fail to generalize to questions such as “ What is the population of the largest city in USA? ”.
This leads to large performance drops on data splits designed to measure compositional [Table] generalization (compositional splits), in contrast to the generalization abilities of humans.
To improve compositional generalization in semantic parsing (compositional semantic parsing), prior work focused on incorporating inductive biases directly to models or data augmentation.
From the model perspective, some work used neuralsymbolic models (Chen et al., 2020), generated intermediate discrete structures (Herzig and Berant, 2020; Zheng and Lapata, 2020), or conducted meta-learning (Lake, 2019).
From the data perspective, Jia and Liang (2016) proposed to recombine data with simple synchronous context-free grammar (S CFG ), despite not for compositional generalization.
Andreas (2019) used some simple rules for data augmentation, where tokens with the same context were considered as exchangeable.
Such techniques are still limited since they only leveraged superficial grammars or rules, and failed when there are linguistically rich phrases or clauses.
To fill this gap, we propose to augment the training data of semantic parsing with diverse compositional examples based on induced or annotated (semantic and syntactic) trees.
Specifically, we propose to exchange subtrees where roots have similar meaning functions.
Since we consider all hierarchies in all trees, deep structures and complex phrases or clauses are considered for data augmentation, which is key for compositional generalization.
For instance, in Figure 1, if we exchange subtrees with “ largest ” as meaning function of its root, composition of “ population of the ” and “ largest city in the smallest state in the USA ” results in a new augmented structure “ population of the largest city in the smallest state in the USA ”.
Although certain substructure substitution methods were explored in other NLP tasks (Shi et al., 2021), subtree substitution with fine-grained meaning functions has been under-explored.
Our experiments showed that such augmented data led to significantly better performance on S CAN (Lake and Baroni, 2018) and G EO Q UERY , and reached new SOTA on compositional split of G EO Q UERY .
Methods Span trees
Suppose training set is { ( x i , z i ) } Ni =1 , where x i is a natural language utterance and
z
i is the corresponding program.
An utterance x can be mapped to a span tree T , such that program( T )=
z , where the deterministic function program( · ) maps span trees to programs (Herzig and Berant, 2020).
As shown in Figure 1, a span tree T is a tree where each node covers a span ( i , j ) with tokens
x i : j = ( x i , x i +1 , · · · , x j ).
A span subtree can be viewed as a mapping from every span ( i , j ) to a single category c ∈ C , where C is a set of domainspecific categories representing domain constants, which include entities (e.g. countryid#usa in Figure 1) and predicates (e.g. loc_2 in Figure 1).
The final program can be computed from the span tree deterministically by the function program(·).
Concretely, program( T ) iterates over the nodes in T bottom-up, and generates a program
z i : j for each node covering the span ( i , j ).
For a terminal node, z i : j = c .
For an internal node, z i : j is determined by composing the programs of its children, z i : s and z s : j where s is the split point.
As in Combinatory Categorical Grammar, composition is simply function application, where a domain-specific type system is used to determine which child is the function and which is the argument.
Span trees can be induced by a hard-EM algorithm or semi-automatically annotated.
We refer the reader to Herzig and Berant (2020) to see how to obtain span-trees.
2. Subtree Substitution (S UBS )
As shown in Figure 1, we consider span subtrees with similar semantic functions as exchangeable.
Formally, func(·) maps a subprogram to a semantic category, and subtrees with the same semantic categories have similar semantic functions.
For two data points ( x 1 , z 1 ) and ( x 2 , z 2 ), if func( z 1 i 1 : j 1 ) =
func( z 2 i 2 : j 2 ), we obtain a new augmented ( x ′ , z ′ ): x ′ =
x 1: i 1 + x 2 i 2 : j 2 + x 1 j 1 : , z ′ = z 1 \ z 1 i 1 : j 1 /z 2 i 2 : j
Definition of func(·) may vary in different dataset.
One straightforward way is to extract the outside predicate in z i : j as its semantic category, which is used on G EO Q UERY , such as func( largest ( state ( all ) ) ) )
= largest .
2. Semantic Parsing
After getting augmented data by subtree substitution, we then combine augmented data and the original training data to train a seq2seq semantic parser, where we choose LSTM models with attention (Luong et al., 2015) and copying mechanism (Gu et al., 2016), or pretrained BART Large (Lewis et al., 2020) as the seq2seq model architecture.
Experiments and Results
Dataset We first use S CAN (Lake and Baroni, 2018) as a diagnostic dataset to test the performance of subtree substitution in compositional semantic [Table]
[Table] S CFG (Jia et al., [Table] G ECA (
[Table]
[Table]
S [Table] ours [Table]
S
[Table]
ours
[Table]
parsing.
S CAN is a synthetic dataset, which consists of simple English commands paired with sequences of discrete actions.
We use the program version of Herzig and Berant (2020).
For instance, “ run right after jump ” corresponds to the program “ i_after ( i_run ( i_right ) , i_jump ) ”.
Also, semi-automatically annotated span trees from Herzig and Berant (2020) are used for subtree substitution.
To test compositional semantic parsing, we use the Primitive right (R IGHT ) and Primitive around right (A ROUND R IGHT ) compositional splits from Loula et al.
(2018), where templates of the form Primitive right and Primitive around right (respectively) appear only in the test set.
In these templates, Primitive stands for jump , walk , run , or look .
For simplicity, func(·) is defined only on i_right and i_left , where func( i_right ) =
func( i_left ) = direction .
That is, all “ i_right ” and “ i_left ” appear as leaf nodes in span trees and they are exchangeable.
We use G EO Q UERY dataset to test the performance of subtree substitution in both i.i.d.
and compositional generalization for semantic parsing.
G EO Q UERY contains 880 questions about US geography (Zelle and Mooney, 1996).
Following Herzig and Berant (2020), we use the variable-free FunQL formalism from Kate et al.
The i.i.d.
split (Question), which is randomly sampled from the whole dataset, contains 513/57/256 instances for train/dev/test set.
The compositional split (Query) contains 519/54/253 examples for train/dev/test set, where templates created by anonymizing entities are used to split the dataset, to make sure that all examples sharing a template are assigned to the same set (Finegan-Dollak et al., 2018).
As for span trees, we use semi-automatically annotated span trees (gold tree) released by Herzig and Berant (2020).
Alternatively, we use the span trees induced by Herzig and Berant (2020)’s span-based semantic parsing, without any human labour.
3. Diagnostic Results
Results of diagnostic experiments on S CAN dataset are shown in Table 1, where we use LSTM parser without data augmentation as the baseline.
We can see that on the R IGHT split, LSTM seq2seq semantic parser could only achieve zero exact-match accuracy without any data augmentation techniques, which means that the model’s compositional generalizibility on the R IGHT split is very poor.
After adding our augmented data with subtree substitution, we achieve an exact-match accuracy of 100%.
Actually, we got 6660 augmented examples besides the original 12180 training examples.
Among all augmented examples, 3351 examples are in the test set, which means 74.87% of 4476 test examples are recovered by subtree substitution.
On the
A ROUND R IGHT split, using LSTM seq2seq semantic parser could already achieve 100% exact-match accuracy, which means that the model learned from
Primitive right and Primitive opposite right generalize to Primitive around right well in our program format “ i_primitive ( i_around ( i_right ) ) ”.
After adding our augmented examples, the parser converged to 100% exact-match accuracy faster, where our method requires around 800 updates to converge while baseline model requires 2800 updates with the same batch size 64.
3. Main Results
Table 2 shows the results of experiments on G EOQ UERY dataset, where we examined both seq2seq
LSTM and BART Large parsers.
LSTM and BART parsers without any data augmentation are simplest baselines.
We also compare with other two data augmentation methods as additional baselines, recombining data with simple S CFG (Jia and Liang, 2016) or using simple rules for Good Enough Data
Augmentation (G ECA ) (Andreas, 2019), which were proven useful for compositional semantic parsing.
We can see that on the Question split, adding augmented data from (gold) subtree sub- [Table] [Table] stitution leads to improvements for both LSTM and BART seq2seq models, suggesting that subtree substitution as data augmentation helps i.i.d generalization for semantic parsing.
On the Query split, (gold) subtree substitution achieves more substantial improvements over seq2seq baseline models (absolute 21% and 3% improvements of the exactmatch accuracy for LSTM and BART respectively), achieving state-of-the-art results.
Moreover, our methods are also better than the two data augmentation baselines.
Therefore, subtree substitution is a simple yet effective compositional data augmentation method for compositional semantic parsing.
With (induced) subtree substitution, S UBS still achieves improvements for LSTM models.
Analysis of Augmented Data We further examine why subtree substitution could achieve much better performance by analyzing its augmented data.
As shown in Table 3, G ECA only identifies and exchanges very simple structures, where the average and maximal length of exchanged segments are 1.93 and 4.
A closer look at these augmented data shows that nearly all of these segments are simple entities (e.g. S TATE : “Illinois”, “Arizona” etc.) or other Nouns (e.g. “area”, “population” etc.).
In contrast, subtree substitution can identify and exchange much more complex structures, where the average and maximal length of exchanged segments are 5.99 and 25.
For example, largest city in the smallest state in the USA and largest state are identified as exchangeable.
As a result, subtree substitution could produce more complex utterance and program pairs, where the average and maximal length of these resulted utterances are 10.43 and 26, compared with the average (8.53) and maximal (18) length of utterances returned by
G ECA .
Moreover, subtree substitution could generate much more augmented instances, because it can identify more complex structures besides those simple ones identified by G ECA .
Compared with
S CFG , S UBS could also identify complex structures automatically with subtrees, while S CFG only handle simple phrases defined by rules.
Effect of Training Data Size Table 4 shows that with more training examples, models’ performances improve.
In all settings, using (gold) subtree substitution boosts the performance of BART.
When there are 100 and 200 training examples, the improvement is more significant, demonstrating the effectiveness of S UBS in the few-shot setting.
Related Work
Several data augmentation methods have been introduced for (compositional) semantic parsing.
Jia and Liang (2016) recombined data by S CFG , and
Andreas (2019) used some simple rules to exchange tokens with the same context.
However, they leveraged only superficial grammars or rules, which has limited capacity to identify complex structures.
Akyürek et al.
(2020) learned to recombine and resample data with a prototype-based generative model, instead of using rules.
Certain substructure substitution methods have been explored for data augmentation in other NLP tasks (Shi et al., 2021).
Dependency tree cropping and rotation within sentence was used in low-resource language POS tagging (¸Sahin and Steedman, 2019) and dependency parsing (Vania et al., 2019).
Dependency tree swapping was explored in low-resource language dependency parsing (Dehouck and Gómez-Rodríguez, 2020), and Universal Dependency features was used for zero-shot cross-lingual semantic parsing (Yang et al., 2021).
However, subtree substitution with fine-grained meaning functions has not been examined.
Some rule-based data augmentation methods were also explored in table semantic parsing (Eisenschlos et al., 2020; Yang et al., 2022).
To the best of our knowledge, we are the first to explore tree manipulation for semantic parsing.
Conclusion
This work proposed to use subtree substitution to compositionally augment the data of semantic parsing to help the compositional generalization.
Our method achieved significant improvements over seq2seq models, other data augmentation methods and span-based semantic parsing. Acknowledgements
The authors would like to thank reviewers for their helpful insights and feedback.
This work is funded in part by a grant from Amazon and Salesforce
A Training Details
We adapted OpenNMT (Klein et al., 2017) for
LSTM model with attention and copying mechanism, while used fairseq (Ott et al., 2019) to implement BART model.
We manually tune the hyper-parameters.
For
LSTM models, we use one-layer bidirectional
LSTM in the encoder side and one-layer unidirectional LSTM in the decoder side.
We use dropout with 0.5 as dropout rate and Adam optimizer with a learning rate of 0.001.
We use MLP attention and reuse attention scores as copying scores.
On
G EO Q UERY , the batch size is set to 1 sentence without augmented data and set to 64 sentences with augmented data.
On S CAN , all batch sizes are 64 sentences.
For BART models, we use BART large models.
We use Adam as optimizer with a learning rate 1e-5.
We use dropout and attention dropout with 0.1 as dropout rate.
Also, we use label smoothing with a rate 0.1.
Batch sizes aretokens.
Besides, we employ a weight-decay rate 0.01.
All the parameters are manually tuned based on the dev performance.
We train all models on NVIDIA A100 [-SXM4 
-]{+SXM
+}
GB GPU.
We set the max training epoch to beand select the best performed epoch according to dev performance.
Training process on each clause or whole sequence could be finished within 3 hours.
For baselines with other data augmentation methods, we reran G ECA and S CFG on this FunQL formalism of G EO Q UERY and these splits with annotated span trees.
That’s why our results are a little different from the reported results in the original paper.
We got similar results with their source code and our code on our data, in order to make sure that there is no problem with our results and code.
We got the same denotation accuracy as reported by Herzig and Berant (2020), but we reported exactmatch accuracy on Table 2 for fair comparison.