Representations of Syntax [MASK] Useful: E ffects of Constituency and Dependency Structure in Recursive LSTM s (2020)

Abstract: Sequence-based neural networks show significant sensitivity to syntactic structure, but they still perform less well on syntactic tasks than tree-based networks. Such tree-based networks can be provided with a constituency parse, a dependency parse, or both. We evaluate which of these two representational schemes more effectively introduces biases for syntactic structure that increase performance on the subject-verb agreement prediction task. We find that a constituency-based network generalizes more robustly than a dependency-based one, and that combining the two types of structure does not yield further improvement. Finally, we show that the syntactic robustness of sequential models can be substantially improved by fine-tuning on a small amount of constructed data, suggesting that data augmentation is a viable alternative to explicit constituency structure for imparting the syntactic biases that sequential models are lacking.

## introduction natural language syntax is structured hierarchically, rather than sequentially @xcite . one phenomenon that illustrates this fact is english subject-verb agreement, the requirement that verbs and their subjects must match in number. the hierarchical structure of a sentence determines which noun phrase each verb must agree with; sequential heuristics such as agreeing with the most recent noun may succeed on simple sentences such as (1a) but fail in more complex cases such as (1b): we investigate whether a neural network must process input according to the structure of a syntactic parse in order for it to learn the appropriate rules governing these dependencies, or whether there is sufficient signal in natural language corpora for low-bias networks (such as sequential lstms) to learn these structures. we compare sequential lstms, which process sentences from left to right, with tree-based lstms that process sentences in accordance with an externally-provided, groundtruth syntactic structure. we consider two types of syntactic structure: constituency structure @xcite and dependency structure @xcite . we investigate models provided with either structure, both structures, or neither structure (see we find that building in either type of syntactic structure improves performance over the bilstm baseline, thus showing that these structures are learned imperfectly (at best) by low-bias models from natural language data. of the two types of structure, constituency structure turns out to be more useful. the dependency-only model performs well on natural language test sets, but fails to generalize to an artificially-constructed challenge set. after fine-tuning on a small dataset that is designed to impart a strong syntactic signal, the bilstm generalizes more robustly, but still falls short of the tree-based lstms. we conclude that for a network to robustly show sensitivity to syntactic structure, stronger biases for syntactic structure need to be introduced than are present in a low-bias learner such as a bilstm, and that, at least for the subject-verb agreement task, constituency structure is more important than dependency structure. both tree-based model structure and data augmentation appear to be viable approaches for imparting these biases. ## related work prior work has shown that neural networks without explicit mechanisms for representing syntactic structure can show considerable sensitivity to syntactic dependencies @xcite @xcite , and that certain aspects of the structure of the sentence can be reconstructed from their internal representations @xcite @xcite . @xcite showed that sequential models still have substantial room for improvement in capturing syntax, and other work has shown that models with a greater degree of syntactic structure outperform sequential models on syntax-sensitive tasks @xcite @xcite , including some of the tree-based models used here @xcite . one contribution of the present work is to tease apart the two major types of syntactic structure to see which one imparts more effective syntactic biases. ## experiment 2: fine-tuning in experiment 1, tree-based models dramatically outperformed the bilstm in the presence of attractors. this difference may have arisen because most natural language sentences are simple, and thus they do not generate enough signal to illustrate the importance of tree structure to a low-bias learner, such as a bilstm. recent work has shown the effectiveness of syntactically-motivated fine-tuning at increasing the robustness of neural models @xcite . would our models generalize more robustly if we added a few training examples that do not lend themselves to non-syntactic heuristics? to provide the model with a stronger signal about the importance of syntactic structure, we fine-tuned our models on a dataset designed to impart this signal. we used a variant of the pcfg (see appendix f) from section 5 to generate a 500sentence augmentation set. this augmentation set cannot be solved using word cooccurrence statistics, and contains some sentences with attractors. the models were then fine-tuned on the augmentation set for just one epoch over the 500 examples. see appendix c.2 for training details. results: the head-lexicalized model and the bilstm benefited most from fine-tuning, with the head-lexicalized model now matching the performance of the constituency lstm, and the bil-stm showing dramatic improvement on sentences with multiple attractors (figure 1b ; see appendix e, ## discussion overall, we found that neural models trained on natural language achieve much more robust performance on syntactic tasks when syntax is explicitly built into the model. this suggests that the information we provided to our tree-based models is unlikely to be learned from natural language by models with only general inductive biases. in experiment 1, the network provided with a dependency parse did the best on most of the natural language test sets. this is unsurprising, as the task is largely about a particular dependency (i.e., the dependency between a verb and its subject). at the same time, as demonstrated by the constructed sentence test, the syntactic capabilities of the dependency lstm are inherently limited. thus, it must default to non-robust heuristics in cases where the unlabeled dependency information is ambiguous. in future work, these syntactic limitations may be overcome by giving the model typed dependencies (which would distinguish between a subject-verb dependency and a verb-object dependency). one might expect the head-lexicalized model to perform the best, since it can leverage both syntactic formalisms. however, it performs no better than the constituency model when trained on natural language, suggesting that there is little benefit to incorporating dependency structure into a constituency lstm. in some cases, the headlexicalized model without fine-tuning even performs worse than the constituency lstm. when fine-tuned on more challenging constructed examples, the head-lexicalized model performed similarly to the constituency lstm, suggesting that there is not enough signal in the natural language training set to teach this model what to do with the heads it has been given. our results point to two possible approaches for improving how models handle syntax. the first approach is to use models that have explicit mechanisms for representing syntactic structure. in particular, our results suggest that the most important aspect of syntactic structure to include is constituency structure, as constituency models appear to implicitly learn dependency structure as well. though the models we used require parse trees to be provided, it is possible that models can learn to induce tree structure in an unsupervised or weakly-supervised manner @xcite @xcite . another effective approach for improving the syntactic robustness of neural models is data augmentation, as demonstrated in experiment 2. with this approach, it is possible to bring the syntactic performance of less-structured models closer to that of models with explicit tree structure, even with an augmentation set generated simply and easily using a pcfg. future work should further explore both of these approaches. our conclusions about the importance of explicit mechanisms for representing syntactic structure can be strengthened by developing different formulations of the tree lstms. it seems particularly promising to explore alternative formulations of the dependency lstm (as mentioned above) and the effect of learning embeddings of non-terminal symbols for the constituency lstm. finally, future work should investigate whether data augmentation can fully bridge the gap between low-bias learners and structured tree lstms, and whether our conclusions apply to other syntactic phenomena besides agreement.