How Much Syntactic Supervision is “Good Enough”? (2023)

Abstract: In this paper, we explore how much syntactic supervision is “good enough” to make language models (LMs) more human-like. Specifically, we propose the new method called syntactic ablation, where syntactic LMs, namely Recurrent Neural Network Grammars (RNNGs), are gradually ablated from full syntactic supervision to zero syntactic supervision (≈ unidirectional LSTM) by preserving NP, VP, PP, SBAR nonterminal symbols and the combinations thereof. The 17 ablated grammars are then evaluated via targeted syntactic evaluation on the SyntaxGym benchmark. The results of our syntactic ablation demonstrated that (i) the RNNG with zero syntactic supervision underperformed the RNNGs with some syntactic supervision, (ii) the RNNG with full syntactic supervision underperformed the RNNGs with less syntactic supervision, and (iii) the RNNG with mild syntactic supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. Those results may suggest that the “good enough” approach to language processing seems to make LMs more human-like.

## introduction in the literature on targeted syntactic evaluation @xcite , recurrent neural networks (rnns) such as lstms have been demonstrated to implicitly learn syntactic structures of natural language (e.g., subject-verb agreement), despite the lack of explicit syntactic supervision (cf. @xcite . moreover, those rnns also turned out to benefit from explicit syntactic supervision. rnns integrated with explicit syntactic supervision, namely recurrent neural network grammars (rnngs; @xcite , have received considerable attention for their cognitive plausibility and outperformed † currently affiliated with leapmind inc.: noji@leapmind.io. ## discussion in summary, we performed the syntactic ablation experiments where rnngs were gradually ablated from full syntactic supervision to zero syntactic supervision (≈ unidirectional lstm), and then evaluated via targeted syntactic evaluation on the syntaxgym benchmark. in this section, the results of our syntactic ablation experiments will be discussed in the broader context of the computational psycholinguistic literature. ## conclusion in this paper, we explored how much syntactic supervision is "good enough" to make language models more human-like. specifically, we performed the syntactic ablation experiments where rnngs were gradually ablated from full syntactic supervision to zero syntactic supervision (≈ unidirectional lstm), and then evaluated via targeted syntactic evaluation on the syntaxgym benchmark. the results demonstrated that the rnng with mild syntactic supervision achieved the best performance comparable to the state-of-the-art gpt-2-xl. we hope that the "good enough" approach to language processing @xcite provides the promising direction for future research.