An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks (2022)

Abstract: Though linguistic knowledge emerges during large-scale language model pretraining, recent work attempt to explicitly incorporate human-defined linguistic priors into task-specific fine-tuning. Infusing language models with syntactic or semantic knowledge from parsers has shown improvements on many language understanding tasks. To further investigate the effectiveness of structural linguistic priors, we conduct empirical study of replacing parsed graphs or trees with trivial ones (rarely carrying linguistic knowledge e.g., balanced tree) for tasks in the GLUE benchmark. Encoding with trivial graphs achieves competitive or even better performance in fully-supervised and few-shot settings. It reveals that the gains might not be significantly attributed to explicit linguistic priors but rather to more feature interactions brought by fusion layers. Hence we call for attention to using trivial graphs as necessary baselines to design advanced knowledge fusion methods in the future.

## introduction recently large-scale pretrained language models @xcite @xcite have shown to gain linguistic knowledge from unlabeled corpus and achieve strong performance on many downstream natural language processing (nlp) tasks. though probing analysis indicate that, to some extent, they can implicitly capture syntactic or semantic structures @xcite @xcite , whether they can further benefit from more explicit linguistic knowledge remains an open problem. attempts have been made to inject syntactic biases into language model pretraining @xcite @xcite or infuse finetuning with semantic information @xcite , and positive results are reported on downstream tasks. however, the concerns about the effect or viability of linguistic knowledge have been raised. on the one hand, the performance gains highly rely on the availability of human-annotated dependency parsers @xcite or oracle semantic graphs @xcite , which limits the real-world applications. developing accurate semantic graph parsers is yet challenging @xcite . on the other hand, incorporating trees induced from pretrained language models @xcite can outperform the ones fused with dependency-parsed trees for aspect-level sentiment analysis @xcite . this discovery is in line with the similar findings of trivial trees for tree-lstm encoders in sequence modeling tasks @xcite . in this work, we push the envelop and answer the following two questions. do knowledge fusion methods in @xcite benefit from trivial graphs that contain no linguistic information? if that's the case, where might the performance gains come from? with the above questions, we empirically revisit the effectiveness of linguistic knowledge fusion in language understanding tasks. motivated by @xcite , we compare the performance between original dependency-parsed trees and balanced trees for syntax fusion, and compare the results between parsed semantic graphs and sequential graphs for semantic fusion. to our surprise, trivial graphs outperform syntactic trees or semantic graphs in full-supervised setting and achieve competitive results in few-shot setting. all the evidence again shows that the linguistic inductive bias might not be the major contributor of consistent improvements over baselines. additional analysis gives some clues that the possible reasons are extra model parameters and feature interactions from fusion modules. this work encourages future research to add trivial graphs as necessary baselines when designing more advanced knowledge fusion methods for downstream tasks. our experimental code is available at https://github.com/hkust-knowcomp/ revisit-nlu-linguistic-knowledge . ## study design in this section, we briefly introduce two linguistic graphs, i.e., syntactic dependency trees and semantic graphs. as a comparison, we manually construct two trivial graphs to infuse with task-specific finetuning. ## conclusion our study demonstrates that glue tasks can benefit from both trivial graphs and linguistic graphs, indicating that the performance gains of previous fusion methods should not be attributed to linguistic bias entirely. we argue that comparisons merely between methods with and without knowledge fusion may not be able to capture the whole picture. for example, without baselines considering trivial graph structures, the quality of various fused knowledge may not be accurately assessed. more careful evaluations of the effectiveness claims in existing work @xcite @xcite , inter alia) may be encouraged in the same spirit. in addition, tasks and evaluation benchmarks are also crucial to investigate when linguistic structures help. our study contributes to the broader question of how to accurately evaluate models that integrate external knowledge for downstream tasks, such as world or commonsense knowledge @xcite .