Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models (2023)

Abstract: In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel adapter modules encoding different linguistic structures are combined using a novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates are used to determine the importance of these modules at each layer of the model. To reduce the number of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their important scores. Our experiment results with three different pre-trained models show that our approach can outperform state-of-the-art PEFT methods with a comparable number of parameters. In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.

## introduction in recent years, pre-trained language models have become the de facto instrument for the field of natural language processing (nlp) @xcite @xcite @xcite . this shift is largely due to the emergence and success of transformer-based models @xcite where large-scale pre-training helps the model to learn the syntactic and semantic structure of a language without explicit supervision. at the same time, there are good reasons to question whether these models can be said to understand a language in any meaningful and interpretable way @xcite . to address this conundrum, probing studies have demonstrated, to a certain extent, that it is possible to infer linguistic structures from the representations within these models @xcite @xcite . however, the precise connection between the existence of structures and their benefits to task performance is yet to be firmly established. on the other hand, while the conventional way of fine-tuning has found success in a wide array of nlp tasks, its applicability has increasingly diminished due to the associated computational expense with the recent shift towards larger and more complex models @xcite . while some have argued that pre-training on unstructured text alone equips the model with sufficient capacity to comprehend the meaning of language, others @xcite have asserted that mapping the model's behavior onto human-comprehensible structures offers more dependable evidence of its ability to tackle tasks beyond merely exploiting superficial cues. specifically, studies in this area have yielded successful attempts to inject syntactic and semantic structures into the pre-trained language models @xcite @xcite , with positive results reported on downstream tasks. however, despite recent efforts, no existing work has addressed the problem of where and how to effectively inject multiple different structures in an efficient manner. the conventional approach of fine-tuning pretrained nlp models involves optimizing the full set of model parameters for each task. however, this results in a separate copy of fine-tuned model parameters for each task and has become increasingly infeasible due to the recent trend of pre-training larger and larger models. to address these concerns, a surge of recent work has been dedicated to the study of parameter-efficient fine-tuning (peft) methods @xcite , where only a small portion of task-specific trainable parameters are tuned while keeping the rest of the model frozen. while these studies have achieved impressive performance even comparable to the full fine-tuning, they have been mostly focused on either determining the subset of model parameters for tuning @xcite or finding the location to insert additional trainable parameters @xcite @xcite . no existing work has addressed the problem of whether linguistic structural priors can be incorporated into these trainable parameters under the peft setting. in this work, we align the two research areas of injecting linguistic structures and peft by proposing a strategy of effectively combining multiple linguistic structures into pre-trained nlp models in a parameter-efficient fashion. to combine multiple linguistic structures, we propose a novel architecture inspired by mixture-of-experts models @xcite , where relational graph convolutional networks (rgcn) modules @xcite encoded with different linguistic trees are aggregated using learnable gumbel-softmax @xcite gates, and inserted between each layer of the pre-trained model. to reduce the number of parameters, we propose a pruning strategy where we first tune the full set of rgcn modules before pruning all but the top "experts" based on the importance score learned from the gates. to demonstrate the benefits of our approach, we perform experiments on the glue benchmark with three different pre-trained nlp models and compare the results with state-of-theart peft methods @xcite . further, we perform additional analysis to understand which types of linguistic structures are kept at each layer of the model and provide insights for future work on injecting knowledge through peft methods. in short, our contributions can be summarized as the following: ## related works we organize this section based on the two research areas that our work seeks to align. in §2.1, we provide an overview of techniques to inject linguistic structure, while §2.2 summarizes recent trends in parameter-efficient fine-tuning. ## model architecture in this section, we describe the architectures of our mixture-of-linguistic adapters (figure 1 ). we start by first introducing the relational graph convolutional network (rgcn) modules for incorporating linguistic structures ( §3.1) before describing the method used for combining multiple rgcn ( §3.2). finally, we discuss how the adapters are inserted into the pre-trained model ( §3.3). ## training strategy while the architecture proposed in section 3 allows us to aggregate multiple adapter modules at each pre-trained layer, it significantly decreases the efficiency due to the number of task-specific parameters used during training and inference. to address this issue, we propose a pruning strategy to reduce the number of experts. in order to decide which expert to keep at each layer, we first fine-tune the full set of expert modules using our mixture-of-linguistic-experts architecture (figure 1 ) for a fixed small number of steps. after the gates have converged, importance score from the gates can be used to determine which experts to keep. while an iterative pruning strategy @xcite @xcite can also be used, it is less efficient due to the requirement of more training steps. finally, after the pruning process, we restart the training process and fine-tune the resulting model with one expert module per layer. ## experiments we describe in detail the experimental settings and results in this section. we start by providing a brief summary of the linguistic graphs ( §5.1) before describing the datasets ( §5.2) models ( §5.3), and the hyperparameters settings ( §5.4). finally, we present the results in §5.5. ## analysis in this section, we provide an analysis of the model's behavior by first examining the linguistic expert used for each model ( §6.1) before examining the convergence rate of gumbel-softmax gates at different layers of the model ( §6.2). ## conclusion and future work in this work, we introduce an approach that combines the two popular research areas of injecting linguistic structures and parameter-efficient fine-tuning (peft). to start, we introduce a novel framework that combines multiple linguistic structures in an architecture inspired by the mixtureof-experts model, where gumbel-softmax gates are used to learn the importance of these experts at different layers of the model in a small fixed number of training steps. finally, we reduce the parameter count by pruning all but one expert at each layer such that the resulting number of trainable parameters is comparable to state-of-the-art peft methods. after running experiments with three different pre-trained models on the glue benchmark, the results show that our method can achieve the best overall performance while significantly outperforming the baselines on high-resource tasks. finally, we examine the experts selected by each model and the convergence rate of the gumbel-softmax gates to gain a better understanding of the models' behavior and provide valuable insights for future studies on knowledge injection. for future work, we plan to perform further experiments to determine the relative advantage of different linguistic knowledge and study how the quality of graphs affects model performance on downstream tasks. one significant challenge is to efficiently incorporate relation types of dependency trees, which we will explore in future work. in addition, we plan to further improve the efficiency of our approach by incorporating findings from other recent works, such as dropping adapters in the lower layers @xcite . lastly, we plan to extend our approach to inject linguistic structures (including discourse graphs) into decoderonly architectures @xcite and perform studies on larger model variants @xcite .