Multi-Task Learning with Adapters for Plausibility Prediction: Bridging the Gap or Falling into the Trenches? (2024)

Abstract: We present a multi-task learning approach to predicting semantic plausibility by leveraging 50+ adapters categorized into 17 tasks within an efficient training framework. Across four plausibility datasets in English of varying size and linguistic constructions, we compare how models provided with knowledge from a range of NLP tasks perform in contrast to models without external information. Our results show that plausibility prediction benefits from complementary knowledge (e.g., provided by syntactic tasks) are significant but non-substantial, while performance may be hurt when injecting knowledge from an unsuitable task. Similarly important, we find that knowledge transfer may be hindered by class imbalance, and demonstrate the positive yet minor effect of balancing training data, even at the expense of size.

## introduction the ability to distinguish between plausible and implausible events represents a crucial building block for natural language processing (nlp). while existing models include classical transformer-based approaches @xcite , researchers also devise world-knowledge features @xcite , and examine lexical abstraction chains @xcite in order to integrate relevant but yet missing information. in contrast, our work tackles the prediction of plausibility from a novel perspective, by testing whether knowledge from different tasks may be used to fill knowledge gaps and to improve plausibility models in low-to mid-size resource scenarios. leveraging adapters @xcite @xcite as an efficient multi-task learning framework, we train 53 task adapters categorized into 17 tasks ranging from syntactic problems such as parsing to lexical semantics tasks such as abstractness prediction as well as sentence-and discourse-level semantics problems such as question answering. across four plausibility datasets in english of varying size and linguistic constructions, we compare how models perform without external information (singletask adapters) in contrast to models provided with knowledge from other tasks (multi-task learning with adapter-fusion). in particular, the main goal of this paper is not to improve state-of-the-art results for each dataset but to explore whether task transfer through adapter-fusions works better than single-task adapters. more specifically, we are interested in the relationships between the source tasks (e.g., abstractness prediction or parsing) and the target task (plausibility prediction), and investigate which kind of knowledge is potentially relevant but yet missing for successfully predicting whether a given event is plausible or implausible. we first train single-task adapters for plausibility using each datasets' training data, and then explore the impact of additional within-task data regarding class balance as a potential factor. this is relevant insofar as language models (lms) are commonly pretrained on mainly plausible training data and should thus be expected to perform better for plausible than implausible data. in a second step, we train and evaluate a range of adapter fusion models. our results indicate that (i) depending on the dataset, single-task adapter models can represent a viable alternative to full fine-tuning, (ii) knowledge from different tasks does not substantially improve and even hurt performance, depending on task, dataset, and training data setting, and (iii) adding indomain data and removing class imbalance sustains plausibility prediction across datasets. analyzing task categories reveals minimum negative impact from syntactic tasks, followed by discourse-level and lexical semantics tasks. we thus conclude that given the prerequisite of class balance, knowledge transfer through adapter fusion does not lead to substantial improvements for plausibility prediction when leveraging complementary tasks and might be even hurt in case of more closely related tasks. in our work, we address the challenge of modelling plausibility from a novel angle, and test whether leveraging knowledge from other tasks improves plausibility prediction with a standard-sized transformer model through providing information from closely related vs. vastly different tasks, thus exploring which knowledge gaps need to be filled. multi-task learning with adapters adapters @xcite have been introduced as a parameter-efficient fine-tuning approach foot_1 for trans-formers @xcite with comparable performance. they consist of sets of additional task-specific parameters that are introduced at every layer of a transformer and updated during finetuning, while the remaining plm parameters are kept frozen. since adapters can be used in a modular fashion, they are particularly well-suited for multi-task and cross-lingual transfer learning @xcite @xcite as well as to inject external knowledge sources to solve downstream tasks @xcite . we use adapters @xcite @xcite as our framework; it enables both training task-specific adapters, i.e., knowledge extraction, and combining the trained adapters in a second step through knowledge composition in a non-destructive way. ## datasets we harness four english datasets for plausibility: pep-3k @xcite consists of 3,062 subject-verb-object events in english that focus on highly concrete concepts, e.g., lion-destroy-house. events have been judged plausible or implausible by five crowd-sourced annotators. 20q foot_2 comprises a collection of 20 question-style games played by crowd-sourced workers. one player tries to guess a topic by asking questions to the other player (who knows the topic) that lead to a discrete answer. possible answers are {always, usually, sometimes, rarely, never}. we use the dataset version adapted for binary plausibility classification by @xcite . @xcite encompasses 16,115 english sentence pairs differing only in an adjective modifying a noun, e.g., {a horse goes away ø a dead horse goes away}. the dataset was collected for predicting changes in plausibility within a multi-class setting; the set of labels is {impossible, less likely, equally likely, more likely, necessarily true}. to train and evaluate on this dataset, we map every s1 from the sentence pairs xs1, s2y to the label plausible. for sentences s2 we map the labels impossible and less likely to implausible, and the labels equally likely, more likely and necessarily true to plausible. ellie @xcite is a small dataset composed of 575 english elliptical constructions, i.e., the dataset was constructed to evaluate the effect of argument thematic fit when resolving ellipses. instances are labeled typical, atypical, or violating selectional preference regarding agents and patients. we map the labels typical and atypical to plausible, and instances violating selectional preference to implausible. while we add ellie data for training, our main use of the dataset is for in-domain evaluation, to assess generalization to complex linguistic constructions. for an overview of dataset statistics, training setting sizes, dataset splits, and details regarding the conversion of selectional preference datasets such as ellie for plausibility modeling, we refer to app. b. ## models single-task adapters to establish baseline performance for predicting plausibility without knowledge from additional tasks, we train single-task (st) adapters. 4 to further explore the influence of adding within-task knowledge and class imbalance, we experiment with training (i) on the train portion of each target dataset (train); (ii) on all full datasets except for the target dataset, and evaluate on the target datasets' dev and test, with and without removing class imbalance (w/o train, w/o train+b); (iii) on all datasets, including train of the target dataset, and evaluate on target datasets' dev and test, with and without removing class imbalance (w/ train, w/ train+b). to compare results to previous work, we test models trained on the respective other datasets (w/o train, w/o train+b) and evaluate on pep-3k and 20q dev and test set splits as used by @xcite . we conduct an intermediate error analysis on our st adapter models, in order to understand how training data choices influence model performance. for this, we calculate error overlap at instance level and compute spearman's ρ across training settings. in case of substantial overlap between wrongly predicted instances, we assume low influence of training data. in the reverse case, we assume that training data does make a difference. more details and results are presented in app. c, fig. 1 with observations indicating that additional training data leads to different types of errors and may thus add relevant knowledge. furthermore, removing class imbalance alters sets of errors significantly, in case of a previously imbalanced dataset. adapter fusion we make use of 53 sourcetask adapters trained on 17 tasks categorized into syntactic, lexical-semantic, and sentence/discourse level (for an overview see to incorporate knowledge from other tasks, we train task-based adapter fusions using all task adapters belonging to a task, plus a task adapter for plausibility prediction. experimental setup we use roberta @xcite (roberta-base) as the backbone transformer for all models. we train st adapters for our target task of predicting whether a text input is plausible or not by using a task-specific prediction head, thus following the training setup recommended by @xcite . we pick the best model based on development set results optimizing for macro f1. to train adapter fusions, we use the three best-performing target task adapters based on st performance. we consider three training data settings to explore knowledge transfer (i) in lowresource settings and high class imbalance (train), (ii) in cases where no train portion might be available or included (w/o train), and (iii) for balanced datasets (w/ train+b). our hypothesis is that training with small and imbalanced datasets may particularly benefit from knowledge transfer. the training setup mirrors the single-task setup, except for using a smaller learning rate and a larger batch size as in @xcite , with models optimized for macro f1. for more details, we refer to app. d. ## results in the following, we present our results comparing fusion-based against single-task adapter models for the target task of assessing plausibility. we use the almost stochastic order (aso) test @xcite as implemented by @xcite to assess which training and does knowledge transfer through adapter fusion improve models of plausibility? does adding in-domain data improve models of plausibility? ## limitations and future directions events based on s-v-o events or comparably simple constructions have been successfully leveraged for exploring selection preference and thematic fit tasks @xcite @xcite . however, the addition of context could potentially resolve potential ambiguities in the s-v-o triples and thus improve plausibility prediction. furthermore, while we train and evaluate our models on datasets such as adept coming with sentence-level contexts, high class imbalance leads to a relatively small proportion of implausible sentences which are particularly relevant as lms are usually pretrained on mostly plausible data and expected to inherently perform better for plausible expressions. we hope future research extends this work by collecting plausibility ratings for more complex constructions within broader contexts. here, @xcite and @xcite present interesting work exploring the generation of implausible and less plausible but relevant outputs to complement their dataset with the goal of increasing model performance and assist humans in well-balanced decision-making, respectively. further, experiments with a wider variety of (larger) models represent a relevant future task to explore whether the presented negative results are specific to the used underlying transformer backbone or prevalent across model sizes and families. finally, in this work, we follow previous research @xcite @xcite regarding the formulation of plausibility prediction as a binary classification task to discern plausible from implausible events. plausibility can, however, also be captured in a graded way using more finegrained labels that allow for graded classification such as the label set {impossible, less likely, equally likely, more likely, necessarily true} adopted by @xcite for modeling change in semantic plausibility between two sentences. we thus encourage further research on modeling plausibility from a graded perspective to capture the phenomenon at a more fine-grained level. ## conclusion we tackled the task of discerning plausible from implausible events by adopting a multi-task learning perspective and exploring whether knowledge transfer from different tasks improves performance and reveals insights about relevant knowledge. using 53 adapters categorized into 17 tasks, we found that complementary knowledge sustains but not substantially improves performance, while choosing a "wrong" task might seriously hurt the results. we further demonstrated that knowledge transfer may be hindered by class imbalance, and that balancing training data shows a significant positive yet non-substantial effect, even at the expense of size.