In- B o XBART : Get Instructions into Biomedical Multi-Task Learning (2022)

Abstract: Single-task models have proven pivotal in solving specific tasks; however, they have limitations in real-world applications where multi-tasking is necessary and domain shifts are exhibited. Recently, instructional prompts have shown significant improvement towards multi-task generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain. Motivated by this, this paper explores the impact of instructional prompts for biomedical MTL. We introduce the BoX, a collection of 32 instruction tasks for Biomedical NLP across (X) various categories. Using this meta-dataset, we propose a unified model termed as In-BoXBART, that can jointly learn all tasks of the BoX without any task-specific modules. To the best of our knowledge, this is the first attempt to propose a unified model in the biomedical domain and use instructions to achieve generalization across several biomedical tasks. Experimental results indicate that the proposed model: 1) outperforms single-task baseline by ~3% and multi-task (without instruction) baseline by ~18% on an average, and 2) shows ~23% improvement compared to single-task baseline in few-shot learning (i.e., 32 instances per task) on an average. Our analysis indicates that there is significant room for improvement across tasks in the BoX, implying the scope for future research direction.

## introduction for long, task-specific models have played a central role in achieving state-of-the-art performance in both general and biomedical nlp @xcite . during 2017-2019, pre-train and fine-tune paradigm @xcite became the prevalent approach in nlp. due to success of language models (lms) in the biomedical in this approach, a model is allowed to utilize tasks to get familiar with instructions and use them to map a given input to its corresponding output. figure 1 shows the overview of our proposed multitasking approach for various biomedical nlp tasks. recently, prompt-based models have been widely used because of their ability to achieve generalization instead of task-specific models @xcite . @xcite ; @xcite and @xcite show the effectiveness of instructional prompts in generalizing on seen as well as unseen general-domain nlp tasks. in this paper, we adapt this instructional promptbased approach for the first time to achieve generalization across various biomedical nlp tasks. to this extent, this paper introduces a collection of 32 instruction tasks for biomedical nlp across (x) various categories (box) and proposes a unified model that can generalize over 32 different biomedical nlp tasks. the proposed unified model (i.e., in-boxbart) is trained on the instruction-based meta-dataset (i.e., box) and evaluated on each task individually from the box. to evaluate the proposed approach, we compare our model (i.e., in-boxbart) with two baselines: multi-task learning owing to the problems associated with single-task learning in terms of their space and time requirements, several multi-task learning approaches have been proposed over the years. decanlp @xcite built a multi-tasking model by converting format of each tasks to question answering format. several other works have followed similar approach, for example, by converting tasks to reading comprehension @xcite and textual entailment format @xcite . the multitasking model t5 @xcite was built with the help of a unified framework that converts all text-based language problems into a text-to-text format. sci-five @xcite involved building a textto-text model for the biomedical literature. @xcite introduced pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning to improve multitask learning performance. models empowered by multi-task learning have achieved sota in many different tasks, e.g., question answering (qa) @xcite , commonsense reasoning @xcite and structured knowledge grounding tasks @xcite . instruction learning the turking test @xcite ) was proposed to measure the efficacy of models to follow instructions. studies have been made to investigate the effect of natural language instructions on model performance @xcite @xcite . moreover, @xcite proposed natural instructions which break down each task to multiple sub-tasks that help models in following instructions and subsequently generalize to unseen tasks (i.e., cross-task generalization). flan @xcite and t0 @xcite models were built by leveraging instruction/prompt-tuning on diverse range de-identification (1 task) ner (15 tasks) anatem bc2gm bc4chemd bionlp 2009 bionlp11epi bionlp13ge ex-ptm jnplba linnaeus ncbi bc5cdr bionlp11id bionlp13pc bionlp13cg craft n2c2 -de-identification 2006 pos tagging (1 task) genia relation extraction (2 tasks) drug-drug interaction chemprot question answering (4 tasks) bioasq-8b yesno sentiment analysis (1 task) bioasq-8b factoid bioasq-8b list pubmed qa systematic review (5 tasks) hormone replacement therapy covid cooking acromegaly accelerometer sentiment analysis medical drugs document classification (1 task) hallmarks of cancer risk factor identification (2 tasks) n2c2 -risk factors heart disease 2014 yesno n2c2 -risk factors heart disease 2014 time of tasks and achieving zero-shot generalization on target unseen tasks. task reframing @xcite proposed several guidelines to reframe task instructions to improve model response to follow instructions. analysis introduced to understand in-context learning better on a large set of training tasks @xcite . instructgpt model @xcite is proposed, which is fine-tuned with human feedback to follow natural instructions. furthermore, many works focused on investigating whether lms understands meaning of natural language and prompts @xcite . weller et al. (2020) and ye and ren (2021a) use task descriptions to achieve generalization to new tasks. puri et al. (2022) introduced instruction augmentation to improve model performance and sample complexity. wang et al. (2022) has developed instruction-based multi-task framework for few-shot named entity recognition (ner) task. prasad et al. (2022) introduced gradient-free instructional prompt search (grips) for improving task instructions for large lms. recently, many approaches have been proposed to improve model performance using instructions @xcite @xcite . ## box we use 29 existing, widely adopted biomedical nlp datasets collected from various challenges, platforms and organizations to create box. we define the box as a benchmark dataset for biomedical mtl across 9 different categories. in the box, category # of training samples ner 82503 de-identification 106 pos tagging 16323 qa 5778 re 23359 sentiment analysis 2860 systematic review 5761 document classification 3119 risk factor identification 986 total 140795 ## ----------------------------average effect of instruction in few-shot learning we have compared the average rouge-l of in-boxbart with a single-task baseline for fewshot setting. figure 5 shows the relative performance of in-boxbart compared to single-task baseline. we have shown results for all few-shot learning experiments in appendix d. from the results, we see that in-boxbart achieves on an average 60.64% rouge-l and the single-task model achieves 37.31% for 32 instances per task. in-boxbart significantly outperforms the single-task baseline by 23.33%. from figure 5 , we can see that in-boxbart consistently perform better compared to the baseline. as we know, obtaining a large annotated dataset in the biomedical domain is difficult, time-consuming and costly. from fewshot learning, we can see that instructions are beneficial in achieving high performance compared to task-specific models. ## discussion can we design better instructions? since instruction teach the model how to solve a given task, domain specific information rich instructions can improve model performance. one potential way is to use the knowledge of domain experts. however, designing a good biomedical instruction can be one research direction. how to handle long-context input? training instances of many biomedical datasets consist electronic health records (ehrs) or discharge summaries of patients. because of this, these instances are long and exceed the maximum input length of lms such as bert, bart. in this scenario, encoding extra information in terms of prompts or instructions becomes difficult. one potential solution is to use longformer @xcite , and another solution is to use t5 kind of models which use relative position embeddings so that the inference length can be longer @xcite . how to handle multi-class classification tasks? multiple classes cause an issue while creating biomedical instructions because we can not present one example per class. if we do that, the encoding of bi and input will exceed the maximum length of lms. a naive solution is to select examples of a few labels or remove the examples. however, this will cause a label bias issue or performance degradation. potential future research direction can be designing a methodology to handle multi-class classification tasks. how far we are from the sota? we have presented preliminary comparison of our results w.r. ## summary and conclusions this research shows the impact of instructions in mtl for the first time in the biomedical domain. to this extent, we introduced the box, a first benchmark dataset consisting of 32 instruction tasks across various biomedical nlp domains. using this meta-dataset, we proposed a unified model, i.e., in-boxbart which outperforms single-task baseline and vanilla-boxbart by ∼ 3% and ∼ 18%, respectively. our proposed approach also shows an effective performance for a few-shot setting which is more beneficial in the biomedical domain where obtaining large annotated datasets is difficult. we hope that the box benchmark, in-boxbart, and experimental results encourage future research into more unified models for biomedical nlp.