An Exploration of Encoder-Decoder Approaches to Multi-Label Classification for Legal and Biomedical Text (2023)

Abstract: Standard methods for multi-label text classification largely rely on encoder-only pre-trained language models, whereas encoder-decoder models have proven more effective in other classification tasks. In this study, we compare four methods for multi-label classification, two based on an encoder only, and two based on an encoder-decoder. We carry out experiments on four datasets—two in the legal domain and two in the biomedical domain, each with two levels of label granularity— and always depart from the same pre-trained model, T5. Our results show that encoder-decoder methods outperform encoder-only methods, with a growing advantage on more complex datasets and labeling schemes of finer granularity. Using encoder-decoder models in a non-autoregressive fashion, in particular, yields the best performance overall, so we further study this approach through ablations to better understand its strengths.

## introduction multi-label classification constitutes the task of predicting multiple labels for an input as opposed to a single (possibly binary) one. the labels are drawn from a set of up to several hundred classes, often with the added challenge of class imbalance. while the order in which labels are predicted is irrelevant, there can be interdependence between subsets of labels. the task is commonly approached with a classification model based on a pre-trained encoder followed by a multi-output classification head. encoder-decoder models, like t5 (raffel et al., 2020), have taken over recent nlp literature with state-of-the-art results on various tasks, such as question-answering (qa), summarization, singlelabel classification, etc. raffel et al. (2020) showed that any given nlp task could be reformulated as a text-to-text task and solved with conditional * equal contribution. generation, i.e., generating a text sequence that represents the desired output, be that a span of text in qa, a text summary, a label descriptor, etc. liu et al. (2021) presented an alternative use of encoder-decoder models for classification tasks in particular, wherein t5's decoder is used in a nonautoregressive fashion to obtain output representations, which are then fed to a classification head. the application of encoder-decoder methods to multi-label classification is currently limited to one experiment in the work of liu et al. (2021), who compare a text-to-text approach and their nonautoregressive approach on a single dataset, including an encoder-only baseline built off of a different pre-trained model, bert (devlin et al., 2019). they obtain results favorable to the two encoderdecoder methods, but since the focus of their work is not multi-label classification in particular, their evaluation is insufficient to draw hard conclusions about this task, and analysis on the contribution of different model components to performance on the task is missing altogether. in this work, we carry out an extensive study of encoder-decoder approaches to multi-label classification. to ensure the thorough and fair evaluation of all methods: we release our code base to assure reproducibility and let others extend our study by experimenting with new methods and more datasets. foot_0 ## related work class imbalance is a critical issue in multi-label classification, with researchers searching for the best method to handle rare (less represented) labels. encoder-only approaches snell et al. (2017) introduced the idea of a prototype label vector, obtained by averaging over all instances of a given class and used to add inductive bias to their prototypical network for multi-label classification. in a similar vein, mullenbach et al. (2018) developed the label-wise attention network (lwan) architecture, in which label-wise document representations are obtained by learning to attend to the most informative input words for each label, using trainable label vectors as keys. chalkidis et al. (2020) systematically studied the effects of different language encoders (cnns, bigrus, bert) and several variants of lwan with regards to the representation of prototype labels. experimenting with three datasets (eurlex, mimic-iii, and amazon), they showed that better language encoders counter-play the positive effect of the lwan module, i.e., a standard bi-gru classifier outperforms cnn-based lwans (mullenbach et al., 2018), and a standard bert outperforms bigru-lwan, respectively. moreover, bert-based lwans offer minor overall improvements compared to a vanilla bert classifier, wherein bert's cls token representation is passed to a classification head (devlin et al., 2019). chalkidis et al. (2021) were the first to explore the use of a t5 model for multi-label classification, although they only considered an encoder-only classifier, disregarding the model's decoder. they followed the now standard approach of a classification head on top of @xmath0. in experiments with mt5 (xue et al., 2021), they showcased improved results compared to xlm-r (conneau et al., 2020) on a newly introduced multilingual dataset, multieurlex. encoder-decoder approaches text-to-text approaches, which utilize the full encoder-decoder model, have proven effective for binary and singlelabel classification tasks (raffel et al., 2020; chung et al., 2022). the key to such approaches are label verbalizers, words in natural language which verbalize the underlying semantics of a given class. label verbalizers are represented in the embedding space of pre-trained models and in this way benefit from the model pre-training. this can be more optimal especially for few-and zero-shot labels, in comparison to head-based classification methods where randomly initialized parameters have to be learned from scratch. liu et al. (2021) presented an alternative use of the full t5 model for non-autoregressive tasks, e.g. single-label and multi-label classification, wherein the decoder is used to obtain label-wise representations informed by the input document, which in turn are fed to label-specific binary classification heads. liu et al. (2021) performed one set of experiments on the eurlex-57k dataset (chalkidis et al., 2019), in which they compared their nonautoregressive approach to a t5-based text-to-text approach and a standard bert-based classifier. they found that both t5-based approaches outperformed the encoder-only classifier, the nonautoregressive method performing best. nonetheless, the encoder-only classifier had less than half the parameters of the t5 model (110m vs 222m). encoder-decoder approaches thus seem to carry potential for multi-label classification, still with insufficient empirical evidence, however. ## methods we experiment with four methods for multi-label classification, encoder+head, lwan, seq2seq, and t5enc, basing their implementation on the t5 model (raffel et al., 2020). t5 is a transformerbased encoder-decoder model (vaswani et al., 2017), which encodes a string of input tokens and generates a string of output tokens. all methods discussed below use t5's encoder to represent input documents, a document being denoted as [x 1 , x 2 , . . . , x n ], where n is the document length in terms of t5 subword tokens. some methods further use the model's decoder-we introduce decoder notation where needed. encoder+head in this case, we use only the encoder of t5 in the standard classification setting, as introduced by devlin et al. (2019). we feed the lwan in this case, we use a label-wise attention network (lwan) (mullenbach et al., 2018) on top of the t5 encoder, as done in chalkidis et al. (2020). we feed the document to the encoder, and use one attention head per label to generate l label-wise document representations d @xmath1, i.e., l weighted averages of the contextualized token representations. intuitively, each head focuses on possibly different tokens of the document relevant to the corresponding label. lwan employs l linear layers (o @xmath2, to produce l scores (logits), one per label. ## datasets we experiment with four datasets from the legal and biomedical domains, each with two different label granularities, i.e., label sets including more abstract or more specialized concepts. uklex united kingdom (uk) legislation is publicly available as part of the united kingdom's national archives. @xcite most of the laws have been categorized in thematic categories (e.g., healthcare, finance, education, transportation, planing), which are stated in the document preamble and are used for archival indexing purposes. the uklex dataset (chalkidis and søgaard, 2022) comprises 36.5k uk laws. the dataset is chronologically split in training (20k, 1975-2002), development (8k, 2002-2008), and test (8.5k, 2008-2018) sets. eurlex european union (eu) legislation is published on the eur-lex website. all eu laws are annotated by eu's publications office with multiple concepts from eurovoc, a thesaurus maintained by the publications office. @xcite eurovoc has been used to index documents in systems of eu institutions. we use the english part of the dataset of chalkidis et al. (2021), which comprises 65k eu laws (documents). the dataset is chronologically split in training (55k, 1958-2010), development (5k, 2010-2012), and test (5k, 2012-2016) sets. it supports four different label granularities. we use the 1st and 2nd level of the eurovoc taxonomy. bioasq the bioasq (task a) dataset consist of biomedical articles from pubmed, foot_4 annotated with concepts from the medical subject headings (mesh) taxonomy (tsatsaronis et al., 2015; nentidis et al., 2021). @xcite mesh is a hierarchicallyorganized vocabulary produced by the national library of medicine. the current version of mesh contains more than 29k concepts referring to various aspects of the biomedical research (e.g., diseases, chemicals and drugs). it is primarily used for indexing, cataloging, and searching of biomedical and health-related information. we subsample 100k documents from the period 2000-2021 in the latest version (v.2022) of the dataset, and split those chronologically for training (80k, 1964-2015), development (10k, 2015-2018), and testing (10k, 2018-2020). we use the 1st and 2nd levels of the mesh taxonomy. mimic-iii the mimic-iii dataset (johnson et al., 2017) contains approximately 50k discharge summaries from us hospitals. each summary is annotated with one or more codes (labels) from the icd-9 hierarchy, which has eight levels in total. @xcite . the international classification of diseases, ninth revision (icd-9) is the official system of assigning codes to diagnoses and procedures associated with hospital utilization in the united states. documents in mimic-iii have been anonymized to protect patient privacy, including chronological information (e.g., entry/discharge dates). hence, ## conclusions in this work, we compared four approaches to multi-label classification, two based on an encoder only and two based on an encoder-decoder. we experimented with 4 datasets from 2 different domains (legal and biomedical), which support two different label granularities. we found that encoderdecoder methods outperform encoder-only methods, in line with findings in other nlp tasks. we further found that the non-autoregressive use of an encoder-decoder model performs better than using it for conditional generation. we found that decoder depth, width and self-attention are all key contributors to the success of this best approach. in future work, we will consider prompt-based approaches as well, specifically instruction-based fine-tuned models (wei et al., 2022), currently limited by the excessive computational cost of encoding the full label set as part of the input string.