Length-Aware Multi-Kernel Transformer for Long Document Classification (2024)

Abstract: Lengthy documents pose a unique challenge to neural language models due to substantial memory consumption. While existing state-of-the-art (SOTA) models segment long texts into equal-length snippets (e.g., 128 tokens per snippet) or deploy sparse attention networks, these methods have new challenges of context fragmentation and generalizability due to sentence boundaries and varying text lengths. For example, our empirical analysis has shown that SOTA models consistently overfit one set of lengthy documents (e.g., 2000 tokens) while performing worse on texts with other lengths (e.g., 1000 or 4000). In this study, we propose a Length-Aware Multi-Kernel Transformer (LAMKIT) to address the new challenges for the long document classification. LAMKIT encodes lengthy documents by diverse transformer-based kernels for bridging context boundaries and vectorizes text length by the kernels to promote model robustness over varying document lengths. Experiments on five standard benchmarks from health and law domains show LAMKIT outperforms SOTA models up to an absolute 10.9% improvement. We conduct extensive ablation analyses to examine model robustness and effectiveness over varying document lengths.

## introduction lengthy documents widely exist in many fields, while the input limit of transformer models prevents developing powerful pre-trained language models on those long documents, such as bert @xcite and roberta @xcite . for example, a recent study shows that clinical documents have grown over 60% longer in a decade @xcite . truncation is a common strategy to handle long documents and fit the input limit of bert-based classifiers, however, the method may lose many critical contexts beyond the first 512 tokens and hurdle model effectiveness. auto-regressive large language models (llms), such as chatgpt @xcite show their great ability at processing long documents, however the training object of these llms is to prediction the next token, which is inconsistent with the text classification task. in other words, supervised fine-tuning on these domain specific data may not improve the performance of llms on these classification task. therefore, researchers focus on prompting methods @xcite @xcite @xcite or decoding strategies @xcite rather than fine-tuning @xcite to help the llms categorize text better. this reality makes llms limited in this scenario. compare with these methods, developing discriminative transformer models that can model long documents is a more direct and effective solution to handle the long document classification task. among existing transformer-based models, long document modeling has two major directions, hierarchical transformer and sparse attention @xcite . the hierarchical approach @xcite @xcite @xcite splits document into small text chunks (e.g., 128 tokens) so that long document models can take shorter input per step. as the self-attention in transformer-style models causes quadratic complexity o(n 2 ), the sparse attention aims to lower the complexity to linear and reduce context fragmentation caused by the segments @xcite @xcite . for example, sparse attention in longformer @xcite lifts up the input limit from 512 tokens to 4096 tokens. popular evaluation benchmarks also switch from social media data (e.g., imdb and amazon reviews @xcite dataset length-quantile l-mean size |label| splits 25% 50% 75% train valid test diabetes 408 608 945 720 1,265 10 885 190 190 mimic 1,432 2,022 2,741 2,200 11,368 50 8,066 1,753 1,729 ecthr a/b to understand the length effects and encounter the long document challenges, we conduct extensive analysis and propose length-aware multi-kernel transformer (lamkit) for robust long document classification. lamkit diversifies learning processes by a multi-kernel encoding (mk) so that the model can capture contexts from different perspectives. the mk contains multiple neural encoders with diverse kernel sizes and can relieve context fragmentation caused by a unique segment encoder on short text chunks. lamkit promotes model robustness over varying-length documents by a length-aware vectorization (lav) module. the lav encodes length information in a hierarchical way, position embedding on segment and length vectors on document level. we compare lamkit with 8 domain-specific models on five datasets @xcite , sco-tus @xcite , ecthr-a @xcite and ecthr-b @xcite , diabetes @xcite ) from health and legal domains evaluated by f1 and auc metrics. additionally, we also conduct a case study on the performance of chatgpt in these tasks. classification results demonstrate that our lamkit approach's outperforms competitive baselines by an absolute improvement of up to 10.9%. we conduct further experiments on the length-varying effects and ablation analysis to examine the effectiveness of our individual modules. ## data we have retrieved five publicly available dataset, diabetes @xcite , mimic-iii @xcite , ecthr-a @xcite , ecthr-b @xcite and sco-tus @xcite , which are popular benchmarks for the long document classification. we obtained diabetes @xcite from the 2018 national nlp clinical challenges (n2c2) shared task with a collection of longitudinal patient records and 13 selection criteria annotations. we exclude 3 annotations due to less than 0.5 inter-rater agreements and discard documents with fewer than 40 tokens. mimic-iii (medical information mart for intensive care) @xcite is a relational database that contains patients admitted to the intensive care unit (icu) at the beth israel deaconess medical center from 2001 to 2012. we follow previous work @xcite to select discharge summaries and use the top 50 frequent labels of international classification of disease codes (9th edition, icd-9), which are types of procedures and diagnoses during patient stay in the icu. ecthr-a collects facts and articles from law case descriptions from the european court of human rights' public database @xcite . each case is mapped to the articles it was found to have violated in the echr, while in ecthr-b @xcite , cases are mapped to a set of allegedly violated articles. scotus is a data collection of us supreme court (the highest us federal court) opinions and the us supreme court database (scdb) @xcite with cases from 1946 to 2020. scotus has 14 issue areas, such as criminal procedure, civil rights, and economic activity. we summarize data statistics and splits in while studies @xcite have achieved improving performance overall to encode more contexts beyond the 512 token limit, there is very few work examining the effects of varying document lengths over model robustness. to answer the question, we conduct an exploratory analysis of existing state-of-the-art (sota) models and evaluate their performance. %(57 /rqjiruphu %ljelug +%(57 0rgho 0lfur) 4xduwhu 4 4 4 4 figure 1: average performance on quarter splits for four state-of-the-art baselines. the length boundaries of quarters are shown in our exploratory analysis follows existing studies @xcite @xcite to split data, includes three state-of-the-art transformer classifiers (bigbird, longformer, and hierarchical bert (h-bert)) for long document and a bert classifier, and evaluates models performance by f1-micro (f1-µ) score. we refer to the details of experimental settings and sota baselines under the experiments section. for each quarter, we maintain similar data sizes and run the classifier multiple times to take average performance scores. finally, we visualize the relation between model performance and document lengths in figure 1 . figure 1 shows that model performance varies across document lengths, posing a unique challenge to build robust models on varying lengthy data. for example, while the sota classifiers achieve better scores on mid-lengthy texts, the performance drops significantly in either short (e.g., 400 tokens) or super long (e.g., 10k tokens) documents. the consistent observations can suggest that: 1) varying length can be a critical factor to make models perform better; 2) length-based splits are important to understand the capacity of classifiers on long documents. the findings inspire us to propose the length-aware multi-kernel transformer (lamkit) to encounter the length factor. ## length-aware multi-kernel transformer this section presents our length-aware multi-kernel transformer (lamkit) for robust long document classification in figure 2 . lamkit consists of three major modules, 1) multi-kernel encoding, 2) length-aware vectorization, and 3) hierarchical integration, aiming to solve context fragmentation and augment model robustness on lengthy documents. we deploy different encoding kernels to diversify text segments with various contexts. incorporating length as vectors can adapt classifiers across varying-length documents. finally, we elaborate on how to learn robust document representations via a hierarchical integration. ## experiments we follow the previous studies @xcite @xcite on lengthy document to preprocess data and split data into training, validation, and test, as in our evaluation presents performance comparisons and ablation analysis to understand the length effects and the models better. more details of the hyperparameter settings for the baselines and lamkit are in the appendix a, which allows for experiment replications. ## result analysis this section reports the performance of sota baselines and lamkit in terms of f1 and auc metrics, both micro (µ) and macro (m) modes. besides the overall performance, we examine varyinglength effects and conduct ablation analysis on our individual modules (e.g., mk and lav). the results show that lamkit not only surpasses the baselines by a large margin on long documents from both health and legal domains but also shows more stable performance on documents of varying lengths. ## case study on chatgpt large language models (llms) have achieved impressive performance on many generative tasks, such as long text summarization or long text qa. however, long text classification is a natural language understanding task, which makes fine-tuning the large model on such a task not a guaranteed improvement in classification accuracy. thus the dominant paradigms for text classification in llms are zero-shot learning and few-shot learning @xcite . to examine the ability of llms on the long document classification task, we utilize representative gpt-3.5-turbo via chatcompletion api foot_0 in a zero-shot prompting strategy with multiple templated instructions summarized by @xcite @xcite , and report the best performing template results. due to privacy concerns and data usage agreement, we do not test chatgpt (openai, 2022) on mimic and diabetes. the results in ## conclusion in this study, we posit that for long document classification tasks, the length of the text might be a pivotal determinant for model performance. our exploratory experiments demonstrate that the current state-of-the-art models display inconsistent results across samples of differing lengths, suggesting their lack of robustness and affirming our hypothesis. to address this issue and the inherent problem of context fragmentation in long-text models, we propose length-aware multi-kernel transformer. through extensive experiments, lamkit consistently outperforms all baseline models across five standard long document classification benchmarks. moreover, we follow our exploratory experiments to examine model robustness over varying document lengths. we also conduct ablation studies on two modules. the results show that lamkit exhibits better robustness and stability across different lengths. additionally, the case study on chatgpt (ope-nai, 2022) reveals that llms still underperform discriminative models on long document classification tasks, suggesting that the paradigm of solving classification problems through generation still needs to be enhanced.