Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study (2023)

Abstract: Text classification is an area of research which has been studied over the years in Natural Language Processing (NLP). Adapting NLP to multiple domains has introduced many new challenges for text classification and one of them is long document classification. While state-of-the-art transformer models provide excellent results in text classification, most of them have limitations in the maximum sequence length of the input sequence. The majority of the transformer models are limited to 512 tokens, and therefore, they struggle with long document classification problems. In this research, we explore on employing Model Fusing for long document classification while comparing the results with well-known BERT and Longformer architectures.

## introduction text classification is one of the critical tasks in natural language processing, which refers to finding the suitable label/ labels to a particular input text @xcite . it has a wide range of applications in different domains such as sentiment analysis @xcite , fake news detection @xcite @xcite and offensive language identification @xcite . these tasks are generally referred to as sentence classification tasks since the input text is typically in the form of sentences. in recent years, transformer models such as bert have provided state-of-theart results in these text classification tasks @xcite . while most of the text classification tasks are sentence classification, several domains require classifying lengthy texts into labels typically referred to as document classification. specifically, domains such as legal and medical often contain long documents that need document classification methods @xcite . however, adapting the transformer models that produced state-of-the-art results in sentence classification to document classification is challenging @xcite . the most common transformer models, such as bert @xcite , have a limitation of 512 tokens in their input layer, which means the tokens in a lengthy document exceeding this limit will be truncated in the tokenisation step. the limitations outlined above have attracted significant attention from the research community, leading to the exploration of new document classification architectures. one widely adopted approach is to leverage transformer models that can process longer sequences. notably, the longformer (beltagy et al., 2020) and bigbird @xcite transformer models have demonstrated exceptional performance in document classification tasks, with the capacity to accommodate up to 4,096 tokens. however, training transformer models that can process longer sequences is a resource-intensive task, and it may not be feasible for less-resourced domains and languages @xcite . in an effort to mitigate this challenge, researchers have attempted to adapt existing pre-trained transformer models to accommodate longer sequences. notably, two such approaches are hierarchical bert @xcite and cogltx @xcite , both of which propose innovative strategies for adapting bert to long document classification. taking this research further, we propose, a method to adapt bert-like transformer models to long document classification using model fusion. while the methods such as hierarchical bert @xcite and cogltx @xcite ) mainly focus on tackling longterm dependencies using different attention mechanisms to reduce their computational complexity, we explore a novel idea with model fusing to the long document classification task. model fusion refers to the idea of combining several fine-tuned models @xcite . the motivation behind using model fusion is that multiple models can identify different patterns using different parts of their network, and it is possible to merge multiple models into one model, which will be capable of having all information compressed into a single model. to implement this idea, we divide long documents into multiple parts and use these parts to train part-wise models. finally, we fuse all part-wise models to create a single model capable of handling lengthy sequences. our evaluation of this approach on four popular document classification datasets shows that while our hypothesis is strong, model fusion does not improve stateof-the-art document classification. nonetheless, we report our results with the aim of helping researchers avoid repeating unsuccessful experiments in the future. furthermore, this paper identifies potential flaws in experimental design, enabling researchers to refine their methods and improve future studies that employ model fusion in long document classification foot_0 . our main contributions of the paper are, 1. we present the first study in using model fusion in long document classification. the rest of the paper is organised as follows. section 2 highlights the recent work on long document classification and model fusing. section 3 describes the datasets we used. section 4 explains data preparation for experiments, sub-model training, model fusing and prediction on test data. section 5 presents the results and discusses possible problems in the results and ideas for improvements. section 6 summarises our main experimental findings and conclusions. the datasets, specific long-text processing methods fail to outperform these simple baselines. limsopatham (2021) experimented with the effective usage of bert for long document classification by parsing the front part of the document and the rear part of the document separately and experimenting with the results. despite numerous efforts to address challenges in long document classification, the results still fall short compared to sentence classification, demanding further dedication from the research community. 2020 ) employed fusing multiple models for visual question answering. ## data we evaluated our approach with four popular document classification datasets; echr @xcite , echr anon @xcite 20newsgroups @xcite and case-2022 @xcite . we describe each of them below. the distribution of the number of words in each dataset is also shown in echr @xcite ) european court of human rights (echr) hears allegations that a state has breached human rights provisions of the european convention of human rights. the dataset contains approx. 11.5k cases from echr's public database. we use the dataset for documentlevel binary violation tasks; given the facts of a case, the task is to classify whether there has been any human rights violation or not. echr anon @xcite this dataset contains an anonymised version of the echr with demographic data being anonymised. to achieve this, all named entities in the text have been replaced with corresponding tags. 20newsgroups @xcite the dataset is composed of 18,828 news articles, which are classified into 20 different categories. the goal of this task is to perform multi-class classification to accurately identify the category of each article. to evaluate our model's performance, we reserve 20% of the data for the test set. @xcite this dataset is from the shared task on socio-political and crisis events detection case -subtask 1. the task is a document classification to detect whether a news article contains information about a socio-political event or not. the dataset features 9,384 news articles in the training set, and we have utilised 20% of it as the test set since the gold labels in the test set are not released. ## methodology we divide our method into five stages, which we describe below. ## results and discussion ## conclusion this paper presents an empirical study on the effectiveness of model fusing in long document classification, with the aim of comparing its performance to that of state-of-the-art models such as longformer @xcite . our results indicate that longformer @xcite outperforms our experimental setup across all datasets. while we identify several drawbacks of the method, we believe that there is still potential for further exploration in this area. although our average fusing approach did not yield improved performance in long document classification, there is a need for more research on different fusing methods and their efficacy in various tasks.