Transformer-based Models for Long-Form Document Matching: Challenges and Empirical Analysis (2023)

Abstract: Recent advances in the area of long document matching have primarily focused on using transformer-based models for long document encoding and matching. There are two primary challenges associated with these models. Firstly, the performance gain provided by transformer-based models comes at a steep cost – both in terms of the required training time and the resource (memory and energy) consumption. The second major limitation is their inability to handle more than a pre-defined input token length at a time. In this work, we empirically demonstrate the effectiveness of simple neural models (such as feed-forward networks, and CNNs) and simple embeddings (like GloVe, and Paragraph Vector) over transformer-based models on the task of document matching. We show that simple models outperform the more complex BERT-based models while taking significantly less training time, energy, and memory. The simple models are also more robust to variations in document length and text perturbations.

## introduction matching long documents (e.g.: research papers, wikipedia articles, patents, etc.) is an important task that can help understand the (dis)similarity between documents for downstream tasks like long document search. the first step towards better document matching is obtaining meaningful long document representations. recent advances in this area have primarily focused on using transformer-based models for long document encoding and matching @xcite @xcite . we use the term transformers to mean pre-trained transformers. despite promising results, there are two primary challenges associated with such models. first, the performance gain provided by the huge transformerbased language models (lms), like bert @xcite , gpt-2 @xcite , and longformer @xcite come at a steep cost -both in terms of the required training time, and the resource (memory and energy) consumption. for example, the smaller bert base model has 110 million parameters, whereas the bigger bert large model has a total of 340 million parameters and fine-tuning a single bert base model on gpu can take hours. the second major limitation of transformer-based models is their inability to handle more than a pre-defined input token length at a time (512 tokens for bert, and 4096 tokens for longformer). this is a big drawback as they cannot handle long documents like research papers, patents, long articles, etc., without using aggregation techniques @xcite . in this work, we empirically demonstrate that embeddings obtained from glove @xcite , and paragraph vectors @xcite along with simple neural models, such as feed-forward networks, and cnns, outperform several transformer-based models for the document matching task. we define these models as simple as they take significantly less time to train and consume less memory and energy overall when compared to complex transformer-based models. our long document matching setting is fundamentally different from long-form question answering and sentence similarity tasks. for the latter tasks the query is 'short', unlike the long document matching task where both the query and the target text are 'long'. we experiment with three different kinds of semi-structured long document datasets in english: (i) acl anthology research papers, (ii) english wikipedia articles, and (iii) patents from us patent and trademark office (uspto). our primary contributions are summarized as follows: early work on long document matching focused on clustering techniques @xcite @xcite . recently, @xcite proposed a deep learning based architecture for ad-hoc retrieval when comparing documents. some works have also used convolutional networks @xcite @xcite , with weighting mechanism @xcite to generate a final query-document score. mitra et al. (2017) propose a combination model that uses weighted sum representation-based and interaction-based results. yang et al. (2016b) propose han, a hierarchical attention network for document matching. jiang et al. (2019) propose smash, a multi-depth attention based hierarchical recurrent neural network for long-document matching. however, yang et al. (2020b) pre-train smith, a transformer based hierarchical model for text matching that outperforms smash across multiple datasets. jha et al. (2022) use supervised contrative learning for interpretable long document matching. we compare several of these models on the required training time and resources. a growing body of literature has used transformer based model for long document encoding @xcite @xcite @xcite . longformer @xcite adapts transformers to use an attention mechanism that scales linearly with sequence length. big bird @xcite uses a sparse attention mechanism that reduces bert's quadratic dependency on the sequence length to linear. cogltx @xcite ) uses text blocks for rehearsal and decay over key sentences to overcome the insufficient longrange attentions in bert. transformer-xl @xcite and compressive tranformers (rae 2019 ) compress the transformers to use attentive sequence over long text. although promising, we demonstrate that transformer-based models are not considerably better than simple neural models on the task of long document matching. ## empirical evaluation here we provide details of the simple and the transformer-based models and present an empirical comparison between them based on their overall performance, training time, resource consumption, and robustness on the document matching task. task formulation we define the task of document matching as follows. given a source document s, and a set of target documents d t , the goal is to estimate the semantic match ŷ = sim(s, t), where @xmath0, t). similar target documents will have a higher similarity score. the document matching problem can be seen as a binary classification task, where the model predicts 1 for similar documents, and 0 for dissimilar documents. we use the term "matching" in the broad sense of document relevance (see appendix a.2). the models take as input a pair of documents (source and target), and compute the cosine similarity between the encoded document representations. if the cosine similarity is greater than a similarity threshold θ, they are considered similar; otherwise they are considered dissimilar. models we pick a representative set of models from different categories and compare them by building their siamese versions (shown in figure 1 ). the siamese network has three primary components: (i) input (source and target document), (ii) document encoder, and (iii) loss function. the source and target document encoder networks share weights. we experiment with three simple neural models: (i) dssm: a simple feedforward network @xcite , (ii) arc-i: a cnn-based model @xcite , and (iii) han: an rnn-based hierarchical attention network @xcite bert @xcite , (ii) long: longformer @xcite , and (iii) smith: siamese multi-depth transformer based hierarchical encoder @xcite . we report the mean precision, recall, f1, and accuracy over 5 folds for the best performing hyper-parameters. the code can be found here: https://github.com/ akshitajha/simplemodelsforlongdocumentmatching . ## conclusion we empirically demonstrate the trade-off of using transformer-based models for semi-structured long english documents like research papers, wikipedia articles, and patents. transformer-based models have an overall strong performance and smaller variability across datasets. however, we observe that for the task of long document matching, using contextual embeddings do not provide any added advantage. a simple feed-forward network or a cnn-based model using glove or doc2vec embedding outperforms several state-of-the-art pre-trained transformer-based models at a fraction of their overall training time and resources (memory and energy). these simple neural models are also more robust to changes in document length and text-perturbation.