C o SAE mb: Contrastive Section-aware Aspect Embeddings for Scientific Articles (2024)

Abstract: Research papers are long documents that contain information about various aspects such as background, prior work, methodology, and results. Existing works on scientific document representation learning only leverage the title and abstract of the paper. We present CoSAEmb, a model that learns representations from the full text of 97402 scientific papers from the S2ORC dataset. We present a novel supervised contrastive training framework for long documents using triplet loss and margin gradation. Our framework can be used to learn representations of long documents with any existing encoder-only transformer model without retraining it from scratch. CoSAEmb shows improved performance on information retrieval from the paper’s full text in comparison to models trained only on paper titles and abstracts. We also evaluate CoSAEmb on SciRepEval and CSFCube benchmarks, showing comparable performance with existing state-of-the-art models.

## introduction scientific papers contain dense detailed information in the paper text. however, existing works on scientific document representation learning are restricted to either the title and abstract of the paper or encode sentence-level information. scibert @xcite and biobert @xcite are the earliest transformer-based representation learning models for scientific texts, both of which use the bert model as base architecture. both of these models operate at the sentence level. models such as specter @xcite , oag-bert @xcite , siamese-scibert @xcite , and scincl @xcite encode the title and the abstract of the paper. @xcite learn multiple aspect-specific vectors of research papers at the sentence level. we posit that the title and the abstract embeddings or sentence vectors fail to capture the intricate details of the paper. in contrast to existing methods, our training strategy (described in section 3) takes into consideration the full-text of the paper in a contrastive learning setup. we propose to learn section embeddings for a paper, which are clustered together in the embedding space. the relative distance of dissimilar papers and similar papers with respect to a candidate is enforced to be larger than a margin. we leverage the positive pairs in contrastive learning to minimize the distance between the section embeddings of the same paper, ensuring that intra-paper similarity is preserved. likewise, the negative instances in the contrastive loss ensure dissimilar papers are farther away. this setup ensures that section embeddings of the same paper are closer to each other in the embedding space than embeddings of other papers, preserving the local information context. similarly, similar papers are closer in the embedding space in comparison to dissimilar papers, preserving interdocument relations. we present a schematic of the embedding space in figure 1 . ## related works scibert @xcite and biobert @xcite train a bert architecture model on the semantic scholar corpus and biomedical corpora, respectively. both models take sentences as input and learn sentence representations. these are not specifically trained to represent research papers. specter @xcite is initialized from scibert and uses a citation-based triplet loss for fine-tuning the model. it encodes the paper's title and abstract. it leverages citations to construct triplets for contrastive loss, i.e., w.r.t. to a query paper (q), a positive paper (k + ) is one of its cited papers. similarly, a negative paper (k -) for q is a paper that is not cited by q (but could be cited by k + ). specter is evaluated on scidocs (cohan et al., 2020) benchmark consisting of seven tasks categorized into document classification, user activity prediction, citation prediction, and recommendation. our proposed approach builds upon specter by introducing a technique to utilize existing 512 context length limited transformers for large documents of any length. scincl @xcite extends the specter model with controlled nearest neighbor sampling over the citation graph that avoids collision between positive and negative samples. oag-bert @xcite encodes paper titles, abstracts, and heterogeneous entities such as authors, research fields, venues, and affiliations using a bert model. they design strategies such as entity-type embeddings to denote heterogeneous entities and entity-aware 2d positional embeddings to demarcate inter and intra-entity token boundaries. the model also uses span-aware entity masking instead of bert's random word masking to make the model learn entities. ferosa @xcite is a faceted recommendation system for scientific articles. it recommends papers from the induced subnetwork of the candidate paper (built from cited papers and papers with highly similar content) for the facets: background (introduction), alternative approaches (related works), method, and comparison (results and conclusion). edges in the network are assigned facets based on the section in which the paper is cited in the candidate paper. ferosa supports faceted recommendation, while our framework is focused on learning faceted representations for scientific articles that can be further used for downstream tasks such as faceted recommendation. aspect embeddings, i.e., embeddings for different facets of the paper, are learned in @xcite and @xcite . embeddings for facets such as task, method, and dataset, are learned in @xcite ; however, these are again learned from the title and abstract of the paper. @xcite use the pwc dataset @xcite , which contains task, method, and dataset labels for papers. the best-performing model siamese-scibert is finetuned on the pwc dataset based on aspect labels. aspire @xcite , on the other hand, leverages co-citation sentences as a source of document similarity. however, unlike our proposed method, both these models use only the title and the abstract to learn aspect vectors. we also present a comparison of cosaemb with other models model input contrastive citation information entity information scibert sentence sbert sentence ✓ specter title & abstract ✓ ✓ oag-bert title & abstract ✓ siamese-bert title & abstract ✓ scincl title & abstract ✓ ✓ aspire sentence ✓ ✓ cosaemb (ours) full-text paragraphs ✓ ✓ in as outlined in @xcite , works on efficient transformers address the quadratic time complexity by using one of the techniques: (i) sparsify the attention matrix using fixed patterns, (ii) learnable patterns that learn the access pattern in a data-driven fashion (rephrase, arrange tokens based on similarity), (iii) introducing memory modules, (iv) low-rank approximation of the self-attention matrix, (v) kernels, (vi) recurrence, or a combination of these. multiple variants of efficient transformers such as linformer @xcite , longformer @xcite , bigbird (zaheer et al., 2020), reformer @xcite , performer (choromanski et al., 2021) have been proposed. while multiple efficient long document transformers are proposed (e.g., @xcite discuss thirty-two efficient transformers), only longformer has been adapted for scientific papers and evaluation on the qasper benchmark @xcite shows significant scope for adoption of these models for the scientific domain. the performance and effectiveness of other efficient transformer models on scientific documents are largely unexplored. unlike works on efficient transformers, we focus on reusing the existing models such as scibert and specter with contrastive fine-tuning to learn full-text document representations. ## cosaemb in this section, we present cosaemb (pronounced ko-saam-bee), contrastive section-aware aspect embeddings. cosaemb is a supervised contrastive training framework for long documents using margin gradation. scientific articles are long documents containing information on multiple aspects. we learn aspect representations for long documents instead of relying on a single document vector to capture all the information from various aspects. we briefly discuss the intuition behind our document representation framework next. ## experiment details training dataset cosaemb requires full-text of research papers and citation information to construct the section, abstract, and inter-document triplets for training the model. for a fair comparison of our methodology with specter, we use the training dataset of specter, which consists of 684100 triplets in the train set. based on the citation information, these triplets are constructed from around 146000 query papers sampled from the semantic scholar corpus @xcite . however, these triplets are constructed only from the title and abstracts of the papers (format similar to our inter-document triplets). we use the semantic scholar corpus to construct section triplets by extracting the full-text of the papers present in specter's training set. we extract full-text of 97402 papers from the semantic scholar corpus. we use keyword matching to categorize section headings into five classes: introduction, related works, method, results, and conclusion. we use the dataset of 25,482 section headings organized manually into previously mentioned five categories by @xcite to collect seed keywords. to facilitate the categorization of new section headings in full texts, we developed a straightforward algorithm that assigns one of the following labels to each heading: introduction, related works, results, conclusion, or methods based on the dataset curated by @xcite . section headings that do not match any of category headings in the dataset are automatically assigned to the methods category. we present the distribution of different section headings in the dataset in training details we implement our model in pytorch and train our model on three nvidia v100 gpus. the model is trained for two epochs with a slanted triangular learning rate of 2e-5 with warmup steps equal to 0.1 fractions of the total steps. we use a batch size of 32, which fits on our three gpus. we run ablations for margin values m t i , m tg , and m ts : (0.5, 1, 1.5), (1, 2, 3), and (2, 4, 6). the best results are obtained with m t @xmath0, m @xmath1, and m @xmath2. ## evaluation and results the description of datasets used for evaluating the capability of the cosaemb model are as follows: csfcube @xcite csfcube is an annotated corpus of 50 query abstract-facet pairs from the acl anthology. the candidate set for each query is constructed from the s2orc corpus using six methods, including tf-idf and averaged word2vec. the three facets in the dataset are background, method, and result. as this dataset supports query by aspect, it is a relevant database to evaluate the model in an aspect-based retrieval scenario. scirepeval @xcite ) scirepe-val is a benchmark for evaluating scientific document representations, which consists of twenty-five tasks across four formats: classification (e.g., field of study, mesh descriptors classification, etc.), regression (e.g., predict tweet mentions, peer-review rating, and maximum h-index of authors), proximity (e.g., same author detection, highly influential citations, etc.), and ad-hoc search (e.g., rank a candidate set for a query on trec-covid, cord-19 datasets). full the order of distances among different paper section embeddings, on papers unseen during training. we set up a simple retrieval setup, where section embedding is used as a query and embeddings from a candidate set are retrieved based on euclidean distance. the candidate set is composed of other section texts from the same paper as well as other papers. we evaluate if cosaemb embeddings retrieve other section texts from the same paper. we also evaluate a model trained with m t @xmath7, or if the gain is observed because of increased training data. while the cosaemb models perform comparable to existing state-of-the-art models on cs-fcube and scirepeval benchmark, it shows an improved performance in retrieving the full-text paragraphs. on the csfcube dataset, cosaemb performs slightly better on r-precision metric, for most aspects. unlike aspire, cosaemb is not trained on any aspect-specific data and is just trained on full-text sections data only. for the csfcube dataset, similar to other models we use the title and abstract text for query. however, for each of the specific facets, we separate the title and abstract with our special tokens in addition to the [sep] token (for facet background, we use [intr] and [rw]; for method facet we use token @xcite , and for result facet we use @xcite ). in comparison to specter, which is the fairest baseline to cosaemb, improved performance can be observed for the 'method' aspect. this could be attributed to the full-text information available to cosaemb during training. on the scirepeval dataset, we observe slightly worse performance of cosaemb. it could denote that it is not always required to utilize full-text information for the tasks present in scirepeval benchmark. @xcite pretrain a new model similar to specter to increase the domain coverage with data from 23 fields, which leads to a 15-point increase for in task performance, which also indicates that training the model on diverse domain data would be helpful. ## conclusion we present cosaemb, a technique to learn section-aware embeddings with any existing encoder-only transformer model without retraining it from scratch. it can be adapted to several domains that deal with long documents such as legal and research papers. we use contrastive loss to represent multiple section embeddings of papers in a compact cluster, and maximize the distance to dissimilar and minimize the distance to similar papers. margin gradation introduces a hierarchy of distances. the representations can be used for various downstream applications, such as the generation of related works for manuscripts, predicting appropriate submission venues for manuscripts, and predicting missing citations or comparisons.