S ym T ax: Symbiotic Relationship and Taxonomy Fusion for Effective Citation Recommendation (2024)

Abstract: Citing pertinent literature is pivotal to writing and reviewing a scientific document. Existing techniques mainly focus on the local context or the global context for recommending citations but fail to consider the actual human citation behaviour. We propose SymTax, a three-stage recommendation architecture that considers both the local and the global context, and additionally the taxonomical representations of query-candidate tuples and the Symbiosis prevailing amongst them. SymTax learns to embed the infused taxonomies in the hyperbolic space and uses hyperbolic separation as a latent feature to compute query-candidate similarity. We build a novel and large dataset ArSyTa containing 8.27 million citation contexts and describe the creation process in detail. We conduct extensive experiments and ablation studies to demonstrate the effectiveness and design choice of each module in our framework. Also, combinatorial analysis from our experiments shed light on the choice of language models (LMs) and fusion embedding, and the inclusion of section heading as a signal. Our proposed module that captures the symbiotic relationship solely leads to performance gains of 26.66% and 39.25% in Recall@5 w.r.t. SOTA on ACL-200 and RefSeer datasets, respectively. The complete framework yields a gain of 22.56% in Recall@5 wrt SOTA on our proposed dataset. The code and dataset are available at https://github.com/goyalkaraniit/SymTax.

## introduction citing has always been the backbone of scientific research. it enables trust and supports the claims made in the scientific document. the ever-growing increase in the amount of scientific literature makes it imperative to ease out the author's task of finding a list of suitable papers to follow and cite @xcite @xcite . citation recommendation is such a process that helps researchers to be aware of the relevant research in respective domains. there are two different approaches to recommend citations: local @xcite @xcite , and global @xcite @xcite . local citation recommendation is the task of finding and recommending the most relevant prior work, mainly corresponding to a specific text passage (also known as citation context), making it context-aware. on the other hand, global citation recommendation recommends a list of suitable prior art for the entire document, mainly given the title and abstract or the whole document. in this paper, we solve the task of local citation recommendation, which is more fine-grained and provides a solution to the actual challenge the author faces. for example, consider the below citation excerpt: 1 this can have extreme consequences in real-life scenarios such as autonomous cars citx. examining the above context in isolation makes it challenging to predict the specific article cited at citx. however, leveraging global information such as title, abstract, and taxonomy narrows down the search space while at the same time utilizing symbiotic relationship provides the model with an enriched pool of the most suitable candidates. unlike acl-200 and refseer datasets with curated contexts of fixed size, we curate richer contexts by incorporating complete information of adjoining sentences with respect to the citation sentence. to summarise, we make the following contributions: ## related work local citation recommendation has drawn comparatively less interest than its global counterpart until recently. @xcite introduced the task of local citation recommendation by using tf-idf based vector similarity between context and cited articles. @xcite extracted hand-crafted features from the citation excerpt and the remaining document text, and developed a system to recommend citations while the document is being drafted. the neural probabilistic model of @xcite determines the citation probability for a given context by jointly embedding context and all articles in a shared embedding space. @xcite proposed neural citation network based on encoder-decoder architecture. the encoder obtains a robust representation of citation context and further augments it via author networks and attention mechanism, which the decoder uses to generate the title of the cited paper. @xcite utilised stacked denoising autoencoders for representing cited articles, bidirectional lstms for citation context representation and attention principle over citation context to enhance the learning ability of their framework. @xcite proposed a bert-gcn model which uses @xcite to obtain embeddings for context sentences, and graph convolutional network @xcite to derive embeddings from citation graph nodes. the two embeddings are then concatenated and passed through a feedforward neural network to obtain relevance between them. however, due to the high cost of computing gcn, as mentioned in @xcite , bert-gcn model was evaluated on tiny datasets containing merely a few thousand citation contexts. it highlights the limitation of scaling such gnn models for recommending citations on large datasets. @xcite suggested the use of global information of articles along with citation context to recommend citations. it computes semantic matching score between citation context and cited article text, and bibliographic score from the article's popularity in the community to generate a final recommendation score. @xcite perform neighbourhood contrastive learning over the full citation graph to yield citation embeddings and then uses k-nearest neighbourhood based indexing to retrieve the top recommendations. the most recent work in local citation recommendation by @xcite proposed a two-stage recommendation architecture comprising a fast prefetching module and a slow reranking module. we build upon work of @xcite by borrowing their prefetching module and designing a novel reranking module and another novel module named enricher that fits between prefetcher and reranker. we name our model as symtax (symbiotic relationship and taxonomy fusion). ## proposed dataset motivation. citation recommendation algorithms depend on the availability of the labelled data for training. however, curating such a dataset is challenging as full pdf papers must be parsed to extract citation excerpts and map the respective cited articles. further, the constraint that cited articles should be present in the corpus eliminates we organise the constructed dataset into a directed graph. nodes within the graph encapsulate a rich array of attributes, encompassing abstracts, titles, authors, submitters, publication dates, topics, categories within cs, and comments associated with each paper. edges within graph symbolise citations, carrying citation contexts and section headings in which they appear. this provides a format that offers better visualisation and utilisation of data. unlike previously available datasets, which use a 200-character length window to extract citation context, we consider one sentence before and after the citation sentence as a complete citation context. we create a robust mapping function for efficient data retrieval. since every citation does not contain a digital object identifier, mapping citations to corresponding papers is challenging. the use of several citation formats and the grammatical errors adds a further challenge to the task. to expedite title-based searches that associate titles with unique paper ids, we devise an approximate mapping function based on lcs (longest common substring), but the sheer size of the number of papers makes it infeasible to run directly, as each query requires around 10 seconds. finally, to identify potential matches, we employ an approximate hash function called minhash lsh (locality sensitivity hashing), which provides the top 100 candidates with a high probability for a citation existing in our raw database to be present in the candidate list. we then utilise lcs matching with a 0.9 similarity score threshold to give a final candidate, thus reducing the time to a few microseconds. finally, our dataset consists of 8.27 million citation contexts whereas the largest existing dataset, refseer, consists of only 3.7 million contexts. the dataset is essentially comprised of contexts and the corresponding metadata only and not the research papers, as is the case with other datasets. even after considering a relatively lesser number of papers as a raw source, we curated significantly more citation contexts (i.e., final data), thus showing the effectiveness of our data extraction technique. this is further supported empirically by the fact that our dataset has significantly higher values of average local clustering coefficient and average degree with respect to the other datasets (as shown in technical merits. arsyta offers the following merits over the existing datasets: (i) as shown in ## symtax model we discuss the detailed architecture of our proposed model -symtax, as shown in figure 2 . it comprises a fast prefetching module, an enriching module and a slow and precise reranking module. we borrow an existing prefetching module from @xcite whereas an enriching module and a reranking module are our novel contributions in the overall recommendation technique. the subsequent subsections elaborate on the architectures of these three modules. ## experiments and results this section illustrates the various baselines, evaluation metrics and datasets used to benchmark our proposed method followed by the performance comparison. baselines. we consider evaluating various available systems for comparison. bm25 @xcite : it is a prominent ranking algorithm, and we consider its several available implementations and choose elastic search implementation 8 as it gives the best performance with the highest speed. scincl @xcite : we use its official implementation available on github 9 . scibert @xcite as lm and vectorbased taxonomy fusion. specter and scibert are two state-of-the-art lms trained on scientific text. sciv performs as the best model on acl-200, fulltextpeerread, refseer and arsyta on all metrics. specg performs best on arxiv(hatten) on all metrics and results in a marginally less r@20 score than sciv. we observe the highest scores on fulltextpeerread followed by acl-200. it is due to the fact that these datasets lack diversity to a large extent. e.g. fulltextpeerread is extracted from papers belonging to artificial intelligence field, and acl-200 contains papers published at acl venues. in contrast, we observe the lowest scores on arsyta followed by arxiv(hatten). ## analysis we conduct extensive analysis to assess further the modularity of symtax, the importance of different modules, combinatorial choice of lm and taxonomy fusion, and the usage of hyperbolic space over euclidean space. furthermore, we analysed the effect of using section heading as an additional signal (shown in appendix a). ## conclusion in this paper, we present a model for local citation recommendation that leverages the notion of symbiosis from biology, and we draw its analogy with human citation behaviour. we propose the notion of taxonomy fusion for learning rich concept representations and project them into hyperbolic space to derive a latent feature. we introduce a novel dataset that is comparatively large, dense, recent and more challenging than other existing datasets. through several experiments and analyses, we prove our model as highly modular, which can run on datasets with comparatively few signals and accommodate additional signals as well. our model consistently outperforms sota by huge margins for all evaluation metrics across all datasets.