Clustering-based Sampling for Few-Shot Cross-Domain Keyphrase Extraction (2024)

Abstract: Keyphrase extraction is the task of identifying a set of keyphrases present in a document that captures its most salient topics. Scientific domain-specific pre-training has led to achieving state-of-the-art keyphrase extraction performance with a majority of benchmarks being within the domain. In this work, we explore how to effectively enable the cross-domain generalization capabilities of such models without requiring the same scale of data. We primarily focus on the few-shot setting in non-scientific domain datasets such as OpenKP from the Web domain & StackEx from the StackExchange forum. We propose to leverage topic information intrinsically available in the data, to build a novel clustering-based sampling approach that facilitates selecting a few samples to label from the target domain facilitating building robust and performant models. This approach leads to large gains in performance of up to 26.35 points in F1 when compared to selecting few-shot samples uniformly at random. We also explore the setting where we have access to labeled data from the model’s pretraining domain corpora and perform gradual training which involves slowly folding in target domain data to the source domain data. Here we demonstrate further improvements in the model performance by up to 12.76 F1 points.

## introduction keyphrases are a set of words that convey the most salient topics of an article or a document, and identification of such keyphrases can be very useful in extracting key information from the long documents through summarization @xcite , semantic and faceted search @xcite and document retrieval @xcite . recently, a lot of work has been done in using language models (lms) for extracting keyphrases * indicates equal contribution using generative models through keyphrase generation @xcite @xcite @xcite @xcite . however, in this work we focus on encoder-only keyphrase extraction @xcite @xcite , specifically framing the task as a sequence tagging in the bio schema format @xcite . kbir @xcite showed that the task and domainspecific pre-training helps in learning rich representations of the keyphrases and leads to better downstream keyphrase extraction performance compared to models that are pre-trained using a task-agnostic objective like masked language modeling. task-specific pre-training of lms for keyphrase extraction requires abundance of supervised data with documents and their corresponding keyphrases. obtaining human annotated data can be a very expensive, error-prone and an inefficient process, hence a majority of the labelled datasets for keyphrase extraction are from the scientific domain @xcite @xcite @xcite , as authors provide keywords with their scientific article to improve discoverability. however, pre-training on domain-specific data often results in poor downstream keyphrase extraction performance on out of domain data. fine-tuning with a sufficiently large dataset typically allows the model to generalize well beyond the pre-training domain. however, for lowresource domains, such data can be difficult to obtain at scale. few-shot learning is a setup extensively explored with very large language models and typically in-context @xcite @xcite , however we focus on the more niche setup of few-shot learning using fine-tuning for sequence tagging with encoderonly models. keyphrase-aware plms are trained to build strong representations for keyphrases in text and we hypothesize that we are able to leverage these embeddings to bootstrap a model by finetuning it only a few-samples from the target domain in order to obtain satisfactory performance. in this work we investigate what sampling strategy, given a limited budget of up to 100 annotations, allows us to select data points from a lowresource target domain for annotation that would be the most effective few-shot samples for fine-tuning. we further explore if we can leverage access to scientific-domain pre-training data oagkx @xcite used by the present state-of-theart keyphrase extraction model, kbir @xcite to bootstrap model performance. the main contributions of this work are summarised below: ## few-shot keyphrase extraction in this work, we investigate if we can effectively sample data from target domains d t having n documents, to be annotated and used for fine-tuning in a few-shot setting. in line with prior work @xcite , we setup keyphrase extraction as a sequence tagging task using the bio schema (b-key, i-key, o) using huggingface @xcite . given a sequence of tokens x @xmath0, ..., x n i }, the model is trained to predict a sequence of labels y @xmath1, ..., y n i }, where each y j @xmath2, i-key, o} label represents whether the j th input token of the i th document in d t is either a beginning of the keyphrase (b-key), inside of the keyphrase (i-key), or outside of the keyphrase (o). we further quantify the impact of obtaining labeled data in the source (pre-training) domain d s having m documents. as our sampling strategies do not rely on labels we simulate low-resource domains in largescale labelled data allowing us to train on a few data points but evaluate on a large number of highquality test points. the use of the labeled data is considered the equivalent of an annotation and we don't conduct any annotation ourselves. ## experimental setup in this work, we investigate the generalization capability of the kbir model and its vanilla counterpart roberta, on the keyphrase extraction task on out-of-domain datasets with respect to the scientific domain-specific oagkx (çano and bojar, 2020) dataset on which kbir was pre-trained. train validation test openkp 134k 6.6k 6.6k stackex 300k 16k 16k manner: we use manner @xcite , a cross-domain few-shot support and query-based architecture in an n-way k-shot sequence tagging framework as a strong baseline. we conducted a thorough literature review of cross-domain few-shot setups to find similar setups for named entity recognition in manner @xcite that we had to make minor adjustments to serve as a strong baseline. @xcite leverages a support and query based architecture to setup an n-way k-shot cross-domain sequence tagging framework that has demonstrated to be very effective outperforming previous sota such as @xcite and l-tapnet @xcite . a major caveat is that they use significantly more data (> 1000 samples) in their few-shot experiments and even more data to conduct source domain training. we recreated these experiments by maintaining the number of data points seen across the training as @xmath3. we do so in both settings where source domain data is and isn't available for training. full-fine tune: we use the aforementioned 22k uniformly sampled data points from a given target dataset in order to fine-tune the model for upper bound performance. random sampling (r): we select k few-shot samples uniformly at random only from the target domain as the few-shot samples (section 3.1.1). clustering-based sampling (c): we select k top-scoring samples only from the target domain as the few-shot samples, based on the scoring function defined in the equation 2 (section 3.1.2). ## cross-domain generalization we evaluate model performance on the source domain data to analyze whether the model is able to generalize across domains and not catastrophically forget the source domain. we do so by evaluating against the kp20k @xcite corpus which consists of scientific articles as seen in we observe that both the model despite being trained in a cross-domain setting remain fairly competitive against a fully-fine tuned model on the source domain data. demonstrating that our proposed framework does not degrade the model's generalization performance. ## data saturation we also explored if scaling up the value of k allows us identify the point at which random (r) outperforms our proposed methods in ## conclusion & future work in this work, we explored the generalization capabilities of the kbir for keyphrase extraction across different domains using few-shot fine-tuning. we proposed a novel clustering-based few-shot sampling approach that uses sub-domain information as topics for extracting high-quality few-shot samples in a cross-domain setting, which leads to a significant gain in performance compared to randomly sampling few-shot samples. we also demonstrated that the gradual training regime in a fewshot setting performs better than its counterparts. we conducted a case study of similarity metrics and prompts that could enhance clustering-based sampling to quantify improvements to the gradual training regime. further exploration is required on heuristics that could further improve data diversity and if these findings hold true for in-context learning settings for keyphrase generation.