Few- TK : A Dataset for Few-shot Scientific Typed Keyphrase Recognition (2024)

Abstract: Scientific texts are distinctive from ordinary texts in quite a few aspects like their vocabulary and discourse structure. Consequently, Information Extraction (IE) tasks for scientific texts come with their own set of challenges. The classical definition of Named Entities restricts the inclusion of all scientific terms under its hood, which is why previous works have used the terms Named Entities and Keyphrases interchangeably. We suggest the rechristening of Named Entities for the scientific domain as Typed Keyphrases (TK), broadening their scope. We advocate for exploring this task in the few-shot domain due to the scarcity of labeled scientific IE data. Currently, no dataset exists for few-shot scientific Typed Keyphrase Recognition. To address this gap, we develop an annotation schema and present Few-TK, a dataset in the AI/ML field that includes scientific Typed Keyphrase annotations on abstracts of 500 research papers. To the best of our knowledge, this is the introductory few-shot Typed Keyphrase recognition dataset and only the second dataset structured specifically for few-shot NER, after Few-NERD. We report the results of several few-shot sequence-labelling models applied to our dataset. The data and code are available at https://github.com/AvishekLahiri/Few_TK.git

## introduction the recent past has witnessed an explosion in the amount of scientific literature available to us, especially with the advent of the web and scholarly search engines. the expansiveness and variations in even a single scientific domain today requires a wide-ranging set of information extraction tools and datasets. named entity recognition (ner) is the information extraction task of identifying references to rigid designators @xcite and is the basic building block for a great number retrieval augment generation (rag) is a recent advancement in open-domain question answering (odqa). rag has only been trained and explored with a wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. in this paper, we evaluate the impact of joint training of the retriever and generator components of rag for the task of domain adaptation in odqa. we propose rag-end2end, an extension to rag... ner in scientific domain is frequently referred to as keyphrase extraction, which is due to the constrictive nature of the classical definition of named entities, which states that proper nouns are the only words that can be allowed as named entities @xcite . keyphrase extraction often singularly refers to the span detection of scientific terms, and not assigning types to them. therefore, we term the scientific ner task as the typed keyphrase recognition task. we present the reasons in detail in section 2. we aim to study typed keyphrase recognition for the scientific domain in a more challenging lowresource context, specifically the few-shot setting, alongside the standard supervised setting. there is a substantial amount of research available on deep learning-based approaches to classify named entities @xcite . but the difficulty with these approaches is that they are data-intensive approaches. the stumbling block is the collection of such an inordinately large amount of labeled data. this is where few-shot learning comes into the picture. few-shot learning enables the generalization of the model to new unseen classes based on only a few labeled samples. therefore, we introduce the task of few-shot scientific typed keyphrase recognition and design a novel annotation schema for the same. we use this schema to annotate scientific paper abstracts. both coarse-grained and fine-grained keyphrase types have been included in the annotation schema to get an extended keyphrase type set. this helps us in the few-shot scenario because the latter requires testing and validation on unseen class types that we can easily get from the large number of keyphrase types. a sample annotation is shown in in summary, we make the following contributions: (a) we present the first human-annotated dataset, called few-tk, for few-shot typed keyphrase recognition in scientific domain that is focused on the ai/ml literature. this dataset serves to mitigate the scarcity of labeled scientific ie data to some extent. to the best of our knowledge, ours is only the second few-shot dataset for ner, following few-nerd @xcite . we present this as a challenge dataset to the community because detecting scientific typed keyphrases from such an expanded label set is notably more challenging than in typical scenarios. (b) we introduce a new annotation schema for the scholarly domain that is portable to other scientific domains. this schema differs significantly from previous su-pervised ner schemata in terms of entity types and facilitates a broader coverage of entities. (c) we demonstrate the challenging nature of our dataset using several state-of-the-art deep neural models that have been developed, both for the standard supervised setting and the few-shot setting. ## scientific typed keyphrases the term "named entity" was first coined at muc-6 @xcite , with its scope primarily limited to proper nouns @xcite . in standard texts, named entity types such as person, organization, and location exclusively pertain to proper names. however, there has been an unwritten agreement among researchers regarding the inclusion of temporal and numerical expressions as named entities @xcite . previous studies in the scientific domain have attempted to address this by framing the task simply as scientific ner @xcite @xcite @xcite ). yet, this definition of named entities limits the coverage of scientific terminology because scientific literature often uses many terms that are indispensable in terms of the semantic meaning they provide but that do not qualify as proper names. for example, in in this paper, we investigate the task of extraction of keyphrases and their classification, which is similar to ner, but use the term "keyphrases" instead of named entities, to give the task a broader scope. previous works using the "keyphrase" term @xcite @xcite @xcite have predominantly focused on the keyphrase extraction task alone. in contrast, we amalgamate the classification and extraction tasks into a single task -typed keyphrase recognition. we have used the term "keyphrase" in a manner that is more consistent with its usage in @xcite . the authors categorized keyphrases into three types, namely, process (including methods, equipment), task, and material (including corpora, physical materials); this is similar to our attributing types to keyphrases. thus, we introduce the terminology "typed keyphrases", which denotes words or phrases that have significance in the given scholarly text. they have a wider scope in the scientific domain as compared to named entities. this gives rise to the task of scientific typed keyphrase recognition. ## problem definition in this section, we first offer a brief overview of few-shot learning, followed by a definition of fewshot typed keyphrase recognition. ## dataset creation we annotate scientific keyphrases on 500 abstracts from four sub-domains of artificial intelligence/machine learning in the broad spectrum of computer science. namely, these sub-domains are natural language processing (nlp), classical artificial intelligence (ai) together with machine learning (ml), data mining together with information retrieval and computer vision (cv). the reason behind annotating at the abstract-level is that by taking abstracts instead of either only titles (d'souza and auer, 2021) or full texts @xcite , we get a distilled representation of the entire paper. besides, the sentence length in abstracts is found to be substantially longer, yet not too long to resist processing by typical deep neural models used in nlp. ## keyphrase types there are 9 coarse-grained keyphrase types in our dataset few-tk. the focus keyphrase type mainly refers to the area of interest or problem that is being tackled in the article, i.e., the center of attention of the article. the proposed technique keyphrase type alludes specifically to the name of the modus operandi put forward in the paper. the algorithm/tool keyphrase type chiefly refers to any pre-existing concept that has been used in the paper. the allied terms keyphrase type primarily refers to all those phrases which do not fall in the above coarse-grained categories. the study domain keyphrase type calls attention to those phrases that refer to a particular discipline 1 http://brat.nlplab.org/ or subject area. the supplementary material (code/library) keyphrase type principally refers to any auxiliary resource that has been provided along with the paper. the dataset keyphrase type refers to any dataset that has been used in the research article. the metric keyphrase type essentially contains all those measures that are used in the ai/ml domain for evaluating various types of learning approaches. the performance keyphrase type accommodates the results reported in the scientific text. the full set of typed keyphrases (coarse-grained and fine-grained) are described in appendix b and presented in tabular format in appendix c. the allied terms category is very significant because it includes those entities that would have been easily overlooked in any other scientific schema; these entities often have considerable importance attached to their occurrence in the scientific text. for example, in artificial intelligence/ machine learning literature, we encounter the word "training" several times, and it is quite an important term in this literature. however, existing entity schemata often overlook it or categorize it as a generic entity. through our schema, such terms are afforded greater refinement with fine-grained types that better capture their nuanced meanings. if we closely examine the pattern of our finegrained keyphrase types, we observe that for most of the coarse-grained types, the fine-grained categories are predominantly theoretical ai/ml, nlp, computer vision, and data mining/information retrieval, because these are the leading areas of study within artificial intelligence. we argue that the coarse-grained keyphrase types can be used in other scientific domains like physics and chemistry. similar to our fine-grained categories, the physics domain can be divided into astrophysics, nuclear physics, thermodynamics, biophysics, etc; the chemical domain may be divided into physical chemistry, organic chemistry, inorganic chemistry, analytical chemistry, and biochemistry. by incorporating keyphrase types relevant to these specialized fields, we can greatly assist downstream tasks such as question answering and knowledge graph construction. ## experiments we demonstrate the challenging nature of our dataset by evaluating it on sota ner models that have been developed previously both for the fullysupervised settings and the few-shot settings. we now briefly describe these models. ## results the results for both the fully-supervised and fewshot frameworks are detailed out below. 4 shows the results of the tagging model using two bert-type models. we see that scib-ert @xcite gives better results than bert @xcite when used in the tagging model. however, the results on our dataset are significantly worse than that achieved on scierc @xcite , underscoring the challenging nature of our dataset even in the fully-supervised setting. this difficulty may primarily stem from the expanded keyphrase set present in our dataset. figure 2 shows the category-wise f1 scores for each fine-grained type. there are some classes which have very low or zero f1 score. this may be attributed both to the nature of the phrases in those classes and the low count of samples available for those classes as seen in figure 1 . ## error analysis we present our analysis for the nnshot model in tables 7 and 8 for the span and type errors respectively. we use the 3-way 1-shot setting for our analysis. we consider two types of errors that occur when few-shot models try to classify typed keyphrases. if a model fails to detect the span of a typed keyphrase correctly, it is considered as a span error. if a token that must be included in a typed keyphrase is not done so by the model, it is called a false negative (fn) case, while if a token is incorrectly included as part of a typed keyphrase, it is called a false positive (fp) case. when the span of a keyphrase has been correctly identified, if the model makes a misclassification while predicting the type of the keyphrase, it is termed a type error. if the model correctly predicts the coarse-grained type but fails to predict the fine-grained type accurately, it is termed a within error. on the other hand, if the model inaccurately predicts the coarse-grained type, it is referred to as an outer error. ## discussion the following points have come to our notice while creating the dataset. generally, when considering both pure ai/ml literature and its sub-areas, we observe that abstracts from journals in allied ai fields, such as nlp or cv, often contain a considerable number of entities originating from the context of pure artificial intelligence, machine learning, or deep learning. however, in the reverse scenario, 2 https://paperswithcode.com/ where abstracts from pure ai journals are examined, the presence of entities from allied ai areas is significantly less common. the tkdd journal was found to contain representations from all four domains, with the pure ai domain being the least dominant among them. an inherent challenge we discovered with the annotation of scientific documents is that quite often a term is presented in a descriptive manner, which makes specifically demarcating the keyphrases quite a challenging task. in the fully supervised setting, we have additionally evaluated the ability of supervised models to detect span-level mentions by tasking the model with predicting only the keyphrase spans. we observe in ## conclusion we have developed a unique dataset tailored for the task of few-shot scientific keyphrase recognition within the scientific domain. we have also evaluated various models on it to assess its credibility as a challenging dataset. we hope that this dataset will be used as a cornerstone in research on scientific typed keyphrase recognition.