SEKE : Specialised Experts for Keyword Extraction (2025)

Abstract: Keyword extraction involves identifying the most descriptive words in a document, allowing automatic categorisation and summarisation of large quantities of diverse textual data. Relying on the insight that real-world keyword detection often requires handling of diverse content, we propose a novel supervised keyword extraction approach based on the mixture of experts (MoE) technique. MoE uses a learnable routing sub-network to direct information to specialised experts, allowing them to specialise in distinct regions of the input space. SEKE, a mixture of Specialised Experts for supervised Keyword Extraction, uses DeBERTa as the backbone model and builds on the MoE framework, where experts attend to each token, by integrating it with a bidirectional Long short-term memory (BiLSTM) network, to allow successful extraction even on smaller corpora, where specialisation is harder due to lack of training data. The MoE framework also provides an insight into inner workings of individual experts, enhancing the explainability of the approach. We benchmark SEKE on multiple English datasets, achieving state-of-the-art performance compared to strong supervised and unsupervised baselines. Our analysis reveals that depending on data size and type, experts specialise in distinct syntactic and semantic components, such as punctuation, stopwords, parts-of-speech, or named entities. Code is available at https://github.com/matejMartinc/SEKE_keyword_extraction.

## introduction keyword extraction, i.e., extraction of words that represent crucial semantic aspects of the text, is crucial for efficiently summarizing, indexing, and retrieving relevant information from large textual corpora, making it one of the most fundamental nlp tasks @xcite . while the first automated solutions have been proposed more than two decades ago @xcite @xcite , the task is far from solved, and the scientific community is still actively trying to improve the performance of the keyword extractors (see section 2 for details). recently, the so-called foundation models have completely revolutionised the way natural language processing (nlp) is done, and the development of large language models (llms) has enabled taskagnostic architectures to solve downstream nlp tasks in a zero-shot fashion without the need for labelled data @xcite . while recent related studies @xcite and experiments conducted in this study show that llms' keyword extraction performance is still not comparable to specialised supervised approaches, their zero-shot performance nevertheless remains impressive. it was made possible, to a large extent, by the integration of novel training regimes and architectural components, such as low-rank adaptation (lora) @xcite and mixture of experts (moe) @xcite , which make models more efficient, while requiring fewer computational resources. while these components have been successfully employed in an unsupervised language modelling setting with abundant data, there are less studies exploring their effectiveness in a supervised setting with much fewer resources. the main reason for this is that these components were developed with a specific objective of improving the scalability of the models @xcite and therefore their effect on the performance is somewhat neglected or measured only indirectly, that is, studies focus on how the use of these components allows for greater scalability, which in turn leads to performance gains. in contrast, in this study, we focus on the performance improvements obtained from these components without increasing the size of the backbone model. the hypothesis we want to test is whether it is beneficial for keyword extraction to allow specific parts of the network to specialise for specific types of tokens. since the "keywordiness" of a token is to a large extent determined by its specific contextual role, we explore if introducing a gating network that assigns different parts of the sequence to specialised layers, which would only attend to specific subsequences and tokens according to their semantic and grammatical role, could improve the current state-of-the-art (sota) in the field. in addition, we explore what type of specialisation different experts undertake and how much data is required for a successful specialisation. we test our approach on several datasets with different amounts of training data available, to determine the data threshold that still allows the convergence of expert layers. finally, we examine whether synergy can be achieved between moe and bidirectional long short-term memory (bilstm) network that would lower this threshold and improve overall performance by amplifying the sequential information lost by the context slicing caused by the moe strategy. the contributions of the paper are as follows. ## results and analysis in for the supervised baselines, copyrnn, cat-seqd, gpt-2, gpt-2+bilstm-crf, and tnt-kid, we list results reported by @xcite , for diff-kpe, we list results reported by @xcite , for jointkpe, we list results from @xcite , and for hybridmatch, we list results from @xcite . note that gpt-2, gpt-2 + bilstm-crf and tnt-kid are sequence labelling approaches, same as ours. since they return a non-predetermined number of keywords (i.e., the approaches learn during training how many keywords they should return), they require adaptation to each specific keyword labelling regime for optimal performance and were therefore fine-tuned on the validation datasets. in contrast, jointkpe, diffkpe nad hybridmatch baselines return a predetermined number of best ranked candidates (3, 5 or 10, depending on the evaluation criteria) and do not require validation set fine-tuning for adaptation. same applies to the two generation baselines, copyrnn and catseqd, which tend to generate a large set of predicted keywords, from which a fixed number of top predicted phrases (3, 5 or 10) is taken to compare with the ground truth. finally, besides testing our custom sequence labelling heads (containing moe, bilstm, or both), we also report results for the unmodified pre-trained de-berta model with a standard feed-forward token classification head. in the validation set training scenario, by adding the moe on top of the deberta model (see configuration deberta moe), we improve the performance of the vanilla model (see configuration deberta) with a feed-forward token classification head on four out of six datasets according to all evaluation criteria. the improvement is not observed for the two smallest datasets, krapivin and inspec. on the other hand, adding the additional bilstm head to the vanilla deberta model improves the performance of the base model on all datasets (see configuration deberta bilstm) and according to almost all criteria (the only exception being the f1@1 criterion on the openkp dataset). the results therefore indicate a synergy between the moe and bilstm layers, since the model with both moe and bilstm layers manages to substantially improve the base model and also outperforms the deberta moe and deberta bilstm models on four out of six datasets according to all criteria. in the full training scenario, the deberta moe model outperforms the vanilla deberta on three out of six datasets according to all evaluation criteria. interestingly, the deberta moe model fails on all computer science datasets, which suggests that expert specialisation might be harder on this domain. on the other hand, the deberta moe bilstm model outperforms the vanilla one on all but one dataset and criterion, the f1@1 criterion on the openkp dataset. in this scenario, we also manage to substantially improve the sota on the inspec, kptimes, and jp-times datasets according to both criteria. while we outperform the best-performing model joinkpe in f1@10 on the kp20k and the krapivin datasets, the results also show that the unsupervised models in general achieve much worse performance than the supervised models. while the llm-based approaches outperform other unsupervised baselines, their performance lags behind supervised approaches. out of the four tested llms, the two proprietary models, gpt4o-mini and gemini-1.5flash-8b, perform the best, with gemini-1.5-flash-8b being the best model on most datasets. ## conclusion in summary, we investigated the potential of moe for keyword extraction tasks. we introduced seke, a novel architecture that combines moe and bil-stm built upon the deberta model. this approach achieved superior performance compared to existing sota models on standard datasets for keyword extraction. the study revealed that adding bilstm can help mitigate the data limitations, showing the feasibility of the approach even on small training datasets. in the future, we will try to pinpoint the exact dataset size thresholds, down to which the proposed approach is still feasible. additionally, we will expand the evaluation to more benchmarks to further strengthen the analysis. in this work, our primary goal was to evaluate the general effectiveness of our proposed framework using a consistent and reasonable configuration across all experiments. to maintain a fair comparison and avoid overfitting to specific datasets, we used commonly adopted default values (e.g., 4 experts, top-@xmath0. in the future, a more exhaustive hyperparameter search will be conducted, which could yield additional performance gains and further insights into the model behaviour. the analysis revealed that specialisation is domain-specific. current results indicate that in text with higher complexity (i.e., scientific papers) experts to a larger extent rely on syntactical patterns than in the less complex text, where experts to a larger extent specialise for processing of semantic clues. this hypothesis will be further tested in the future by applying the approach in new domains and languages other than english. finally, we can observe that despite the impressive improvements in the development of foundational llms, keyword extraction task can still not be sufficiently solved by employing these models, which lag behind supervised approaches on most datasets. since the biggest advantage of supervised approaches is their ability to adapt to the specifics of the syntax, semantics, content, and keyword tag-ging regime of the specific corpus @xcite , in the future we will adapt llms to the specific corpora and keyword extraction regimes with fine-tuning and in-context learning.