A rabic Keyphrase Extraction: Enhancing Deep Learning Models with Pre-trained Contextual Embedding and External Features (2022)

Abstract: Keyphrase extraction is essential to many Information retrieval (IR) and Natural language Processing (NLP) tasks such as summarization and indexing. This study investigates deep learning approaches to Arabic keyphrase extraction. We address the problem as sequence classification and create a Bi-LSTM model to classify each sequence token as either part of the keyphrase or outside of it. We have extracted word embeddings from two pre-trained models, Word2Vec and BERT. Moreover, we have investigated the effect of incorporating linguistic, positional, and statistical features with word embeddings on performance. Our best-performing model has achieved 0.45 F1-score on ArabicKPE dataset when combining linguistic and positional features with BERT embedding.

## introduction keyphrases are the phrases that best represent a document. they play an essential role in many natural language processing (nlp) and information retrieval (ir) tasks, such as indexing, summarization, categorization, and opinion mining @xcite . manual extraction of keyphrases is time-consuming and requires experts' knowledge; thus, the extraction needs to be automated @xcite . although many studies have been proposed to address automatic keyphrase extraction and generation, the performance is still moderate due to the task's difficulty @xcite . several approaches have been proposed; one of the earliest approaches is the two-step ranking, in which candidate phrases are extracted with several heuristics and then ranked using supervised or un-supervised methods @xcite . another approach is the classification approach, in which candidate phrases are classified as keyphrases or not @xcite . a more recent approach is formulating keyphrase extraction as a sequence labeling task in which each word in the documents is labeled as part of a keyphrase or not @xcite . another recent approach is to consider formulating the task as a generation task utilizing sequence-to-sequence models in order to be able to generate keyphrases that are not available in the source text, i.e., keyphrase generation @xcite . word embeddings prove their effectiveness in many nlp tasks. several word embeddings are proposed, such as word2vec @xcite and fasttext @xcite . the earliest proposed word embeddings generate the same vector for the word regardless of the word context hence called static word embeddings @xcite . recently, several word embeddings generate different embeddings for the word depending on its context hence called contextualized word embeddings (pilehvar and camacho-collados) such as bert @xcite , and elmo @xcite . several studies have utilized various types of word embeddings into supervised and unsupervised keyphrase extraction, and they positively affect performance. arabic has its own characteristics that pose many challenges on any ir or nlp task @xcite . thus, it is crucial to investigate the performance of state-of-the-art techniques of keyphrase extraction on arabic, which might differ in terms of performance from other languages. several datasets are available for arabic keyphrase extraction; the arabic keyphrase extraction corpus (akec) @xcite , arabic dataset proposed by @xcite , wikiall foot_0 from arabic wikipedia documents, and arabickpe @xcite . during our investigation of the keyphrase extraction studies, we have found that studies on arabic keyphrase extraction are falling behind in applying state-of-the-art technologies. for example, only a few studies have utilized word embeddings; @xcite have investigated using word2vec and semantic similarity to generate keyphrases for three documents only. @xcite have investigated using word2vec and bidirectional-long short term memory (bi-lstm) in keyphrase extraction. to fill this gap, we aim to apply deep learning approaches to keyphrase extraction utilizing the static and contextualized word embeddings for arabic keyphrase extraction. thus, we have formulated the task as a sequence labeling task and have used a bi-lstm classifier with token representation extracted from two types of pre-trained word embeddings. additionally, we aim to investigate the effect of incorporating statistical, positional, and linguistic features with static and contextual embeddings. in this study, we have used bidirectional encoder representations from transformers (bert) in arabic keyphrase extraction and have compared it to word2vec embedding. additionally, we have investigated three ways of utilizing bert for arabic keyphrase extraction; extracting the output of the last encoder of the bert model and using it as a feature, concatenating the output of the last four encoder layer of bert, and fine-tuning bert. to the best of our knowledge, this is the first study incorporating contextualized word embedding from bert into the arabic keyphrase extraction task. we have found that utilizing contextual embeddings vastly enhances the performance of arabic keyphrase extraction model. moreover, adding features to the arabic keyphrase extraction bi-lstm model, in general, has a positive effect on the model performance. the rest of the paper is organized as follows: we present the related works in section 2, section 3 presents our methodology, and section 4 presents experiments results and discussion. our conclusion is presented in section 5. ## related work keyphrase extraction has two general approaches: unsupervised and supervised @xcite . supervised approaches are powerful and perform better than unsupervised approaches. however, the unsupervised approaches are less expensive @xcite . several unsupervised keyphrase extraction studies have been conducted. @xcite have proposed yake!, an unsupervised keyphrase extraction system based on statistical features extracted from a single document. their approach depends on six features; term frequency within the document, normalized term frequency, term relative position (sentence index), term relatedness to context, term case, and how often a term appears in different sentences (term different sentence). they have evaluated the system on 20 datasets in five languages. yake! proved its effectiveness generally compared to other systems with large text and performed well with shorter text on different domains and different document types. moreover, the frequency feature has a more positive impact while removing term relatedness from the context and term different sentence features improves performance. meanwhile, the term frequency feature is more useful when the document size increases, while the position feature is more beneficial in shorter texts. the case feature is more useful with mid to larger documents, while the term different sentence feature is better with short to mid documents. @xcite have leveraged word embedding for unsupervised graphbased keyphrase extraction. their model selects candidate words based on their part-of-speech (pos) tag; they have only selected nouns and adjectives. they have built three graphs; a word-word graph based on the word co-occurrence, a wordtopic graph that connects words to their topics, and a topic-topic graph that is constructed when the same word appears in different topics. they have also proposed a modified random-walk model to rank candidate words and a new scoring model for candidate phrases based on the cosine similarity of the generated word embedding and the modified page rank score. the top scoring phrases are considered document keyphrases. evaluating the type of word embeddings used shows that their embedding outperforms other embeddings on this task. they have reported that their model performs the best on all the tested datasets compared to other baselines. @xcite have utilized word embedding with graph-based unsupervised keyphrase extraction along with document embedding. they have used a pre-trained sent2vec @xcite model trained on wikipedia to create word embedding. the embedding vector is created by averaging all document words and n-gram embeddings. they have found that using the word as a node is better when dealing with a short text dataset and using a phrase as a node is better when dealing with a long text dataset. several studies have formulated the keyphrase extraction task as a sequence labeling task. @xcite have proposed a deep learning model for automatic keyphrase extraction using bi-lstm and pre-trained gloveembedding @xcite . their model outperforms copyrnn @xcite model on the same dataset. @xcite have utilized bi-lstm and crf for keyphrase extraction from scientific documents using 100-dimension pre-trained gloveembedding for embedding initialization. they have studied the role of each model layer on the performance and have compared crf only, forward-lstm, and bi-lstm. they have found that removing the bi-lstm layer negatively affects the recall, while removing the crf layer increases the recall and decreases the precision. hence, it indicates that bi-lstm can capture long-distance semantics and cause extraction of more gold-standard keyphrase. they have also found that crf can capture the dependencies between labels leading to higher model precision. combining bi-lstm and crf has the best performance among the three created models. additionally, the model outperforms copyrnn @xcite . many studies have used encoderdecoder architecture to generate absent and present keyphrases. the most popular study is the work by @xcite . they have proposed a generative model for keyphrase generation based on encoder-decoder architecture. they have used bidirectional gated recurrent units (bi-gru) for the encoder and forward-gru for the decoder and incorporated attention mechanism @xcite (rnn-model) and copy mechanism @xcite (copyrnn-model) to deal with out-ofvocabulary words. copyrnn model outperforms all the models they have compared by an average of almost 20% @xcite . moreover, the copyrnn model outperforms rnn in predicting both present and absent keyphrases. (kehua yang, 2019) encoder-decoder model is entirely based on the self-attention mechanism. they have incorporated semantic similarity between keyphrases. the model outperforms all baselines in predicting the present keyphrase. additionally, it outperforms copyrnn in predicting absent keyphrases. targeting the problem of overlapping phrases generated by sequence-to-sequence models, @xcite have proposed (paranet). the model consists of two parallel encoders; one to encode the text and the other to encode the linguistic constraints introducing coverage attention. they have used multi-task learning on two parallel decoders to generate the keyphrase and pos tag for each word in the keyphrase. they have tested different settings for combining the vector of the words and their syntactic tags, using the hyperbolic tangent function, using tree-lstm, and adding coverage attention to the previous two. on the evaluation of present keyphrases, all their model settings outperform the extraction and generation methods baselines, including copyrnn @xcite . their best performing setting is when using tree-lstm to combine vectors along with coverage attention. several studies have used bert contextualized word embedding in two strategies; feature-based strategy or fine-tuning-based strategy. word feature is extracted from the pre-trained bert in the feature-based strategy. in fine-tuning based strategy, bert model parameters are fine-tuned with the new smaller dataset for the downstream task adding one fully connected layer on top of it @xcite . @xcite have utilized bert embedding in multi-task learning for keyphrase extraction. @xcite have finetuned bert and scibert @xcite for keyphrase extraction. they have found that the best performance happens within the first three epochs of fine-tuning and that scibert performs better than bert on scientific datasets. @xcite have experimented with four neural network architectures based on bi-lstm and multi-head attention on top of the transformer models bert and scibert. a recent study has combined graph embedding and bert embedding for keyphrase extraction is phraseformer @xcite . they have concatenated the resulting graph embedding and word embedding for each word and have used the resulting encoding as input. another way of utilizing bert for keyphrase extraction using a feature-based technique is using it in ranking candidate phrases @xcite . @xcite have incorporated different types of features with bert extracted features for the chinese medical keyphrase extraction. the task is considered as a character-level labeling task. they have incorporated pos feature and lexicon feature using two techniques: concatenation and feature embedding. they have used the model without feature as a baseline and have tested the effect of features (pos only, lexicon only, and their combination) and the effect of different feature incorporation techniques (concatenation, embedding, and their combination). their results show that incorporating the lexicon feature has a more positive impact than the pos feature, regardless of the incorporation techniques. furthermore, the best incorporation technique is the embedding technique. @xcite have utilized contextualized word embeddings and compared them to static word embedding in sequence labeling keyphrase extraction. they have used bi-lstm-crf and bi-lstm architectures with several embeddings; bert, scibert, elmo, transformerxl @xcite , openai-gpt @xcite , openai-gpt2 @xcite , roberta @xcite , glove, fasttext, and word2vec. they have found that contextualized embeddings are better than static embedding, and bert is the best among them since it uses bi-directional training. studies on arabic keyphrase extraction followed several approaches. rule-based approaches (el- @xcite @xcite (alotaibi and ahmad, 2019) @xcite , ranking approachs @xcite , using a graph-based model as a base for ranking @xcite , utilizing bag-of-concept (awajan, 2015) @xcite , machine learning approachs (ali and omar, 2015) (armouty and tedmori, 2019) @xcite and deep learning approachs @xcite . @xcite have combined statistical and machine learning methods and have formulated the keyphrase extraction task as a classification task. they have used term frequency, first occurrence, sentence count, c-value for multi-word nested terms, and tf-idf statistical features to construct a feature vector. they have trained linear logistic regression, linear discriminant analysis, and support vector machine (svm) classifiers. @xcite have used tf-idf and the first occurrence weight of the term with support vector machine (svm), naïve bayes, and random forest classifiers. @xcite have used graph centrality measures along with term frequency and pos tags as input features to multi-layer perceptron, naïve bayes, random forest, and oner algorithms. @xcite have proposed a deep learning-based model and a large-scale dataset for keyphrase extraction task. they have used aravec @xcite to represent each token and a bi-lstm model. ## conclusion this study uses two types of pre-trained word embeddings for arabic keyphrase extraction task: static and contextualized word embedding (word2vec and bert). several features are incorporated into the models to test their effect on performance. we have found that contextualized word embedding has vastly enhanced the performance of arabic keyphrase extraction. moreover, incorporating features with static embedding has more effect than incorporating features with contextualized embedding. different features and features combinations affect the performance differently depending on the used embeddings. for future work, we consider trying the effect of adding more features to the models. moreover, investigate the best combination of layers to select from bert for keyphrase extraction.