Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training (2022)

Abstract: Keyphrase generation is the task of automatically predicting keyphrases given a piece of long text. Despite its recent flourishing, keyphrase generation on non-English languages haven’t been vastly investigated. In this paper, we call attention to a new setting named multilingual keyphrase generation and we contribute two new datasets, EcommerceMKP and AcademicMKP, covering six languages. Technically, we propose a retrieval-augmented method for multilingual keyphrase generation to mitigate the data shortage problem in non-English languages. The retrieval-augmented model leverages keyphrase annotations in English datasets to facilitate generating keyphrases in low-resource languages. Given a non-English passage, a cross-lingual dense passage retrieval module finds relevant English passages. Then the associated English keyphrases serve as external knowledge for keyphrase generation in the current language. Moreover, we develop a retriever-generator iterative training algorithm to mine pseudo parallel passage pairs to strengthen the cross-lingual passage retriever. Comprehensive experiments and ablations show that the proposed approach outperforms all baselines.

## introduction keyphrases are single or multi-word lexical units that best summarize a piece of text. as such, they are of great importance for indexing, categorizing, and mining in many information retrieval and natural language processing tasks @xcite @xcite . keyphrase generation is the task of automatically predicting keyphrases given a piece of long text. existing works on keyphrase generation mostly focus on english datasets @xcite while keyphrase generation for languages other than english is still under-explored. since search engines usually provide services to customers using different languages, multilingual keyphrase generation becomes a significant problem while it is still unknown how well existing keyphrase generation approaches perform in non-english languages. nevertheless, there are two challenges we will face regarding multilingual keyphrase generation. first, to the best of our knowledge, there is no largescale dataset publicly available for training and benchmarking multilingual keyphrase generation models. building keyphrase datasets at a sufficient scale is difficult and costly. second, compared with the existing datasets in english, which can contain millions of data examples and cover a wide diversity of topics, the data resources in non-english languages are inherently scarce. for example, in the domain of e-commerce, marketplaces using english have abundant customer queries to be used for keyphrase mining, while queries in some languages are relatively less than in english, which is probably because of a smaller size of user population or a shorter operation time. we start tackling these challenges by contributing two new datasets for multilingual keyphrase, which cover six languages and two domains. the first dataset ecommercemkp is collected from a real-world major e-commerce website. the product descriptions are used as the source text while the target keyphrases are collected from user search queries. this dataset contains a total of 73k data examples, covering four different languages (spanish, german, italian and french). the second multilingual keyphrase dataset academicmkp lies in the academic domain, in which titles and abstracts are used as the source text and the author-provided keyphrases are deemed targets. a total of 2,693 aca-demic papers in chinese and korean are included in academicmkp. to overcome the resource scarcity challenge in training multilingual models, we propose a retrieval-based method to leverage the keyphrase knowledge in large-scale english datasets. by investigating multilingual keyphrase data, we observe that data in different languages may talk about similar topics. therefore, we conjecture that passage-keyphrases pairs in english can be of help as an external knowledge base for multilingual keyphrase generation. to be specific, given a passage in low-resource language xx, we propose to use a retrieval model to find multiple top-related passages in english. these retrieved english passages provide high-quality english keyphrases that can be used as hints for generating keyphrases in other languages. after that, the generator takes the code-mixed inputs, including the passage in language xx and retrieved english keyphrases, and predicts keyphrases in language xx. in the cross-lingual retrieval training, parallel passage-keyphrases pairs between english and other languages are extremely limited. for example, in the e-commerce domain, only a small fraction of products have both english and non-english descriptions (being sold in multiple countries). such a data scarcity issue weakens the ability of cross-lingual knowledge acquisition from high-resource english keyphrases as intermediary, and finally hinders the potential of the retrievalaugmented keyphrase generation. to mitigate the problem, we propose a retriever-generator iterative training (rgit) algorithm to automatically mine pseudo training parallel pairs from unlabeled data. concretely, the retriever can dynamically adjust in terms of the current variations of generation performance between the proposed retrieval-augmented generator and the base one without the aid of retrieved english keyphrases. starting from insufficient seed parallel pairs, if the retrieved pseudo passage-keyphrases pairs in the current iteration can bring in higher generation results as the generator's feedback, those pseudo parallel data will be regarded as high quality and incorporated into the seed ones to further boost the retriever. such cycle providing positive effects can be repeated until the increasing generation performance stopped. we conduct extensive experiments on ecom-mercemkp and academicmkp and demonstrate that large-scale english datasets do provide use-ful knowledge for multilingual keyphrase generation. the proposed retrieval-augmented method outperforms traditional extraction-based models, sequence-to-sequence neural models, and its variants. moreover, the rgit algorithm boosts the retrieval performance significantly by mining over 20k pseudo-parallel passage pairs. we also conduct detailed analyses to investigate the effectiveness of retriever-generator iterative training. ## related work keyphrase generation. the advance of neural language generation enables models to freely generate keyphrases according to the phrase importance and semantics, rather than extracting a list of sub-strings from the text @xcite @xcite . @xcite propose the first keyphrase generation model copy-rnn, which not only generates words based on a vocabulary but also points to words in the source text -overcoming the barrier of predicting absent keyphrases. following this idea, chen et al. ( 2018 2020 ) propose using gan for kpg. @xcite introduce hierarchical decoding and exclusion mechanism to prevent models from generating duplicate phrases. @xcite propose to dynamically align target phrases to eliminate the influence of order, as highlighted by @xcite . @xcite al. (2020); liu et al. (2020a) ; park and caragea (2020) use pre-trained language models for better representations of documents. retrieval augmented text generation (rag) recently shows great power in knowledge-intensive nlp tasks such as open-domain question answering, fact checking and entity linking @xcite @xcite . in rag, a retriever (either sparse @xcite or dense @xcite ) searches for useful non-parametric knowledge from a knowledge base, then a generator combines the non-parametric retrieved knowledge with its parametric knowledge, learned during pre-training, for solving the task. different from these tasks, keyphrase generation is not a knowledge-intensive task but we treat the english passage-keyphrase training data as our knowledge. similar approaches have been investigated in neural machine translation @xcite , dialogue @xcite , and knowledge-base qa @xcite . in keyphrase generation, @xcite ; @xcite ; @xcite retrieve similar documents from training data to produce more accurate keyphrases. however, their retrieval module is a non-parametric model and cannot be generalized in the multilingual setting due to the large vocabulary gap between languages. ## task definition in this paper, we aim to tackle the keyphrase generation task in a multilingual setting, which means one model of desire is capable of generating keyphrases in any language that it has been trained with. the benefits of having a single keyphrase generation model for multiple languages are threefold: (1) collecting keyphrase annotation for individual language can be prohibitively expensive; (2) training and deploying separate models for each language is laborious; (3) joint training of multiple languages can alleviate the resource scarcity by utilizing rich monolingual data. formally, we define the multilingual keyphrase generation task as follows. given a piece of text p xx in language xx, our goal is to predict its corresponding keyphrases k xx 1 , k xx 2 , ..., k xx n in language xx, where n is the total number of target keyphrases for this text p xx . in this study, xx can be german (de), spanish (es), italian (it), french (fr), korean (ko) or chinese (zh). ## model scarcity of resources is one of the topmost challenges for multilingual tasks, which is also the case for multilingual keyphrase generation. one may find it difficult to collect enough text data in languages other than english, much less the annotation of keyphrases in specific domains. to overcome this problem, we propose a retrieval-augmented approach to make use of the relatively rich resources in english. the motivation for our proposed retrieval-augmented approach comes from an observation from data: texts and keyphrases expressed in different languages usually share common topics or knowledge concepts. for example, in e-commerce websites, it is often the case that the same products are sold in different marketplaces/countries. thus these products as well as their keyphrases, though exhibited in different languages, can share a high semantic similarity. in other words, given a text in language p aa , if we could find a similar text in language p bb , its associated keyphrases k bb may serve as a good hint for the to-be-generated keyphrases k aa in language aa. since english has the most abundant text-keyphrases pairs in both e-commerce and academic papers domains, its resource can be treated as a non-parametric keyphrase knowledge base, which provides texts in english covering a wide range of topics and concepts, as well as the associated high-quality keyphrases. as shown in glish passage p en j has its associated n j english keyphrases k en j,1 , k en j,2 , ..., k en j,n j . these retrieved english keyphrases are taken as external knowledge for keyphrase generation in step 2. ## conclusion in this study, we investigate a novel task setting -multilingual keyphrase generation -and contribute two new multilingual keyphrase generation datasets covering multiple domains and languages. furthermore, we propose a retrieval-augmented multilingual keyphrase generation framework with retriever-generator iterative training. results show that the proposed approach outperforms a wide range of baselines.