Incorporating Multimodal Information in Open-Domain Web Keyphrase Extraction (2020)

Abstract: Open-domain Keyphrase extraction (KPE) on the Web is a fundamental yet complex NLP task with a wide range of practical applications within the field of Information Retrieval. In contrast to other document types, web page designs are intended for easy navigation and information finding. Effective designs encode within the layout and formatting signals that point to where the important information can be found. In this work, we propose a modeling approach that leverages these multi-modal signals to aid in the KPE task. In particular, we leverage both lexical and visual features (e.g., size, font, position) at the micro-level to enable effective strategy induction and meta-level features that describe pages at a macro-level to aid in strategy selection. Our evaluation demonstrates that a combination of effective strategy induction and strategy selection within this approach for the KPE task outperforms state-of-the-art models. A qualitative post-hoc analysis illustrates how these features function within the model.

## introduction we present a novel multi-modal approach to keyphrase extraction (kpe), which is the task of automatically extracting salient phrases from a given document. the kpe task is a foundational task, which plays a facilitating role in many information retrieval (ir) tasks, including classification, summarization, and document indexing @xcite . specifically, the kpe task requires accurate selection of the phrases that best capture the web document's topic. wellperforming approaches take advantage of document structure and entity co-occurrences. over the history of work in this area, there have been a variety of benchmarks @xcite @xcite and an equally wide variety of both non-neural @xcite @xcite and neural modeling approaches @xcite @xcite . the earliest kpe approaches were mainly limited to domain-specific keyphrase extraction. the recent release of openkp @xcite , a large-scale feature-rich dataset specifically developed for open-domain web-page keyphrase extraction, has encouraged further research related to the kpe task. a novel characteristic of this data set is the inclusion of features related to visual properties. visual properties of words and web page layout offer a kpe model utility in at least two respects. first, micro-level features operating at the word level, including lexical features as well as features related to size, font, color, and position of words, signal relative importance of words within extended texts. intuitively, texts that are highlighted with colored, bold or bigger font or placed in more obvious places within a web page are more likely to be important, and should correspondingly be given higher probability as a keyphrase. next, macro-level features describing the layout and type of page (e.g., news, storefront, etc.) are related to the distribution of keyphrases. for example, for a news or blog website, the title or the first paragraph of its main content are likely locations for finding keyphrases, while for an indexing page, the important phrases will more likely be found in lists. important information may also be listed underneath or beside pictures. based on these insights, we propose a multimodal framework, strategy-based multimodal architecture for keyphrase extraction (smart-kpe), addressing the web kpe task in two steps: multimodal strategy induction to apply specific extraction tactics with a refined use of micro-level features and strategy selection to choose results from different tactics using macro-level features. in our evaluation, we compare smart-kpe with several state-of-the-art baselines, where smart-kpe shows its better ability to locate and extract keyphrases. we offer post-hoc case studies and ablation studies to illustrate model strengths and weaknesses. in addition to the improvement over sota baselines for the kpe task, to the best of our knowledge, strategy-based multimodal architecture for keyphrase extraction is the most comprehensive treatment of multimodality in open-domain kpe. ## conclusions and future work in this work we propose a strategy-based multimodal architecture for keyphrase extraction (smart-kpe) as a new state-of-the-art method for multimodal web-page keyphrase extraction. different from traditional keyphrase extraction models mainly focusing on text, smart-kpe illustrates the advantage of incorporating other modalities to help keyphrases location and salience prediction. our proposed model outperforms several state-of-the-art baselines with the introduction of multimodal information. through several case studies, we further illustrate how micro and macro-level features lead to the model's correct or incorrect selections. as a first attempt to introduce macro-level meta-features for strategy selection, we believe there is much potential to refine and improve our approach. one high-level idea is to add further supervision to the current selector model based on empirical web page clustering, to better train the model to develop a set of more distinct keyphrase prediction strategies, and more effectively adjust the respective selector weights. we also plan to add more types of meta-features to generate richer multimodal representations. furthermore, the smart-kpe framework can be easily adapted to other nlp tasks, and we believe there is much potential in combining smart-kpe with different models to further boost performance on opendomain kpe and other web-related tasks.