Leveraging Structured Metadata for Improving Question Answering on the Web (2020)

Abstract: We show that leveraging metadata information from web pages can improve the performance of models for answer passage selection/reranking. We propose a neural passage selection model that leverages metadata information with a fine-grained encoding strategy, which learns the representation for metadata predicates in a hierarchical way. The models are evaluated on the MS MARCO (Nguyen et al., 2016) and Recipe-MARCO datasets. Results show that our models significantly outperform baseline models, which do not incorporate metadata. We also show that the fine-grained encoding’s advantage over other strategies for encoding the metadata.

## introduction question answering (qa) is a long-standing task in nlp and ir. having qa systems that perform well on real-world questions is of significant value for search engines and intelligent assistants. while some of the earliest work tackled the task of answering questions based on a large corpus @xcite @xcite (albeit mostly focusing on simple fact-oriented questions), much of the recent work on qa has focused on answering questions in a less realistic setting -drawing the answer from a paragraph of text @xcite , which is commonly referred to as machine reading comprehension (mrc). in this work, we tackle the more realistic problem -candidate answers passages selection/reranking for real-world questions on the web. in contrast to both mrc and early work on qa from a large corpus, web pages often provide an additional source of knowledge. in particular, and thanks in part to the semantic web initiative (berners-lee recipe cooktime preptime recipeinstructions recipeyield etc. et @xcite , it is estimated that a non-trivial portion of web pages contain metadata annotations that provide a deeper understanding of the website content. the web data commons project @xcite estimates that 0.9 billion html pages out of the 2.5 billion pages (37.1%) in the common crawl web corpus foot_0 contain structured metadata. figure 1 shows an example of this metadata which comes in the form of objectpredicate pairs annotated with schema.org tags -a set of tags/predicates defined in the schema.org foot_1 hierarchy. in the example, the hierarchical metadata is used to add more structure to the web page of a recipe, providing meaning to the otherwise unstructured content. this makes several aspects of the recipe explicit -the preparation time (preptime), cooking time (cooktime), ingredients (ingredients), etc. figure 2 shows the "recipe" object in schema.org; it contains several properties such as cooktime, preptime, we hypothesize that leveraging this metadata, in addition to the textual content, will improve the performance of qa systems on the web. to date, the end-to-end web qa systems have not made use of this metadata information. we first explore how to incorporate (and the effect of incorporating) semantic web hierarchical metadata into statistical nlp models for web-based qa. more specifically, we introduce a fine-grained encoding method for metadata predicates, to better leverage the semantic information in it. we evaluate the models on the answer passage selection/re-ranking task of ms marco @xcite , that contains real user queries sampled from the bing search engine, with the answer passages extracted from real-world web pages. results show that our approaches outperform the baseline systems substantially, with more significant gains on the subset of queries whose candidate passages contain richer metadata tags. our work demonstrates the importance of encoding metadata information for qa, and verifies our hypothesis that the metadata knowledge can significantly benefit the performance of the neural models. we also provide qualitative analysis that includes performance comparisons across domains. our findings further provide motivation for webmasters to annotate their web pages with semantic schema.org markup and for question answering systems developer to leverage them. ## related work our work is related to several directions of work in semantic web, nlp and ml. ## leveraging metadata for answer passage selection in our setting of answer passage selection, the input to the system is a set of candidate passages p 1 , ..., p n , and a query q, the goal is to identify the passage that best answers the question. for each candidate passage p i , we have the url i of the web page from where it is extracted. the web document from url i , may contain a list of metadata object-predicate pairs (obj 1 , pred 1 ), ..., (obj m , pred m ). the detailed approach of obtaining the pairs is presented in section (3.1). each predicate pred j consists of a root r j and a property pro j (e.g., recipe and cook-time for /recipe/cooktime, respectively). we denote the path between r j and pro j as pt j . ## experiments and analysis this section first presents the qa dataset that is used for evaluation, and then describe results comparing different methods (with or without leveraging the metadata information). ## conclusion we demonstrate benefits of incorporating metadata information from web pages for improving answer passage selection model. we describe methods for obtaining metadata and decorating passages with metadata object-predicate pairs, and a finegrained encoding strategy for leveraging metadata information in neural models. for future work, we'll investigate metadata for other tasks such as web entity linking and extraction.