Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning (2020)

Abstract: Open attribute value extraction for emerging entities is an important but challenging task. A lot of previous works formulate the problem as a question-answering (QA) task. While the collections of articles from web corpus provide updated information about the emerging entities, the retrieved texts can be noisy, irrelevant, thus leading to inaccurate answers. Effectively filtering out noisy articles as well as bad answers is the key to improve extraction accuracy. Knowledge graph (KG), which contains rich, well organized information about entities, provides a good resource to address the challenge. In this work, we propose a knowledge-guided reinforcement learning (RL) framework for open attribute value extraction. Informed by relevant knowledge in KG, we trained a deep Q-network to sequentially compare extracted answers to improve extraction accuracy. The proposed framework is applicable to different information extraction system. Our experimental results show that our method outperforms the baselines by 16.5 - 27.8%.

## introduction numerous entities are emerging everyday. the attributes of the entities are often noisy or incomplete, even missing. in the field of electronic commerce, target attributes (e.g., brand, flavor, smell) of new products are often missing @xcite . in medical analysis, attributes like transmission, genetics and origins of a novel virus are often unknown to people. even in dbpedia, a well-constructed and large-scale knowledge base extracted from wikipedia, half of the entities contain less than 5 relationships @xcite . a method that is capable of supplementing reliable attribute values for emerging entities can be highly useful in many applications. although information extraction methods have been extensively studied, the task of open attribute value extraction remains challenging. first, the emerging entities may have new attribute values that are absent in the existing kg. under such circumstances, the prediction methods under the closed-world assumption and the methods that cannot utilize external information are not well suited due to their limited recalls. second, while web corpus can be used as a good resource to provide relatively updated and relevant articles for large varieties of emerging entities, the articles retrieved from web corpus can be noisy and/or irrelevant, which in turn leads to a limited precision. finally, even when articles are relevant, the extracted answers might still be inaccurate due to the errorprone information extraction model. to effectively filter out noisy answers that are obtained either due to the irreverent articles or the errors incurred by the information extraction system, we pose the following two questions: first, how many articles should we collect from the enormous web corpus? second, how to select the most reliable value out of the pool of all the possible answers extracted from the articles? there is no common answer to the first question that works for all triplets because of the inconsistent degrees of difficulties in finding the correct attribute values. the decision of when to stop querying more external articles needs to be made after successive evaluations of the candidate answers. thus the decision making process is inherently sequential. reinforcement learning (rl) is a commonly adopted method to deal with sequential decision problems and has been widely studied in the field of robotic and game @xcite . but there are not many researches on open attribute value extraction with rl. one existing literature of rl-based method for value extraction is proposed by @xcite . in their work, a rl framework is designed to improve accuracy of event-related value extraction by acquiring and incorporating external evidences. however, their approach requires a great amount of context information about the specific event of interest during the training process. it is not trivial to extend their framework for open attribute value extraction, because we would need to collect context words and train a new model with annotated data for each emerging attribute. therefore, their framework cannot be generalized to open attribute value extraction task when various entities and attributes are involved. while using the context words to construct the states in rl is not suitable in our task, our solution is to leverage the rich, well-organized information in kg, which is not only informative but also generalizable. such information can be leveraged in answer comparisons, which addresses our second question. for example, to fill the incomplete @xmath0, display resolution, ?>, from the kg we may find that the attribute values "display resolutions" of an entity that is under category "phone" is commonly expressed in the format of "xxx by xxxx pixels", where x stands for some digit. the typical instances of the attribute values for entities under the same category provide valuable background information. in this paper, we propose a knowledge-guided rl framework to perform open attribute value extraction. the rl agent is trained to make good actions for answer selection and stopping time decision. our experiments show that the proposed framework significantly boosts the extraction performance. to the best of our knowledge, we are the first to integrate kg in a rl framework to perform open attribute value extraction in summary, our contribution are in three folds: ## overview problem definition we denote the entity-attribute-value triplet @xmath1, r, v >. the goal is to find the attribute value in an incomplete @xmath2, r, ? >. to achieve this purpose, we pose a question generated with a pre-defined template to search engine to obtain relevant articles. for example, to fill the incomplete @xmath3, core code, ?>, we retrieve articles with the query "what is the core code of gtx1080?". an information extraction system, such as qanet @xcite , is used to extract a candidate answer with a certain confidence score from an article. however, due to the inconsistent qualities of the online articles and the inevitable errors caused by the information extraction system, the results of extracted from only one online article is not satisfactory in many cases. another source of information that can be leveraged help to fulfill such a task is a kg. while it is hard to find out the attribute values for an emerging entity given the existing ones, the kg can serve as the background knowledge about the attributes. we approach the problem using a reinforcement learning framework that is illustrated in the next section. ## reinforcement learning for open attribute value extraction the attribute value extraction task is modeled as a markov decision process (mdp), where the rl agent is actively engaged in the decision making process to maximize the reward, which measures the correctness of the extracted attribute values. the mdp is modeled as a tuple (s, a, t, r), where @xmath4; @xmath5; t (s |s, a) refers to a transition function that maps the domain of state and action to a probability distribution of states; r(s, a) is a reward function that maps the domain of state and action to a real number, which is encoded such that the higher value the better. we describe our rl methodology by illustrating these components as follows. action and transition at each decision stage, the agent will observe two candidate answers from two articles and make decisions to answer the two questions: (i) which answer is better out of the two? (ii) should the agent stop at the current best answer or continue querying more articles? at the initial decision point, two candidate answers are obtained from two articles simultaneously queried from the web, where we arbitrarily assign one of them to be the current best answer and the other the new candidate answer. we define the following three actions in a: at all subsequent decision points, we will retain or replace the current best answer and continue comparing with the new candidate answers queried from the web until the action is "stop". state at each decision point, the state is constructed by concatenating the following three components, where different sources of information are combined. ## experiments in this section, we compare our proposed rl framework to the state-of-the-art extraction-based baselines, demonstrating its robustness and ability to obtain accurate answers for missing attribute values. our codes are publicly available online. 3 ## conclusion and discussion this paper presents a novel rl framework to perform open attribute value extraction. through a set of experiments, we observe that the most of the computation cost is incurred by training the information extraction system. the remaining computation cost from rl framework is comparably small during both the training and the prediction process. specifically, during our experiments, we trained a three-layer deep neural network model, which has much fewer parameters compared to the information extraction system. the proposed rl method demonstrates promising performance, where the kg showed its ability to provide guidance in open attribute extraction task. our framework also contributes to areas of knowledge graph completion and automatic question-answering for attribute values. kg has huge potential to provide rich background information in many nlp applications. our solution for attribute value extraction can be extended to other nlp tasks. a potential attempt might be to use kg to design the reward in the rl framework to provide weak supervision. we leave this as our future work.