Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers (2022)

Abstract: Identifying and integrating missing facts is a crucial task for knowledge graph completion to ensure robustness towards downstream applications such as question answering. Adding new facts for a knowledge graph in real world system often involves human verification effort, where candidate facts are verified for accuracy by human annotators. This process is labor-intensive, time-consuming, and inefficient since only a small number of missing facts can be identified. This paper proposes a simple but effective human-in-the-loop framework for fact collection that searches for a diverse set of highly relevant candidate facts for human annotation. Empirical results presented in this work demonstrate that the proposed solution leads to both improvements in i) the quality of the candidate facts as well as ii) the ability of discovering more facts to grow the knowledge graph without requiring additional human effort.

## introduction a knowledge graph (kg) is an efficient way of storing information and relations between different types of entities, and is an integral part of many real-world applications such as question answering @xcite . however, incompleteness is a well-known issue that inherently exists in a kg caused by missing facts and entities, and it significantly limits the capability of the downstream applications @xcite . since richness of the facts in a kg can directly impact its quality, identifying and collecting missing facts is a crucial task. additionally, it is essential to ensure that the facts being added to kg are verified to be highly accurate. there is research work published on how to automatically identify missing facts using machine learning models @xcite . many of them use embedding models over the entities and relations in a kg to train link prediction models to * work done during an internship at apple. predict missing facts within the kg @xcite , while some use a searchbased question-answering approach to get candidate answers from the web and then identify the correct missing fact by aggregating the found answers @xcite . these approaches can automatically identify a large number of missing facts, but there was usually no human verification (which is also infeasible given the volume of the identified facts) to make sure that the newly discovered facts meet real-world-system-level accuracy. collecting high-quality new facts for a live kg is a very complicated process that often requires human effort. for example, the success of finding the birth date of a person from the web heavily depends on the popularity of the person and the uniqueness of their name. on the other hand, open access to the web makes it hard to protect the accuracy of its information; even wikipedia suffers from vandalism @xcite . this influenced our initial design of a fact collection framework with a human-inthe-loop component. the system starts with an unanswerable query, leverages state-of-the-art technologies to identify the intent and entity of the input query, retrieves a few candidate answers via web search using a natural language based questionanswering (qa) system, then instead of using a machine learning model to aggregate the results, e.g., @xcite , we direct the candidate facts to human annotators for review. while this pipeline is successfully deployed in a real world system, there is an opportunity for improvements to be made. we observed that the natural language-based qa system sometimes returns no candidate answer for a query, leading to a low coverage problem. on the other hand, even if the qa system returns some candidate answers for certain queries, many of them do not contain the correct answers. in such cases, the human annotations ended up with negative responses, indicating a waste of human labor. to address this problem, one option is to train a better qa system either by using more data or adopting more complicated models, which needs expensive engineering efforts. another option is to increase the number of candidates returned by the qa system and involve more human annotators for fact verification, but because of the low-correct-ratio nature of the candidate answers, this approach would mean an even more severe waste of human effort. in this paper a new human-in-the-loop fact collection framework is proposed that addresses the aforementioned problems without training new qa systems nor adding additional human annotators. to solve the low coverage problem of the qa system, the input query is perturbed by generating semantically equivalent variations of it. these variations are then input to the qa system to get diverse answer sets (similar to what has been done in @xcite ). then, to address the lowcorrect-answer-ratio problem, a candidate answer selector is introduced, inspired by the well-known uncertainty-based active sampling strategy queryby-committee (qbc) @xcite @xcite , to choose the most relevant answers for human annotation. to summarize, the main contributions of this work are: the rest of this paper is organized as follows: sections 2 describes our human-in-the-loop fact collection framework. section 3 demonstrates experimental results. section 4 reviews related work. finally, concluding remarks and future works are included in section 5. ## methodology architecture of the proposed system is illustrated in figure 1 . user query answering is generally carried out by a two-stage model that first extracts the relevant information relating to the intent of the query from a web page, then ranks and selects the most relevant articles that match the entity and the intent of the query. this process can be improved by fine tuning the model used, though it requires quality data and is time-consuming. the proposed framework does not rely on fine tuning but makes changes to the input of the model. the system leverages the sensitivity of language based models to the input and combines it with a selector that works on the principle of query-by-committee. the rest of the section shall illustrate each component of the proposed pipeline via a concrete example. ## related work in @xcite , the authors adopted a very similar idea of perturbing the input query, then they used a search-engine with open web access to identify diverse answers, and finally aggregated the answers using scoring model to identify the most likely answers. although this work is in same spirit, the proposed approach is different from theirs in two ways: first, a much simpler aggregation mechanism without the need to train a scoring model is used, thus the proposed approach is easier to implement and can work with any qa system; second, human-in-the-loop verification is included to ensure the high quality of identified facts. in addition to improving the human annotation effectiveness, reducing human annotation efforts is another related research topic. a lot of the work try to solve this problem by introducing a deep learning or machine learning model @xcite . certain papers also aim to reduce the annotation work by selecting only samples that are optimal to training the model @xcite . the proposed solution is different from these works as this solution does not train new machine learning models to reduce annotation efforts rather uses selective sampling. the use of qbc to identify relevant answers is also related to the ensemble learning paradigm, where the goal is to improve the predictive performance of a single model by training multiple variations and combining their predictions @xcite . in fact, qbc can be viewed as a way to create the ensemble with the specific purpose of using it to identify uncertain examples for active learning @xcite . both qbc and ensemble learning require creating a group of slightly different models so that a diverse predictions or decisions can be later combined or integrated to produce more accurate prediction. the proposed approach was inspired by this idea, and unlike qbc whose primary focus is to identify uncertain examples for human annotation to decrease the uncertainty of the model @xcite , the goal here is to identify certain examples. ranking web search results is an important topic in elevating user experience. there are various approaches that try to solve this problem. aggregated search results and re-ranking based on similarity is one of the solutions @xcite . while this work takes inspiration from the aggregated search results, the proposed solution differs in the re-ranking aspect. similarity of web pages is not checked for, instead the voting mechanism in the qbc selector to find the best potential answers. ## conclusion & future work this paper proposes a new framework that aims to reduce the efforts of human annotators in the fact collection process for enriching a knowledge graph. empirical observations of the proposed solution when compared against a baseline framework demonstrate that the proposed framework has better recall, and can identify much more relevant facts. this framework is easy to implement, provides an additional automated and scalable layer of enrichment to web answer retrieval, and gives a significant boost to the fact collection process in terms of quality and coverage. future work shall include testing the proposed framework within other domains such as open domain question answering. moreover, this framework can significantly boost the recall, which can help in collecting more labeled data to train a model to predict the reliability of a website given a query's intent. in this way, the qa system can be enhanced to provide more relevant candidate answers.