Extraction of Common-Sense Relations from Procedural Task Instructions using BERT (2021)

Abstract: Manipulation-relevant common-sense knowledge is crucial to support action-planning for complex tasks. In particular, instrumentality information of what can be done with certain tools can be used to limit the search space which is growing exponentially with the number of viable options. Typical sources for such knowledge, structured common-sense knowledge bases such as ConceptNet or WebChild, provide a limited amount of information which also varies drastically across different domains. Considering the recent success of pre-trained language models such as BERT, we investigate whether common-sense information can directly be extracted from semi-structured text with an acceptable annotation effort. Concretely, we compare the common-sense relations obtained from ConceptNet versus those extracted with BERT from large recipe databases. In this context, we propose a scoring function, based on the WordNet taxonomy to match specific terms to more general ones, enabling a rich evaluation against a set of ground-truth relations.

## introduction lately, ai-based methods advanced rapidly in their capabilities leading to the tackling of ever more challenging tasks. this progress is expected to continue and to potentially culminate in general-purpose intelligence within a few decades @xcite . however, current systems are designed for highly specific tasks and lack crucial capabilities for their application in a broader context. in particular, they have insufficient common-sense knowledge and reasoning capabilities. common-sense knowledge are essential facts humans acquire throughout our life and which are frequently applied for everyday tasks, mostly in a subconscious manner. however, such knowledge is rarely expressed as it is unnecessary to state the obvious, making it highly elusive. this paper addresses this challenge and focuses on the acquisition of common-sense knowledge. concretely, we are interested in manipulation-relevant knowledge that can support action planning, answering questions such as "what do i need to cut a bread?" or "what can i do with a knife?". the acquisition is framed as a relation-extraction task where we focus on the instrumentality relations between tools, actions and objects in the kitchen domain. the classical approach to acquire common sense knowledge is to query publicly available knowledge bases. however, since common-sense information is scarcely expressed, there exist only a few dedicated sources @xcite . the most prominent one is the conceptnet knowledge graph @xcite , which is widely used in various applications @xcite @xcite . it connects words and phrases of natural language with labeled assertions, so-called predicates. the main issue of conceptnet and similar sources is that the provided amount of information is rather limited, due to the fact that crowd-sourcing is an ineffective strategy for collecting common-sense information. the incentive for the public to contribute is weak as the information by definition is common sense and widely known. furthermore, the amount and granularity of the information varies substantially across different topics, making it rather unclear to assess its relevance for specific applications. recently published language models such as bert or gpt2 @xcite constitute a drastic leap forward in the domain of natural language processing (nlp) as they distinctly improved benchmarks in the tasks of language translation, question answering, named entity recognition and many more. these large neural networks are pre-trained on a massive amount of unsupervised data and can be fine-tuned to different tasks based on comparably few labeled examples. considering their success story, it is interesting to investigate their effectiveness in the extraction of common-sense information from widely available text databases. such an approach is scalable as the models can easily process additional sources. in this paper, we apply this concept to acquire instrumentality relations from procedural task instructions, more specifically recipes. procedural task instructions are one of the few sources where common-sense knowledge is made explicit because they aim to instruct humans to perform a task they are potentially unfamiliar with. concretely, we finetune bert to learn the relation extraction using a few labeled examples and compare the yielded relation set against the one of conceptnet. the evaluation is based on a set of ground-truth relations, which we collect in a study. in this context, we propose a scoring function to match specific terms to more general ones based on the wordnet taxonomy. the extensive evaluation underlines the effectiveness of bert, leading to distinctly more relations with an acceptable proportion of false relations that can flexibly be adjusted with standard filtering techniques. ## related work the lack of common-sense knowledge and reasoning capabilities was recently addressed in darpa's "machine common sense" initiative @xcite and led to an increased attention within the research community. various new sources for common-sense knowledge have been established since. atomic @xcite , for example, provides a database with causes and effects of common everyday actions such as making a coffee. webchild @xcite is a commonsense knowledge graph that provides in contrast to other sources also comparative knowledge using relations such as "larger than" between different concepts. it does not rely on crowd-sourcing, but instead uses different algorithms to accumulate the knowledge on the basis of large text corpora. databases for visual common-sense have been recently proposed by @xcite . sources derived from wikipedia such as db-pedia @xcite , yago @xcite or wikidata @xcite have often been used to extract commonsense knowledge from. jebbara et al. (2019) proposed multiple score-functions in to rank relations that encode prototypical locations of objects. they relied on crowd-sourcing, dbpedia and annotated image databases to generate ground truth relations to evaluate their methods. manipulation-related knowledge is particularly interesting in the field of robotics, where explicit action representations are based on relations between one action and the manipulated object @xcite . such relations are extracted from video @xcite ), text data @xcite or even multiple modalities @xcite . common-sense knowledge is tackled in a broad range of topics. zhou et al. tackled temporal common-sense by proposing dedicated datasets and a specific language model that outperforms bert on the task of classifying typical events according to their temporal properties @xcite . common-sense properties of word embeddings were extracted by @xcite using a zero-shot learning approach. this enables a property-based comparison of entities to answer questions like "is an elephant bigger than a tiger?". @xcite augmented the entities contained in the squad dataset @xcite with common-sense knowledge from con-ceptnet and wordnet, allowing them to answer a variety of additional questions about the entities. recent work focuses on the extraction of action effects, i. e. how does the object state changes when certain actions are applied @xcite . in this regard, the action context is often encoded as well @xcite @xcite , which is similar to the linguistic concept of verb semantics @xcite . verb semantics describe the meaning of a verb within a context depending on the "agent" (the one executing the action), the "patient" (here the object on which the action is applied on) and an instrument (the tool used for manipulation). in contrast to the mentioned work, our contribution provides three novel aspects. first, it in-vestigates the viability of common-sense relation extraction using pre-trained models that are finetuned with a very small amount of labeled examples for a specific application. second, it measures the relevance of the extracted relations based on their coverage of a ground-truth relation set, thereby proposing a scoring function to consider the matching between specific and more general terms. lastly, it compares the relation set against the one contained in conceptnet, which provides insight into conceptnet's practical relevance for the specific application. ## approach we are interested in acquiring instrumentality relations for the kitchen domain. specifically, we want to know the relevant tools for certain tasks. for instance, a knife can be used for cutting bread, but a cutting board may be helpful as well. a relation @xmath0, a, o) is a triplet consisting of three strings, where t encodes one tool and the associated task is described by action a and object o. some examples for relevant relations are (knife, cut, bread), (fridge, cool, food), and (bowl, mix, salad). we use bert to extract such information from large text corpora, where the text is loosely structured. in the past, powerful models required a large amounts of labeled data to achieve a reasonable performance, making such approaches not applicable for most applications. however, pre-trained models have drastically reduced the label-burden and simultaneously increased their performance. the main advantage of this approach in comparison to the extraction from structured database is its scalability. once the model is trained it can easily be applied on vast amounts of available text to harness the desired information. hence, more relations can be extracted with a higher language variety as are contained in current common-sense databases. furthermore, it can be applied on other domains as long as text corpora cover the relevant relations. the disadvantage is the necessity of annotating some examples for the specific application. in the following, we describe the approach in detail and also propose an evaluation metric to measure the match between two relation sets. this is crucial to determine the relevance of the extracted relations according to a set of ground-truth relations. ## experiments initially, we discuss the relation extraction from recipes using bert. this evaluation is based on the small set of manually labeled instructions. subsequently, we analyze how well these extracted relations match the set of ground-truth relations in comparison to those of conceptnet. ## conclusion we explored whether bert can be used to extract common-sense relations from procedural task instructions as an alternative to querying public databases. we fine-tuned bert for the relation extraction from recipes based on very few labeled instructions and extracted the relations from the large recipe1m+ dataset. to assess their relevance we collected a set of ground-truth relations in a study and proposed an evaluation measure that utilizes the wordnet hypernym graph to incorporate matches between specific and general terms. the matching granularity can naturally be adjusted, allowing a diverse analysis. the experiments highlight various advantages of the bert based approach. it does not only yield a very large amount of unique relations (28k versus 1.3k) and correspondingly matches a large portion of the groundtruth relations, but the recurrence of the relations can also be used to reduce the proportion of false relations. therefore, we regard the extraction of common-sense relations from text as a competitive and complementary approach, particularly considering the ongoing and rapid advance of nlp techniques.