Automatic Entity State Annotation using the V erb N et Semantic Parser (2021)

Abstract: Tracking entity states is a natural language processing task assumed to require human annotation. In order to reduce the time and expenses associated with annotation, we introduce a new method to automatically extract entity states, including location and existence state of entities, following Dalvi et al. (2018) and Tandon et al. (2020). For this purpose, we rely primarily on the semantic representations generated by the state of the art VerbNet parser (Gung, 2020), and extract the entities (event participants) and their states, based on the semantic predicates of the generated VerbNet semantic representation, which is in propositional logic format. For evaluation, we used ProPara (Dalvi et al., 2018), a reading comprehension dataset which is annotated with entity states in each sentence, and tracks those states in paragraphs of natural human-authored procedural texts. Given the presented limitations of the method, the peculiarities of the ProPara dataset annotations, and that our system, Lexis, makes no use of task-specific training data and relies solely on VerbNet, the results are promising, showcasing the value of lexical resources.

## introduction question answering in reading comprehension tasks focusing on procedural texts (i.e. texts describing processes) is particularly challenging in natural language processing (nlp), because this type of text describes a changing world state @xcite @xcite @xcite . tracking the state of entities in such texts is an important task to enable proper question answering in reading comprehension tasks. the challenging part in such question answering tasks is not where answers are explicitly mentioned in a sentence @xcite . for example, in figure 1 , the sentence with bold elements tells us explicitly that urea and carbon dioxide are located in kidneys by the end of paragraph: blood delivers oxygen in the body. proteins and acids are broken down in the liver. the liver releases waste in the form of urea. the blood carries the urea and carbon dioxide to the kidneys. the kidneys strain the urea and salts needed from the blood. the carbon dioxide by product is transported back to the lungs. q: does blood enter the kidney? a: yes. this sentence. however, the fact that the blood itself will also be located at the kidneys is implicit in the semantics of the verb "carry", in which the agent of the action (mover) moves along with the theme (thing moved). therefore, answering the question in figure 1 requires knowledge of the particular semantics of this verb. this is the kind of inference our system, lexis, is able to model using verbnet. this type of reasoning requires event extraction as a first step, and event participant state extraction as a second step. our method covers both steps, but is at this point limited to sentence-level inference. in section 2, we provide an overview of the work related to this research. section 3 introduces the dataset used to evaluate lexis, as well as the details of the methods we have used. section 4 presents our experimental settings, followed by section 5 which illustrates the results of each setting. section 6 discusses the advantages of lexis, and provides an error analysis to illuminate the sources of weakness. in section 7, we conclude and briefly discuss our future work. ## data and methodology the main contribution of this work is bringing to the forefront the advantages of using lexical resource based methods. in particular, in this work, these methods are used for automatic annotation of text with labels regarding the location and existence states of entities in a given sentence. ## experiments the text spans that represent arguments are either noun phrases (for entities, e.g. sediment from the ocean in the sentence the waves contain sediment from the ocean.) or prepositional phrases (for location state tracking, e.g. in sediment in the sentence they are buried in sediment.). a human annotator labels sediment as the destination of they, but the parser labels the whole phrase in sediment as the destination. on the other hand, there are cases of conjunction, where several entities are conjoined and one predicate is stated about them. in that case, we should be able to separate the conjoined entities and generate a triple or tuple for each. for example, in the sentence algae and plankton die, two change of existence tuples should be generated: (algae, destroyed), and (plankton, destroyed). such syntactic problems were solved using the spacy @xcite dependency parser. nouns were lemmatized, the nominal heads of the noun phrases in prepositional phrases were extracted, the conjoined noun phrases were disjoined. ## results tables 2, 3, and 4 illustrate the evaluation results. in addition to the cumulative/general state tracking results, we have evaluated location tracking and existence tracking separately in order to monitor the sources of error. we have also provided the results for the relaxed setting in as expected, the inclusion of propbank argument roles (in setting 2) increases recall from 0.29 ( while performing the human judgment task for the relaxed setting, we also came across cases of gold label errors (see 'gle' in within the allen ai leaderboard for the propara task (see ## conclusion and future work the goal of this work was demonstrating how lexical resource-based methods using existing nlp resources could be utilized in automatic annotation of text for tracking entity states. the results were quite encouraging and demonstrated the potential of leveraging pre-existing rich lexical resources for challenging inference tasks. the limitations of our approach were examined, including verbnet's lack of coverage compared to propbank, another semantic role labeling lexical resource. we also found certain verbnet predicates that were particularly opaque and benefited from expansion. in addition, and as expected, using the verbnet semantic parser does not always yield a 100% accurate parse, resulting in downstream errors. in addition, there is a clear need for document-level entity state tracking, which requires automatic reference resolution. parser discussed in section 6.2, we can expect to increase the recall by backing-off to a propbank semantic parser (e.g. @xcite , that has been trained on much larger training data. this is the next step we plan to undertake. however, it is important to note that the reason we chose vnsp for this task is that it is the only semantic parser that generates the rich semantic representations in the predicate logic format that we employed in our method. therefore, another possibility that looks more promising (but is a much longer-term solution) is continuing to update the verbnet lexical resource based on the error analysis introduced in this work, annotating new data to cover examples from these updates, and re-training the verbnet semantic parser. another area for planned future work is to use the generated entity states to augment the propara data and feed it into a machine learning model to assess performance improvement. finally, as proposed in section 6.2, we will be adding a stateof-the-art reference resolution system to achieve document-level entity state tracking. all of these additions will be evaluated and reported on.