Structured Prediction for Joint Class Cardinality and Entity Property Inference in Model-Complete Text Comprehension (2020)

Abstract: Model-complete text comprehension aims at interpreting a natural language text with respect to a semantic domain model describing the classes and their properties relevant for the domain in question. Solving this task can be approached as a structured prediction problem, consisting in inferring the most probable instance of the semantic model given the text. In this work, we focus on the challenging sub-problem of cardinality prediction that consists in predicting the number of distinct individuals of each class in the semantic model. We show that cardinality prediction can successfully be approached by modeling the overall task as a joint inference problem, predicting the number of individuals of certain classes while at the same time extracting their properties. We approach this task with probabilistic graphical models computing the maximum-a-posteriori instance of the semantic model. Our main contribution lies on the empirical investigation and analysis of different approximative inference strategies based on Gibbs sampling. We present and evaluate our models on the task of extracting key parameters from scientific full text articles describing pre-clinical studies in the domain of spinal cord injury.

## introduction while there has been significant progress on information extraction tasks with a comparably low level of structural complexity such as entity recognition @xcite , relation extraction @xcite , and co-reference resolution @xcite , there is not much progress on capturing the comprehensive meaning of a text with respect to a given semantic model in terms of a given vocabulary of classes and properties. we refer to this task as model-complete text comprehension (mctc) which requires to put all the above mentioned classical nlp-tasks into a larger context. the goal of mctc is to capture all the information in the text that is expressible with respect to the semantic model, while ignoring those meaning aspects which are not. this can be framed as a structured prediction problem consisting in inferring the most plausible instance of the semantic model. one challenging problem in mctc lies in the prediction of the correct number of individuals for each class, hereinafter referred to as cardinality prediction, that is answering the question(s): "how many (and which) individuals of a class are mentioned in the text?". in essence, this can be approached by grouping mentions of known realworld entities into equivalence classes, which has widely been addressed under the heading of coreference resolution @xcite . however, in many problem domains, we need to identify equivalence classes of entities that are priorily unknown (in terms of not referring to a specific real-world entity). thus, explicit mentions in text such as naming variations etc. can not be directly mapped to a set of existing entities. to the contrary, such entities are only distinguishable on the basis of their describing properties. take the case of scientific publications concerning pre-clinical studies containing a variable number of experimental groups each of which is described by an injury model, an animal species, treatments etc. here, mentions of experimental groups do not refer to existing real-world entities and they need to be inferred/grouped on the basis of their identifying properties that are mentioned in the text. we refer to the prediction of how many distinct individuals 1 of a particular class are (indirectly) mentioned in a text as cardinality prediction and solve it jointly 1 we refer to mentions of entities in a text as entities and to the denotation of such entities in a given model of the text as individuals with the prediction of the properties of each individual. we model this joint task as the task of predicting a (logical) model of the text, which involves making choices as to which individuals exist for each class. towards capturing the dependence of class cardinalities and properties, we propose a joint inference approach that infers equivalence classes of entities in a text while at the same time predicting the properties of each equivalence class. we model this task as a statistical inference problem, relying on a factorized posterior conditional distribution p( y | x) as implemented in crfs to approximate the true distribution over possible instantiations @xmath0. applying maximum-a-posteriori inference, we infer the most likely instance of the model that captures the whole meaning of the text as expressible by the semantic model. this includes the determination of the number of distinct equivalence classes (thus solving cardinality prediction) as well as predicting the properties for each equivalence class. our approach is evaluated on text comprehension of research articles describing pre-clinical studies in the domain of spinal cord injury. capturing correct key-parameters of the study protocol can be modeled as an mctc problem as it requires a comprehensive understanding of the text rather than extracting single binary relations only. in this domain, we focus in particular on the extraction of experimental groups and their properties as described in section 4.1. the data set foot_0 and the source code foot_1 are public available. in this work, we answer the following research questions: ## related work there are a number of traditional natural language processing tasks related to model-complete text comprehension. in this section, we briefly discuss each task and provide some pointers to systems addressing the corresponding task, focusing on the bio-medical domain. entity recognition and linking (ner+l) describes the task of finding entity mentions in a text and linking them to unique concepts in some knowledge base. the task originated in the context of information extraction, consisting of identifying persons, company names etc. @xcite but has also received prominent attention in the biomedical field focusing on entities such as genes, diseases, treatments, etc. @xcite . ner+l is an important preliminary step in many downstream applications as it identifies core informational units that are needed for more complex analysis levels including relation extraction, slot filling, and mctc. relation extraction (re) describes the task of detecting relations between entities mentioned in a text @xcite . while many models rely on a pipeline architecture predicting entities first and then predicting relations, more recent works model both tasks jointly @xcite . although there has been notable progress on re in the last years, the task has been typically restricted to extracting binary relations within single sentence boundaries only @xcite . with our work, we strive to go beyond such simplifications towards document-level text interpretation with respect to a more complex model. co-reference resolution (crr) describes originally the task of finding nouns and pronouns that refer to the same underlying entity @xcite . when applying crr to the medical field, the task shifts towards the resolution of mentions of diseases, tests, compounds, groups, treatments, etc. @xcite . cardinality prediction in isolation can be modeled as a crr problem, where the number of distinct non co-referring entities need to be found. with regard to the goal of comprehensive text understanding, classical co-reference resolution is clearly not enough, as also the properties of each entity need to be extracted. while singh et al. @xcite have attempted to model the tasks of entity recognition, relation extraction and co-reference resolution jointly, in their approach the interaction between relation extraction and co-reference resolution is not modelled directly, only via entity tags. in our approach we model the joint interaction between inducing equivalence classes (resolving co-references) while extracting the properties of entities/individuals as a basis to inform the decision about whether two individuals are the same (thus co-refer) given their properties. durret et al. @xcite propose a global inference entity-level modeling for classical co-reference resolution based on a rich factor graph. in the unrolled factor graph, each factor refers to one entity property defined on a semantic or syntactic linguistic basis. in contrast to this work where properties of an individual/entity are pre-defined by the semantic model. thus, our focus lies in their joint exploration while learning their interplay during inference in order to decide whether the properties belong to the same individual or not. @xcite propose an unsupervised generative model incorporating several linguistic properties of the entity and its mention. in contrast, our work does not rely on entities that are explicitly mentioned in text. instead, our model follows the schema of a semantic model to reason about the existence of individuals that can be inferred from the text and groups these individuals into groups by way of inferring the properties of these individuals. the task of slot-filling (sf) was first introduced in the message understanding conference @xcite . it is concerned with predicting an entity-centric structure having a set of relations to other entities as it can be found e.g. in ontology-based information extraction (sanchez-cisneros and aparicio gali, 2013; @xcite or extracting info-boxes from wikipedia articles @xcite . contrary to mctc, classical slot-filling requires the prediction of a single structure per document only, which heavily reduces relational complexity and does not include nested individuals. there are many approaches to sf ranging from relying on distant supervision as described by @xcite to, more recently, neural approaches as described by zhang et al. @xcite . finally, sf can be seen as an upstream process for (cold-start) knowledge base population as described by ter horst et al. @xcite . our work is highly related to information extraction systems in the (bio-) medical field. when it comes e.g. to the prediction of key parameters of clinical studies, most work focuses on the extraction of pico-concepts: patient/problem (p), intervention (i), comparison (c) and outcome (o). summerscales et al. @xcite have applied conditional random fields to extract key parameters from abstracts of clinical studies including treatments, experimental groups, and outcomes. contrary to our approach, the task is defined as an ner+l problem, not aiming at capturing the semantic relations and concepts. trenta et al. @xcite have proposed to rely on a maximum entropy classifier jointly extracting fine grained pico elements from abstracts. brujin et al. @xcite combined an svm-based text classifier with regular expressions to extract pico elements. further, ferracane et al. @xcite aim to leverage co-reference resolution to identify experimental groups (patients) from medical abstracts. however, none of these works aims at deeper extraction of arms/experimental groups and their properties. in general, most approaches in the literature focus on sentence extraction and classification only @xcite @xcite rather than on predicting a semantic structure. ## method structured prediction describes a variety of tasks with the goal of predicting a pre-defined target structure that is extracted from an unstructured input text @xcite . we formulate the mctc problem as a structured prediction task, where the structure to be predicted is an instance of the semantic model capturing the meaning of a text. this involves the task of predicting the number of individuals of each class (cardinality prediction) as well as predicting the values of the key properties of each individual. our proposed method relies on probabilistic graphical models i.e. conditional random fields (crfs; @xcite ) as their application is well established in many structured prediction tasks in the context of nlp. encoding semantic models: an instance of the semantic model is encoded as a nested vector y containing as many elements as there are classes and properties in the model. thus, given a set of classes {c 1 , . . . , c n } and a set of properties @xmath1, . . . , p m }, y can be written as { v c 1 , ..., v cn } where each v c i has the form [|c i |, i i 1 , . . . , i i m ]. |c i | represents the cardinality of class c i , i.e. the number of individuals of class c i mentioned in the text. i i j ⊆ p is a vector describing an individual of class c i in terms of its properties. example: consider a semantic model consisting of two classes c 1 and c 2 where individuals of class c 1 have properties hasa, hasb, hasc, and individuals of class c 2 have properties hasd, hase. one specific instance of the semantic model would be represented as: [[2, [a 1 , b 1 , c 1 ], [a 1 , b 1 , c 2 ]], [1, [d 1 , e 2 ]] ## experiments model-complete text comprehension aims at the automatic instantiation of a semantic model based on information extracted from a natural language text. such an instance contains information about individuals of equivalence classes, their cardinality and their properties. thus the overall task of mctc can be evaluated towards i) the correct prediction of the number of individuals, and ii) the prediction of properties for each individual. in the following, we describe our use case application, the experimental procedure and results. ## conclusion we have proposed an approach to the task of modelcomplete text comprehension (mctc) that relies on a learned model of the posterior distribution of instances of a semantic model given a text to infer the most likely instance of a semantic model that captures the meaning of the text best. we have relied on crfs to model the conditional distribution in a factorized way and empirically investigated the impact of different approximate inferences strategies on our problem. our experiments on the task of predicting the structure of experimental groups from scientific full text articles describing pre-clinical studies in the field of spinal cord injury show that modeling the mctc task as a joint inference problem, extracting the cardinality in combination with predicting the properties of the individuals, outperforms a number of reasonable baselines predicting the cardinality alone. in future work, we intend to investigate combinations of our inference strategies, relying on the result state produced by our par + inference strategy to seed the rss inference method to re-sample the property values, expecting to see an overall gain in both cardinality prediction and entity property prediction over both inference strategies.