Predicting Informativeness of Semantic Triples (2021)

Abstract: Many automatic semantic relation extraction tools extract subject-predicate-object triples from unstructured text. However, a large quantity of these triples merely represent background knowledge. We explore using full texts of biomedical publications to create a training corpus of informative and important semantic triples based on the notion that the main contributions of an article are summarized in its abstract. This corpus is used to train a deep learning classifier to identify important triples, and we suggest that an importance ranking for semantic triples could also be generated.

## introduction subject-predicate-object triples are used in numerous natural language processing areas, including question answering (e.g. @xcite ), ontology building (e.g. @xcite ) and literature based discovery (e.g. @xcite ). while they can be thought of as representing the minimum unit of semantic expression, there is a large degree of variability in the amount of new (not commonly known) content they convey. on the one hand, they sometimes represent what can be termed background knowledge, for example "new zealand -isa -country" or "pharmaceutical services -treats -health personnel", while on the other, they may describe very specific findings such as pimobendan treats hypertrophic cardiomyopathy or lcn2 protein, human -asso-ciated with -chronic kidney disease. we use biomedical publications to test the hypothesis that training data consisting of such, important, triples can be created from abstracts, and train a deep learning algorithm to identify these high importance triples from a list of all triples appearing in a paper. the system could also be adjusted to output a weight instead of a binary decision, allowing for an importance ranking of semantic triples within an article. the paper begins with an overview of related work in section 2, the experimental set-up follows in section 3, with the results and discussion in section 4 and conclusions drawn in section 5. ## background a number of tools for automatically extracting semantic relations -(subject, relation, object) triples -from unstructured text exist @xcite . however, as @xcite point out, the majority of works incorporating these do not perform much pre-or post-processing and therefore include many potentially uninformative triples, and works proposing to extend currently existing collections of semantic relations often speak of extending the set of relations, not refining the relations present (e.g. @xcite ). evaluations of semantic relation extraction systems are often very comprehensive, e.g. @xcite present a detailed independent evaluation of semrep -a biomedical domain tuned triple extraction tool -and discover common sources of error for this tool, but such evaluations do not quantify the quality of the triple that is retrieved by the system. it is unclear whether the incorrectly extracted triples are uninformative, or the opposite. while not phrased as focusing on informative / important triples, existing works often restrict to particular types of relations: @xcite evaluate the extraction of health claims, defined as a relation between something that is being manipulated and something that is being measured (e.g. the relation between a substance and a disease). @xcite restrict to drug-drug interaction, protein-protein interaction, and medical concept relation extraction, while @xcite focus on mechanisms, i.e. activities, functions and causal relations. such restrictions are likely to increase the overall quality of the remaining triples: removing the isa relation alone eliminates a large quantity of background knowledge. the closest to our work is due to @xcite who filter out uninformative triples computationally, based on the difference between triples' expected and observed frequencies. ## results and discussion a 5-fold cross validation was performed, and each explored model was trained on (a possibly balance adjusted version of) the training portion giving rise to an accuracy or f-measure on the test portion. this allows an average to be computed and the best model to be determined. the results are presented in with under-sampling, the accuracies for @xmath0, supporting the hypothesis regarding frequent use of synonyms. to avoid a uniform assignment of the majority class, the f-measure metric (which rewards both precision and recall) is used in models without under-sampling. an f-measure of 1 represents perfect precision and recall, and the highest f-measure achieved is 0.975. shapley additive explanations (shap) @xcite uses ideas from game theory to explain feature contributions to machine learning decisions. figure 1 depicts the feature contributions on a randomly selected sample of 100 triples for the best model without under-sampling. each dot represents a single triple, with the intensity (@xmath1. the horizontal position indicates whether the contribution caused the prediction to go up -towards being classified as an important triple -or down. the top three rows show expected results: that high values in the number of relations in the document, very frequently occurring cuis or relations arising from sentences low in importance ranked by textrank (giving a high rank) impact the prediction very negatively. unsurprising positive contributors are: 1) the frequency of the triple in the document: a new contribution may be reiterated in the document, 2) the triple appearing in the conclusion: this often contains a summary of contributions, 3) the triple including the treats relation: the filtering ensures this is a new triple and being treatment specific, is likely the focus of the work, 4) the triple appearing in the introduction, where the novelties of the work are often highlighted. the contributions of a higher depth value is also, as expected, positive. contributions are also due to the cuis' umls source vocabulary (indicated by source ). in some cases, these categorize the cui: for example, aod (alcohol and other drug thesaurus) and psy (psychological index terms) are not unexpected. surprising may be the pair mshnor and mshjpn, representing the norwegian and japanese translations of medical subject headings, as they appear to have opposite effect. however, mshjpn's contribution is very limited, suggesting that its completeness may not match that of mshnor. ## conclusions and future work we have demonstrated that a dataset of semantic triples created from full articles based on similarity between triples in the body of the text and triples in the abstract can be used to train a deep learning classifier to make predictions about a semantic triple's importance. an analysis of feature contributions was also performed. while a direct prediction of the similarity score appeared difficult with the quantity of data available, converting the similarity scores into categorical values may be trainable and would provide the basis of a ranking. again with greater quantity of data, features based on medical subject headings of each cui could be beneficial indicated by the success of the umls source vocabulary features. the work undertaken was in the biomedical domain based on a tool tuned for biomedical domain grammatical relation extraction. porting the approach to another domain, where subject-verbobject triples would need to be extracted using a generic grammatical relation extraction algorithm and some features would require re-engineering, would also form an interesting extension of the work.