R el D iff: Enriching Knowledge Graph Relation Representations for Sensitivity Classification (2021)

Abstract: The relationships that exist between entities can be a reliable indicator for classifying sensitive information, such as commercially sensitive information. For example, the relation person-IsDirectorOf-company can indicate whether an individual’s salary should be considered as sensitive personal information. Representations of such relations are often learned using a knowledge graph to produce embeddings for relation types, generalised across different entity-pairs. However, a relation type may or may not correspond to a sensitivity depending on the entities that participate to the relation. Therefore, generalised relation embeddings are typically insufficient for classifying sensitive information. In this work, we propose a novel method for representing entities and relations within a single embedding to better capture the relationship between the entities. Moreover, we show that our proposed entity-relation-entity embedding approach can significantly improve (McNemar’s test, p <0.05) the effectiveness of sensitivity classification, compared to classification approaches that leverage relation embedding approaches from the literature. (0.426 F1 vs 0.413 F1)

## introduction more than a hundred countries have established freedom of information (foi) regulations that require public organisations, such as governments, to release their official documents to the public @xcite , for example the freedom of information act 2000 in the uk. 1 such regulations exempt the release of documents that contain sensitive information, for example personal or confidential information. therefore, all government documents must be sensitivity reviewed to identify any potentially sensitive information before the documents can be considered for public release. there is a growing need for automatic sensitivity classification approaches to assist govern-1 https://www.legislation.gov.uk/ukpga/2000/36/contents ment reviewers to sensitivity review large collections of digital documents, to comply with foi laws @xcite . however, automatically classifying foi sensitivities is a challenging task @xcite , since sensitivity is often context-dependent. for example, information about an employee's salary details may, or may not, be sensitive depending on the role of the employee (e.g., a company director's salary may be in the public domain, whereas a regular employee's salary is usually considered to be personal information). therefore, entities and the relations between entities can be an important indicator of sensitive information @xcite . we hypothesise that representing entity-relations in an embedding space can provide useful information for sensitivity classification and, in-turn, enable a sensitivity classifier to classify contextdependent sensitivities more effectively. studies such as @xcite showed that the relational information between entities in a knowledge graph can be effectively utilised to learn entity and relation embeddings. however, learning separate entity and relation embeddings may not be the most effective approach for sensitivity classification, since an entity or a relation alone is not a reliable indicator of sensitivity. this is illustrated in the example above, where the mention of a salary is potentially sensitive depending on whose salary is being discussed. therefore, to capture the context of a potentially sensitive entity-relation, we argue that there is a need to capture the whole entity-relation-entity relationship (e.g., person-isdirectorof -company) in a single embedding space. in this work, we propose reldiff : a novel approach for generating entity-relation-entity embeddings within a single embedding space. reldiff adopts two fundamental vector algebraic operators to transform entity and relation embeddings from knowledge graphs into entity-relation-entity embeddings. we show that the reldiff embeddings can be leveraged to improve the effectiveness of sensitivity classification. moreover, we leverage six popular knowledge graph embedding (kge) methods from the literature to compute reldiff embeddings and compare the effectiveness of reldiff against each of these kge methods for sensitivity classification. the contributions in this paper are three-folds: (1) we evaluate the importance of entity-relation embeddings for classifying sensitive information; to the best of our knowledge, this is the first work that effectively leverages entity-relation information for sensitivity classification. on a collection of government documents with real sensitivities (hereafter denoted as govsensitivity), we show that integrating our reldiff embeddings into sensitivity classification significantly improves (mcnemar's test, @xmath0.05) classification effectiveness, compared to several approaches from the literature that learn separate embeddings for entities and relations -e.g., rotate @xcite (f 1 0.426 vs 0.413). ## related work we now discuss related work on sensitivity classification and entity-relation representations. ## entity-relation embeddings in this section, we first present our proposed reldiff approach for generating entity-relation-entity embeddings in section 3.1. second, in section 3.2 we present knowledge graph embedding (kge) approaches that we use as baselines for the evaluation of reldiff. ## classification pipeline in this section, we present our architecture pipeline for integrating entity-relation representations into sensitivity classification. the pipeline, illustrated in figure 2 , takes two inputs: a knowledge graph with pre-trained embeddings and the govsensitivity collection containing sensitive and non-sensitive documents. the pipeline has five components: (1) the relation extraction component extracts entities and relations from the document collection and prepares a graph from the extracted relations. we present details about this component in section 4.1. ## results in this section, we present the results of our sensitivity classification experiments. ## analysis we now provide an analysis of the findings from our experiments. we discuss the effect of regularisation in the ensemble classifiers in section 7.1. in section 7.2, we describe the contribution of individual relation types on the effectiveness of sensitivity classification. in section 7.3, we discuss the importance of the improvements by reldiff in sensitivity classification effectiveness for sensitivity review. ## conclusions we proposed a method, reldiff, to represent entityrelation-entity triples in an embedding space for automatic sensitivity classification. we compared the reldiff embedding features with embeddings from popular and sota knowledge graph methods (kge) and term features from documents enriched with entity-relations. in general, all relation representation methods we evaluated, consistently improved the effectiveness of sensitivity classification over baseline text classifiers. however, we showed that the kge methods are insufficient to effectively represent entity-relation information for sensitivity classification. on the other hand, our proposed approach reldiff can leverage these existing kge methods to produce an effective entityrelation representation for sensitivity classification. from the different configurations of kge methods, we found that the reldiff features can significantly improve the performance of sensitivity classification (0.739 bac & 0.426 f 1 , reldiff rotate ) in comparison to a baseline text classifier (0.728 bac & 0.409 f 1 ) and kge baselines (0.730 bac & 0.412 f 1 , concat rotate ), according to the mcnemar's test, @xmath2.05. moreover, while the overall classification performance varies according to the l 2 regularisation penalty in an ensemble classifier, the classifier with the kge features never outperforms the classifier with the reldiff features. furthermore, since false positive (fp) predictions can affect the speed of sensitivity reviewers @xcite , reldiff classifiers can markedly increase the sensitivity reviewers' speed due to the notably lower fps compared to the text classification baseline (up to 53% speed gain for 8.03% documents). we also showed that various relation types, such as person/place_of_birth and founder/organisation, have different effects on the classification performance. we will investigate this important and interesting research as future work.