Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities (2022)

Abstract: Much of the existing work on text novelty detection has been studied at the topic level, i.e., identifying whether the topic of a document or a sentence is novel or not. Little work has been done at the fine-grained semantic level (or contextual level). For example, given that we know Elon Musk is the CEO of a technology company, the sentence “Elon Musk acted in the sitcom The Big Bang Theory” is novel and surprising because normally a CEO would not be an actor. Existing topic-based novelty detection methods work poorly on this problem because they do not perform semantic reasoning involving relations between named entities in the text and their background knowledge. This paper proposes an effective model (called PAT-SND) to solve the problem, which can also characterize the novelty. An annotated dataset is also created. Evaluation shows that PAT-SND outperforms 10 baselines by large margins.

## introduction novelty/anomaly detection has been an active research area for decades @xcite @xcite . recently, it has received increased attention in nlp. broadly speaking, there are two main types: (1) topic-based novelty detection, which classifies a given text to a training/known class (topic) or reject/detect it as belonging to some unknown classes @xcite @xcite ; and (2) semantic novelty detection, which determines whether a given text represents a semantically/contextually novel phenomenon. for example, @xcite studied a semantic novelty detection problem -detecting semantically novel scene descriptions (e.g., "a person walks a chicken in the park" is a novel scene, whereas "a person walks a dog in the park" is normal one). this task is more fine-grained and requires factual reasoning over text as compared to that of (1), which has been studied extensively. (2) has only been introduced recently and is the focus of this paper. this paper proposes a new semantic novelty detection task: given a factual text d containing two named entities foot_0 , we want to classify whether d represents a semantically novel fact or a normal one with respect to the entity pair. for example, consider the text d 1 and an entity pair underlined in d 1 in figure 1 . d 1 represents a normal fact as it is natural for an actor (johnny galecki) to act in a sitcom or tv show (the big bang theory). however, d 2 in figure 1 depicts a novel fact with respect to the underlined entity pair because a ceo of a technology company (elon musk) acting in a sitcom (the big bang theory) is quite surprising and novel. factual text appears in diverse media sources, such as news articles, blog posts, reviews etc. detecting semantically novel facts involving popular real-world (named) entities has many applications because anything novel is always of interest and can trigger readers' curiosity. for example, a mobile newsfeed application can increase user engagement by recommending novel news/facts of named entities and promoting news articles with novel facts. although novelty is subjective and personal, there exist some novel facts that the majority of people agree. in this work, we restrict our study to this consensus-view of semantic novelty and leave the personalized novelty for future work. solving the proposed task requires joint finegrained reasoning over (1) the relationship between the pair of entities in the textual context and (2) the background knowledge of the entities. for example, considering d 2 in figure 1 , we first need to detect that the entity pair ("elon musk", "the big bang theory") in d 2 has the "cast-member" relation and then, leverage the interaction of the relation with the background knowledge of the entities (i.e., "elon musk" is a tech entrepreneur and "the big bang theory" is a tv show) to infer the semantic novelty (because, a tech entrepreneur does not normally act in a tv show). we utilize the external knowledge repository (kr) -wiki-data @xcite to extract the named entity's background knowledge, which is a list of property-value pairs. for example, elon musk's background knowledge contains propertyvalue pairs: [(a) (occupation, entrepreneur), (b) (gender, male), (c) (field of work, tech entrepreneurship) ....]. however, not all propertyvalue pairs is useful for inference (e.g., only (a) and (c) are useful for d 2 in figure 1 ). thus, a solution for automatic selection of the useful property-value pairs is needed (see sec. 4). in fact, the useful property-value pairs provide a reason or characterization for the novelty. problem definition: given (1) a set of training factual text d @xmath0, d 2 , .....d n }, with each d @xmath1 to solve the task, we propose a new model, called pat-snd (property attention network for semantic novelty detection) to detect novel factual text. additionally, pat-snd also provides the characterization (or reason) for the novelty (unlike @xcite ). pat-snd first employs an existing relation classification technique to identify the relation between the entity pair. the identified relation is then used in a novel relation-aware property attention network (pat) module that leverages the attention mechanism to select the useful background knowledge from the kb k to perform semantic reasoning for novelty detection. the learned attention knowledge in pat is also used to provide the characterization for the novelty (see sec. 4). pat-snd is evaluated using our newly created nftd (novel factual text detection) dataset. we leverage a distant supervision technique @xcite with the wikipedia foot_1 as the corpus and wikidata as the kr to build a large training dataset. evaluation results show that pat-snd outperforms the 10 latest novelty detection baselines by very large margins. our main contributions are as follows: ## related work novelty or anomaly detection has been studied extensively over the years. early representative works include one-class svm (ocsvm) @xcite , support vector data description (svdd) @xcite and hybrid approaches @xcite that learn features using deep learning and then apply ocsvm or svdd to build one-class classifiers. more recent deep learning approaches are based on auto-encoders @xcite @xcite , gan @xcite , neural density estimation @xcite , multiple hypothesis prediction @xcite , robust mean estimation @xcite and regularization @xcite . chalapathy and chawla (2019 novelty detection has also been studied in outof-distribution (ood) detection or open-set recognition @xcite @xcite . however, these methods work in the multi-class setting. ours is an one-class classification problem. there are also works on topical novelty detection @xcite @xcite @xcite . they differ from ours as we focus on fine-grained semantic novelty detection. our work is also related to semantic plausibility (spl) that studies the problem of whether an event is plausible or not @xcite @xcite @xcite and selectional preference (spr) that deals with the "typicality" of an event @xcite @xcite @xcite @xcite @xcite . these works differ from ours as (1) conceptually, spl and spr are related but different from novelty, (2) our task demands the use of background knowledge in the named entities for semantic reasoning. however spl and spr only perform reasoning on the surface form of objects in the text, and (3) they use fully labeled data @xcite while we have only normal (one-class) data in training. commonsense reasoning is remotely related to our work. existing works have built multi-choice commonsense reasoners @xcite , studied the commonsense knowledge contained in language models @xcite @xcite and knowledge graph @xcite , and constructed new datasets for better evaluation @xcite . several researchers also investigated physical commonsense reasoning @xcite @xcite and affordance of entities @xcite . they do not perform novelty detection. trivia fact mining @xcite @xcite @xcite @xcite @xcite is also relevant, but it is mainly about interestingness. some trivia facts are interesting because they are rare, but not necessarily novel. existing papers use labeled training data for learning, or rely on wikipedia structure to retrieve interesting facts using information retrieval methods @xcite . we have only normal data but no novel data. our proposed model is based on an attention network. related nlp works using attention techniques include @xcite @xcite @xcite ). but they solve different problems, such as sentiment analysis and argument mining and are not about novelty detection. their approaches also differ from ours. ## dataset collection and annotation to build a large factual text dataset annotated with named entities, we leverage the distant supervision technique in @xcite . we create our training and test datasets, using wikipedia as the corpus and wikidata @xcite as the external knowledge repository (kr). we choose wikidata as kr for extracting background information of the entities, because the good community collaboration and contribution of wikidata makes it a high-quality kr compared to other krs @xcite . wikidata encodes real-world knowledge in the form of triples: (e 1 , r, e 2 ), which means entity e 1 and entity e 2 have a relation r. for instance, (the big bang theory, cast-member, johnny galecki). the named entities in the wikipedia corpus are linked to the wikidata. we can find unambiguous mappings between entity mentions in the text and wikidata entities. for example: in the wikipedia source: " @xcite ] is an american television sitcom, filmed in front of a live audience, training test # instances (factual text) 251,619 (nr) 1000 (nr), 1000 (nv) avg. text length 41.35 26.02 stars [[johnny galecki]] et al.", the named entities in bracket [[.]] have an unique one-to-one mapping to the entities in wikidata. training dataset preparation. the distant supervision technique can be briefly described as follows: for a piece of text d from wikipedia involving e 1 and e 2 (with hyperlink uniquely mapping to wikidata entities), if there is a triple (e 1 , r, e 2 ) in the kr, we assume that the textual information in d expresses the relation r between e 1 and e 2 . in this case, we automatically annotate (e 1 , r, e 2 , d) as a distantly supervised instance and add it to our training dataset. for entity pairs (e 1 , e 2 ) with more than one relation, we discard them because they bring ambiguity in our dataset. due to the budgetary constraints, we can not evaluate on all relations in the wikidata. we create our training data related to 20 human related relations. the details of these 20 relations are in appendix sec. a. with distant supervision, we allow noise to exist in the training dataset because this process requires no human annotation, and scales up the learning of more relations. we split the whole dataset created via distant supervision into two parts: train set and the test set pool, making sure that there is no overlapping in either text or entity pairs between these two parts. this test set pool is used for test dataset preparation. ## proposed approach our proposed pat-snd model works in two steps: (1) entity relation classification, and (2) triple semantic novelty scoring (sns). given a factual text d containing a pair of entities (e 1 , e 2 ), pat-snd first identifies the relation r between (e 1 , e 2 ) in d in step (1) [sec. 4.1]. next, the background knowledge of the entities e 1 and e 2 retrieved from the kb k together with the predicted relation r are fed to the sns module to score the semantic novelty of d with respect to (e 1 , e 2 ) and k in step 2 [sec. 4.2]. as our training data d tr consists of only normal class examples (as discussed in sec. 1), it's not possible to train sns solely with d tr . thus, we propose a kb-based contrastive data generator (cdg) to generate pseudo-novel examples. the sns module is then trained with both normal class examples in d tr as well as the generated pseudo-novel examples in a supervised learning manner. we will discuss more about it in sec. 4.3. ## conclusion this paper proposes a new semantic novelty detection problem -semantic novelty detection in factual text involving named entities. a novel attention-based network pat-snd is proposed to solve the problem. a new dataset nftd is created and released as a benchmark for the nlp community. experimental results showed that pat-snd outperforms 10 baselines by very large margins.