Semantic Novelty Detection in Natural Language Descriptions (2021)

Abstract: This paper proposes to study a fine-grained semantic novelty detection task, which can be illustrated with the following example. It is normal that a person walks a dog in the park, but if someone says “A man is walking a chicken in the park”, it is novel. Given a set of natural language descriptions of normal scenes, we want to identify descriptions of novel scenes. We are not aware of any existing work that solves the problem. Although existing novelty or anomaly detection algorithms are applicable, since they are usually topic-based, they perform poorly on our fine-grained semantic novelty detection task. This paper proposes an effective model (called GAT-MA) to solve the problem and also contributes a new dataset. Experimental evaluation shows that GAT-MA outperforms 11 baselines by large margins.

## introduction novelty or anomaly detection has been an important research topic since 1970s @xcite due to numerous applications @xcite . recently, it has also become important for natural language processing (nlp). many researchers have studied the problem in the text classification setting @xcite @xcite @xcite . however, these text novelty classifiers are mainly coarse-grained, working at the document or topic level. given a text document, their goal is to detect whether the text belongs to a known class or unknown class. this paper introduces a new text novelty detection problem -fine-grained semantic novelty detection. specifically, given a text description d, we detect whether d represents a semantically novel fact or not. this work considers text data that describe scenes of real-world phenomena in natural language (nl). in our daily lives, we observe different real-world phenomena (events, activities, situations, etc.) and often describe these observations (referred as "scenes" onwards) in nl to others or write about them. it is quite natural to observe scenes that we have not seen before (i.e., novel scenes). for example, it is a common scene that "a person walks a dog in the park", but if someone says "a man is walking a chicken in the park", it is quite unexpected and novel. detecting such semantic novelty requires complex conceptual and semantic reasoning over text and thus, is a challenging nlp problem. note that conceptually, the judgement of the novelty of a scene is subjective and might differ from person to person. however, there are some scenes for which a majority of people have agreement about their novelty. a good example of such majority-view of novelty is the widely-spread meme pictures on social media, which contain novel interactions between objects. in this work, we restrict our research to this majority-based view of novelty and leave the personalized novelty view angle for the future work. in this work, we leverage the captions of images from popular datasets like @xcite , to build a semantic novelty detection dataset (sec. 3), 1 where we consider an image as a scene and the corresponding image captions as different nl descriptions of the scene. detecting text describing semantically novel observations have many applications, e.g., recommending novel news, novel images & videos (based on their text descriptions), social media posts and conversations. the problem of semantic novelty detection is defined as follows. problem definition: given a set of natural language descriptions @xmath0, d 2 , .....d n } of common scenes, build a model m using d to score the semantic novelty of a test nl description d with respect to d, i.e., classifying d into one of the two classes {normal, novel}. "normal" means that d is a description of a common scene and "novel" means d is a description of a se-mantically novel scene. as the detection model m is built only with "normal" class data, the task is an one-class text classification problem. we are unaware of any existing work that can effectively solve this problem. although existing novelty/anomaly detection and one-class classification algorithms are applicable, since they are coarse-grained or topic-based, they perform poorly on our task (see sec. 5). note that although we focus on solving the problem of semantic novelty detection of nl descriptions of scenes, the proposed task and solution framework are generally applicable to other applications. this paper proposes a new technique, called gat-ma (graph attention network with max-margin loss and knowledge-based contrastive data augmentation) to identify nl description sentences of novel scenes. since our task is at the sentence level and fine-grained, we exploit graph attention network (gat) on the parsed dependency graph of each sentence, which fuses both semantic and syntactic information in the sentence for reasoning with the internal interactions of entities and actions. to enable the model to capture long-range interactions, we stack multiple layers of gats to build a deep gat model with multi-hop graph attention. we also create the pseudo novel training data based on the given normal training data through contrastive data augmentation. thus, gat-ma is trained with the given original normal scene descriptions and the augmented pseudo novel scene descriptions (sec. 4). gat-ma is evaluated using our newly created novel scene description detection (nsd2) dataset. the results show that gat-ma outperforms a wide range of latest novelty or anomaly detection baselines by very large margins. our main contributions are as follows: whereas the existing work focuses on coarse-grained document-or topiclevel novelty, our task requires fine-grained sentence-level semantic & syntactic analysis. 2. we propose a highly effective technique called gat-ma to solve the proposed semantic novelty detection problem, which is based on gat with dependency parsing and knowledgebased contrastive data augmentation. 3. we create a new dataset called nsd2 for the proposed task. the dataset can be used as a benchmark dataset by the nlp community. our work is closely related to anomaly, outlier or novelty detection. earlier approaches include one-class svm (ocsvm) @xcite or support vector data description (svdd) @xcite . in recent years, deep learning approaches dominated. @xcite and @xcite learned features using deep learning and then applied ocsvm or svdd to build one-class classifiers. many recent approaches are based on autoencoders @xcite @xcite , gan @xcite , neural density estimation @xcite , multiple hypothesis prediction @xcite , robust mean estimation @xcite and regularization @xcite . see the surveys @xcite for more details. our gat-ma is based on stacked graph attention neural networks, parsing and data augmentation. novelty detection has also been studied in outof-distribution (ood) detection @xcite @xcite @xcite . however, these methods work in the multi-class classification setting. our work focuses on one-class classification. our work is also related to document or sentence topical novelty detection @xcite @xcite @xcite @xcite . these tasks differ from our problem setting as we focus on fine-grained semantic novelty detection. our work is also related to semantic plausibility (spla) and selectional preference (spre). spla is concerned with whether an event is plausible, and spre is about the "typicality" of an event. for spla, existing models employ pretrained language models @xcite and manually elicited entity property knowledge @xcite to model physical plausibility in the supervised setting. other related work includes creating datasets with plausibility ratings @xcite and dealing with multi-event inference @xcite . for spre, the early works include @xcite @xcite @xcite @xcite . the performance is improved by neural networks @xcite @xcite . our work is different: (1) conceptually, spla and spre are related but different from novelty, (2) they are mostly based on structured subject-verb-object triples, rather than natural language sentences, and (3) they use fully labeled data @xcite while we do novelty detection with only normal data in training. the work of commonsense reasoning is remotely related to our work. existing works build multi-choice commonsense reasoners @xcite , study the commonsense knowledge contained in language models @xcite @xcite and knowledge graph @xcite , and build new datasets for better evaluation @xcite . several researchers also investigated physical commonsense reasoning @xcite @xcite and affordance of entities @xcite . they do not perform novelty detection. our work is also related to trivia fact mining @xcite @xcite @xcite @xcite @xcite . however, trivia is more related to interestingness. some trivia facts are interesting because they are rare, but not necessarily novel. existing works use labeled training data for learning, or rely on wikipedia structure to retrieve interesting facts using information retrieval methods @xcite . we have only normal data but not novel data. our proposed model learns text representation using a graph neural network and leveraging dependency parsing. other works in nlp that use graph neural networks and dependency structures include @xcite @xcite @xcite , etc. but they solve different problems, such as sentiment analysis and argument mining. their approaches are also different from ours and do not do novelty detection. ## dataset collection and annotation as there is no semantic novelty detection dataset available for text, we build a new dataset. as our proposed task requires learning of latent semantic knowledge in text, such as capturing the interaction among entities and verbs (e.g. "person" and "food" are related to each other by verb "cook"); the actions (verbs) that an entity can support (e.g., only a person can perform action "cook"); actions an entity can be applied on (e.g. "cook" can be applied on entity "vegetables"), etc., we aim to build a corpus rich in such knowledge. text data like news articles, social media posts, reviews, etc., generally contain such knowledge in low density and thus, are not very suitable. instead, we leverage image captions to built our dataset, which we found to be suitable for our task. image caption data collection. we found that the captions of non-iconic images (depicting multiple objects and their interactions) meet the aforementioned dataset requirements. we chose three popular benchmark image caption datasets: coco @xcite , flickr30k @xcite and visual genome @xcite to build our dataset. coco consists of 616,435 captions of flickr images. flickr30k contains 158,915 captions about people and animals, and visual genome contains 5.4 million captions describing interactions among various objects. to ensure we have a diverse dataset to learn interactions among entities and verbs, we merge the 3 datasets into one large dataset. ## the proposed gat-ma model the proposed gat-ma model consists of two main components: (i) knowledge-based contrastive data generator (cdg), and (ii) text semantic novelty scorer (sns). given a set of nl descriptions d @xmath1, d 2 , ..., d n } of normal scenes in the training data, cdg dynamically generates pseudo-novel descriptions by perturbing the normal scene descriptions in d tr utilizing the lexical knowledge base wordnet foot_1 @xcite . the normal descriptions in d tr are augmented with these pseudo-novel descriptions (used as novel class examples in training) to learn a sns. the sns is a deep gat model that learns to score an input text to measure its semantic novelty with respect to d tr . to capture the semantic and syntactic information in an input text d, gat-ma parses d into a dependency graph and feeds the graph enriched with additional word-level features to the sns, which is then trained to assign higher score to a normal scene description compared to that of a novel one. ## error analysis the auc score in (.) for each verb is as follows: pull (0.99), carry (0.99), push (0.99), drive (0.97), travel (0.97), hit (0.95), throw (0.95), kick (0.94), climb (0.94), look (0.93), build (0.93), cook (0.92), walk (0.92), ride (0.87), fly (0.84), cut (0.82), swim (0.81), jump (0.73), drink (0.73), and eat (0.69). we carried out error analysis on our test data and found that the errors are mainly due to the following factors. the first factor is the pretrained word embeddings' quality. the quality of the word embedding is critical for gat-ma to effectively do reasoning. gat-ma makes mistakes when the pretrained word embedding is not of good quality. for example, the "talapoin" in "the talapoin at the zoo is leaning down to drink some water". the second factor is the limitation of knowledge acquired by gat-ma during training. gat-ma relies on the taxology information in wordnet to generate contrastive novel descriptions during training. however, sometimes the reasoning of novel description requires more complex world knowledge. for examples, "two kids are sitting in the bar drinking spirit" is novel and requires knowledge that kids is not old enough to drink any alcohol. another example "a dog is eating onions on the ground" is novel and requires the world knowledge that onions is poisonous to dogs foot_5 . ## conclusion novelty detection is an important problem because anything novel is of interest. this paper proposed a semantic novelty detection problem and designed a graph attention network based approach (called gat-ma) exploiting parsing and data augmentation to solve the problem. as there is no existing evaluation dataset for the proposed task, an evaluation dataset has been created. experimental comparisons with a wide range of baselines showed that gat-ma outperforms them by very large margins.