A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution (2023)

Abstract: State-of-the-art natural language processing models have been shown to achieve remarkable performance in ‘closed-world’ settings where all the labels in the evaluation set are known at training time. However, in real-world settings, ‘novel’ instances that do not belong to any known class are often observed. This renders the ability to deal with novelties crucial. To initiate a systematic research in this important area of ‘dealing with novelties’, we introduce NoveltyTask, a multi-stage task to evaluate a system’s performance on pipelined novelty ‘detection’ and ‘accommodation’ tasks. We provide mathematical formulation of NoveltyTask and instantiate it with the authorship attribution task that pertains to identifying the correct author of a given text. We use amazon reviews corpus and compile a large dataset (consisting of 250k instances across 200 authors/labels) for NoveltyTask. We conduct comprehensive experiments and explore several baseline methods for the task. Our results show that the methods achieve considerably low performance making the task challenging and leaving sufficient room for improvement. Finally, we believe our work will encourage research in this underexplored area of dealing with novelties, an important step en route to developing robust systems.

## introduction recent advancements in natural language processing (nlp) have led to the development of several pre-trained large-scale language models such as bert @xcite , roberta @xcite , and electra @xcite . these models have been shown to achieve remarkable performance in closed-world settings where all the labels in the evaluation set are known at training time. however, in real-world settings, this assumption is often violated as instances that do not belong to any known label ('novel' instances) are also * equal contribution, contact email: hgupta35@asu.edu observed. this renders the ability to deal with novelties crucial in order to develop robust systems for real-world applications. the topic of novelty is getting increased attention in the broad ai research @xcite @xcite . also, in nlp, the 'novelty detection' task in which novel instances need to be identified is being explored @xcite ; related problems such as anomaly detection (chalapathy and chawla, 2019), out-of-domain detection, and openset recognition @xcite @xcite are also being studied. in addition to the task of 'detection', dealing with novelties also requires 'accommodation' that pertains to learning from the correctly detected novelties. despite having practical significance, this aspect of dealing with novelties has remained underexplored. furthermore, dealing with novelties is a crucial step in numerous other practical applications such as concept learning, continual learning, and domain adaptation. to initiate systematic research in this area of 'dealing with novelties', we formulate a multi-stage task called noveltytask. initially, a dataset consisting of examples of a set of labels (referred to as 'known labels') is provided for training and then sequential evaluation is conducted in two stages: novelty detection and novelty accommodation. both these stages include distinct unseen evaluation instances belonging to both 'known labels' (labels present in the training dataset) and 'novel labels' (labels not present in the training dataset). in the first evaluation stage i.e. the novelty detection stage, the system needs to either identify an instance as novel or classify it to one of the 'k' known labels. this is the same as the (k + 1) class classification problem (where k corresponds to the number of known labels) used in standard anomaly/ood detection tasks. this evaluation stage is followed by a feedback phase in which the initially, examples of a set of k labels ('known labels') are provided for training a classification system. the first evaluation stage i.e. the 'novelty detection' stage consists of evaluation instances from the k known labels and 'n ' novel labels. for each instance, the system needs to either classify it to one of the k known classes or report it as novel (not from any of the k known classes) i.e. the system is evaluated on a (k + 1) class classification problem. this stage is followed by a feedback phase in which the ground truth label of the novel instances that the system correctly reports as novel is revealed. the system then needs to leverage these new examples (of the novel labels) for the second evaluation stage (novelty accommodation) in which it is evaluated on a (k + n ) class classification problem. in addition to the initially provided training examples of the k known labels, the system can leverage these new examples of the novel labels for the next evaluation stage, the novelty accommodation stage. this stage also has evaluation instances from both the known and the novel labels (distinct and mutually exclusive from the detection stage); however, in this stage, the system needs to identify the true label of the evaluation instances, i.e. it's a (k + n ) class classification problem where n corresponds to the number of novel labels. we summarize this multi-stage task in figure 1 . we note that noveltytask is a controlled task/framework for evaluating a system's ability to deal with novelties and not a method to improve its ability. it is intuitive that the ability to deal with the novelties should be directly correlated with the ability to detect the novelties; our two-stage pipelined formulation of noveltytask allows achieving this desiderata as higher accuracy in correctly detecting the novelties will result in more feedback i.e. more examples of the novel labels that will eventually help in achieving higher performance in the accommodation stage. however, in the detection stage, the system needs to balance the trade-off between reporting instances as novel and classifying them to the known labels. consider a trivial system that simply flags all the evaluation instances of the detection stage as novel in order to get the maximum feedback; such a system will get the true groundtruth label (novel label) of all the novel instances present in the detection stage and will eventually perform better in the accommodation stage but it would have to sacrifice its classification accuracy in the detection stage (especially on instances of the known labels). we address several such concerns in formulating the performance metrics for noveltytask (section 3). in this work, we instantiate noveltytask with authorship attribution task in which each author represents a label and the task is to identify the correct author of a given unseen text. however, we note that the formulation of noveltytask is general and applicable to all tasks. we leverage product reviews from amazon corpus @xcite for the attribution task. we explore several baseline methods for both detection and accommodation tasks (section 4). in summary, our contributions are as follows: 1. we define a unified task for 'dealing with novelties' consisting of both novelty detection and novelty accommodation. 2. we provide a controlled evaluation framework with its mathematical formulation. 3. we instantiate noveltytask with the authorship attribution task. 4. we study the performance of several baseline methods for noveltytask. ## background and related work in this section, we first discuss the related work on novelty/ood/anomaly detection tasks and then detail the authorship attribution task. ## noveltytask noveltytask is a two-stage pipelined framework to evaluate a system's ability to deal with novelties. in this task, examples of a set of labels (referred to as known labels) are made available for initial training. the system is sequentially evaluated in two stages: novelty detection and novelty accommodation. both these stages consist of distinct unseen evaluation instances belonging to both 'known' and 'novel' labels. we define a label as novel if it is not one of the known labels provided for initial training and all instances belonging to the novel labels are referred to as novel instances. we summarize this multi-stage task in figure 1 . in this section, we provide a mathematical formulation of novelty-task, detail its performance metrics, and describe the baseline methods. ## conclusion and discussion to initiate a systematic research in the important yet underexplored area of 'dealing with novelties', we introduce noveltytask, a multi-stage task to evaluate a system's performance on pipelined novelty 'detection' and 'accommodation' tasks. we provided mathematical formulation of novelty-task and instantiated it with the authorship attribution task. to this end, we also compiled a large dataset (consisting of 250k instances across 200 authors/labels) from amazon reviews corpus. we conducted comprehensive experiments and explored several baseline methods for both detection and accommodation tasks. looking forward, we believe that our work opens up several avenues for interesting research avenues in this space, such as improving performance of detecting the novel instances and leveraging the feedback in a way that helps the system adapt with just a few examples of the novel labels.