BARLE : Background-Aware Representation Learning for Background Shift Out-of-Distribution Detection (2022)

Abstract: Machine learning models often suffer from a performance drop when they are applied to out-of-distribution (OOD) samples, i.e., those drawn far away from the training data distribution. Existing OOD detection work mostly focuses on identifying semantic-shift OOD samples, e.g., instances from unseen new classes. However, background-shift OOD detection, which identifies samples with domain or style-change, represents a more practical yet challenging task. In this paper, we propose Background-Aware Representation Learning (BARLE) for background-shift OOD detection in NLP. Specifically, we generate semantics-preserving background-shifted pseudo OOD samples from pretrained masked language models. We then contrast the in-distribution (ID) samples with their pseudo OOD counterparts. Unlike prior semantic-shift OOD detection work that often leverages an external text corpus, BARLE only uses ID data, which is more flexible and cost-efficient. In experiments across several text classification tasks, we demonstrate that BARLE is capable of improving background-shift OOD detection performance while maintaining ID classification accuracy. We further investigate the properties of the generated pseudo OOD samples, uncovering the working mechanism of BARLE.

## introduction most state-of-the-art nlp models are evaluated with the assumption that the training data and testing data is drawn from the same distribution. however, when models are deployed in real-world settings, this assumption can be easily violated, and current nlp models tend to suffer from drastic performance drops on out-of-distribution (ood) data @xcite @xcite . identifying ood samples and distinguishing them from in-distribution (id) ones, known as ood detection, plays an essential role in a wide range of nlp applications @xcite @xcite . existing ood detection methods in nlp mostly focus on identifying semantic-shift ood samples (e.g., samples from unseen classes) @xcite @xcite . however, @xcite point out that it is rare to encounter semantic-shift ood inputs in real-world settings. in practice, background-shift oods may be more pervasive. these are samples that belong to the same task as the id data, but with a shift on background features, such as changes in the domain or the style of the text @xcite . for instance, a topic classification model trained on news articles (id) vs. tweet messages (ood), or a sentiment classification model trained on movie reviews (id) vs. product reviews (ood). background-shift ood detection represents a more practical yet challenging task. for example, consumer-facing manufacturers are routinely interested in building sentiment classification models to understand consumer sentiment for product-related issues in online reviews. however, such reviews also contain non-product design/aspect related reviews (such as retailer shipping and customer service experiences). if the sentiment classification model is applied to this review dataset without identifying the background-shift oods (i.e., non-product design/aspect related reviews), the model may result in a lower prediction performance and thus negatively affect the company's decision on product-related issues. other applications of background-shift ood include psychometric nlp tasks such as inferring users' trust, anxiety, and literacy across domains like health and finance @xcite . in this work, we propose an efficient and effective approach for background-shift ood detection. first, our approach does not require any external data. prior ood detection methods often rely on external text corpora to simulate the ood sam-ples and learn id-specific representations @xcite @xcite . however, it is difficult to decide which external data to use, and the choice of the external dataset is critical for successful ood detection @xcite . moreover, incorporating prior knowledge in choosing ood data for training may introduce inductive bias to the ood detector @xcite . in our work, we utilize pretrained language models (e.g., distilbert @xcite , bert @xcite ) and perform a masked language modeling heuristic to generate semantics-preserving background-shifted pseudo samples from the id data. for instance, a sentiment classification id example "a sane and breathtakingly creative film" might be mapped to a pseudo example "a massive and perfectly executed painting", where the background features (film) are replaced but the positive semantics is preserved. by taking advantage of pretrained language models, we can obtain better quality pseudo oods in a more principled manner. second, we design a background-aware contrastive loss to push the id training samples apart from their pseudo ood counterparts. combined with another semantic contrastive loss, the learned representations are not only semantically distinguishable (which is important for the main task) but also encode rich id background information (which is important for ood detection). we then employ an existing ood scoring mechanism @xcite @xcite to map the learned background-aware representation to a scalar that indicates the ood likelihood. our approach is named barle, short for background-aware representation learning for identifying background-shift oods. in experiments across several text classification tasks, we show that barle achieves superior performance in identifying background-shift ood samples while maintaining the id task performance. this implies that barle can be used as a 2-in-1 model which not only delivers desirable performance for the id task, it can also detect any suspicious ood samples. we also investigate the properties of generated pseudo ood samples to better understand the working mechanism of barle, which may shed light on future ood detection work. the code is publicly available via github. 1 1 https://github.com/hduanac/barle . our implemen-2 related work ## conclusion in this work, we propose a simple yet effective method for background shift ood detection. our method leverages pretrained language models to synthesize semantics-preserving backgroundshifted pseudo ood samples from the id training data. by contrasting the id training samples with the their pseudo ood counterparts, our proposed training scheme learns background-aware representations and improves ood detection performance. additional analyses on the properties of the generated pseudo ood samples also validates our design effectiveness. we believe our work sheds new light on ood detection for building robust nlp systems. as noted, possible applications include an array of nlp-based user modeling tasks including inferring psychometrics, text-based personality detection in forums and social media @xcite , and stylometric authorship identification where style is the primary task and topics and genres are background @xcite .