D rift W atch: A Tool that Automatically Detects Data Drift and Extracts Representative Examples Affected by Drift (2024)

Abstract: Data drift, which denotes a misalignment between the distribution of reference (i.e., training) and production data, constitutes a significant challenge for AI applications, as it undermines the generalisation capacity of machine learning (ML) models. Therefore, it is imperative to proactively identify data drift before users meet with performance degradation. Moreover, to ensure the successful execution of AI services, endeavours should be directed not only toward detecting the occurrence of drift but also toward effectively addressing this challenge. % considering the limited resources prevalent in practical industrial domains. In this work, we introduce a tool designed to detect data drift in text data. In addition, we propose an unsupervised sampling technique for extracting representative examples from drifted instances. This approach bestows a practical advantage by significantly reducing expenses associated with annotating the labels for drifted instances, an essential prerequisite for retraining the model to sustain its performance on production data.

## introduction the recent advancements in machine learning (ml) and deep learning (dl) have propelled the emergence of diverse natural language processing (nlp) ai solutions featuring cutting-edge ml and dl models. nonetheless, their exclusive proficiency in inductive reasoning has given rise to substantial challenges when applied in practical business contexts. one such challenge is a data drift, an inconsistency between reference (i.e., training) and production data distributions @xcite . as the alterations in data distribution violate the fundamental assumption of ml, the iid condition that posits an identical distribution between training and test data, the occurrence of data drift has the potential to aggravate the accuracy of previouslytrained models and ultimately damage the quality of ai services. consequently, it is crucial to detect data drift and provide an updated model before customers experience a degradation in performance. the ml community has classified the data drift into two principal categories @xcite @xcite . assume an input x, target y , and the groundtruth relationship between x and y as f , so that @xmath0. the first type of data drift is covariate drift @xcite , which implies the change in the input feature distributions (i.e., @xmath1. the second category is concept drift @xcite , where the underlying relationship f changes (i.e., @xmath2. these two types of data drift readily occur in practical applications, such as introducing instances with unseen target labels or emerging new words/phrases under existing target labels. however, previous studies regarding data drift detection solely focused on a singular drift type, either covariate drift @xcite @xcite or concept drift @xcite @xcite . furthermore, these studies primarily centred on the identification of drift, but from a practical viewpoint, it is equally crucial to effectively address the challenge of upholding the model's performance and the quality of ai services. the conventional and straightforward approach involves annotating the drifted instances and incorporating them in a training batch for model retraining. however, employing human annotators for labelling a substantial volume of data points constitutes a resource-intensive undertaking. to this end, we propose a system called drift-watch, which detects both covariate and concept drift in text data. regarding the detector for covariate drift, we ascertained that using both semantic and syntactic features is beneficial over the exclusive reliance on either. regarding the detector for concept drift, we investigated multiple approaches, including the incorporation of large language mod-els (llms), but found that conventional and simpler methods outperform the llm-based approach in practical applications. in addition to this, we built a sampling methodology that autonomously extracts representative drifted instances, along with their corresponding importance rankings. this unsupervised approach can significantly reduce the effort to annotate labels for drifted instances, a necessity in the re-training ml model. the main contributions of this paper can be summarised as follows: 1) we introduce an auditor capable of detecting both covariate and concept drift, 2) we propose an effective sampling approach for the extraction of representative samples from drifted instances, which offers the practical advantages by significantly reducing the effort required for annotating labels, 3) our sampling methodology provides importance rankings for the drifted instances, facilitating prioritising annotation orders in the situation of limited resources, 4) we ascertain that contemporary llms may not necessarily outperform traditional approaches when implemented in practical applications. ## components of proposed solution the overall process of driftwatch solution is illustrated in figure 1 . first, models consisting of our covariate and concept drift detectors are trained using the reference dataset. next, production instances affected by both covariate and concept drift are predicted using the trained models. finally, representative sampling is introduced to address a practical issue where enough human labourers to annotate drifted instances are unavailable. finally, newly annotated instances are integrated into the reference data. ## related works several studies have been conducted on covariate drift detection. @xcite generated sentence embeddings and performed statistical tests to detect changes in embedding vectors of reference and production data. @xcite introduced a similar approach but used maximum mean discrepancy (mmd) test @xcite . @xcite employed an autoencoder, assuming that instances with high reconstruction errors are classified as outliers. they used changepoint model (cpm) @xcite to monitor whether a significant change in the reconstruction error of production data has occurred. analogously, @xcite proposed a framework named detail, which leverages sentence embedding vectors for density modelling and detects covariate drift along with explanations. @xcite introduced a linguistic covariate drift detector that identifies changes in vocabulary usage, syntactic structure, and semantic meanings. another line of works focused on identifying concept drift. a conventional approach is to use the confidence score of a winning label, which is generated by a pre-trained classifier, with a statistical testing @xcite or cpm @xcite . @xcite implemented a supervised detection method that employs the performance of production data. @xcite proposed an integrated framework for detecting and alleviating the data drift issue by finding a training batch that is the most similar to the production data and employing the model trained with the batch. ## conclusion this paper introduces driftwatch, a tool designed for the automated detection of data drift and the extraction of representative instances affected by such drift. the practical advantages of driftwatch extend to industrial practitioners by facilitating proactive identification of data drift and reducing resources required for the annotation process for model re-training.