Combining Counting Processes and Classification Improves a Stopping Rule for Technology Assisted Review (2023)

Abstract: Technology Assisted Review (TAR) stopping rules aim to reduce the cost of manually assessing documents for relevance by minimising the number of documents that need to be examined to ensure a desired level of recall. This paper extends an effective stopping rule using information derived from a text classifier that can be trained without the need for any additional annotation. Experiments on multiple data sets (CLEF e-Health, TREC Total Recall, TREC Legal and RCV1) showed that the proposed approach consistently improves performance and outperforms several alternative methods.

## background information retrieval (ir) systems often return large numbers of documents in response to user queries and screening them for relevance represents a significant effort. this problem is particularly acute in applications where it is important to identify most (or all) of the relevant documents, for example, systematic reviews of scientific literature @xcite and legal ediscovery @xcite . technology assisted review (tar) develops methods that aim to reduce the effort required to screen a collection of documents for relevance, such as stopping rules that inform reviewers that a desired level of recall has been reached (and no further documents need to be examined) @xcite . one of the most commonly applied approaches to developing stopping rules is to identify when a particular level of recall has been reached by estimating the total number of documents in the collection (either directly or indirectly). the majority of algorithms using this approach operate by obtaining manual judgements for a sample of the unobserved documents and inferring the total number of relevant documents, e.g. @xcite @xcite . however, these approaches may not account for the fact that most relevant documents appear early in the ranking, information shown to be useful for stopping algorithms @xcite , so they may examine more documents than necessary. counting processes counting processes, stochastic models of the number of occurrences of an event within some interval @xcite , can naturally model changes to the frequency of relevant document occurances within a ranking. they have been used to develop stopping rules that ensure that a desired level of recall is reached while also minimising the number of documents that need to be reviewed and found to outperform a range of alternative approaches @xcite @xcite . however, in these methods the estimate of the total number of relevant documents is only based on the examined initial portion of the ranking and no information is used from the remaining documents. classification another approach has been to use a text classifier to estimate the total number of relevant documents @xcite . this method has the advantage of using information about all documents in the ranking by observing the classifier's predictions for the documents that have not yet been examined. however, the classifier alone does not consider or model the occurrence rate at which relevant documents have already been observed. these methods were found to be effective, although each was only tested on a single dataset. this paper proposes an extension to stopping algorithms based on counting processes by using a text classifier to inform the estimate of the total number of relevant documents. this approach makes use of information about the relevance of documents from the entire ranking without increas-ing the number of documents that need to be examined for relevance, since the text classifier is trained using information already available. ## approach we begin by describing the existing counting process approach and then explain how the classifier is integrated. counting process stopping rule @xcite the approach starts by obtaining relevance judgements for the n highest ranked documents. a rate function is then fitted to model the probability of relevant documents being encountered in the (as yet) unexamined documents later in the ranking. the counting process uses this rate function to estimate the total number of relevant documents remaining. based on this estimate, the algorithm stops if enough relevant documents have been found to reach a desired recall. if not then more documents are examined and the process repeated until either the stopping point is reached or all documents have been examined. fitting an appropriate rate function is an important factor in the success of this approach. a set of sample points are extracted from the documents for which the relevance judgements are available (i.e. top n), and the probability of a document being relevant at each point is estimated. this set of points is then provided to a non-linear least squares curve fitting algorithm to produce the fitted rate function. for each sample point, the probability of a document being relevant is computed by simply examining a window of documents around it and computing the proportion that is relevant, i.e. integrating text classification since the existing counting process approach only examines the top-ranked documents, it relies on the assumption that the examined portion of documents provides a reliable indication of the rate at which relevant documents occur within the ranking. however, this may not always be the case, particularly when only a small number of documents have been manually examined or when relevant documents unexpectedly appear late in the ranking. this problem can be avoided by applying a text classifier to the unexamined documents at each stage. the information it provides helps to ensure that the rate function is appropriate and that the estimate of the total number of relevant documents provided by the counting process is accurate. the text classifier is integrated into the stopping algorithm in a straightforward way. we begin by assuming that the ranking contains a total of n documents. as before, relevance judgements are obtained for the first n. these judgements are used to train a text classifier which is applied to all remaining documents without relevance judgements (i.e. n + 1 . . . n ). the rate function is now fitted by examining all n documents in the ranking (rather than only the first n), using manual relevance judgements for the first n and the classifier's predictions for the remainder (i.e. n + 1 . . . n ). as before, the algorithm stops if it has been estimated that enough relevant documents are contained within the first n to achieve the desired recall. otherwise, the value of n is increased and the process repeated. see figure 1 . classlabel: the class labels output by the classifier are used directly, i.e. using eq. 1 with relevance determined by the classifier. classscore: alternatively, the scores output by the classifier is used, subject to a threshold of 0.5, i.e. eq. 1 is replaced by ## experiments experiments compared three approaches. cp: baseline counting process approach without text classification, i.e. @xcite . cp+classlabel and cp+classscore: counting process combined with the text classifier (see §2). the counting process uses the most effective approach previously reported @xcite . a power law @xcite was used as the rate function with an inhomogeneous poisson process since its performance is comparable with a cox process while being more computationally efficient. parameters were set as follows: the confidence level was set to 0.95 while the sample size was initialised to 2.5% and incremented by 2.5% for each subsequent iteration. an existing reference implementation was used @xcite . the text classifier uses logistic regression which has been shown to be effective for tar problems @xcite . the classifier was based on scikit-learn using tf-idf scores of each document's content as features, the content is title and abstract for clef datasets, email messages for trec datasets (including title and message body), and news articles content for the rcv1 dataset. the classification threshold was set to 0.5 (model default) by optimising f-measure for relevant documents using a held-out portion of the dataset. text classifiers based on biobert @xcite and pubmedbert @xcite were also developed. however, neither outperformed the linear logistic regression model on clef datasets, a result consistent with previous findings for tar models @xcite . tar problems are often highly imbalanced, with very few relevant documents. however, tar methods successfully place relevant documents early in the ranking @xcite @xcite . consequently, the prevalence of relevant documents in the training data is unlikely to be representative of the entire collection, particularly early in the ranking. to account for this cost-sensitive learning was used during classifier training. the weight of the minority class within the training data was set to 1, while the weight of the majority class was set to the imbalance ratio (i.e. count of the minority over the majority class in training data) @xcite . ## results and analysis experiments were carried out using a range of desired recalls {0.9, 0.8, 0.7} with results shown in ## conclusion this paper explored the integration of a text classifier into an existing tar stopping algorithm. experiments on six collections indicated that the proposed approach was able to achieve the desired recall level with a statistically significant lower cost than the existing method based on counting processes alone. integrating the classifier output in the form of labels or scores made little difference, with improvements observed using either approach. the text classifier provides the stopping algorithm with information about the likely relevance of the documents that have not yet been manually examined, allowing it to better estimate the number remaining.