Using a novel evaluation toolkit that simulates a human reviewer in the loop, we compare the effectiveness of three machine-learning protocols for technology-assisted review as used in document review for discovery in legal proceedings. Our comparison addresses a central question in the deployment of technology-assisted review: Should training documents be selected at random, or should they be selected using one or more non-random methods, such as keyword search or active learning? On eight review tasks -four derived from the TREC 2009 Legal Track and four derived from actual legal matters -recall was measured as a function of human review effort. The results show that entirely non-random training methods, in which the initial training documents are selected using a simple keyword search, and subsequent training documents are selected by active learning, require substantially and significantly less human review effort (P < 0.01) to achieve any given level of recall, than passive learning, in which the machine-learning algorithm plays no role in the selection of training documents. Among passive-learning methods, significantly less human review effort (P < 0.01) is required when keywords are used instead of random sampling to select the initial training documents. Among active-learning methods, continuous active learning with relevance feedback yields generally superior results to simple active learning with uncertainty sampling, while avoiding the vexing issue of "stabilization" -determining when training is adequate, and therefore may stop.
For example, Cormack and Grossman REF presented an approach called continuous active learning and showed that selecting initial training documents through a simple keyword search, and subsequent training documents by continuous active learning, required significantly less (according to a sign test with p < 0.01) reviewing effort to achieve any given level of recall, compared to when the learning algorithm did not implement an active learning strategy to select the documents to use for training.
Evaluation of machine-learning protocols for technology-assisted review in electronic discovery
Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval
Computer Science