A Comparative Analysis between Human-in-the-loop Systems and Large Language Models for Pattern Extraction Tasks (2022)

Abstract: Building a natural language processing (NLP) model can be challenging for end-users such as analysts, journalists, investigators, etc., especially given that they will likely apply existing tools out of the box. In this article, we take a closer look at how two complementary approaches, a state-of-the-art human-in-the-loop (HITL) tool and a generative language model (GPT-3) perform out of the box, that is, without fine-tuning. Concretely, we compare these approaches when end-users with little technical background are given pattern extraction tasks from text. We discover that the HITL tool performs with higher precision, while GPT-3 requires some level of engineering in its input prompts as well as post-processing on its output before it can achieve comparable results. Future work in this space should look further into the advantages and disadvantages of the two approaches, HITL and generative language model, as well as into ways to optimally combine them.

## introduction creating custom ai models for natural language processing (nlp) tasks is not an easy feat: it typically involves labeling large datasets, selecting appropriate ml architectures/models, and training them. to lower the barrier of entry in nlp model creation, the scientific and industrial community has proposed human-in-the-loop (hitl) systems @xcite @xcite . while they differ in the nlp tasks they target (e.g., classification, extraction, question answering) and the techniques they employ (e.g., active learning, custom algorithms), their operation from the perspective of the end user is similar: the user labels a few examples, which are then used by the system to build a first version of the model and then to ask back the user for additional targeted feedback. this feedback is in turn used to iteratively refine the model and continue the loop by asking for further feedback. by asking for targeted feedback at each iteration, the goal is to lower the effort required to build a model, enabling fast convergence to a performant model while using a small number of labeled examples. while such tools are gaining popularity, the nlp community has also recently proposed several pretrained generative language models, such as gpt-3. the largest versions of such models have shown incredible few-shot performance on several tasks @xcite . thus the question arises: given the out-of-the-box performance of such models, is it possible to feed them with a few examples and get similar performance to that of more complex hitl systems? in this paper, we answer this question for the task of text pattern extraction. in this task, the goal is to extract text instances that follow a similar pattern. examples include extracting crime incidents from crime reports, e.g. 4,556,123 incidents or 5,193,927 incidents , or revenue from financial press releases, e.g. revenue was @xmath01.3 million . to answer the question, we compare pattern induction @xcite , an hitl tool tailored to pattern extraction tasks, against gpt-3 @xcite . we evaluate the two approaches based on how an end-user, who does not necessarily have a technical background, would use these tools to accomplish an nlp task, in this case pattern extraction. unlike nlp experts, end-users typically use tools out-of-the-box, that is, without writing code and without fine-tuning parameters or modifying and re-training the layers of the ai model. our preliminary results show that the use of simple techniques to prompt gpt-3 does not yet lead to the same performance as that of the tailored hitl tool. in this work, we describe these results and present further research directions on the relationship of models such as gpt-3 to hitl tools. the following summarizes the contributions of this paper: we start with literature review in section 2, provide empirical study in section 3, discuss each approach pros and cons in section 4, and conclude in section 5. ## related works in this work, we focus on few-shot learning systems (fsl) @xcite , which can learn from a handful of examples. this paradigm appeals to non-technical users given the small number of required examples to fine-tune a system, thus removing the need to label and maintain large labeled training datasets. ## discussion our experiments show that the hitl method for ie results in higher precision while in the large generative model, given a structured prompting and post-processing step, gpt-3 gives higher recall. gpt-3 is able to contextualize the prompts and learns a more general model. yet, the downside of gpt-3 for ie is that in of itself does not perform the ie tasks. to get comparable results to the hitl model, we had to (1) engineer and design the structure of the prompts to leverage gpt-3's powerful language abilities and (2) post-process the string output from gpt-3. additionally, the hitl model (1) elicits targeted user feedback and (2) allows for an iterative approach to building the underlying rule-based model. these two aspects are not found in gpt-3. in light of these results, how do we then combine the advantages of each approach, human-in-the-loop and traditional ai models? how do we leverage the human-machine interactions to help increase both metrics? we leave these questions for future work. ## conclusions & future work in this short work, we evaluate information extraction tasks on a human-in-the-loop, rule-based approach against a state-of-the-art generative language model, gpt-3. our results show that the rule-based model outperforms gpt-3, when used out-of-the-box similar to how an end-user might use it to perform an nlp task. there are potentially better ways of constructing the prompts for gpt-3, but we wanted to better understand the performance of both the rule-based models and generative language model out-of-the-box. future work in this area should look into enabling end-users with little to no technical expertise to accurately and quickly build nlp models. for instance, one possibility is to go beyond user simulations and study how actual end-users create prompts. another possibility is to leverage previous works in automatically generating prompts and then designing ways to elicit user input to craft better prompts @xcite ). an important future direction is to identify disadvantages and advantages of each approach, traditional large language models and rule-based models, and look into how combining such approaches would better enable end-users in nlp.