CICL e: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification (2024)

Abstract: Contaminated or adulterated food poses a substantial risk to human health. Given sets of labeled web texts for training, Machine Learning and Natural Language Processing can be applied to automatically detect such risks. We publish a dataset of 7,546 short texts describing public food recall announcements. Each text is manually labeled, on two granularity levels (coarse and fine), for food products and hazards that the recall corresponds to. We describe the dataset and benchmark naive, traditional, and Transformer models. Based on our analysis, Logistic Regression based on a TF-IDF representation outperforms RoBERTa and XLM-R on classes with low support. Finally, we discuss different prompting strategies and present an LLM-in-the-loop framework, based on Conformal Prediction, which boosts the performance of the base classifier while reducing energy consumption compared to normal prompting.

## introduction food-bourne illnesses and contaminated food pose a serious threat to human health and lead to thousands of deaths @xcite . natural language processing (nlp) solutions based on machine and deep learning (ml, dl) or large language models (llms) enable fast responses to new threats by generating warnings from publicly available texts on the internet @xcite . these texts, however, as we show, can be noisy and are characterized by thousands of classes. categorization of incidents into these classes is needed to properly address food risks, as different food risks require different countermeasures with different urgency. for example, while a fraudulent import of foodstuffs usually imposes a very low risk to consumer health, listeria or salmonella contamination is a serious risk to consumer health and can even lead to death. automatic classification of the hazard involved in food recalls can help estimate the severity of food risks and lead to faster and more targeted measures. while we are aware of existing text datasets on the topic of food-bourne illnesses @xcite , they focus only on the detection of such illnesses and not on their categorization. publicly available official food recall announcements, and specifically their shorter titles, contain detailed information on food risks and the specific products involved and may be used as a basis for data-driven ml approaches. acknowledging this research gap, this paper is specifically addressing the following research questions: r1: are titles of food recall articles sufficient for classification of the food hazards and food products involved? as llms like gpt-4, and llama @xcite represent the current state-of-the-art in nlp, but are also known for their high resource consumption, we further explore the following question: r2: can llms be used to address or improve automatic food hazard classification? in this work, we address the above task of classifying food hazards and the involved products from food recall texts and provide the following main contributions: research on ml with food data has traditionally focused on tabular and image data @xcite @xcite , where food risk classification is usually presented as a detection task: rather than identifying specific hazards, the task is formulated as the binary classification of the presence or absence of an incident @xcite @xcite . focusing on the text modality, we found literature exploring such detection tasks based on amazon reviews matched with fda food recall announcements @xcite , as well as twitter data, labeled using instances from the us national outbreak reporting system @xcite . in both of these papers, the data is highly unbalanced towards the negative class (i.e. no incident). furthermore, the description of the method for labeling the twitter texts in @xcite lacks important details, as it is unclear on what grounds texts and events are matched. we only found one study on the actual classification of hazards: @xcite which uses a hierarchical transformer model to classify chinese news texts into four categories of hazards. the authors do not share further details on their data, which makes an assessment of the paper's validity hard. overall, the field of food risk classification based on text lacks reproducible work on fine-grained prediction of hazards based on openly accessible data. ml models based on the attention mechanism by @xcite and the self-supervised masked language modeling pre-training paradigm generally outperform traditional methods both on monolingual @xcite and multilingual @xcite data. although such transformers were origi-nally described as an encoder-decoder architecture @xcite , mapping an input-text to an output-text, models intended for classification tasks usually only employ an encoder @xcite @xcite , followed by several task-specific layers. recently, llms such as llama 2 @xcite , @xcite , and gptfamily models @xcite have been shown to exceed the capabilities of smaller transformers even without further fine-tuning. that is, by simply providing a context of a few labeled samples per class, llms can predict the classes of unseen samples within this context. as llms are usually text-to-text transformers, the context is provided directly in each prompt. this paradigm is commonly referred to as few-shot prompting or in-context learning @xcite . since the discovery of the few-shot capabilities of llms, newer llms are often designed for high few-shot performance @xcite . current work on few-shot prompting focuses mostly on creating/finding the optimal few-shot samples from the training data. @xcite proposed a workflow for automatically finding similar samples from the pool of labeled "training" samples for code summarization. @xcite focused on automatic generation of chain of thought (cot) labels for samples in reasoning tasks. cot is another prompting paradigm that asks the llm to provide a chain of reasoning before delivering the prediction and has been shown to drastically improve reasoning performance @xcite . nevertheless, to the best of our knowledge, there is no previous work on how to leverage few-shot prompting in llms for multi-class prediction problems with a high number of classes. while prompting with long contexts may "confuse" the llm @xcite , the context window may even be too small to support a view of all classes in such cases. furthermore, the previous approaches focus on selecting specific samples rather than relevant classes for the prompt. the cp framework @xcite is a method for associating predictions of a classification algorithm with confidence guarantees. essentially, cp selects sets of classes from the predicted labels of a classifier or regressor that statistically contain the true label with a predefined probability. we note, however, that the standard cp setting pro-vides guarantees for the individual class labels; i.e., errors for one of the labels can, hence, exceed the set guarantees. a formulation, called mondrian of cp, can make the guarantees hold also for the individual labels. note that cp does a very good job of keeping the output (set of labels) small, while the prediction guarantees hold. it can be applied to any classification (or regression) model, as long as a calibration set is set aside together with a measure of the model's prediction error with regard to the true labels. this measure is referred to as the "nonconformity function". in our context, the stronger the model, and the better the non-conformity function, the fewer labels (hence, shots) will be produced with the same guarantee. we point to the work of @xcite ; johansson et al. (2014 ## the food recall dataset in this section, we first describe the dataset used and shared in this paper, then discuss the labeling process and quality, as well as class distribution. ## methodology this section describes the problem setting and the methods utilized in our empirical evaluation. ## empirical analysis in this section, we present the experimental setup and the experimental results. ## discussion our results suggest that a general reduction of fewshot examples, taken from a population that is very likely to contain the true class, can outperform traditional ml classifiers and transformers in terms of f 1 even without using other prompting techniques, e.g., example matching @xcite or cot. this makes simple selection of few shots based on cosine similarity (gpt-simk) a powerful approach, especially for tasks with a high number of classes or extraction tasks. in this setting, we leverage the well-documented strong in-context reasoning capabilities of llms @xcite @xcite , which do not necessarily need a full view of all possible classes. however, in cases where we are able to train a decently performing base classifier, cicle is a better choice: it boosts the performance of the base classifier by detecting insecure samples using cp and redirecting them to an llm. further analysis (see appendix d.3) shows that cicle's strength is mainly based on better performance on low-support classes compared to the base classifier. this is most probably owed to the low demand for "training data" in few-shot prompting. in general, the influence of the base classifier and llm on cicle's performance is controlled by the α value: low values favor the llm, while high values favor the base classifier. lastly, cicle is subject to a trade-off between predictive and temporal performance: it allows sacrificing accuracy (e.g. due to hardware and time constraints) while keeping the context for the llm as concise as possible. this makes it a resource-friendly alternative to prompting with all classes. our results further suggest, in line with @xcite , that sample order and context length are highly influential on prompting performance. we therefore see a need for future work on optimal ordering of few-shot samples and tuning of the α parameter within cicle. ## conclusions we present the first dataset for large-scale multiclass classification of short texts describing food recalls, which we release publicly under a cc by-nc-sa 4.0 license. the dataset contains labels annotated by experts on food products and hazards on two levels of granularity, coarse (tens) and fine (hundreds). answering our research question r1, we find that the titles of food recall announcements mostly hold sufficient information for classification, although not all of them include information on both product and hazards (r1.1). a strong indication for this statement is that we are able to reach above random performance with multiple classifiers: we carry out benchmarking, analysing the performance of naive, traditional, and deep-learning classifiers for all four classification tasks in the proposed dataset, showing the potential of traditional classifiers on this dataset. our results further indicate that products are harder to classify than hazards, which we attribute to the often creative choice of product names (r1.2). moving on to r2, we also experimented with prompting using one few-shot example per class. here we see that gpt-3.5 performs comparably or better than the best non-llm classifiers (r2.1). we also show that resource-efficient prompting at comparable predictive performance is possible by proposing a novel llm-in-the-loop framework that leverages conformal prediction (r2.2).