Assessing In-context Learning and Fine-tuning for Topic Classification of G erman Web Data (2024)

Abstract: Researchers in the political and social sciences often rely on classification models to analyze trends in information consumption by examining browsing histories of millions of webpages. Automated scalable methods are necessary due to the impracticality of manual labeling. In this paper, we model the detection of topic-related content as a binary classification task and compare the accuracy of fine-tuned pre-trained encoder models against in-context learning strategies. Using only a few hundred annotated data points per topic, we detect content related to three German policies in a database of scraped webpages. We compare multilingual and monolingual models, as well as zero and few-shot approaches, and investigate the impact of negative sampling strategies and the combination of URL & content-based features. Our results show that a small sample of annotated data is sufficient to train an effective classifier. Fine-tuning encoder-based models yields better results than in-context learning. Classifiers using both URL & content-based features perform best, while using URLs alone provides adequate results when content is unavailable.

## introduction text classification of webpages is used to understand information consumption by categorizing large collections of individuals' browsing histories (e.g., @xcite . by categorizing webpages, researchers can identify patterns of online news consumption @xcite and quantify exposure to populist sentiments @xcite . analyzing browsing histories by topic often necessitates "finding the needle in the haystack", as typically just a small fraction of webpage visits correspond to a given domain, such as news sources @xcite . therefore, identifying the few relevant pages among numerous unrelated visits makes manual labeling impractical. machine learning classifiers are often used as an automated and scalable alternative @xcite . since the introduction of the transformer architecture, fine-tuning pre-trained language models (plms) such as bert @xcite has seen widespread adoption in text classification tasks. applications include classifying public opinions about policies in digital media @xcite and identifying protest-related content in newspaper articles @xcite . further applications encompass sentiment analysis on social media posts @xcite and advertising @xcite . however, finetuning classifiers still requires hundreds to thousands of manually labeled documents. given the multilingual nature of the web and the noisy data resulting from the scraping process, compiling a representative training set remains a complex and time-consuming task. generative models such as llama @xcite and mistral @xcite are often inherently multilingual and can generalize to completely unseen tasks without the need for fine-tuning, potentially making them a promising alternative. in this study, we investigate the use of large language models (llms) for the task of binary topic classification across a corpus of scraped webpages. we evaluate our approach by identifying webpages that provide information on three specific german policies discussed during data collection: (1) a policy introduced to combat child poverty, (2) the promotion of renewable energy, and (3) the amendment of cannabis legislation. we compare the classification accuracy between multilingual @xcite and monolingual @xcite pre-trained language models by fine-tuning them on manually labeled data. our analysis extends to generative models @xcite , evaluating few-shot prompting for document classification and assessing the impact of demonstrator sampling strategies. ## related work political and social sciences researchers increasingly use topic classification to filter large collections of webpages derived from browsing histories @xcite . this task is commonly modeled as binary or multiclass classification, assigning text segments to one or more predefined categories. until recently, researchers in these applied fields have relied on traditional nlp methods such as naive bayes classifiers @xcite and logistic regression models @xcite . the adaptation of bert models created new opportunities by improving classification accuracy. for instance, @xcite fine-tuned bert models to classify opinions on policies in digital media. similarly, @xcite explored the use of bert variants for classifying sentences in newspaper articles to detect protest-related content. @xcite applied a logistic regression model for classifying the topics of parliamentary speeches. research on webpage classification also includes the use of url features @xcite , extracted content @xcite , graph representations @xcite , and visual features @xcite . ## dataset for our experiments, we use a corpus of scraped webpages annotated by topic. we describe the data collection and annotation process in section 3.1. the topic labels correspond to three german policies that were of interest during the period of data collection: (1) basic child support policy (kindergrundsicherung), introduced to combat child poverty, (2) energy transition policy (förderung erneuerbarer energien), designed to promote renewable energy, and the (3) cannabis legalization amendment (cannabislegalisierung). we refer to these policies as the children, energy, and cannabis policies throughout this paper. our dataset contains substantially more topic-unrelated than relevant webpages. this exemplifies a common challenge in the social, political, and communication sciences: finding relevant content within a vast database of unrelated webpages. ## methods we model the detection of topic-related content as a binary classification task for each of the three topics. we compare the f1-scores of fine-tuned encoder models (supervised) and in-context learning strategies (few/zero-shot) against suitable baselines. figure 1 shows a schematic overview of the supervised training and classification pipeline. the evaluated llms are listed in to aggregate the predicted labels for chunks into document level labels during inference, we assign a positive label to webpages if the label of at least one chunk is predicted to be topic-relevant. ## conclusion we compare the performance of fine-tuned encoder models against in-context learning strategies for the classification of topic-related content. using only a few hundred positively annotated data points per topic, we detect content related to three german policies in a database of scraped webpages. the best supervised classifier, gelectra-large, using url & content features, achieves an average f1 score of 0.430 over all topics, performance varies by topic. it performs well on the children and cannabis topics but performs suboptimal in terms of precision for the energy topic. all fine-tuned models achieve strong performance on the high-quality balanced test set, re-gardless of using url or content-based features. however, performance declines substantially on lower-quality and unbalanced data, with high recall but lower precision due to more pages being falsely labeled as topic-related. while recall remains high across all topics and test sets, precision drops considerably, leading to a substantial number of false positives, which indicates that the model is overly sensitive to keywords that are topic-related but also occur in other contexts. webpage content proved to be a strong signal for classification over urlbased baselines, and classifiers that combined url & content-based features perform best. in cases where content-based analysis is infeasible, urlbased classifiers can provide an adequate baseline performance, although the precision-recall tradeoff in settings with real-world data requires a careful approach. however, a manual error analysis revealed that the classifiers struggle to distinguish between weak and strong relations to the topic, with url-based labels leading to incorrect associations of boilerplate texts with the topic. an investigation of more elaborate chunk pooling and combination strategies in future work is needed. additionally, incorporating loosely topic-related negative examples into the training data would likely improve classifier precision by enabling better differentiation between relevant and non-relevant instances. for instance, online shops that advertise cannabis or solar panels are relevant to the topic in general but not in the sense of political policy discussion. our evaluation shows high accuracy for zeroand few-shot prompting without fine-tuning, indicating their potential in data-constrained situations. few-shot learning can be viable when runtime is less critical, but labeled data is expensive. however, fine-tuning encoder-based models generally yields better results and should be given preference over in-context learning for annotating large datasets. future work. it is likely that classifier precision can be enhanced by filtering out topic-unrelated chunks and training a content-only classifier to remove unrelated content. to address the limited number of positive examples, data augmentation appears like a fruitful addition to the pipeline. for in-context learning, advanced prompting methods such as prompt chaining and chain-of-thought prompting are likely to enhance llm reasoning.