Detecting Continuously Evolving Scam Calls under Limited Annotation: A LLM -Augmented Expert Rule Framework (2025)

Abstract: The increasing prevalence of scam calls, particularly on online platforms for recruitment, ride-hailing, and delivery services, has become a significant social and economic issue. Traditional approaches to scam call detection rely on labeled data and assume a static distribution of scam narratives. However, scammers continuously evolve their tactics, making these methods less effective. In this paper, we propose a novel approach leveraging large language models (LLMs) to detect continuously evolving scam calls. By abstracting scam and normal call rules based on expert knowledge, we develop a hierarchical few-shot prompting framework. This framework consists of a discrimination module to identify scam characteristics, a reflection module to reduce false positives by comparing with normal call features, and a summary step to synthesize the final detection results. Our method is evaluated on real-world and synthesized datasets, demonstrating superior performance in detecting evolving scam calls with minimal labeled data. Furthermore, we show that the framework is highly adaptable to new scam detection scenarios, requiring only modifications to the expert rules.

## introduction with the development of network and communication technologies, voice calls have become an indispensable part of our daily lives. while bringing great convenience, they are also exploited by malicious actors to conduct scam activities, causing severe economic and social problems. @xcite beyond conventional inter-personal calls, voice calls are also widely used by many online platforms specialized for recruiting, car-hailing, food/package delivery etc., under which relevant parties are often required to use platform-embedded calls to communicate with each other for better service assurance. however, * corresponding author due to the exposure of phone numbers, a surge of scam activities are observed on these platforms, e.g., defrauding job seekers or e-commerce shoppers by pretending to be a recruiter or package deliverer. hence, given the huge volume of daily calls, it is of great importance to detect scam calls automatically on these platforms. to detect scam calls, @xcite ; @xcite ; @xcite propose to first convert calls into texts and then train a text classifier to discriminate between normal and scam calls. despite promising results have been reported, these methods are generally built on two assumptions: 1) sufficient annotation of scam and normal calls; 2) training and testing datasets coming from the same distribution. but in practice, due to the high labeling cost and low occurrence rate of scam calls, assuming a large number of labeled scam calls is often unrealistic. more seriously, scammers do not stick with the same deceptive narrative over time, but instead will change them from time to time to enhance their credibility. tabel 1 show two examples of a type of scams, in which scammers first pretend to provide opportunities for job seekers to switch to a well-paid industry and then persuade them to accept the associated job training scheme. as seen in the table, scammers in period 1 frequently mention "xxx software technology company" and "software testing", but later shift to narratives of providing opportunities in live-streaming marketing in period 2 when live-streaming gains popularity. when the classifier is trained on data from period 1, coupling with the availability of only a small number of labeled examples (e.g., tens or hundreds), it easily renders the classifier to wrongly believe that "software" is highly relevant to scams, making it hard to detect the scams in the narrative from period 2. obviously, this kind of methods fall short in recognizing continuously evolving deceptive narratives, even if the core deceptive trick is unchanged. example 1 (period 1) 我们是xxx软件技术公司的，我们这边是做软件测试的，你之前 有了解过这个行业么？没做过也没关系，我们会提供一个培训服务， 我们这儿将来也会是一个重要行业，赚的也很多。 trans: we are from xxx software technology company and we are engaged in software testing. have you ever known about this industry? it doesn't matter if you haven't done it before. we will provide training services. this will be an important industry in the future and earn a lot of money. example 2 (period 2) 我们是xxx公司，您这块做过直播带货的工作么，想要尝试一下么？ 我们前期会有老员工带带你，教你商品介绍、推销话术之类的，对 对对相当于一个培训的过程。 trans: we are xxx company. have you worked in live-streaming marketing before, or would you like to give it a try? we will have experienced employees guide you, teaching you product introductions, sales pitches, and so on. yes, it's essentially a training process. to have the methods better adapt to continuously evolving scenarios, a simple way is to couple with domain adaptation methods @xcite @xcite to learn domain-invariant features. but due to the significance of changes in scam narratives as well as the limited annotations, as seen in our experiments, these methods are largely ineffective, too. on the other hand, there also exist some methods that first predefine a list of keywords relevant to scams and then use the keyword-matching method to detect scams @xcite . but due to the existence of countless ways to express the same meaning, these methods often miss to detect lots of scam calls. recently, with the rise of llms, @xcite explored the ways of utilizing gpt-3.5 or gpt-4 to detect scam calls by simply providing the model with several normal and scam call demonstrations. however, it primarily leverages llms' few-shot classification ability, rarely taking advantages of llms' powerful generalization and reasoning capability. inspired by our human beings recognizing scam calls by checking whether the calls comply with a set of scam rules, in this paper, we propose to use llms to replace human beings' role in the process. to this end, we first abstract a set of judgment rules, which are expressed in natural language, based on experts' prior knowledge on scams for each type of scam calls. the set of rules are required to capture the main characteristics of scam calls. in addition to the scam rules, to promote the distinguishability, we also abstract a set of rules for normal rules. to better make use of these rules, we decompose the scam detection task into a series of reasoning steps, leading to a hierarchical few-shot prompting detection framework. specifically, we first design a discrimination module to judge whether a call shows some characteristics of a specific type of scam calls by instructing llms to refer to the scam rules. then, due to the high similarity between scam and normal calls, to reduce the false positive rate, we further propose a reflection module, which essentially asks llms to refer to normal call rules. finally, a summary step is added to output the structured comprehension results. we experiment with our method on a real-world scam call dataset from an online recruitment platform and a synthesized scam call dataset, with the results showing significantly better performance in detecting continuously evolving scam calls with few annotations. we also demonstrate that our framework can be easily transferred to a new scam detection scenario by simply changing the expert rules. we publicize the synthesized dataset, which is modified from real calls, to foster the development of this valuable task. prompting method large language models have demonstrated strong reasoning abilities, improving logical reasoning without requiring parameter updates @xcite @xcite . chain of thought prompting method improves the model's perfor- types key features 1 offering many job positions without details about the roles 2 encouraging people to switch to a new industry, often associated with deceptive training courses 3 offering free meals and accommodation for positions that typically do not provide such benefits 4 offering jobs close to the recruits' locations, regardless of where they are 5 claiming the company has many branches across the country ## problem formulation to mimic the task of detecting scam calls under continuously evolving scenarios, we suppose the training and testing call datasets d t rain and d t est come from two different distributions. in practice, the two datasets could be obtained by collecting calls at two different time periods (e.g., separated by several months). both datasets contain a proportion of scam and normal calls, with the number of normal calls generally much larger than that of scam calls. the scam calls generally include several types, with the number denoted as k. for instance, scam calls collected from an online recruitment platform for low-end jobs include scam types: 1) persuading to accept job training by pretending to provide opportunities to switch to a new well-paid industry; 2) providing jobs with locations at any places as you want, etc. more types are shown in ## methodology to accurately detect continuously evolving scam calls, we find that it is necessary to use large llms with huge parameters, partially because of their more powerful understanding and reasoning abilities, instead of using some simple and small llms. but due to the large volumes of daily calls, if large llms are employed to handle every call, the computational cost would be extremely expensive. to alleviate this issue, we note that most of the calls can be easily judged as normal with some relatively simple models. thus, to reduce the computation cost, we propose to use some light-weight models to first filter out some certainly normal calls, and then only employ large llms to handle the difficult-to-judge calls, as seen in the overall model diagram in fig. 1 . ## conclusion this paper identifies the data distribution shift caused by the evolving nature of scam calls and introduces a framework using large language models (llms) to detect these evolving scams. by leveraging expert-defined scam and normal call rules in a hierarchical few-shot prompting approach, our method ensures stable performance. experimental results show that our approach outperforms existing methods, providing an effective solution to the detection problem. the framework's adaptability to new scam scenarios through simple rule modifications enhances its real-world applicability. we also release the synthesized dataset to support future research.