Evaluating LLM -Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketing (2025)

Abstract: The rise of influencer marketing has blurred boundaries between organic content and sponsored content, making the enforcement of legal rules relating to transparency challenging. Effective regulation requires applying legal knowledge with a clear purpose and reason, yet current detection methods of undisclosed sponsored content generally lack legal grounding or operate as opaque “black boxes.” Using 1,143 Instagram posts, we compare gpt-5-nano and gemini-2.5-flash-lite under three prompting strategies with controlled levels of legal knowledge provided. Both models perform strongly in classifying content as sponsored or not (F1 up to 0.93), with Gemini favouring recall (0.93) and GPT favouring precision (0.95), though performance drops by over 10 points on ambiguous cases. We further develop a taxonomy of reasoning errors, showing frequent citation omissions (28.57%), unclear references (20.71%), and hidden ads exhibiting the highest miscue rate (28.57%). While adding regulatory text to the prompt improves explanation quality, it does not consistently improve detection accuracy. The contribution of this paper is threefold. First, it makes a novel addition to regulatory compliance technology by providing a taxonomy of common errors in LLM-generated legal reasoning to evaluate whether automated moderation is not only accurate but also legally robust, thereby advancing the transparent detection of influencer marketing content. Second, it features an original dataset of LLM explanations annotated by two students who were trained in influencer marketing law. Third, it combines quantitative and qualitative evaluation strategies for LLM explanations and critically reflects on how these findings can support advertising regulatory bodies in automating moderation processes on a solid legal foundation.

## introduction and background the rapid rise of social media has made influencer marketing a central strategy for brands seeking to shape followers' purchasing decisions through influencers' reach and credibility @xcite . while effective at enhancing trust and engagement, this strategy is often opaque, as influencers generally avoid disclosures to maintain authenticity or protect follower engagement. consequently, sponsored content is frequently hidden or inadequately disclosed @xcite , limiting the consumers' ability to recognise advertising foot_0 and making regulatory oversight difficult. distinguishing ads from organic posts can be ambiguous (figure 1 ); tagged brands may signal sponsorship or merely personal preference. even with close scrutiny, regulators can misjudge cases, risking unfair penalties for legitimate influencers and causing complaints, as seen in @xcite before the dutch self-regulatory body stichting reclame code (src) foot_1 , where an independent jury justified its decision using legal reasoning. the lack of transparency in influencer marketing is the largest issue consistently identified by self-regulatory bodies @xcite @xcite . self-regulators are industry organisations that make private rules for businesses. the main challenge for such bodies trying to measure compliance with their own rules in practice is the sheer amount of social media posts that can potentially contain commercial content. the fact that social media platforms do not allow anyone to thoroughly search their databases further complicates the enforcement of transparency standards. for practitioners, separating organic content from ads is the first step in assessing the compliance of influencer marketing with advertising law and selfregulatory codes. this is a laborious process that requires experts to spend their time viewing social media posts that might not contain any advertising. commercially available software platforms aid in this process by using keyword filters 3 , which are usually not accurate enough to eliminate all organic posts from a sample. in response to these challenges, computational research has sought to automate the detection of undisclosed ads @xcite @xcite @xcite , but current methods face two limitations: (1) they often lack a solid legal foundation, exposing regulators to pushback in relation to their decisions, and (2) they prioritise accuracy over explanation @xcite . without a reasoning process, detection systems risk crossing boundaries that may conflict with free speech or other protected interests @xcite . large language models (llms) may offer promising ways to address both of these gaps. they can be prompted to reference relevant legal rules and provide explanations for their outputs, which makes the results more transparent and easier to interpret @xcite . at the same time, llms are prone to errors (e.g. hallucinations, weak grounding @xcite ). this paper explores the potential of llms for detecting undisclosed influencer marketing by examining how they identify hidden advertising and evaluating the quality of their accompanying legal reasoning. our main contributions are (1) a taxonomy of common errors in legal reasoning generated by llm, extending previous research to a complex domain-specific context, namely, the detection of undisclosed advertisements on social networks, which also serves as a broader example of automatic compliance monitoring; (2) an original dataset of llm explanations annotated by two students who were trained in influencer marketing law; ## study design and methodology in this study, we evaluate how different llms classify influencer content and produce legal reasoning to justify their identification of advertising in the dutch context. to this end, we created a dataset consisting of three types of content: disclosed advertisements, hidden advertisements, and organic posts (details are provided in the following section). the dataset is first fed into three different llms under three prompting strategies. each model produces two outputs: (1) a binary classification indicating whether the post constitutes an advertisement, and (2) an accompanying explanation with legal reasoning to justify the decision. then, for all posts and each type of content, we use two methods to examine the outputs: quantitative evaluation: we assess advertisement/organic content classification performance using standard classification metrics. this enables performance comparisons both within and across categories, and allows us to select the two bestperforming models to proceed to the next step. limiting further evaluation to these top-performing models helps avoid redundant comparisons and streamlines the analysis process. as a baseline, we use a tf-idf (unigrams, bigrams) representation combined with logistic regression, employing an 80:20 train-test split. we did not include other deep learning models, such as bert, as prior work suggests that they perform even worse in this context @xcite . qualitative evaluation: we manually select balanced representative cases from each content type. research assistants review the explanations by rating their helpfulness and annotating error types. this reveals systematic patterns linking specific errors to content types and prompting strategies. we also provide a case analysis, where a senior legal researcher reflects on the textual quality of a selection of outputs. ## results we first evaluate the classification performance of three llms under three prompting strategies across the entire dataset in a zero-shot setup (i.e., without fine-tuning). based on these results, we select the two best-performing llms for subsequent tasks, which include evaluating classification performance on each type of content and examining the quality of their explanations. ## discussion and conclusion this study examined how large language models (llms) can be applied to detect undisclosed advertising on social media while providing legal reasoning. unlike prior research, which focused almost exclusively on classification accuracy, our work systematically evaluates both the quality of classification and the legal soundness of llm explanations. this dual lens highlights critical gaps in current practice and suggests pathways toward more transparent and accountable automated moderation systems. starting from the classification task, both gpt-5nano and gemini-2.5-flash-lite achieve high overall accuracy in identifying advertising content, but model choice strongly influences both classification strength and error profile: gemini is more effective for recall-oriented tasks such as detecting hidden ads, whereas gpt excels in precision. notably, llms are not always superior to simple baselines in overall classification performance; however, they perform better in challenging cases. similar patterns of strength appear in the 95 ambiguous posts, where annotators (ann1 and ann2) disagreed or expressed uncertainty in the advertisement annotation procedure (section 3.1). examining the content further, these patterns of ambiguity align with previous findings, which attribute annotator disagreement to both data-related factors (e.g., various language features, uncertainty in sentence meaning), and annotator-related factors (e.g. various language features, uncertainty in sentence meaning) jiang and de marneffe (2022); plank (2022-12); @xcite . these intrinsic complexities pose challenges for llms, contributing to lower performance in ambiguous contexts. moreover, increasing the amount of embedded legal text does not consistently improve the classification outcomes. while prompts containing full regulatory codes and explanations raise the perceived helpfulness of llm reasoning (e.g., gemini article_explanation reaching 4.37 ± 0.75 versus 4.20 ± 1.10 for gpt), they do not guarantee better moderation outcomes. this indicates that current llms do not simply "read and apply" legal norms; instead, they rely heavily on internal heuristics and contextual associations. in practice, this means that llms are already capable of recognising different forms of advertising because promotional language and stylistic cues are strongly represented in their training data. cues indicating sponsorship, patterns of product placement, or persuasive rhetorical devices can often be detected without direct reference to regulatory codes. in this sense, the models' performance may reflect an underlying competence in identifying pragmatic markers of advertising, rather than understanding and applying legal knowledge as a content moderator. the explanation analysis further reveals systematic weaknesses. citation-related errors, missing (e2, 28.57%), unclear (e3, 20.71%), or wrong interpretations (e1, 8.57%), dominate across settings, particularly when no legal text is provided. even when legal sources are available, models often select irrelevant provisions or fail to structure reasoning in a way consistent with basic legal methodology. more severe hallucinations of legal citations (e4, 2.62%) and content (e5, 2.38%) are rare but concentrated in no_article prompts, where gpt and gemini fabricated legal references at 7.14% and 5.71%, respectively. these patterns suggest that llms tend to approximate legal reasoning rather than reliably apply normative rules, which essentially means that they fail to 'read, understand, and apply.' a closer look by content type further illuminates these limitations. undisclosed ads produce the highest rate of misidentified cues (e6, 28.57%), showing that llms frequently mistake ordinary or ambiguous content for sponsored posts. in contrast, disclosed ads show almost no hallucinations when legal text is provided, indicating that straightforward content allows llms to stabilise their reasoning more reliably. together with the case analysis carried out, these findings confirm that although llms can approximate legal reasoning, they are far from delivering rigorous justifications akin to an expert with domain-specific knowledge. these findings have two broad implications for moderation. first, they demonstrate that high classification accuracy does not ensure trustworthy enforcement. an llm that labels a post correctly but cites irrelevant or fabricated legal provisions cannot satisfy procedural fairness standards. second, explanation quality varies systematically by content type and prompting strategy, meaning that moderation pipelines cannot rely on a one-size-fitsall approach. platforms using llms for detection must pair performance metrics with legal-reasoning audits to ensure that decisions are not only correct but also defensible. in practice, this means building tools that flag cases with high-risk errors (e.g., e4/e5 hallucinations) for human review and calibrating models to reduce over-classification in ambiguous contexts. these findings also connect to broader debates on moderation with llms. as @xcite argues, nlp research must be situated within regulatory studies to avoid regulatory capture and to bridge the "pacing gap" between technological innovation and legal adaptation. our results reflect this concern: models that appear accurate can still misapply or fabricate legal norms, undermining the legitimacy of enforcement. treating moderation as a purely technical task risks obscuring the regulatory standards it is supposed to serve; instead, explanation quality and legal soundness must be foregrounded alongside accuracy. at the same time, our taxonomy of explanation errors resonates with emerging moderation research that highlights the concerns of llms as moderators. @xcite demonstrates that binary safe/unsafe labels miss important gradations of harm. similarly, in our research, not all explanation errors are equally harmful: vague reasoning may be tolerable, but fabricated citations or misapplied provisions threaten procedural fairness. integrating severity-sensitive auditing into compliance monitoring would thus allow regulators to triage high-risk cases while ensuring that enforcement remains both effective and legitimate. the main contribution of this paper is to integrate the quality of legal reasoning in the evaluation of influencer marketing detection systems. by developing a taxonomy of llm explanation errors and showing how these patterns vary by model, prompting strategy, and content type, we provide an actionable framework for regulators and platform designers. instead of treating llm outputs as opaque predictions, our study demonstrates how to assess whether automated moderation is not only accurate but also legitimate. this is particularly valuable for self-regulatory bodies such as stichting reclame code (src), which must justify enforcement decisions in legal terms rather than through statistical metrics alone. more broadly, our multidisciplinary approach, combining computational evaluation with legal analysis, offers a blueprint for building moderation systems that are transparent, explainable, and aligned with rule-of-law principles rather than black-box heuristics.