AIG uard: A Benchmark and Lightweight Detection for E -commerce AIGC Risks (2025)

Abstract: Recent advancements in AI-generated content (AIGC) have heightened concerns about harmful outputs, such as misinformation and malicious misuse.Existing detection methods face two key limitations:(1) lacking real-world AIGC scenarios and corresponding risk datasets, and(2) both traditional and multimodal large language models (MLLMs) struggle to detect risks in AIGC.Towards this end, we introduce **AIGuard**, the first benchmark for AIGC risk detection in real-world e-commerce. It includes 253,420 image-text pairs (i.e., the risk content and risk description) across four critical categories: *abnormal body*, *violating physical laws*, *misleading or illogical context*, and *harmful or problematic message*.To effectively detect these risks, we propose distilling text annotations into dense soft prompts and identifying risk content through image soft prompt matching during inference.Experiments on the benchmark show that this method achieves a 9.68% higher recall than leading multimodal models while using only 25% of the training resources and improving inference speed by 37.8 times.For further research, our benchmark and code are available at [https://github.com/wenh-zhang/aiguard-dataset](https://github.com/wenh-zhang/aiguard-dataset).

## introduction recent advancements in aigc have significantly improved creative workflows in text @xcite , image @xcite , and video generation @xcite , demonstrating substantial commercial potential. ## risk detection problem formulation detecting risky content in ai-generated ecommerce images involves predicting a probability ŷ ∈ [0, 1] for a given image i, representing the likelihood that the image contains risky content. however, purely predicting the label may overfit to specific patterns and lack interpretability, which undermines generalization and contravenes the principles of developing robust detectors. therefore, we formalize the task as interpretable risk detection, where models generate textual explanations @xmath0, w 1 , ..., w t ) that explicitly identify and describe harmful content while aligning with expert annotations. here, t denotes the number of decoding steps. the goal is to learn a function f θ : @xmath1, s} that jointly optimizes classification accuracy (e.g., f1, auc-roc) and explanation verifiability, ensuring that predictions are grounded in causal expert rationales rather than spurious correlations. ## benchmark description this section outlines the construction of the dataset, detailing the workflow for collecting aigc images, the expert annotation procedure, and the construction of the benchmarks. requirement aigc platform product display e-commercial merchants flux (for product) online marketing platform merchant normal product images with new background scenario product images prompts the sneakers are placed on the ground... lora 1 lora 2 lora ••• lora 3 lora ••• cutout tool original images select from: risky product images with new background platforms generated a significantly large volume of images. to filter out normal images, we employ a multi-stage process combining user complaints, manual inspection, and model-based checks. specifically, all user-submitted complaints flagged with risky tags are subjected to expert annotation procedures. for the large unlabeled dataset, we first recruit professional annotators to manually label the images. subsequently, we train a vit-s/16 model @xcite , a small version of the vision transformer, using professionally annotated data. the trained model is employed to filter out images identified as deemed certainly non-risky, ensuring the resulting dataset contains only high-quality samples with potential risks relevant to real-world e-commerce platforms. the final dataset com-prises 253,420 samples, including 43,885 risky and 209,535 normal instances. ## lightweight detection method to balance efficiency and effectiveness, we develop a lightweight detection model based on the blip framework @xcite , which unifies image-text contrastive learning (itc), image-text matching (itm), and language modeling (lm) to achieve strong performance across multimodal tasks. specifically, we distill expert-annotated risk information into soft prompts using image soft prompt matching and language modeling tasks. during inference, we rely solely on image soft prompt matching to reduce detection time. the framework overview is illustrated in figure 3 . ## discussion based on the dataset and experimental results, this section highlights key challenges and emerging research opportunities for advancing aigc safety. (1) advanced risk detection: though lightweight detection methods show promise, significant improvements are still needed. existing approaches struggle particularly with harmful or problematic message risks, where threats are wellhidden and demand more generalizable solutions. furthermore, as detection methods improve, malicious users may adapt their tactics to hide il-12444 legal content, highlighting the need for robust and adaptable detection strategies. (2) mllms and aigc risks: our experiments reveal that current mllms often fail to detect subtle risks in aigc due to their training on standard realworld data. this underscores the need to expose mllms to aigc outputs, particularly adversarial or risky content, to enhance their understanding. with the increasing prevalence of aigc, it is crucial to introduce aigc content to mllms and develop specialized algorithms (e.g., contrastive learning, adversarial training, pre-training), to improve the models' ability to recognize and process aigc content effectively. (3) aigc alignment datasets: current alignment efforts mainly focus on real-world images. as these data sources become limited, using composite data generated by models can enhance model understanding and improve generalization from weak to strong. future work could expand the dataset to aigc data, which can be generated at a low cost and with controlled parameters. our dataset can also serve as a foundation for building alignment datasets to improve models' comprehension of e-commerce aigc content. (4) safer aigc generation: this work provides a real-world scenario for studying the safety of aigc generation. as the source of this problem, we also highlight the need to develop safer and controllable generation methods, which can address risks in aigc content at root. ## conclusion this work introduces aiguard, the first benchmark designed to detect aigc risks within ecommerce contexts. the benchmark comprises 253,420 image-text pairs, each annotated with corresponding risk information. then, we propose an effective and lightweight detection method that distills risk annotations into learnable soft prompts via image-text matching and next-token prediction tasks. experimental results demonstrate the superior performance of our approach and provide insights into future directions for developing robust detection methods in real-world systems.