Training Language Models under Resource Constraints for Adversarial Advertisement Detection (2021)

Abstract: Advertising on e-commerce and social media sites deliver ad impressions at web scale on a daily basis driving value to both shoppers and advertisers. This scale necessitates programmatic ways of detecting unsuitable content in ads to safeguard customer experience and trust. This paper focusses on techniques for training text classification models under resource constraints, built as part of automated solutions for advertising content moderation. We show how weak supervision, curriculum learning and multi-lingual training can be applied effectively to fine-tune BERT and its variants for text classification tasks in conjunction with different data augmentation strategies. Our extensive experiments on multiple languages show that these techniques detect adversarial ad categories with a substantial gain in precision at high recall threshold over the baseline.

## introduction all advertisements on e-commerce and social media platforms must be moderated to ensure regulatory and ethical standards in countries where they are being served. a tiered moderation workflow with automated components like cached lookup, ml models, rule based annotators complement human experts to ensure reliable content moderation for ads created by advertisers while scaling to ecommerce advertising volumes. the advertising platform currently enables ads to be created in various media formats like text, images and videos. in this work, we focus on detecting adversarial ads in one broad class of ads, where engagement is driven primarily through text and images. such ads on e-commerce site serve as a casing for the product being advertised. the casing includes product text and image attributes along with optional custom captions provided by the advertiser. it is under the purview of moderation to check whether * work done when at amazon an ad contains prohibited content. any ad containing prohibited content can have an adverse impact on the shopper experience and hence needs to be prevented from showing up. see section 2.1 for a broad overview of the adversarial ad categories. in this paper, we focus on techniques we use to train nlp models built as a part of this system. training any ml model requires a good quality dataset that is representative of the policy being enforced. the quality of data available to train models targeting a defect, say detection of "adult and objectionable content" depends on several factors. typically occurrences of such products are rare but the impact of such an ad on shopper experience is adverse. the uncommonness of these violations makes curating large in-domain monolingual corpora difficult. this problem is compounded in low resource languages where there are limited linguistic resources and the rarity of these violations are even more skewed. further, it is expensive and time consuming to gather more labeled data. through this paper, we show different ways to train generalised language models when we have limited labeled data. we suggest various ways for data augmentation and empirically provide evidence suggesting when each of the approaches works best. we explore how we can leverage the product catalogue and user behaviour in weak and semi-weak supervision, curriculum learning and multilingual training strategies to train generalised language models like bert @xcite and its variants. our experiments show : ## finetuning bert under low resource constraints we explore various techniques that can be used to train generalised language models(glm) like bert and multilingual variants with significant performance gains over baseline models described in section 2.3. we look at resource constraints during training of machine learning models in a supervised setting attributed to the following cases: we leverage product catalog to source data for weak and semi-weak supervision training in monolingual setting. we also explore how curriculum strategies and multilingual training can benefit training text classifiers for low resource languages. our experiment show that generalised language models like bert or multilingual variants like m-bert can be trained using these techniques with significant performance gains over baseline model described in section 2.3. ## results in all experiments, we track model performance using precision and recall. precision indicates the fraction of ads correctly rejected by model. recall indicates the fraction of true defective products rejected by the model for a particular category. ss and sws for en ## conclusion we have explored multiple ways of training a glm and it's multilingual variant in low resource settings. when large in-domain monolingual corpora is present but labeled data is limited, sourcing weak data applied in semi and semi-weak supervision training improves model performance consistently. curricula are useful in resource constrained settings. multilingual training on a lexical similarity based curriculum is useful when target language is lexically closer to resource rich languages. alternate curriculum like sample hardness is useful in low resource languages which are lexically distant to resource rich language such as en.