Textual Content Moderation in C 2 C Marketplace (2022)

Abstract: Automatic monitoring systems for inappropriate user-generated messages have been found to be effective in reducing human operation costs in Consumer to Consumer (C2C) marketplace services, in which customers send messages directly to other customers. We propose a lightweight neural network that takes a conversation as input, which we deployed to a production service. Our results show that the system reduced the human operation costs to less than one-sixth compared to the conventional rule-based monitoring at Mercari.

## introduction mercari is a c2c marketplace app available in japan and the united states. the application operates along the lines of an online flea market in which any customer can list items for sale and purchase others' listings. to prevent unpleasant customer experiences and unnecessary difficulties, mercari defines some guidelines for using the platform. if a customer violates the guidelines, moderators from the customer support of mercari may give them a warning to protect the safety and trustworthiness of the market. to prevent abuse, platforms try to screen and monitor user-generated content. this is a humanintense task known as content moderation, and some companies like appen, facebook, and pinterest introduced content-based deep learning approaches @xcite to content platforms in recent years. likewise, mercari operates such moderation systems for several domains to maintain a safe and enjoyable marketplace. we observed that most difficulties are encountered while sellers and buyers process with their transactions. in contrast to a normal b2c ecommerce pattern, sellers and buyers need to send * work performed while at mercari, inc. messages to complete transactions, even though they are often both inexperienced customers. for example, customers might send messages to each other to provide specialized delivery information. moderation of user messages is thus a crucial measure to prevent any forms of abuses, e.g. the transmission of offensive messages to others, inducing others to conduct an offline transaction, or even urging buyers to pay upfront via external services. these abuses are commonly detected by keyword-based monitoring or patrol (random inspections made by moderators of the customer support), which has been noted as being extremely limited in terms of scale and efficiency. in this work, we introduce a scalable textual content moderation system based on machine learning models to perform the message moderation task. foot_0 figure 1 shows an overview of our moderation system. in the proposed moderation system, messages with abusive intent or content that violate the terms and conditions could be reported from these three channels: patrol (random inspec-tions), keyword/rule-based monitoring, and predictions made by machine learning (ml) models. in this work, we found that ml models successfully detected potentially-violated messages with a much higher precision compared to the conventional keyword/rule-based monitoring. however, because the type of violations may evolve over time, (e.g., a local government prohibiting the reselling of medical masks during covid-19) updating ml-based methods in a timely manner following the fluctuations of the marketplace is notably difficult. we preserve the rule-based approach to accommodate the latest types of violations, so that moderators can still quickly revise the monitoring set of keywords and rules, which helps the system detect new domains of violations. in addition, a moderation system cannot tolerate false positives of any misjudged violation which might negatively effect the customer experience. all of the messages considered as potential violations by our moderation system are thus checked manually by moderators. this ensures that the customer experience on the marketplace is not compromised, and also creates a perfect human-in-the-loop cycle. to better detect offending messages and improve on the accuracy of existing ml models, we used these human-checked messages as an accurately-labelled dataset for model re-training. the contributions of the present work are summarized as follows. in contrast to typical b2c or b2b2c platforms, in which every item for sale can often be accessed with a single click, communication is inevitable on a c2c platform. therefore, commercial content moderation plays a vital role in the process of communication to complete transactions successfully @xcite . because manual content moderation does not scale to large platforms, it is natural to introduce automated content moderation systems. @xcite ; @xcite investigated the applicability of machine learning-based approaches to the facilitation of content moderation by detecting textual violations using language models. in the same context, to relieve human resources in the task at mercari, @xcite introduced a machine learning-driven item screening system that detects banned listings such as weapons, money, and medicine. ## conclusion in this work, we have proposed a scalable mlbased violation detection system designed to accept multi-round user conversations and some categorical features. a variety of violation domains are served as individual components in an one-vs-rest fashion to retain scalability to high volumes of requests and flexibility on targeting domains. in comparison with conventional rule-based monitoring, we have demonstrated that the proposed ml-driven approach successfully reduced the workloads of moderators in customer support to less than onesixth of their previous levels by automatically detecting abusive messages.