LLM -Based Robust Product Classification in Commerce and Compliance (2024)

Abstract: Product classification is a crucial task in international trade, as compliance regulations are verified and taxes and duties are applied based on product categories. Manual classification of products is time-consuming and error-prone, and the sheer volume of products imported and exported renders the manual process infeasible. Consequently, e-commerce platforms and enterprises involved in international trade have turned to automatic product classification using machine learning. However, current approaches do not consider the real-world challenges associated with product classification, such as very abbreviated and incomplete product descriptions. In addition, recent advancements in generative Large Language Models (LLMs) and their reasoning capabilities are mainly untapped in product classification and e-commerce. In this research, we explore the real-life challenges of industrial classification and we propose data perturbations that allow for realistic data simulation. Furthermore, we employ LLM-based product classification to improve the robustness of the prediction in presence of incomplete data. Our research shows that LLMs with in-context learning outperform the supervised approaches in the clean-data scenario. Additionally, we illustrate that LLMs are significantly more robust than the supervised approaches when data attacks are present.

## introduction product classification plays an important role in international trade and e-commerce. this is because import and export tariffs are assigned based on the category of products. according to the latest report from world custom organization @xcite , in year 2022-2023 more than 1.3 billion declarations are booked through customs worldwide @xcite . this massive workload, a result of trade globalization, can impose a significant burden on human experts such as customs personnel and the companies involved in import, export, and e-commerce. in addition, product classification can often be a complicated task and require subject matter expertise, as there is a wide range of products traded across various industries. as such, for human personnel to become competent in understanding the nuances of different products and how to classify them in compliance with wco guidelines is a nontrivial task and requires several months of training, according to our subject matter expertise. moreover, correct and detailed classification is critical, as incorrect classification can lead to tax liabilities owed to authorities. this can result in fines, penalties, and in some cases, legal repercussions and business discontinuation bans in the jurisdictions affected by a tax breach. managing the increasing workload of product classification in global trade is difficult. this challenge is further compounded by the continuous globalization of e-commerce. additionally, staying accurate and up-to-date as global trade classification guidelines, such as the harmonized system (u.s. department of commerce, 2023), which continuously change, further adds to the challenges of manual product classification. therefore, many organizations active in industry have adopted automated methods of product classification using machine learning @xcite @xcite @xcite . however, the issue with current classification approaches is that they primarily focus on the 'clean' version of data, often ignoring the common data perturbations that happen in realworld product classification during inference time. in this context, 'perturbations' or 'attacks' refer to issues in data that limit the classifier's performance, such as incomplete or abbreviated data. the ability to robustly predict correct product classifications in scenarios where data might be far from perfect is of paramount importance, especially in cases where incorrect classification can lead to incorrect taxation and trade liabilities in international trade under the harmonized system @xcite . therefore, in this research, we aim to understand which models perform better in scenarios with potential data attacks. this not only facilitates more informed model decision-making, but also considers real-life data challenges. consequently, our contributions are as follows: ## background this section provides a review of the related work and essential background that supports our research. ## methodology although product classification is generally tested on datasets free of inaccuracies, in real-world scenarios the data received from users is often very short and abbreviated. as such we define an adversarial attack framework to simulate realistic data from clean data. for data perturbation method, we follow the approach introduced in @xcite . similar to the method explained in gpt3mix approach @xcite , we use gpt-4 (version: 0613) to create perturbations and generate synthetic yet highly realistic datasets to resemble the real-life scenario of the data. we write a prompt that includes the instructions to gpt-4 for different variations of data perturbations. these instructions are then passed to gpt-4 along the origi-nal product description to perform perturbations. in response, gpt-4 completion returns the perturbed product description. additional details on prompt templates are provided in figures 2 and 3 in appendix a. as outlined in the prompts, we instruct gpt-4 to perform controlled data perturbations so that the initial meaning of the descriptions is still mostly preserved and they remain classifiable by a human annotator. ## evaluation in the following, we outline the details of our evaluation. ## llms we experiment with both open-source and proprietary llms, including llama 2 chat with 70b parameters @xcite , gpt3.5, and gpt4 (model version: 0613) (openai, 2023). unlike the supervised approach, we were not able to perform multiple runs and report error ranges for llms due to the excessive cost of inference. however, we set the temperature values to 0 to minimize potential variations in the llm outputs across multiple runs. ## conclusion in this research, we presented a data perturbation framework to simulate the real-world data deficiencies for ml-based product classification. we then proceeded with a comprehensive evaluation of different supervised and llm-based classification approaches in presence and absence of data attacks. our findings show that llm-based approaches are generally more robust against adversarial attacks and more suitable for applications that require high robustness in predictions and misclassification can cause compliance repercussions. as future work, we will further investigate the security robustness of llms in data-critical applications and explore leveraging llms for providing classification rationales in addition to label predictions.