DEFT ri: A Few-Shot Label Fused Contextual Representation Learning For Product Defect Triage in e-Commerce (2022)

Abstract: Defect Triage is a time-sensitive and critical process in a large-scale agile software development lifecycle for e-commerce. Inefficiencies arising from human and process dependencies in this domain have motivated research in automated approaches using machine learning to accurately assign defects to qualified teams. This work proposes a novel framework for automated defect triage (DEFTri) using fine-tuned state-of-the-art pre-trained BERT on labels fused text embeddings to improve contextual representations from human-generated product defects. For our multi-label text classification defect triage task, we also introduce a Walmart proprietary dataset of product defects using weak supervision and adversarial learning, in a few-shot setting.

## introduction in large e-commerce organizations, there are many defects generated periodically with a massive pool of software teams and developers spread across geographies to pick from, each with unique domain specialization. most organizations have a large pool of human triaging agents responsible for routing these product defects across various teams within the organization. however, large-scale software releases are time-sensitive, and effective defect assignments are a critical component in the process that is prone to bottlenecks. determining the most suitable team to own a defect may require several attempts; thus, wasting time to diagnose a defect not in the team's domain of specialty and, overall, negatively impacting the defect resolution throughput. prior industry research work on automated defect triage has primarily focused on using the traditional machine learning approaches. however, with the recent surge of state-of-the-art pre-trained language models, one under-explored field of application is operations in agile software development. in the defect triage, handling scenarios require natural language understanding to utilize the context of the defects logged by human testers, to predict all the teams associated with resolution. the current defect triage process is primarily humanagents driven. this work integrates an automated defect triage framework, deftri using product defect's contextual features to achieve operational excellence within walmart's software development lifecycle. we propose a novel framework, deftri to perform an automated defect triage using contextual representations of human-generated defect texts. we use walmart's proprietary data of product defects curated by product managers, program managers, and beta-testers to train our models. we use domain-specific lexicons to generate labeled training data using weak supervision in a few shot settings. we further use adversarial learning to increase our training sample size while increasing the robustness of our models. we propose our model architecture for fine-tuning pre-trained bert @xcite for our multi-label classification task. finally, we consolidate our experiments, analyze the results and discuss future research work. ## related work prior research work on defect triage @xcite @xcite ) mostly focuses on using traditional machine learning and rnns on word vector representations of text using bow, word2vec, tfidf, etc. another recent research relies on graph representation learning for defect triage @xcite . this paper proposes a graph recurrent convolution network with a joint random walk mechanism-based architecture. also, several recent research on label embedding @xcite @xcite has shown promising results for learning the text and label representation in the same latent 1 space. we further the research by proposing a novel architecture to derive superior contextual text representations using state-of-the-art language model bert for multi-label defect triage. most of these published research benchmarks are on open-source defect report datasets -eclipse and mozilla @xcite . however, these datasets are focused on technical errors generated during system failures and do not mimic our use case. our product defects are comprehensive user testing reviews consisting of natural language, technical and domain-specific text. in the real world, gathering labeled data is hard and expensive. hence, we propose a methodology to generate a robust proprietary multi-label training dataset using weak supervision and adversarial learning. ## data our primary dataset is a proprietary in-house dataset consisting of actual defect reviews generated by beta testers for one of our major software releases. we rely on defect title and description fields to create the text corpus and text labels to identify the teams uniquely. each defect could have multiple associated teams and vice versa. for our research, we have 3485 samples as a train set and 85 samples as the test set with 15 unique team labels for our multi-label dataset. refer ## model the multi-team-labels defect classification task in this research can be summarized with s as the tuple set. d i and t i represents the i th defect denoted as d and its corresponding team-labels denoted as t. n, n and m are the total number of defects, the length of the i th defect text and the number of teams-labels of the i th document, respectively. our framework, deftri aims at assigning teamlabels to its corresponding defects based on the conditional probability p(t i |d i ). defect text corpus (anonymized excerpts) ...for a store only query like xxx i am seeing available for scheduled pickup as the stack title on fe when i don't have a slot booked.this stack title should just reflect the xxx query like ios and web..incorrect xxx mapping (number mapped to xxx.. ...i cant add xxx to my cart from order details from my previous canceled order. there is no actionable cta.there is an add to cart cta for the xxx. see attached video. using ios xxx... ## experiments as baseline and our proposed architecture, we use the pre-trained bert-base-uncased model @xcite . we perform a total of 6 experiments for our models under 3 different settings (1) baseline fine-tuned bert model with no fused labels (2) fine-tuned bert with fused labels without [sep] token and (3) fine-tuned bert with fused labels with [sep] token, using 2 classification heads combinations e.g linear and bilstm. refer measuring accuracy on exact binary vector matching for multi-label classification is too penalizing because of the low tolerance for partial errors. therefore, we divide our predictions by classes. for each of the team-labels in our dataset, we calculate the number of false positives (fp), false negatives (fn), true positives (tp), true negatives (tn). finally, to obtain our accuracy, we sum up the values across each team-labels as below, ## analysis based on our experiments, we observed that labelfused contextual learning-based fine-tuned bert models significantly outperformed the base model using only the context of the defect text. the performance boost over the base bert pre-trained fine-tuned model is because of the context in the label embeddings used in addition to the defect text in the label-fused models, which optimizes on the alignment of features, which makes it possible to classify better. our team labels were short meaningful english words vs abbreviations which made fused embeddings better for classification when paired as a sentence with the defect texts as inputs. we observed that label-fused model without @xcite token performed better that with [sep] token which could have been because of the unnatural formation of sentence a, where a bunch of team labels are concatenated together. also, with the addition of synthetically generated data using adversarial examples for model training, we achieved an average accuracy improvement of 2.69% across our models vs. using the original data only. however, during our experiments we observed that the performance was sensitive towards the choice of text corpus sequence length and perturbation percentage for data augmentation made, during model training. a higher percentage of perturbations combined with a lower sequence length of text corpus negatively impacted performance. ## future work fine-tuning language models with weak supervision definitely solves the challenge of low labeled data availability. however, the models performance definitely suffers from error-propagation of pseudo-labels generated during the process. recent research in contrastive self-regularized selftraining approach @xcite and gan-bert in adversarial setting @xcite have shown promising results for fine-tuning bertbased language models with weak supervision. also, contrastive learning and adversarial learning approaches applied to various nlp tasks have demonstrated improvement over fine-tuning on bert-based models @xcite . to further our research, we would improve upon these approaches. ## conclusion in this work, we proposed a novel framework, deftri for automated defect triage using contextual representations of human-generated defect reviews at walmart. we discussed our methodology of generating a new proprietary labeled dataset by using weak supervision and adversarial learning, in a few shot setting. we presented two label-fused model approaches for fine-tuning pre-trained bert. as hypothesized, the experimental results show that our approach improves the multi-label text classification task for defect triage. we also proposed our future work of implementing contrastive learning for fine-tuning using weak supervision.