What Do Users Care About? Detecting Actionable Insights from User Feedback (2022)

Abstract: Users often leave feedback on a myriad of aspects of a product which, if leveraged successfully, can help yield useful insights that can lead to further improvements down the line. Detecting actionable insights can be challenging owing to large amounts of data as well as the absence of labels in real-world scenarios. In this work, we present an aggregation and graph-based ranking strategy for unsupervised detection of these insights from real-world, noisy, user-generated feedback. Our proposed approach significantly outperforms strong baselines on two real-world user feedback datasets and one academic dataset.

## introduction collecting vast amounts of user feedback on products and services is a common practice these days for a multitude of companies. this can prove to be a rich resource for product owners to improve the quality of their product offerings, correct failures and gauge the general performance of their product from both implicit and explicit signals that might be present in the feedback. however, in most cases, including the one we address in this work, the feedback is unstructured and voluminous, and can therefore remain underutilized for the most part. in our particular use-case, we receive thousands of textual feedback daily on average 1 , and it is time-consuming and laborious for product owners to manually extract actionable insights from them. users leave feedback on a variety of aspects they experience in the course of using the product ( ## related work there is limited work that focuses on uncovering insights from real-world user feedback data. most of the work involves labeled academic datasets, requiring approaches that involve some form of supervision. for instance, @xcite involves a weakly supervised joint sentiment-topic model that detects sentiment and topic simultaneously from text, applied to two labeled academic datasets. approaches that are unsupervised @xcite are largely based on latent dirichlet allocation (lda) @xcite , which yields distribution of unigrams as topics. although @xcite adopt an unsupervised strategy, it has not been explored on user feedback data. these approaches are not directly applicable to our problem setting since we require short phrases focusing on the performance of an action by the user. approaches such as textrank @xcite , singlerank @xcite , ex-pandrank @xcite , topicrank @xcite , topicalpagerank @xcite , positionrank @xcite , bi-lstm-crf sequence labeling @xcite , face @xcite , and mul-tipartiterank @xcite have been applied to the task of key phrase extraction from documents as opposed to theme detection from clusters. for cluster-labeling, there has been work around using keyword extraction from clusters, utilizing wordnet synsets to expand the keywords, followed by a selection procedure to assign the final label @xcite . none of these approaches focus on the extraction of short actionable phrases, a.k.a. themes, as is our use case. in most of these approaches, candidate key phrases are assumed to appear in contiguous positions in a document and are concatenated to form phrases. as described earlier (figure 1 ), it is unrealistic to make such assumptions on real-world user feedback data and our proposed approach considers non-contiguous candidate phrases as well. in this work, we use two internal unlabeled datasets containing user feedback in english on two product offerings. the feedback collection pipeline has opt-in and opt-out mechanisms in place to allow the user to decide whether their data can be used for further analysis. user-specific information has been removed from these datasets for privacy and confidentiality reasons. these datasets vary in content based on the specific product they contain feedback about, and in the number of feedback documents contained in each. we refer to them as prod 1 and prod 2 . prod 1 received orders of magnitude more feedback than prod 2 . for exploration purposes we sampled about 10k documents for prod 1 and 1.5k documents for prod 2 . in addition, intent label document find phone i need your help finding my lost phone book hotel i'm inquiring about the availability of a room that fits 10 people from monday to tuesday in manhattan schedule maintenance please find someone who specializes in cars, my check engine light has turned on the data contains utterances from 10 domains, e.g. banking, travel, kitchen & dining etc. ## methodology in this section, we describe our proposed approach for extracting themes that help in discovering insights from user-generated text (outlined in figure 2 ). in order to discover coherent emerging insights from the vast amounts of user-generated data, we first aggregate semantically similar feedback using clustering algorithms, and extract a representative set of documents per cluster. this is described in section 4.1. thereafter, themes for these clusters are generated as detailed in section 4.2. the entire pipeline is unsupervised. ## experiments in this section, we describe the experimental details of the pipeline, and provide details on the baselines we compare with. ## results model performance on clinc150 dev set: statistical significance tests conducted show that on the dev set, our methodology significantly outperforms random baseline on meteor representative set of documents per cluster p&p ours -the new monitoring tab is very poor. it doesn't show enough data and you can't click on graphs to get detailed views. detailed view view graphs -i cannot see the monitoring of 2 <@xmath0. unreliable reporting of metrics, graphs are sometime unavailable. -i can't even select @xmath1. -need to see @xmath2. ## conclusion & future work this work addresses the problem of discovering actionable insights from unlabeled real-world user feedback data, in an unsupervised fashion. data is clustered into groups containing coherent insights, followed by theme detection per cluster using a graph-based ranking approach. experiments conducted on two real-world user feedback datasets as well as an academic dataset show our proposed approach to significantly outperform baselines by a large margin. in the future, we would expand the scope of our work to datasets with other characteristics and distributions (e.g. review datasets) to study