Forum 4.0: An Open-Source User Comment Analysis Framework (2021)

Abstract: With the increasing number of user comments in diverse domains, including comments on online journalism and e-commerce websites, the manual content analysis of these comments becomes time-consuming and challenging. However, research showed that user comments contain useful information for different domain experts, which is thus worth finding and utilizing. This paper introduces Forum 4.0, an open-source framework to semi-automatically analyze, aggregate, and visualize user comments based on labels defined by domain experts. We demonstrate the applicability of Forum 4.0 with comments analytics scenarios within the domains of online journalism and app stores. We outline the underlying container architecture, including the web-based user interface, the machine learning component, and the task manager for time-consuming tasks. We finally conduct machine learning experiments with simulated annotations and different sampling strategies on existing datasets from both domains to evaluate Forum 4.0’s performance. Forum 4.0 achieves promising classification results (ROC-AUC ≥ 0.9 with 100 annotated samples), utilizing transformer-based embeddings with a lightweight logistic regression model. We explain how Forum 4.0’s architecture is applicable for millions of user comments in real-time, yet at feasible training and classification costs.

## introduction comment sections are omnipresent in today's online environments, for example, on news websites, blogs, online shops, or app stores. in these sections, users submit their feedback and opinion, request features and information, or report issues and bugs. also, in social media such as twitter or facebook, users regularly comment on specific topics, events, products, or services. in many domains, includ-ing e-commerce and journalism, users discuss with each other, read others' opinions to e.g. assess the quality of the service or the product @xcite , and provide feedback to other users and other domain experts like the journalist @xcite , who wrote the article or the developer who created the app @xcite . even though research has criticized phenomena such as "dark participation" @xcite , comments can contain constructive information for different domain experts in different fields @xcite . for example, in app development, vendors use app reviews in app stores to collect new feature ideas, bug reports, or ideas of additional user scenarios for their app @xcite . software vendors consider the reviews to decide which bug or feature request to prioritize in the next development cycle @xcite . in online journalism, media outlets harness user comments to acquire a broader perspective on additional arguments, collect resonance about their articles, or identify and contact experts or persons concerned for follow-up stories @xcite . however, the quality of the comments varies significantly, and their amount is sometimes overwhelming, which makes manual monitoring and analysis a real challenge @xcite . in this work, we propose forum 4.0, an opensource user comment analysis framework to semiautomatically analyze a large number of user comments for domain experts from various domains. forum 4.0 leverages a combination of transfer learning @xcite , human-in-theloop (bailey and chopra, 2018), and active learning @xcite strategies to automatically analyze the comments' content. to enable replication and further research, we share forum 4.0's source code, the scripts, and datasets we used for our research foot_0 and a video, which showcases forum 4.0 foot_1 . ## usage of forum 4.0 we describe exemplary usage scenarios of forum 4.0 for journalists and product managers in their respective online journalism and app development domains and introduce forum 4.0's user interface. ## architecture we describe forum 4.0's container-based architecture and its machine learning pipelines. the comment collector aggregates user comments from various sources, including media sites, app stores, and social media. forum 4.0 currently contains the "one million posts corpus" and imports comments from the google play store and the german news site spiegel online 4 . ## machine learning experiments to preliminary evaluate the applicability of forum 4.0 and the performance of its machine learning models, we conducted experiments with comments from news sites and app stores. for the online journalism domain, we used the one million post (omp) corpus @xcite . it consists of ∼1m german user comments submitted to the austrian newspaper der standard, partly annotated by forum moderators. for the app store domain, we used an existing annotated app review dataset (ard) @xcite . we used 9,336 annotated german comments (1,625 positives and 7,711 negatives) regarding omp's "personal story" label. these user comments share the users' personal stories regarding the respective topic, including experiences and anecdotes. we used 6,406 annotated english app reviews (1,437 positives and 4,969 negatives) regarding the ard's "bug report" label. in bug reports, users describe problems with the app that should be fixed, such as a crash, an erroneous behavior, or a performance issue. we simulated the human annotator, who gradually annotates a batch of user comments, triggering a new training and evaluation cycle. we trained the classifier on the training set and evaluated the model on the remaining comments. we started our first training with ten samples and triggered new training for every ten new annotations. forum 4.0 allows random sampling and uncertainty sampling for new annotations, which we compared in our experiments. with random sampling, we randomly chose and added ten new samples to our training set. with uncertainty sampling, we added the user comments for which the classifier's output is closest to 0.5. we stopped adding more user comments to the training set as soon as the balanced accuracy score converged. we evaluated the classification model on the remaining user comments after each training, using the balanced accuracy, f1-score, and the receiver operating characteristics (roc-auc) metrics. for the comment embeddings, we used two different multi-lingual pre-trained language models to embed the comments: (1) bert @xcite is based on a transformer architecture, which learns contextual relations between sub-(word) units in a text. we used an average token embedding of the four last layers of the bert model as the comment embeddings. (2) sentence-bert (s-bert) (reimers and gurevych, 2019) is based on a modification of the bert network and infers semantically meaningful sentence embeddings. we used a lightweight logistic regression model as a classifier due to performance requirements for quick updates of machine labels during human-in-the-loop coding. to assess the feasibility of our architecture, we further timed the model's training and evaluation. to mitigate the noise of our results, we performed 50 rounds for each experiment. the line plots show the average results of all rounds and the standard deviation. ## experiments results figure 4 shows the balanced accuracy, roc-auc, and f1-scores for all our classification experiments. overall, all classification metrics improve with increasing training data. additionally, the uncertainty sampling strategy outperforms random sampling, and the s-bert embeddings outperform the bert embeddings given the same sampling strategy. all evaluation metrics significantly improve within the first 100 training samples and converge afterward. on the omp dataset, we achieved a balanced accuracy of 0.86 with 100 training samples using uncertainty sampling and s-bert embeddings. with 500 training samples, we reached 0.91. within the first 100 training samples, s-bert embeddings outperformed the bert embeddings. we achieved a similar f1-score as @xcite with ∼50 training samples (0.70) and outperformed their model using 500 training samples with an f1-score of 0.82. on the app review dataset, we achieved a balanced accuracy of 0.92, a roc-auc of 0.96, and an f1-score of 0.85 using 500 training samples. figure 5 shows the time measurements for training the logistic regression model. in all cases, the training size has a linear increase. overall, the training time with the s-bert embeddings (0.1s for 500 samples) takes a shorter time than training with the bert embeddings (0.4s for 500 samples) on both datasets. we also measured the classification time on the remaining test set, which takes less than ∼3ms on the omp (∼8,000 test samples) and the ard (∼6,000 test samples) dataset. ## related work previous work in the app development domain automatically analyzed comments on apps including, app reviews @xcite @xcite and tweets @xcite , to understand and summarize users' needs and support development decisions @xcite . a typical analysis goal is to reduce the noisy user feedback and classify the remaining ones into bug reports, feature requests, and experience reports @xcite . similarly, in online journalism, previous work aimed to reduce noise and hate speech @xcite , identify high-quality contributions @xcite @xcite , summarize the audiences' sentiment @xcite , or identify comments, which address journalistic aspects @xcite . @xcite and @xcite developed a prototype, which supports the analysis of documents and comments regarding a custom concept based on seed terms. forum 4.0 builds upon this previous work and features a domain-independent comment analysis framework for domain experts. domain experts can create or reuse useful labels, annotate user comments regarding these labels, and train machinelearning models, which automatically classify the comments for further utilization. ## conclusion we presented forum 4.0, an open-source framework to semi-automatically analyze user comments in various domains including, online journalism and app store. domain experts can flexibly define or reuse comment analysis dimensions as classification labels in our framework. forum 4.0's architecture leverages state-of-the-art semantic text embeddings with a lightweight logistic regression model to address the labeling flexibility and the scalability requirements for an application to millions of user comments. forum 4.0 starts a new model training after the domain expert annotated additional comments for the concerned label. forum 4.0 evaluates each new model and classifies the remaining user comments for further analysis. we achieved promising results with our machine learning experiments in both domains with different semantic embedding and sampling strategies already after @xmath0.1s). our evaluation suggests that forum 4.0 can also be applied at a larger scale with millions of user comments.