Multi-View Active Learning for Short Text Classification in User-Generated Data (2022)

Abstract: Mining user-generated data often suffers from the lack of enough labeled data, short document lengths, and the informal user language. In this paper, we propose a novel active learning model to overcome these obstacles in the tasks tailored for query phrases–e.g., detecting positive reports of natural disasters. Our model has three novelties: 1) It is the first approach to employ multi-view active learning in this domain. 2) It uses the Parzen-Rosenblatt window method to integrate the representativeness measure into multi-view active learning. 3) It employs a query-by-committee strategy, based on the agreement between predictors, to address the usually noisy language of the documents in this domain. We evaluate our model in four publicly available Twitter datasets with distinctly different applications. We also compare our model with a wide range of baselines including those with multiple classifiers. The experiments testify that our model is highly consistent and outperforms existing models.

## introduction a microblog is a stream of brief updates written by an author over time on social media platforms such as twitter or tumblr @xcite . in such platforms, an important set of tasks tailor for queries @xcite . we demonstrate this by an example. assume we aim to develop a monitoring system for tracking the reports of covid-19 on the twitter website. such a system can be developed in five steps @xcite : 1) collecting a set of general and related query phrases that describe the task. terms like "covid-19", "covid", and "coronavirus" can constitute such a set. 2) collecting the set of tweets that mention these queries. 3) sampling a subset of the collected data to be manually annotated. 4) training a classifier on the annotated documents. 5) using the classifier to label the remaining data. figure 1 : 1a) query-based tasks consist of five steps: 1) obtaining query phrases, 2) collecting the documents that contain the queries, 3) selecting a subset of the collected documents and labeling them, 4) training a classifier, and 5) predicting unseen documents. we use active learning to enhance the document selection in the third step and to consequently improve the classifier in the fourth step. 1b) we extract two views from documents: 1) a document view to encode the entire user posting, and 2) a keyword view to encode the context of the query phrase (earthquake). see section 3.2 for more details. a similar pipeline can be used in other tasks such as detecting suicide ideation @xcite , monitoring press freedom @xcite , certain applications for hate speech detection @xcite , certain applications for churn detection @xcite , and general disease detection @xcite . in this study, we employ active learning @xcite existing studies report various difficulties in mining social media data @xcite @xcite . the costly construction of datasets is a challenge @xcite . to overcome this challenge, we use active learning which is known for reducing the amount of required data by intelligent sampling @xcite . another issue is the typically short length of documents that, along small labeled data, can cause overfitting @xcite . to confront this challenge, we aim to use multi-view learning which is known for reducing the risk of overfitting @xcite by relying on models trained on different sets of features. 1 see figure 1b for an illustration of our algorithm for extracting the views. another challenge is the noisy language 2 of users in social media @xcite . to address this obstacle, we use an active learning technique based on a committee of predictors. ensemble learning is known for addressing noisy data by reducing variance @xcite . furthermore, and again related to noisy data, it is reported that social media users are highly inventive in using words @xcite . to tackle this challenge, we rely on the context of query phrases to identify word semantics @xcite . our study makes three contributions: 1) existing active learning models for classifying social media data traditionally rely on single-view algorithms. in this paper, we propose the first multiview active learning model in this domain. 2) we use pretrained contextual language models to extract our views. to efficiently use these views, we employ the parzen-rosenblatt window method @xcite ) and propose a novel query strategy by integrating the representativeness measure into multi-view active learning. 3) we use an algorithm based on the agreement between predictors to increase the resistance to social media noise. we empirically demonstrate that this step enhances the selection process in active learning. we evaluate our model, which we term ro-caal (robust context-aware active learning), in four distinctly different twitter tasks and demonstrate that it is either the top model or on a par with the best recent methods. ## related work the uncertainty-based sampling model is the most widely used active learning query strategy 3 @xcite @xcite . in this model, the most infor-mative document, i.e., the document that the base learner is uncertain about, is queried and added to the labeled pool. ein-dor et al. (2020) survey numerous methods and report that the performance of the classifiers based on pretrained language models can be enhanced by active learning, however, they also report that existing query strategies yield no significant gain over the uncertainty-based sampling model-measured by the prediction entropy. given the usually satisfactory performance of the uncertainty-based sampling model, the majority of successful applications of active learning in microblogging platforms rely on this model @xcite @xcite . the survey by @xcite confirms this claim, and also reports that language model pretraining can be an additional effective factor to address noisy social media data. therefore, we follow their suggestion and use pretrained encoders in our model and all of the baselines. incorporating the diversity or the representativeness measures into the uncertainty-based query strategy is an active area of research @xcite @xcite . while the uncertainty-based model, which is inherently a single-view model, has shown to be effective, it is well-known that multi-view models have a considerable potential @xcite . to our knowledge, there is no multi-view active learning model for social media data. in this article, we propose such a model by integrating the representativeness measure into multi-view active learning. multi-view active learning @xcite @xcite constructs two views (or sets of features) from input data and trains a base learner on each view. then each base learner is used to label the set of unlabeled data. the data points that are assigned to the opposite classes by the two base learners are detected-these data points are called contention points. one data point from this set is annotated and added to the labeled set. in the results and analysis section, we show that our model outperforms existing single-view models as well as the regular multi-view active learning model. ## proposed model we begin this section by explaining the problem statement, and then, we describe the approach to extract two views from documents. we continue figure 2 : pretrained multi-layer transformers as the encoder in our model. we employ these models-we used bert in our experiments-to extract the document and the keyword representations. a hypothetical short and simple user posting is shown for demonstration. the token [cls] in the bert architecture represents the document encoding. to obtain the keyword representations, we extract the corresponding final pooling layers-see the experimental setup for the details. by explaining the query strategy and the technique to tackle social media noise. finally, we assemble the pieces and provide an overview of the model. ## experimental setup in this section we discuss the datasets, the baselines, and the implementation details. see appendix a for more details on the used datasets. ## results and analysis in this section we report the results and then we provide an empirical analysis. ## conclusions in this paper, we proposed an active learning model for social media tasks tailored for query phrases. we employed an algorithm to derive two views from documents, then, we proposed a new multiview query strategy to aggregate the representativeness and the contention reduction measures. finally, we proposed an algorithm based on the agreement between multiple predictors to tackle noisy content. through an extensive set of experiments in four public datasets we showed that our model outperforms existing baselines. we also reported two ablation studies and extensively analyzed our model.