Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction (2024)

Abstract: Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review, which is the most representative and challenging task in aspect-based sentiment analysis. A key challenge in the ASQP task is the scarcity of labeled data, which limits the performance of existing methods. To tackle this issue, we propose a self-training framework with a pseudo-label scorer, wherein a scorer assesses the match between reviews and their pseudo-labels, aiming to filter out mismatches and thereby enhance the effectiveness of self-training. We highlight two critical aspects to ensure the scorer’s effectiveness and reliability: the quality of the training dataset and its model architecture. To this end, we create a human-annotated comparison dataset and train a generative model on it using ranking-based objectives. Extensive experiments on public ASQP datasets reveal that using our scorer can greatly and consistently improve the effectiveness of self-training. Moreover, we explore the possibility of replacing humans with large language models for comparison dataset annotation, and experiments demonstrate its feasibility. We will release our code and data via GitHub.

## introduction aspect-based sentiment analysis (absa) aims to recognize aspect-level opinions and sentiments from user-generated content @xcite . this problem has consistently attracted interest owing to its proficiency in distilling and summarizing fine-grained opinions from vast data @xcite @xcite . the most representative and challenging task in absa is aspect sentiment quad prediction (asqp) @xcite oh, and the service was spot on. to reduce such mismatches, this paper introduces a pseudo-label scorer for data augmentation. as illustrated in figure 1 , the scorer assesses the degree of match between the review and its pseudolabel. if we have a sufficiently robust scorer, we can filter out all mismatched samples, thereby sig-nificantly enhancing the effectiveness of data augmentation. we propose that the effectiveness and reliability of this scorer hinge on two critical aspects: (1) the quality of the training dataset and (2) its architecture along with the training objective. we discuss these two aspects below. for the first aspect, previous works typically produce negative labels by modifying real labels using heuristic rules @xcite . however, such negative labels are usually simplistic and patterned, limiting the scorer's learning. to overcome this limitation, we create a humanannotated comparison dataset. specifically, we train an asqp model with existing labeled data, use it to infer several pseudo-labels for unlabeled data, and then have human annotators choose the most appropriate pseudo-labels. the labels chosen by annotators are designated as positive labels, while the rest as negative labels. our dataset, in contrast to the rule-based datasets, is more challenging and better aligned with human judgment. for the second aspect, previous works formalize label-scoring as a question-answering problem @xcite or embed the discriminative matching token into the label @xcite . however, our findings suggest that these methods underperform in complex tasks like asqp, due to their limited capacity to model the interactions between reviews and pseudo-labels. recent works in preference optimization reveal that the language model itself can serve as a scorer @xcite . this motivates us to use the conditional likelihoods that a generative model assigns to a pseudo-label as the measure of its quality. compared with the previous methods, this approach enables the scorer to examine the plausibility of a pseudo-label in a token-by-token fashion, thus offering a more comprehensive and effective scoring. we then fine-tune this scorer on our comparison dataset using ranking-based objectives. upon developing this pseudo-label scorer, we apply it in a data augmentation framework, specifically opting for the self-training framework due to its simplicity. we conduct extensive experiments on public asqp datasets to examine its effectiveness and further investigate the following questions: our contributions are summarized as follows: ## comparison dataset we need to construct a comparison dataset to facilitate the training and evaluation of the pseudo-label scorer. this dataset comprises samples each containing a review sentence accompanied by several pseudo-labels, where one is the positive label and the others are negative labels. we train the scorer by requiring it to assign high scores to positive labels and low scores to negative labels. previous works typically produce negative labels using heuristic rules @xcite . they randomly modify elements in the existing positive labels, such as altering boundaries or conducting substitutions. such negative labels are patterned and easily distinguishable, limiting the learning potential of the scorer. therefore, this paper employs human annotators to construct this comparison dataset. ## conclusions in this paper, we introduce a pseudo-label scorer for the aspect sentiment quad prediction (asqp) task to reduce mismatches in data augmentation. we propose that the effectiveness and reliability of this scorer hinge on two critical aspects: the quality of the training dataset and its model architecture. to this end, we create both human-and ai-annotated comparison datasets and propose a scoring method based on a generative model. upon developing this scorer, we apply it to data filtering in a self-training framework and further employ it as a reranker to enhance asqp models. detailed experiments and analysis demonstrate the effectiveness of our comparison datasets and the proposed architecture. furthermore, experimental results on four public asqp datasets reveal that our scorer significantly and consistently improves the performance of existing methods.