ASTE -Transformer: Modelling Dependencies in Aspect-Sentiment Triplet Extraction (2024)

Abstract: Aspect-Sentiment Triplet Extraction (ASTE) is a recently proposed task of aspect-based sentiment analysis that consists in extracting (aspect phrase, opinion phrase, sentiment polarity) triples from a given sentence. Recent state-of-the-art methods approach this task by first extracting all possible text spans from a given text, then filtering the potential aspect and opinion phrases with a classifier, and finally considering all their pairs with another classifier that additionally assigns sentiment polarity to them. Although several variations of the above scheme have been proposed, the common feature is that the final result is constructed by a sequence of independent classifier decisions. This hinders the exploitation of dependencies between extracted phrases and prevents the use of knowledge about the interrelationships between classifier predictions to improve performance. In this paper, we propose a new ASTE approach consisting of three transformer-inspired layers, which enables the modelling of dependencies both between phrases and between the final classifier decisions. Experimental results show that the method achieves higher performance in terms of F1 measure than other methods studied on popular benchmarks. In addition, we show that a simple pre-training technique further improves the performance of the model.

## introduction aspect-sentiment triplet extraction (aste, @xcite is a recent task in aspect-based sentiment analysis that involves the extraction of (aspect phrase, opinion phrase, sentiment polarity) triples from text. the aspect phrase denotes features or attributes of the described object, towards which the sentiment is expressed in the opinion phrase. the categorisation of this sentiment into typically three classes (positive/negative/neutral) is the final element of the triple. for example, in the sentence "the hotel was very good" there is only one aste the room was fine but the staff was rude. the menu is limited and extremely pricy. figure 1 : two examples of input sentences and aste triplets. the spans highlighted in yellow are opinion phrases, whereas spans highlighted in green are aspect phrases. the +/-sign denote positive/negative sentiment, respectively. since aste provides an answer to what? (aspect phrase), how? (sentiment), and why? (opinion phrase) questions regarding sentiment, it is sometimes referred to as a "near complete solution" to sentiment analysis @xcite and has attracted considerable research attention. several types of methods have been proposed, including sequence prediction @xcite , cascade processing @xcite , prompting approaches @xcite or predicting a special word-by-word matrix @xcite . however, the approaches that currently achieve the best predictive performance are span-level approaches @xcite , which are the focus of this work. span-level methods @xcite typically begin by extracting all possible text spans up to a predefined length from a given text. each span undergoes evaluation by a classifier to determine whether it contains an aspect phrase, an opinion phrase, or whether it should be excluded from further processing. the method then considers all possible pairs of identified opinion and aspect phrases, with a secondary classifier examining these pairs to discard false matches and assign sentiment polarity to the valid ones. although several modifications of this scheme have been proposed @xcite @xcite , they share the same property: the prediction of aste triples consists of dozens of independent classifier decisions, and the dependencies between decisions regarding the analysed spans are not modelled. this limits the predictive performance of these techniques, as some task-related knowledge simply cannot be learned. such unexploited properties of structured output include both deterministic rules and probabilistic patterns, some examples of which are given below (see more in app. g). note that learning these properties requires joint modelling of the decisions to extract or link particular phrases. this is impossible to achieve in the current span-based aste frameworks, which often adapt end2end training but make a strong independence assumption and perform multiple independent classifier predictions. in this paper, we address the challenge of modeling dependencies between the extracted phrases and between constructed triples by introducing aste-transformer, a novel architecture for aste. unlike conventional span-based aste approaches, our method does not perform multiple independent classifications to categorize extracted text spans into aspect/opinion/invalid phrases. instead, aspectopinion pairs are formed through a search in a specialized embedding space induced by modified self-attention mechanizm, where each span is represented twice: once as a potential aspect and once as a potential opinion phrase. furthermore, our approach does not construct final triples through independent classifications without considering other candidate triples; instead, for each candidate triple, it produces a representation that depends on all other triples. the contributions of this paper are as follows: the experimental results demonstrated the superior predictive performance of aste-transformer compared to other methods under study. furthermore, the ablation study highlighted the importance of modelling dependencies, which accounted for improvements of up to 5 ppt on f1 score. lastly, the proposed pre-training technique yielded additional, statistically significant performance improvements over previous state-of-the-art aste approaches. first, the input sentence is processed by a masked language model (mlm) composed of standard transformer layers that produce an embedding representation for each token. second, in line with span-based approaches, all text spans up to a certain length are extracted, and their corresponding embedding representations are constructed. next, the spans are analysed by an aspect-opinion pair construction layer, which searches for corresponding aspect-opinion phrases. finally, all candidate aspect-opinion pairs are processed by the triplet construction layer. this layer first computes a representation for each candidate pair, then a classifier assigns them a sentiment polarity or filters them out. importantly, the constructed representation of an aspect-opinion pair depends on all the other candidate pairs. all the above-mentioned steps are realized by a single neural architecture, consisting of three types of transformer-like layers. the network is trained in an end-to-end fashion, by optimizing a loss function measuring the quality of constructed triplets and loss functions of additional intermediary tasks. an overview of the proposed neural network is shown in fig. 2 , and each of its parts is described in the following sections. ## pretraining for aste transformer-based architectures are known to benefit from previous pretraining, especially when dealing with limited supervised data. in the proposed aste-transformer, only the first part (mlm) is pre-trained using standard methods, while all subsequent layers are randomly initialized. therefore, we propose a simple idea of generating abundant noisy aste data from sentiment classification (sc) datasets and employing them for pretraining purposes. note that sc datasets are often much larger than aste datasets because they can be automatically collected from e-commerce platforms, where the consumer's overall product rating can be used as a proxy for opinion sentiment @xcite . the first step of our method is to train aste-transformer model on the original aste dataset and apply it to texts from the sc dataset to produce artificial annotations. as predicting incorrect sentiment polarity is a factor negatively affecting the performance of aste models @xcite , we substitute the sentiment polarity in the generated triples with the gold standard sentiment of the whole sentence as provided in sc dataset. finally, we train a new aste-transformer from scratch, starting from pre-training it on the set of pseudolabelled data, and then combining it with gold standard aste data. the last few training epochs are performed on gold standard aste data only. 16res @xcite @xcite . selected statistics of these datasets can be found in the appendix a. amazon fine food reviews (mcauley and leskovec, 2013) dataset was used for pretraining experiments, except for experiments with 14lap, where amazon review dataset (digital software) @xcite was used. for each domain, we utilize approx. 10,000 reviews. ## related works since the introduction of aspect sentiment triplet extraction (aste) by @xcite , various approaches were proposed for this task. jet @xcite converts the problem into a sequence labelling task using enhanced bioes tagging schema. similarly, pbf @xcite uses three sequential tagging predictors to construct triplets. a different approach is to encode aste triples in a word-by-word matrix with grid tagging scheme @xcite . each element of this matrix is predicted by an independent classifier. in contrast to gts which relies on word-to-word interactions, span-aste @xcite considers all possible text spans from the input sentence, performing multiple independent classifications to construct the output. sbc @xcite uses span representations constructed with a special separation loss and has bidirectional structure to generate aspect-opinion pairs. ftop @xcite divides the input sentence into opinion/aspect phrases using sequence prediction and considers all aspect-opinion pairings by a classifier. recently, episa @xcite explored the possibility of making the predictions dependent while constructing aste triples. in contrast to our method, episa uses a 2-dimensional crf over a decision matrix, which is intractable without making additional assumptions about pair-wise decision independence and approximating potential functions with gaussian kernels. another group of approaches combines spanbased and matrix prediction approaches by predicting a matrix with span-based tags. such approaches include stage @xcite and simstar @xcite . both these approaches predict matrices by applying softmax classifiers independently. finally, @xcite presented a generative approach called gas, which uses prompting of the t5 language model. some of the generative methods also use contrastive learning to learn better representations @xcite , but they do not use search to pair aspects with opinions. a data augmentation technique for aste was proposed in @xcite , but in comparison to our simple pretraining idea, it is rather complex as it employs reinforcement learning and trains additional generator and discriminator models. ## summary in this paper, we have demonstrated the potential of exploiting dependencies between constructed triples in span-based aste approaches. the proposed aste-transformer method showed superior predictive performance on both english and polish benchmarks. additionally, a simple pre-training scheme proved to further improve the performance.