Adaptive Structure Induction for Aspect-based Sentiment Analysis with Spectral Perspective (2023)

Abstract: Recently, incorporating structure information (e.g. dependency syntactic tree) can enhance the performance of aspect-based sentiment analysis (ABSA). However, this structure information is obtained from off-the-shelf parsers, which is often sub-optimal and cumbersome. Thus, automatically learning adaptive structures is conducive to solving this problem. In this work, we concentrate on structure induction from pre-trained language models (PLMs) and throw the structure induction into a spectrum perspective to explore the impact of scale information in language representation on structure induction ability. Concretely, the main architecture of our model is composed of commonly used PLMs (e.g. RoBERTa, etc), and a simple yet effective graph structure learning (GSL) module (graph learner + GNNs). Subsequently, we plug in spectral filters with different bands respectively after the PLMs to produce filtered language representations and feed them into the GSL module to induce latent structures. We conduct extensive experiments on three public benchmarks for ABSA. The results and further analyses demonstrate that introducing this spectral approach can shorten Aspects-sentiment Distance (AsD) and be beneficial to structure induction. Even based on such a simple framework, the effects on three datasets can reach SOTA (state of the art) or near SOTA performance. Additionally, our exploration also has the potential to be generalized to other tasks or to bring inspiration to other similar domains.

## introduction aspect-based sentiment analysis (absa) is designed to do fine-grained sentiment analysis for different aspects of a given sentence @xcite . specifically, one or more aspects are present in a sentence, and aspects may express different sentiment polarities. the purpose of the task is to detect the sentiment polarities (i.e., positive, negative, neutral) of all given aspects. given the sentence "the decor is not a special at all but their amazing food makes up for it" and corresponding aspects "decor" and "food", the sentiment polarity towards "decor" is negative, whereas the sentiment for "food" is positive. early works @xcite @xcite to deal with absa mainly relied on manually designing syntactic features, which is cumbersome and ineffective as well. subsequently, various neural network-based models @xcite @xcite @xcite have been proposed to deal with absa tasks, to get rid of hand-crafted feature design. in these studies, syntactic structures proved effective, helping to connect aspects to the corresponding opinion words, thereby enhancing the effectiveness of the absa task @xcite @xcite @xcite . additionally, some research @xcite @xcite @xcite suggests there should exist task-specific induced latent structures because dependency syntactic structures (following that, we refer to them as external structures for convenience) generated by off-the-shelf dependency parsers are static and sub-optimal in absa. the syntactic structure is not specially designed to capture the interactions between aspects and opinion words. consequently, we classify these structure-based absa models into three categories by summarizing prior research: (1.) external structure, (2.) semi-induced structure, and (3.) full-induced structure. works based on external structures use dependency syntactic structures generated by dependency parsers or modified dependency syntactic structures to provide structural support for absa @xcite @xcite . studies based on semi-induced structures leverage both external and induced structures, merging them to offer structural support for absa @xcite . the first two categories require the introduction of external structures, which increases the complexity of preprocessing, while the third category directly eliminates this burdensomeness. our research is based on full-induced structures. works in this field intend to totally eliminate the reliance on external structures to aid absa by employing pre-trained language models (plms) to induce task-specific latent structures @xcite @xcite . these efforts, however, aim to create a tree-based structure, then convert it into a graph structure and feed it to graph neural networks (gnns) to capture structural information. our research follows this line of thought, but directly from the perspective of the graph, utilizing plms to induce a graph structure for gnns. in addition, studies @xcite have shown that contextual representation contains information about context tokens as well as a wide range of linguistic phenomena, including constituent labels, relationships between entities, dependencies, coreference, etc. that is, there are various scales of information (spanning from the (sub)word itself to its containing phrase, clause, sentence, paragraph, etc.) in the contextual representation. this contextual representational characteristic has rarely been explored in previous studies. therefore, our research investigates the influence of manipulations at informational scales of contextual representation on structure induction with spectral perspective. specifically, we employ graph structure learning (gsl) based on metric learning @xcite to induce latent structures from plms. we investigate three commonly used metric functions (attention-based (attn.), kernel-based (knl.), and cosine-based (cosine)) and contrast their effects on the structure of induced graphs. furthermore, we heuristically explore four types of frequency filters with corresponding band allocations (high, mid-high, mid-low, low) acting on contextual representations, and in this way, we can segregate the representations of different scales at the level of individual neurons. additionally, we introduce an automatic frequency selector (afs) to circumvent the cumbersome heuristic approaches. this allows us to investigate the impact of manipu-lations at scale information for structure induction in contextual representations. we employ three commonly plms: bert base , roberta base , roberta large . our research is based on extensive experiments and yields some intriguing findings, which we summarize as follows: structure induction. by comparing three gsl methods (attention-based (attn.), kernel-based (knl.), and cosine-based (cosine)), we find that the attention-based method is the best for structure induction on absa. ## method to obtain induced graph structure, we propose a spectral filter (flt) approach to select scale information when adaptively learning graph structure. in this section, we introduce a simple but effective approach (flt) to induce graph structures from plms to enhance the performance of absa. the overall architecture is displayed in figure 1 . ## experiment to prove the effectiveness of our approach, we demonstrate experimental results conducted on three datasets for absa and compare them with previous works. we show the details as follows. ## conclusion in this work, we propose utilizing gsl to induce latent structures from plms for absa and introduce spectral methods (flt and afs) into this problem. we also explore the impact of manipulation on scale information of the contextual representation for structure induction. extensive experiments and analyses have demonstrated that the operation of scale information of contextual representation can enhance the effect of gsl on absa. additionally, our exploration is also beneficial to provide inspiration for other similar domains.