SIR - ABSC : Incorporating Syntax into R o BERT a-based Sentiment Analysis Models with a Special Aggregator Token (2023)

Abstract: We present a simple, but effective method to incorporate syntactic dependency information directly into transformer-based language models (e.g. RoBERTa) for tasks such as Aspect-Based Sentiment Classification (ABSC), where the desired output depends on specific input tokens. In contrast to prior approaches to ABSC that capture syntax by combining language models with graph neural networks over dependency trees, our model, Syntax-Integrated RoBERTa for ABSC (SIR-ABSC) incorporates syntax directly into the language model by using a novel aggregator token. Yet, SIR-ABSC outperforms these more complex models, yielding new state-of-the-art results on ABSC.

## introduction aspect-based sentiment classification (absc, @xcite , figure 1 ) is a fine-grained sentiment analysis task that aims to handle the fact that even simple statements such as "the ambience was nice, but service wasn't so great." may express different sentiments towards different aspects (this reviewer is positive about the restaurant's "ambience", but negative about its "service"). in absc, the aspect to be classified is identified by a target string in the input sentence (e.g. "ambience"), and systems have to return the polarity (positive, neutral, negative) of the corresponding sentiment. pre-trained language models (plms) have been shown to work well for absc, presumably because their attention mechanisms capture semantic connections between target and context words @xcite @xcite . @xcite , plms have been supplemented with syntactic features, typically extracted from dependency graphs. this is typically done by using the word embeddings obtained from the plm to initialize the node embeddings of a 1 the code for sir-absc is publicly available at https: //github.com/ihcho2/sir-absc graph neural network (gnn) obtained from the dependency graph @xcite @xcite . however, such combined models have two major limitations: in order to alleviate these limitations, we propose syntax-integrated roberta (sir-absc), a novel framework for effectively augmenting plms with syntactic information. we chose roberta @xcite as our plm baseline due to its notable performance on absc @xcite . instead of stacking multiple gnn layers on top of a plm as in most previous works, we insert a (left bottom) for absc, syntactic distance can be more informative than surface distance. (right) syntactic distances as defined by a dependency tree new [g] token into the plm that acts as a syntax aggregator and effectively replaces the work of gnn layers. a variable distance control (vdc) mechanism allows [g] to focus on capturing syntactic knowledge by using constrained attention masks that reflect the graph structure. a dependency-aware aggregation (daa) mechanism leverages dependency label information. we further improve our model with automatic vdc learning (auto-vdc), which enables sir-absc to capture and focus on important syntactic distances based on the input, an important advantage over gnns. sir-absc outperforms prior approaches, and establishes a new state of the art, on the most widely used absc datasets. our main contributions are summarized below : ## aspect-based sentiment classification in absc, illustrated in figure 1 , the task is to predict the polarity (positive, negative or neutral) of the sentiment in input sentence @xmath0, w 2 , ..., w p , ..., w p+m-1 , ..., w n ] towards a given target aspect t (a substring of the input sentence: t @xmath1, ..., w p+m-1 }). ## sir-absc we present sir-absc (figure 2 ), which incorporates syntactic information into plm designed for absc tasks. we accomplish this goal by augmenting roberta with four components: (1) a single additional input token, named [g], whose attention masks depend on the structure of the input's dependency tree(s), paired with (2) a "variable distance control" mechanism that specifies how the structure of the dependency graph is reflected in [g]'s attention masks, (3) a dependency-aware aggregation which incorporates dependency labels using embeddings, and (4) an "auto-vdc" mechanism that learns the ideal vdc for a given input. input and output after tokenization with roberta's tokenizer, the input to sir-absc is '[s] [g] input sentence [/s] [/s] aspect sequence [/s]', where the aspect sequence is the token "[g]" followed by the target aspect words. [g] uses its own independent dictionary embedding in the input layer but is initialized with the initial [s] embedding. we evaluate this choice in section 5. to obtain the output, the final layers of the [s] and [g] tokens are pooled and fed to a classification layer as follows. first, a global vector c is created by averaging [s] and [g] embeddings followed by a weight w c and an activation function f : then, the final pooled vector (h) is: then, h is fed into a fully connected layer (w f c ) to calculate the final sentiment polarity score @xmath2: the [g] token and distance-based attention masks the [g] token is inserted next to the [s] token. unlike the [s] token that attends to every token in the input, we allow each layer l of [g] to only attend to the subset of input tokens j that are at most a distance d(j) ≤ d l away from the target aspect, allowing us to capture the intuition that the relevance of each word in a sentence to absc depends on its distance to the target aspect words. we achieve this with distance-based attention masks (figure 2 ) that depend either on syntactic or surface distance, and are only used for the [g] token, and the vdc hyperparameters explained below that define the parameter d l . we do not restrict how the input tokens can attend to @xcite . for our baseline model, we assume that each layer l of [g] is associated with one specific distance d l . this yields a layer-specific attention mask m @xmath3,j are 0 if the distance d(j) between token j and the target aspect words is greater than d l , and 1 otherwise). if distance is syntax-based, d(j) is the length of the shortest path between token j and the target aspect (so, if the target aspect consists of multiple tokens, we take the minimum distance to any of its component tokens). if the distance is surface-based, d(j) is simply the token distance to the target aspect. to summarize, all tokens except [g] are updated precisely as in the original plm using the original attention mask m, and [g] is updated using its own attention mask m @xmath4: in and x l out are the input and output token embeddings of the l-th self-attention layer, m is the attention mask that cancels out padding tokens, m @xmath5, q l , k l , v l are the l-th query, key, and value weights, respectively, i is the token index, and g is the [g] token's index. for simplicity, we ignore the relatively minor details, such as the ffn and normalization layers. unlike previous works where plm and gnn operate one after the other, sir-absc is a synchronous and fully-integrated model in that [s] (= semantic aggregator) and [g] (= syntactic aggregator) can attend to and affect each other's embeddings at every layer. we verify that this fullyintegrated characteristic is crucial for sir-absc's performance in section 5. ## analysis we now examine the effect of the design decisions that distinguish sir-absc from roberta. ## conclusion this paper has proposed a novel framework, sir-absc, that effectively incorporates syntactic information directly into a pre-trained large language model (plm) such as roberta for tasks like aspect-based sentiment classification (absc), in which the desired output depends on specific words in the input, and where syntactic distance to the relevant input words may be important. in contrast to prior work, where a separate gnn was added to the output of the plm, in our model, attention masks for new [g] token capture syntactic information, and a new hyper-parameter, named variable distance control (vdc), captures graph structure in a similar fashion. dependency-aware aggregation (daa) mechanism allows sir-absc to use the dependency labels effectively, and finally, the auto-vdc mechanism learns a mixture of multiple distances, allowing the model to identify and focus on important syntactic distances for a given input. to the best of our knowledge, sir-absc is the first model to incorporate syntactic knowledge into roberta for absc without resorting to gnns. experiments show we achieve state-of-the-art performance in semeval-2014 task 4. this demonstrates the efficiency of our approach and suggests a new paradigm for augmenting plm with syntax.