Investigating the Effect of Relative Positional Embeddings on AMR -to-Text Generation with Structural Adapters (2023)

Abstract: Text generation from Abstract Meaning Representation (AMR) has substantially benefited from the popularized Pretrained Language Models (PLMs). Myriad approaches have linearized the input graph as a sequence of tokens to fit the PLM tokenization requirements. Nevertheless, this transformation jeopardizes the structural integrity of the graph and is therefore detrimental to its resulting representation. To overcome this issue, Ribeiro et al. (2021b) have recently proposed StructAdapt, a structure-aware adapter which injects the input graph connectivity within PLMs using Graph Neural Networks (GNNs). In this paper, we investigate the influence of Relative Position Embeddings (RPE) on AMR-to-Text, and, in parallel, we examine the robustness of StructAdapt. Through ablation studies, graph attack and link prediction, we reveal that RPE might be partially encoding input graphs. We suggest further research regarding the role of RPE will provide valuable insights for Graph-to-Text generation.

## introduction earliest works on amr-to-text generation were mostly based on statistical methods. a common practice was to convert amr-to-text task into an already studied problems such as tree-to-text * work done while at orange innovation (lannion, france). now at huawei edinburgh research centre (united kingdom). @xcite , aligned text-to-text @xcite , travel sales problems @xcite or grammatical framework @xcite . recently, most methods are neural-centered with an encoderdecoder architecture @xcite as a backbone @xcite @xcite . unfortunately, this architecture coerces the amr to be linearized as a sequence of tokens. this ends up in structural information loss. to tackle this issue, several strategies have attempted to integrate structure using message propagation @xcite @xcite @xcite . a limitation of those is the absence of pretraining, as demonstrated by @xcite . to this end, @xcite introduced structadapt for lightweight amr-to-text with structural adapters. as linearization and tokenization of the input graph are mandatory steps for plms, structadapt first defines a new graph where nodes are the resulting subwords from the tokenization. as a result, adapter can henceforth include gnn layers operating on the subsequent graph while leveraging pretrained representations. however, although studies have been made to probe position embeddings @xcite @xcite , their role on graph encoding has remained unanswered. in this paper, we are particularly interested in measuring the saliency of rpe with structadapt for amr-to-text generation. our novelty is not in proposing a new method to encode graphs such as @xcite but rather in revealing the interesting behaviours of rpe along with structadapt. ## structadapt: a structural adapter a major issue in amr-to-text, and more generally graph-to-text with transformers @xcite , is the linearization of the input structure. the linearization of the graph returns a sequence of node and edge labels according to a certain traversal of the graph. nonetheless, adjacent nodes in the graph may be at multiple positions away from one another in the final serialization. to counteract this, @xcite introduced structadapt, a structure-aware (encoder) adapter. it solves the problem of segmented nodes labels by reconstructing a new graph from the resulting subwords. more specifically, the relations are primary reified as new nodes in the amr graph. furthermore, labels of those reified relations will be added in the vocabulary as new tokens and therefore will not be decomposed into subwords. however, the labels of the original nodes can still be chunked. to deal with this, each subword node will be connected independently to the reified relation of the initial (non-chunked) node. an example is outlined in figure 1 . as a consequence, the vanilla adapter can now integrate any gnn-based neural network which operates on the new constructed graph (figure 1 ), where nodes are the input tokens. concretely, structadapt replaces the first stacked mlp of vanilla adapter with a gnn-based model as shown in figure 2 . for amr-to-text, only the encoder is equipped with structadapt in order to encode amr structure. the decoder layers adopt vanilla adapters. in our study, we consider three different gnn-based models which are graph convolutional network (gcn) @xcite , graph attention network (gat) @xcite and relational graph convolutional network (rgcn) @xcite . gcn computes a representation for each node a which is a (normalized) aggregation function of representation of its neighbor nodes noted n (a). gat is akin to gcn but differs in that aggregation of neighbors embeddings are weighted using an attention mechanism. unlike gat and gcn, rgcn further captures the type of the relation between multi-head attention add & norm feed-forward adapter vanilla adapter structadapt layer norm. gnn feed-forward layer norm. feed-forward feed-forward figure 2: vanilla adapter vs structadapt two nodes. in our case, as amr relations are reified and stand for new nodes of the graphs, our new relations can either be of direct (a d -→ b) or reverse (a r ←b) connections type as in (ribeiro et al., 2021b). the details of the representations computation for each model can be found in appendix a. the returned nodes embeddings will then be given as input features to the following mlp. ## relative position embeddings instead of adding absolute position embeddings (ape) directly to the token embedding as in standard transformer model, some models such as t5 make use of relative position embeddings inspired from @xcite . as an alternative to ape, rpe offer interesting features. a noteworthy limitation of ape is the need to set a limit of available positions. therefore, long sequences may have ## experiments throughout our experiments, we make use of the ldc2020t02 dataset (amr 3.0 release) 1 and use the t5 base model which employs rpe. the training and evaluation details can be found in appendix b and c, respectively. ## conclusion in this paper, we have explored the effect of relative position embeddings on amr-to-text generation using structural adapters. we have shown that the generation process could be enabled by relative position embeddings when structure is erroneous or missing. in addition, we have demonstrated the capacity of those representations to encode the input graph to some extent. we have further revealed interesting robustness of rgcn model in graph reconstruction ability. for future work, we believe further experiments on other pretrained models and graph-to-text tasks may shed more light on the role of position embeddings.