T - STAR : Truthful Style Transfer using AMR Graph as Intermediate Representation (2022)

Abstract: Unavailability of parallel corpora for training text style transfer (TST) models is a very challenging yet common scenario. Also, TST models implicitly need to preserve the content while transforming a source sentence into the target style. To tackle these problems, an intermediate representation is often constructed that is devoid of style while still preserving the meaning of the source sentence. In this work, we study the usefulness of Abstract Meaning Representation (AMR) graph as the intermediate style agnostic representation. We posit that semantic notations like AMR are a natural choice for an intermediate representation. Hence, we propose T-STAR: a model comprising of two components, text-to-AMR encoder and a AMR-to-text decoder. We propose several modeling improvements to enhance the style agnosticity of the generated AMR. To the best of our knowledge, T-STAR is the first work that uses AMR as an intermediate representation for TST. With thorough experimental evaluation we show T-STAR significantly outperforms state of the art techniques by achieving on an average 15.2% higher content preservation with negligible loss (~3%) in style accuracy. Through detailed human evaluation with 90,000 ratings, we also show that T-STAR has upto 50% lesser hallucinations compared to state of the art TST models.

## introduction a well accepted definition of style refers to the manner (via linguistic elements like word choices, syntactic structures, metaphors) in which semantics of a sentence are expressed @xcite ). text style transfer (tst) is the task of rephrasing an input sentence to contain specific stylistic properties without altering the meaning of the sentence @xcite . we refer the reader to @xcite for a * denotes equal contribution detailed survey of approaches towards tst problem formulation, metrics and models. in the practical scenario, that we consider in this paper, where a large corpus of parallel data is not available @xcite @xcite , two family of approaches have been proposed in the literature @xcite . 1. disentanglement: content and style are disentangled in a latent space and only the style information is varied to transform the sentence. 2. prototype editing: style bearing words in the source sentence are replaced with those corresponding to the target style. the sentence may be further re-arranged for fluency and naturalness. both the above approaches have drawbacks in the way the style agnostic intermediate representation is constructed, described as follows. first, in the disentangling approaches, it is not easy to verify the efficacy of separation between style and content. recent approaches @xcite even state that as content and style are so subtly entangled in text, it is difficult to disentangle both of them in a latent space. consequently, this affects the model's interpretability in that it is hard to attribute an effect we see in the output to the latent intermediate vector or the style vector. second, with prototype editing approaches for lingustic styles (such as author, formality), where the content and style are tightly coupled it is not feasible to segregate style and content carrying words (word examples: cometh, thou). furthermore, style-marker detection is a non-trivial nlp task and needs to be addressed for every new style that is added to the system causing scalabilty concerns @xcite . in this paper, we propose t-star (truthful style transfer using amr graph as intermediate representation) that uses a symbolic semantic graph notation called abstract meaning representation (amr) as the style agnostic intermediate stage. amr @xcite is designed to capture semantics of a given sentence in all entirety while abstracting away the syntactic variations, inflections and function words. in other words, two sentences with the same meaning but written in very different styles will have a very similar amr if not exactly the same (see figure 1 ). this addresses the shortcomings with disentanglement and prototype editing approaches. first, amr being a representation with well-defined semantics, we can inspect, interpret and measure quality of the intermediate representation and the provenance of knowledge transfer between source and target sentence. second, amr being a well known standard has high quality, robust reference implementations, especially for head languages (e.g., english). our contributions are as follows: ## amr as an intermediate representation abstract meaning representations (amrs) is a semantic formalism construct that abstracts away the syntactic information and only preserves the semantic meaning, in a rooted, directed and acyclic graph. in figure 1 , we present certain syntactic variations (changing the voice, and tense) for a sentence without altering meaning. all variations result in the same amr graph. the nodes in the amr graphs ("produce-01", "they", "before", etc.) are concepts (entities/predicates) that are canonicalized and mapped to semantic role annotations present in propbank framesets foot_0 . the edges ("arg0", "time", "duration", etc.) are then relations between these concepts. in other words, amrs aim at decoupling "what to say" from "how to say" in an interpretrable way. we posit that this could be beneficial for text style transfer, where the goal is to alter the "how to say" aspect while preserving "what to say". recently, semantic meaning representation property of amrs has been shown to be useful in other generation tasks. in abstractive summarization, @xcite uses amr as an intermediate representation to first obtain a summary amr graph from document and then generate a summary from it. @xcite use amrs to cover for the lack of explicit semantic modeling in sequence-to-sequence models. in machine translation, @xcite adopted amrs to enforce meaning preservation while translating from english to german. for paraphrase generation, @xcite found that using amrs as intermediate representation reduces the semantic drift. moreover, incorporating symbolic representations as an intermediate representation provides a way to effectively understand the reasons behind a model's shortcomings. we utilize this advantage to analyse the weaknesses of t-star in section 8.2. in order to demonstrate the semantic meaning preservation property of amrs, we design an experiment using three publicly available paraphrase benchmarks, i.e., mrpc @xcite , qqp foot_1 , and sts. mrpc and qqp are sentence pair datasets with each pair labeled yes if they are paraphrases of each other no otherwise. sts dataset assigns a scores from 0 (not similar) to 5 (exactly similar) to a sentence pair. we hypothesize that if amrs are indeed semantic meaning preserving, two sentences with similar meaning should have highly similar amrs. to measure the similarity between two amrs, we use the smatch score @xcite that calculates the number of overlapping triplets between two amrs. we use an off-the-shelf amr parser foot_2 to generate amr given a sentence. we plot the distribution of the smatch scores for mrpc, qqp and sts datasets in figure 2 . for mrpc, we infer that the smatch scores for paraphrases are significantly higher than the smatch scores for non-paraphrases. similarly, for qqp, quartile distribution of smatch scores for paraphrases is higher in comparison to non-paraphrases. for sts dataset, we observe a gradual increase in quartile distribution of smatch scores as we move towards more similar sentences. the experiments above corroborate the claim that amrs can preserve meaning under lexical variations like paraphrasing, tense and voice changes. recent research discussed earlier, have successfully used this property to show task improvements. building on the above qualitative, quantitative and prior research evidence, we further explore the applicability of amr for the tst task. ## proposed solution our proposed model t-star consists of two modules (refer figure 3 ). first, t-star encoder generates an amr given source sentence in style i. second, t-star decoder generates a sentence in style j with similar meaning as preserved in the generated intermediate amr. we take t5-base @xcite ) pre-trained model as our basic seq2seq architecture for both the modules. in order to use amr as a sequence in t5, we borrow the choice of dfs traversal from @xcite , that thoroughly study the effect of various traversals on amr parsing. ## experimental setup in this section we briefly describe the various t-star variations that are analyzed in the subsequent sections, baselines, and the implementation details. the models are validated against the metrics summarized in ## performance on style transfer tasks in this section, we compare our model performance t-star and iterative-t-star to the baselines across two datasets: shakespeare imitation dataset and cds dataset. ## human evaluations automatic metrics are insufficient to thoroughly understand subjective quality measures like content preservation. therefore, we conduct an extensive case study with human evaluations. our analysis is two folds, first we compare strap with our models t-star and iterative t-star on meaning preservation. second, we further understand the various categories of meaning loss failures. for both the human evaluation tasks below, the criteria for choosing annotators were i) proficient in english with a minimum education of diploma. ii) the annotators have to first qualify on two simpler questions, else they are not allowed to continue on the task. each instance is annotated by three taskers, and the final annotation is a majority vote. ## conclusion we explored the use of amr graphs as an intermediate representation for the tst task. we see that the performance of the proposed method t-star surpasses state of the art techniques in content preservation with comparable style accuracy. through qualitative analysis we show that obtaining very high style accuracy scores without altering meaning is indeed a challenging problem.