Dense Paraphrasing for Textual Enrichment (2023)

Abstract: Understanding inferences from text requires more than merely recovering surface arguments, adjuncts, or strings associated with the query terms. As humans, we interpret sentences as contextualized components of a narrative or discourse, by both filling in missing information, and reasoning about event consequences. In this paper, we define the process of rewriting a textual expression (lexeme or phrase) such that it reduces ambiguity while also making explicit the underlying semantics that is not (necessarily) expressed in the economy of sentence structure as Dense Paraphrasing (DP). We apply the DP techniques on the English procedural texts from the cooking recipe domain, and provide the scope and design of the application that involves creating a graph representation of events and generating hidden arguments through paraphrasing. We provide insights on how this DP process can enrich a source text by showing that the dense-paraphrased event graph is a good resource to large LLMs such as GPT-3 to generate reliable paraphrases; and by experimenting baselines for automaticDP generation. Finally, we demonstrate the utility of the dataset and event graph structure by providing a case study on the out-of-domain modeling and different DP prompts and GPT models for paraphrasing.

## introduction two of the most important components of understanding natural languages involve recognizing that many different textual expressions can correspond to the same meaning, and detecting those aspects of meaning that are not present in the surface form of an utterance or narrative. together, these involve broadly three kinds of interpretive processes: (i) recognizing the diverse variability in linguistic forms that can be associated with the same underlying semantic representation (paraphrases); (ii) identifying semantic factors or variables that accompany or are presupposed by the lexical semantics of the words present in the text, through "hidden" arguments (e.g., "stir vigorously."; the argument of stir is not in the surface form); and (iii) interpreting or computing the dynamic consequences of actions and events in the text (e.g., slicing an onion brings about onion slices). the first of these, the problem of paraphrasing, has been addressed computationally since the early days of natural language processing (nlp). the other two mentioned above, however, are more difficult to model with current machine learning approaches, which rely heavily on explicit textual strings to model semantic associations between the elements in the input. many question answering (qa) systems, for example, rely on such syntagmatic forms in the training data for modeling potential associations that contribute to completion or generation task performance. hence, if predicates or arguments are missing, implied, or interpreted from context, there is rarely anything to encode, and consequently little to decode as output, as well. consider the following example from the traditional paraphrasing task. the text difference between the input and output only comes from a lexical substitution, rather than the rephrasing or addition of hidden arguments. to solve this problem, some recent attempts have been made to enrich surface forms that are missing information through "decontextualization" procedures that textually supply information which would make the sentence interpretable out of its local context @xcite @xcite . the focus of the decontextualization is on enriching text through anaphora resolution and knowledge base augmentation, which works well on arguments or concepts that can be linked back to existing knowledge sources, such as wikipedia. consider the following example of the this task. it is able to decontextualize barilla sauce in (2a), but does not reintroduce any semantically hidden arguments from the context in (2b), making inferences over such sentences difficult or impossible. consider the dps of the sentences from examples (1) and (2), illustrated below in (3). compared to the aforementioned tasks, dp aims to recover the semantically hidden arguments that fit the local context of the event (e.g., pan for the saute event) or carry a broader view of the context of the text (e.g., sauted chopped onions shows its transformation through multiple events). ## related work there is a long history in linguistics, dating back to the early 1960s, of modeling linguistic syntagmatic surface form variations in terms of transformations or sets of constructional variants @xcite @xcite . @xcite formally defines this process of preserving the meaning from lexical, phrasal, or sentential expressions e i to e j as paraphrasing. for nlp uses, paraphrasing has been a major part of machine translation and summarization system performance @xcite @xcite @xcite @xcite . in fact, statistical and neural paraphrasing is a robust and richly evaluated component of many benchmarked tasks, notably mt and summarization @xcite , as well as question answering @xcite and semantic parsing @xcite . to this end, significant efforts have gone towards the collection and compilation of paraphrase datasets for training and evaluation. in addition to the meaning-preserving paraphrase strategies mentioned above, there are several directions currently explored that use strategies of "decontextualization" or "enrichment" of a textual sequence, whereby missing, elliptical, or underspecified material is re-inserted into the expression. the original and target sentences are compared and judged by an evaluation as a text generation or completion task @xcite @xcite @xcite @xcite . our work applies both strategies of paraphrasing to the procedural text domain, which is new to the field. unlike typical paraphrase generation tasks @xcite which paraphrase full sentences and favor different wording and structure, our task performs at the entity-level. recent studies in procedural texts focus on tracking the state of events and entities in artificial corpora from arbitrary domains @xcite @xcite . some works also treat recipes as a rich resource for procedural texts. @xcite leverage structured representations of domain-specific action knowledge for modeling a process of actions and their causal effects on entities. other works try to resolve the anaphoric relations between recipe ingredients @xcite . while these works all create corpora suitable for their own problems, our work, in contrast, embeds enriched information of both entities and events in the recipe using dense paraphrasing. enrichment of verbnet predicates can be seen as an early attempt to provide a kind of dense paraphrasing for the verb's meaning. in @xcite , the basic logic of generative lexicon's subevent structure was applied to verbnet classes, to enrich the event representation for inference. the verbnet classes were associated with event frames within an event structure lexicon (esl), encoding the subevent structure of the predicate. if the textual form of the verb is replaced by the subeventual description itself, classes such as change of location and change of possession can help encode and describe event dynamics in the text, as shown in @xcite @xcite . for example, the verbnet entry drive is enriched with the esl subevent structure below: ## method in this section, we detail the procedure involved in creating dps. the dp method can be seen as the method for creating sets of semantically "enriched, but consistent" expressions, that can be exploited by either human consumption (e.g., natural language paraphrases) or machine consumption (e.g., configurable graphs). specifically, we currently adopt a template-based method along with heuristics to generate dps that account for hidden entities and entity subevent structure. sub-event structure dp starts by identifying events from the text. as mentioned above, esl represents an event as having three parts: begin (b e ), inside (i e ), and end (e e ). in our method, we use this subevent structure not only to track the begin and end state of an event, but to create textual redescriptions of the changed event arguments. to illustrate, in hidden arguments dp also recovers hidden arguments that are not present in the surface form of the text to ensure the richness of the subevents. the changed entities associated with the begin or end events can be either hidden or explicit. for example, the bake event from ## experiment 1: dense paraphrasing from annotation we use the text data from the subdomain of cooking recipes to demonstrate the application of the dp. compared to texts of news or narratives, procedural text such as recipes tend to be task-oriented and highly contextualized, allowing the dp to focus on the hidden information and changes that are taking place in the course of a sequence of events in the narrative. specifically, we apply the dp on the existing coreference under transformation labeling (cutl) dataset @xcite . cutl consists of a subset of 100 cooking recipes from a larger recipe-to-video questions (r2vq) dataset @xcite . it contains rich annotation of the cooking-related events and entities (both explicit and hidden), as well as the coreference relations between the entities. ## conclusion in this paper we define dense paraphrasing (dp), the task of enriching a text fragment (lexeme, phrase, or sentence) such that contextual ambiguities are eliminated, contextual anchors or variables are supplied, and any implied arguments are made textually explicit. we outlined our dp procedure that can be applied to enrich the textual dataset, and provided insights on the transformer-based models as baselines for the dp text generation task. we presented the case study for generating dp under the out-of-domain setting, and the analysis on paraphrasing from event graphs, which show the feasibility of modeling dp and the challenges it poses to current large language models. we believe that dp has the potential to help in a broad range of nlp applications. in particular, applications and tasks involving abstractive inferencing can benefit from the dynamic tracking and decontextualized redescriptions of entities appearing in a coreference chain. the notion of following an entity as it changes through a developing narrative or text can be computationally encoded using the technique described here, giving rise to a history or biographical model of an entity. we hope to extend the dp procedure to include creating vector representations of dp that can be fit into a broader range of computational models. we also intend to include reference to the "vertical typing" of an expression (type inheritance) from online resources with definitional texts, such as wikipedia or word-net (e.g., @xmath1, @xmath2. this would further enhance the utility of the resulting dp'ed data for logical inference tasks.