Evaluating and Explaining Natural Language Generation with G en X (2021)

Abstract: Current methods for evaluation of natural language generation models focus on measuring text quality but fail to probe the model creativity, i.e., its ability to generate novel but coherent text sequences not seen in the training corpus. We present the GenX tool which is designed to enable interactive exploration and explanation of natural language generation outputs with a focus on the detection of memorization. We demonstrate the utility of the tool on two domain-conditioned generation use cases - phishing emails and ACL abstracts.

## introduction the capabilities of natural language generation (nlg) models have grown rapidly in recent years, with state-of-the-art models such as @xcite able to produce text that is often indistinguishable from human-written text. despite this progress, there are many remaining challenges in effectively evaluating the quality of machine text generations. most existing evaluation approaches rely on human evaluation of the quality, fluency, and realism of a sample of generated outputs in combination with automated metrics that attempt to replicate these human judgements. however, this focus on text quality disregards several other key evaluation dimensions such as the creativity of the model and the degree of training set memorization. an nlg model that simply reproduces long text snippets from the training data is likely to achieve high quality, but does not represent the ability of the model to creatively generate novel text sequences. this can contribute to an inappropriate belief in the model's sophistication if users are not aware the generated text is copied wholesale from the training data. data scientists developing nlg models are not likely to be familiar enough with a given training corpus to detect this problem from the nlg model output without additional tool support. a second related issue arises more generally when text datasets collected from multiple sources are used to train machine learning models. in this case, identical text substrings can inadvertently end up on both sides of a train-test split. this can lead to artificially inflated model performance metrics, especially in deep learning models, having sufficient parameters to enable input memorization and shortcut generalization. while detection of exact duplicates is straightforward, detection of partial, sub-document duplication is more challenging. to address these issues, we present the genx foot_0 tool which is designed to enable data scientists to understand the provenance of the output of a text generation model. specifically, genx lets users understand which sentences or passages from a generated text output are very similar to sentences in the model's training input. it compares sentences in the output text to text that the model was trained on and renders a marked up version of the text to indicate what parts of the text may have been memorized from the training data. the tool also lets the user find interesting text based on pre-computed statistics related to this potential memorization. for open domain nlg models, datasets such as penn tree bank @xcite or lam-bada @xcite are commonly used for evaluation. however, these datasets cannot help when models are meant to be constrained to a certain domain, and they do not consider long-form text generation, only text completion tasks. another common method is to leverage the trained model for downstream tasks to assess the quality of the language model @xcite . work by @xcite has proposed combining human and statistical evaluation to measure the quality and diversity of generated text. domain-conditioned text generation nlg models can be evaluated by their perplexity calculated on a held out data set. while perplexity is useful for measuring model performance, it has limitations in measuring quality @xcite and is typically is calculated at the model level, without taking in to consideration differences in generated text that result from different decoding strategies that affect the quality of the output. ## genx tool & implementation genx is implemented as a jupyter notebook foot_1 widget foot_2 , which allows for an interactive user experience that is tightly integrated with a popular computational environment for data science. the widget is implemented in two parts: a python side, which performs preprocessing and integration with the jupyter environment, and a javascript side, which handles rendering user interaction. the inputs to genx are pandas foot_3 dataframes for the raw text (each row in the data frame corresponds a sentence), the corresponding sentence-level embedding representation of that text, and an identifier for which document the sentence belongs to. genx requires that the raw text and embeddings are also split into train and test sets. the test set may either be text generated from an nlg model or the test split of the real data. thus genx inputs are train text, train embedding, test text, and test embedding dataframes. by design, genx does not assume a particular embedding technique and requires the user to compute the embeddings. this allows the user to employ whatever method is appropriate for their use-case (e.g. tf-idf, neural network) and does not preclude the adoption of new state-of-theart embeddings methods in the future. for our demonstrations, we use sentence bert @xcite to create the embeddings used in the tool. during preprocessing, i.e., after input but before rendering, the python half of genx computes the cosine distance between each sentence in the train and test embeddings. the 10 nearest neighbors of each sentence in the test split and their respective distances are passed to the javascript half of the widget along with the test sentence dataframe. the tool passes the rows of the train text dataframe that were among the neighbors of any sentence in the test set. when a test sentence is rendered, its nearest neighbor distances are visualized in a bar graph following that sentence. the bars are sorted by distance, with the first nearest neighbor placed on the left, and the last nearest neighbor on the right, so the bars always increase monotonically. they allow the user to get a better understanding of the nearest neighbor distribution, e.g., whether the first nearest neighbor is unique or there are other semantically similar sentences in the train set. furthermore, the tool indicates which parts of the sentences are copied verbatim, or nearly so from the training data. to do so, we align each test sentence against its nearest neighbor in the train set using dynamic time warping @xcite at the token level. we use levenshtein distance @xcite as the token-token distance function foot_4 . we highlight the tokens in the test sentence that are exactly matched to tokens in their nearest neighbor sentence with a strong underline. tokens that are partially matched, i.e. with a levenshtein distance less than 5, have a weaker underline. the remaining tokens are not underlined. the user can mouse over a bar to compare the text of each nearest neighbor against the test sentence. when reading a document, the user may want to get a sense of what documents the nearest neighbor sentences are sourced from. for example, when repeats occur, do they occur together in the same source document? we include a step line chart visualization above the text to illustrate this. the x-axis is the sentence number of the generated sentence, and the y-axis is the source document identifier of the nearest neighbor of that sentence. the y-axis is sorted by first occurrence, so that the chart will increase monotonically unless a source document is revisited, which is clearly visible as dip in the chart. the line chart is also brushable allowing the user find corresponding sentences in the text below. the tool also contains an interactive scatter plot to help the user focus on interesting or problematic examples of generated text and avoid having to page through every document. the axes of the scatter plots are two novel document-level metrics which we refer to as distinctiveness and diversity. for a given document in the test data, distinctiveness is the distance of the first nearest neighbor to each test sentence in a document, averaged across the generated sentences in the document. low distinctiveness means that many sentences in that document were semantically similar to sentences in the training set, and indicate copying for specific phrases and may be symptomatic of memorizing repeated phrases. low distinctiveness for the training set overall may be indicative of broader model issues impacting creativity, potentially caused by sub-optimal parameter settings. each sentence in the training data is found within a particular source document. for a given document in the test data, diversity is the number of unique corresponding source documents for the nearest neighbor of each test sentence in the generated document divided by the number of test sentences. low diversity means the test document has similarity to a single source document, and is indicative of longer length copying from the training set, while a maximum diversity value of 1.0 indicates that the nearest neighbor of each generated sentence is from a different document in the training set. because of the limited prior work on model memorization and lack of existing metrics, we introduce these two new metrics to quantitatively capture the patterns of sentence-level (distinctiveness) and document-level (diversity) memorization by the models. we also average these metrics across documents in the test corpus for corpus-level analysis (see ## use case: phishing email generation phishing emails we initially developed the genx tool when working with a composite dataset of publicly available phishing datasets that contained many duplicates and near-duplicates. this dataset was initially comprised of the aggregation of data made available by @xcite , and @xcite as well as phishing emails provided by industry partners. the initial dataset contained a total of 60,705 emails, however after initial de-duplication efforts using exact string matching only 9,234 emails remained which was split into a train set of 8,311 and a test set of 923. after further de-duplication efforts, aided by genx, the final dataset consists of 5,634 emails in the training set and held out validation and test sets of size 500 each. while we have been rigorous in our efforts to remove emails that are duplicated, the formulaic nature of phishing emails leads to many commonly repeated phrases, sentences, or paragraphs. phishing test/train split while the genx tool was originally designed for the evaluation of memfigure 3: example email from the phishing test set with high overlap with an email from the training set, differing only in the name of the recipient. ## conclusion genx provides a unique capability for interactive evaluation and explanation of nlg model output. the tool goes beyond typical aggregate performance metrics and provides new insight into domain-conditioned nlg model creativity and memorization. across two use cases, we showed this helped distinguish models in situations where aggregate evaluation metrics did not.