Detection and Measurement of Syntactic Templates in Generated Text (2024)

Abstract: The diversity of text can be measured beyond word-level features, however existing diversity evaluation focuses primarily on word-level features. Here we propose a method for evaluating diversity over syntactic features to characterize general repetition in models, beyond frequent n -grams. Specifically, we define syntactic templates (e.g., strings comprising parts-of-speech) and show that models tend to produce templated text in downstream tasks at a higher rate than what is found in human-reference textsWe find that most (76%) templates in model-generated text can be found in pre-training data (compared to only 35% of human-authored text), and are not overwritten during fine-tuning or alignment processes such as RLHF. The connection between templates in generated text and the pre-training data allows us to analyze syntactic templates in models where we do not have the pre-training data.We also find that templates as features are able to differentiate between models, tasks, and domains, and are useful for qualitatively evaluating common model constructions.Finally, we demonstrate the use of templates as a useful tool for analyzing style memorization of training data in LLMs.

## introduction an open question about large language models (llms) is what patterns such models learn from pre-training data @xcite @xcite , and whether the same patterns appear generally across downstream tasks and datasets @xcite . while prior work has focused on the quality of generation @xcite @xcite , and more recently on text generation novelty @xcite , there has been limited work on characterizing the sorts of lexical patterns that are learned by llms. the last black man in san francisco is a poignant, beautifully shot film […] creates a unique and intense viewing experience. ## related work diversity in text generation past evaluations of diversity in llm outputs have primarily focused on token-level diversity @xcite . diverse sampling strategies have been introduced to address the lower token diversity observed in neural text generation @xcite , however it is unclear whether such sampling strategies also increase the diversity of the syntactic structure in llms. beyond lexical diversity, padmakumar and he (2023) extend definitions for measuring content diversity, which has broad applications in downstream tasks such as summarization and creative story generation. recent work has quantified the drop in generated-text diversity specifically relative to the rlhf training process, however this again focused primarily on token level diversity @xcite . our work aligns more closely with the first body of work; we measure the syntactic structure of text rather than its semantic content. most similar to our work is @xcite , which broadly evaluates text repetition metrics at the stylistic, content, and lexical level. our methods do not address repetition in content but rather focus on extending the characterization of lexical and stylistic repetition with text abstractions in llms. @xcite provide a computational approach comprising lexical and syntactic components to describe stylistic elements in model-generated text. the discussion around style in writing has been adopted broadly for a variety of downstream tasks such as author or model attribution @xcite @xcite . while our main goal is to provide measurements and characterizations of repetitive syntactic features in text, definitions of stylistic elements are closely related and help contextualize our findings. one can use our definitions of templates to ask broader questions about the prevailing syntactic style in a given corpus. indeed, recent works adopt various measures of linguistic analysis to address differences in writing style in both human-written and model-generated texts @xcite . ## detecting syntactic templates our goal is to search for abstract representations of texts to capture more subtle repetitions than mere text memorization. repeated strings of literal tokens may not be sufficient for describing such redundancy nor why a summary produced by, e.g., chatgpt, might seem familiar. focusing on syntactic patterns rather than tokens allows us to capture such repetitions for example, a pattern consisting of the part-of-speech sequence dt jj nn in dt jj nn will match to phrases in movie reviews ("a romantic comedy about a corporate executive") and in news summarization rb , dt nn vbd: fiercely , the beast breathed, slowly , the caterpillar inched, vbd in dt nn .: creaked in the wind ., emerged as a butterfly ., inched along the branch ., rattled with each gust ., vbn in jj nns .: diced into perfect cubes ., groaned on rusty hinges ., julienned into thin strips ., minced into fine fragments ., sliced into uniform rounds . ("a humorous insight into the perceived class"), even though these sentences have only one token overlap. ## templates in model-generated text we first evaluate olmo-7b instruct on 3 tasks: open-ended generation, synthetic data generation, and summarization, using both greedy and varying temperature and top-p sampling strategies (rq1). ## searching for templates in pre-training data one hypothesis for the emergence of templates in generated text is that these templates are overrepresented in the training data (rq2). here we interrogate this empirically. ## templates in closed-source models with olmo, we find that 75% of templates are found at high frequencies in the pre-training data ( §6.2). most available models however do not release their pre-training data. here we evaluate the incidence of templates in other closed-source models, which we define as models that do not release their training data. addressing rq1, we characterize the rates of templatic texts in these models, and posit that templates may be indicators of the pre-training data sources models are trained on. ## conclusions in this work, we introduce syntactic templates as a framework for analyzing subtle repetitive characteristics in model-generated text. we show that this analysis can also extend to human-written references and downstream tasks, and find that the pre-training data contains many of these identified templates. we show that evaluating repetition in parts-of-speech sequences is useful for detecting subtle types of data "memorization". our hope is that this work inspires additional research into characterizing where (in data) observed stylistic patterns in llm outputs originate.