Detecting Mode Collapse in Language Models via Narration (2024)

Abstract: No two authors write alike. Personal flourishes invoked in written narratives, from lexicon to rhetorical devices, imply a particular author—what literary theorists label the implied or virtual author; distinct from the real author or narrator of a text. Early large language models trained on unfiltered training sets drawn from a variety of discordant sources yielded incoherent personalities, problematic for conversational tasks but proving useful for sampling literature from multiple perspectives. Successes in alignment research in recent years have allowed researchers to impose subjectively consistent personae on language models via instruction tuning and reinforcement learning from human feedback (RLHF), but whether aligned models retain the ability to model an arbitrary virtual author has received little scrutiny. By studying 4,374 stories sampled from three OpenAI language models, we show successive versions of GPT-3 suffer from increasing degrees of “mode collapse” whereby overfitting the model during alignment constrains it from generalizing over authorship: models suffering from mode collapse become unable to assume a multiplicity of perspectives. Our method and results are significant for researchers seeking to employ language models in sociological simulations.

## background language model performance on arbitrary tasks scale linearly with the number of samples observed during training @xcite . this past year has seen the release of language models trained on datasets containing upwards of two trillion tokens, two orders of magnitude greater than the 300b tokens gpt-3 observed during training @xcite . large training sets are difficult to filter for unsafe language @xcite . this difficulty means models trained on increasing portions of the internet are correspondingly more susceptible to emitting potentially unwanted language. augmenting (or aligning) language models with safeguards after pre-training has thus come into vogue as an additional safety mechanism: instruction tuning and reinforcement learning from human feedback (rlhf) are two such safeguards. 2022 saw openai release a series of models based on instructgpt: a gpt-3 model augmented with both strategies @xcite . both strate-gies involve supervised training. instruction tuning instructgpt was first subject to a supervised fine-tuning process wherein openai trained the model on a series of labelled examples indicating preferred exchanges between two interlocutors. instruction tuning trains the model to follow instructions. rlhf the fine-tuned model was then subject to a process known as reinforcement learning from human feedback, or rlhf @xcite . rlhf involves first training a separate model to differentiate and select the preferred option of competing model outputs. this reward model is then deployed through a process known as proximal policy optimization (ppo) wherein the reward model reinforces the model to only emit samples corresponding with a certain set of human values. ## method we present our experimental design and our large language models of interest. ## discussion one common issue beleaguering older generative adversarial networks (gans) is "mode collapse" wherein overfitting a gan results in the model failing to generalize over their target distribution @xcite . gans suffering from mode collapse consequently becoming more repetitive the more training they receive. our analysis of 4,374 sampled stories reveal the newer gpt-3.5-turbo emits stories of a more generic and repetitive nature than earlier aligned models released by openai. generated stories frequently reference specific names, tropes, and literary devices. the model moreover does not appear to adjust stories according to requested virtual author, indicating gpt-3.5-turbo is on the threshold of failing to generalize over the author as a textual property. we suspect the model suffers from mode collapse due to overalignment. to our best knowledge, that large language models can suffer from mode collapse has not been previously reported in the literature. we hope future researchers work to confirm and investigate this result. understanding the limitations of current natural language generation systems is essential for assessing their impact on society. ## conclusion there is no perfect method for aligning language models, and safeguards like instruction tuning and rlhf remain under active research. @xcite admits instructgpt suffers from an "alignment tax" wherein the model suffers from degraded performance in "several public nlp datasets," but it was unclear whether this degraded performance emerged in out-of-distribution tasks. our study suggests gpt-3.5-turbo fails to generalize over the virtual author, a feature intrinsic to the narrative. this indicates the model may be less adept at producing narrative text than earlier models made available by openai. this result impacts social scientists seeking to use language models to sample demographically-correlated data: instructing gpt-3.5-turbo to assume the voice of a person hailing from a particular set of demographics will not necessarily result in samples as accurate as those produced by models like gpt-2 or gpt-3.