Can Large Language Models (or Humans) Disentangle Text? (2024)

Abstract: We investigate the potential of large language models (LLMs) to disentangle text variables—to remove the textual traces of an undesired forbidden variable in a task sometimes known as text distillation and closely related to the fairness in AI and causal inference literature. We employ a range of various LLM approaches in an attempt to disentangle text by identifying and removing information about a target variable while preserving other relevant signals. We show that in the strong test of removing sentiment, the statistical association between the processed text and sentiment is still detectable to machine learning classifiers post-LLM-disentanglement. Furthermore, we find that human annotators also struggle to disentangle sentiment while preserving other semantic content. This suggests there may be limited separability between concept variables in some text contexts, highlighting limitations of methods relying on text-level transformations and also raising questions about the robustness of disentanglement methods that achieve statistical independence in representation space.

## introduction when computational social scientists analyze text data there are situations where the text is contaminated by a forbidden variable that we want to preclude from our analysis or handle in a special way. for instance, methods in causal inference that use text to correct for unseen confounders @xcite run into difficulties if the text is influenced by the treatment variable @xcite . in other contexts, due to ethical, legal, or robustness considerations, we may want to ensure that models trained on a corpus are not influenced by effects such as demographic factors @xcite , domains @xcite , personal information @xcite , or other sensitivity information @xcite . there are several methods that process text representations to enforce invariance with respect to a forbidden variable @xcite @xcite @xcite . with a few exceptions, these methods operate on a numerical representation of the text, and not directly on the text itself, which makes them less interpretable. furthermore, they typically require a large set of annotated examples of the forbidden variable, which may not always be available or may be costly to obtain. computational social science using text as data is currently being transformed by the introduction of large language models (llms) applied in a zeroshot or few-shot fashion @xcite . in this paper, we consider the question of whether llms have an out-of-the-box ability to disentangle a text: to transform it so that the value of a forbidden variable is hidden while preserving as much as possible of the original text. we apply the llms in a few-shot setup to reproduce the scenario where we have no large-scale annotations available of the forbidden variable. we investigate a variety of prompt-based techniques. if successful, the disentanglement process would also be understandable to a human reader, since the changes are carried out via interpretable transformations of the text itself. we find that this type of disentanglement is challenging for the current generation of llms, as well as human annotators, to carry out. while the most powerful llms (e.g., gpt-4-class models) sometimes transform text so that it is difficult for humans to determine the original value of the forbidden variable, we see only a slight reduction in the accuracy of classifiers predicting the forbidden variable; its statistical presence is clearly maintained. ## related work our work builds on a growing body of literature on removing undesired information from text representations. this line of research has focused on developing algorithms to learn representations that are independent of protected attributes like demographic variables @xcite @xcite . these methods typically leverage adversarial training or projection techniques to encourage models to learn representations orthogonal to the forbidden variable. in addition, some work has also used llms for manipulating text, sometimes with goals related to disentanglement in mind. for example, work has used language models for style transfer and controlled text generation, aiming to modify attributes like sentiment or formality while preserving core content @xcite . other research has leveraged language models for tasks like paraphrasing @xcite , simplification @xcite , and neutralizing biased language through llm fine-tuning @xcite . we build on these works by exploring the use of out-of-the-box language models for the task of forbidden variable removal via text disentanglement. to the best of our knowledge, we are the first to study the effectiveness of llm prompting for the disentanglement task systematically and to compare this approach to human performance. ## defining disentanglement previous work has focused on removing a forbidden variable from a numerical text representation, not from the raw text itself. for a representation x and a forbidden variable z, @xcite defined concept erasure as a process that finds a guarding function h such that h(x) ⊥ z. several approaches have been proposed to find guarding functions; most work has focused on finding projections that optimize guardedness with respect to linear classifiers @xcite . however, this process is not interpretable in that a human cannot easily reason about what the transformation is doing in the numerical representation space. in addition, finding the guarding function requires a collection of annotated training instances. in this article, we use a similar conceptualization, but we transform the text directly instead of working in a latent representation space. we define text train test original reviews llm classifier accuracy train test disentangled reviews classifier @xmath0: the experimental setup for measuring the effectiveness of llms at removing a target variable from the raw text representation. we can trivially satisfy this independence criterion by letting d return a transformed text that is unrelated to the original text w (e.g., an empty text). to avoid such uninformative cases, we also want the transformation to be minimally intrusive. in previous work that relied on linear projections, this condition was satisfied by construction. although the meaning of intrusiveness will vary with research context, we here define it as our ability to measure the effect of disentanglement on our ability to predict other variables represented in the text. looking ahead, we envision that minimal intrusiveness can be defined on a semantic level in terms of maximizing similarity in a representation space or on a superficial level by minimizing string edit distance; at this point, we leave this formalization to future work. ## method the goal of our experiments is to determine whether current out-of-the-box llms are able to disentangle a forbidden variable from text without removing traces of other variables. we test two llms: mistral 7b and gpt-4. we choose mistral 7b as a smaller open-source llm that performs well on benchmarks and can be run on consumer hardware @xcite . we choose gpt-4 as a top-performing commercially available model (as of 2024) @xcite . more specifically, we use mistral 7b v0.2 (instruction-tuned) and "gpt-4-0125-preview" from the openai api. to add context to our results, we also test the performance of mean projection @xcite , a method for removing specific information from representations, as well as, critically, the performance of human annotators. we include human annotators in our experiments to provide a benchmark for the level of forbidden variable removal that can be achieved by careful manual text editing, and to explore the intrinsic challenges that may stem from the entanglement of the forbidden variable and text content in natural language. ## results our results are summarized in the llms performed well at keeping information about topic. furthermore, the paraphrase prompting strategy shows that they successfully keep information about sentiment when rewriting: in the case of gpt-4, the prompt even leads to a slight increase in sentiment classifier accuracy. this indicates that llms may amplify original text while the llms generally struggle to remove statistical associations with the forbidden variable, we also find that human coders face a similar difficulty. this finding implies that there may be, at least for the task described here, limited separability in the text between the forbidden variable and the remainder of the text. this limited separability raises questions about the robustness of disentanglement methods operating on downstream representations as such methods may inadvertently generate representations incongruous with real text. further research is therefore needed to develop techniques that can effectively disentangle the forbidden variable from the relevant semantic content in a way that respects the content of the original text.