Explaining Data Patterns in Natural Language with Language Models (2023)

Abstract: Large language models (LLMs) have displayed an impressive ability to harness natural language to perform complex tasks. We explore whether we can leverage this ability to find and explain patterns in data. Specifically, given a pre-trained LLM and data examples, we apply interpretable autoprompting (iPrompt) to generate a natural language string explaining the data. iPrompt iteratively generates explanations with an LLM and reranks them based on their performance when used as a prompt. Experiments on a wide range of datasets, from synthetic mathematics to natural language understanding, show that iPrompt can yield meaningful insights by accurately finding dataset explanations that are human-interpretable. Moreover, iPrompt is reasonably efficient, as it does not require access to model gradients and works with relatively small models (e.g. ~6 billion parameters rather than >=100 billion). Finally, experiments with scientific datasets show the potential for iPrompt to aid in scientific discovery.

## introduction large language models (llms) have attained an extraordinary ability to harness natural language for solving diverse problems @xcite , often without the need for finetuning @xcite . moreover, llms have demonstrated the capacity to excel at realworld problems, such as mathematics @xcite , scientific question answering @xcite , predicting brain responses @xcite , and classifying proteins and chemical compounds @xcite . in this work, we probe whether we can leverage the learned skills of an llm to discover and explain patterns in a dataset. to do so, we invert dataset input: 3 1 output: 4 input: 4 7 output: 11 … input: 5 9 output: 14 llm ## related work problems related to dataset explanation the problem statement presented in this work closely resembles the widely studied problems of symbolic regression @xcite , program synthesis @xcite ), text/table summarization @xcite , and pattern discovery in data-mining @xcite . iprompt can be viewed as an algorithm for symbolic regression, in which the set of allowable symbols consists of semantically meaningful natural language strings. one recent work proposes the task of inferring prompts that improve supervised prediction @xcite , which we generalize here to diverse use cases for dataset explanation. prompting and autoprompting. with the advent of large-scale models, prompting (i.e. finding the right prompt to use to query an llm for a given task) has exploded as an area of inquiry, often yielding impressive improvements in performance @xcite @xcite ) and spurring a line of work aiming to make prompting easier @xcite @xcite . recently, autoprompting (i.e. automatically searching for a prompt or prompt-embedding via optimization) has emerged @xcite to improve the process of prompting, with methods such as prefix-tuning (li and liang, 2021), p-tuning @xcite , prompttuning with rules @xcite , knowledgeable prompt tuning @xcite and many more @xcite . these strategies use gradient descent to find a set of "adapter" parameters that maximize model performance, but do not require that the new parameters map back to tokens in discrete space, rendering them uninterpretable. a few methods tackle the more difficult problem of searching for prompts that can be expressed in natural language tokens. rlprompt @xcite searches for such a prompt using reinforcement learning and one recent work @xcite queries an llm to produce a prompt. auto-prompt @xcite performs autoprompting via input gradients (see sec. 3). these methods effectively alter a model's predictions, but do not constrain the discovered prompts to be semantically meaningful, resulting in prompts that are difficult to interpret @xcite . another related work directly finetunes an llm to describe the difference between two datasets @xcite . one recent work proposes a method for interpretable autoprompting similar to the one here, with a focus on improving prediction performance rather than on explaining data patterns @xcite . alternative methods for neural-network interpretation a popular method for interpreting neural networks is to inspect an llm's individual predictions via feature importances @xcite , feature-interaction importances @xcite , extractive rationales @xcite , or natural language explanations for individual predictions @xcite . these works can provide meaningful insights for individual predictions, but it is difficult to parse them into an understanding of an entire dataset. alternatively, one can investigate whether an llm's learned representations via probing @xcite or by directly analyzing a model's internal weights and activations @xcite @xcite . however, these approaches are limited in their ability to generate previously unknown descriptions of data. ## experimental setup we consider two sets of experiments. first in sec. 5, we explore iprompt's ability to rediscover a correct and fluent prompt on the variety of simple instruction datasets ( language models for the main set of experiments, we always generate prompts using gpt-j, a 6 billion parameter model @xcite . we restrict prompts to {6,12} tokens for sentiment classification and 6 tokens for the remaining data collections in evaluation metrics our main evaluation measures each prompt's closeness to groundtruth via three metrics: (1) correct -whether the generated explanation contains one of a set of problemspecific keywords. (2) mrr -mean reciprocal rank measuring the rank of the first task-correct prompt. given a set of datasets @xmath0, ..., d n }, we compute: @xmath1, where rank i is the one-indexed rank of the first correct explanation. (3) human -the human evaluation scores between the top-generated explanation and a prespecified groundtruth explanation, when instructed "you are given a groundtruth description along with a generated one. on a scale of 1 (worst) to 5 (best), how interpretable and accurate is the generated description?" foot_4 . the mean human evaluation score (ranging from 1 to 5) is normalized. as a secondary evaluation, we measure generalization ability when we evaluate explanations based on accuracy as a prompt for other models. accuracy is computed following @xcite : using exact matching with beam search, a beam width of 4, and a length penalty of α = 0.6. ## scientific investigations with iprompt we now investigate whether iprompt can explain patterns in scientific datasets. specifically, we analyze the galactica model @xcite with 6.7 billion parameters. we query whether it can describe differences in protein sequence before investigating a neuroscience problem; see appendix a.5 for similar experiments in a chemical toxicity setting. ## conclusion and discussion while we focus on text, iprompt could be applied generally settings where an llm performs well. for example, in computer vision, an interpretable autoprompt may look like a mask of an image, and in vision-language models, an interpretable prompt may be a description of a vision task, e.g. find the largest shape in this image.