Generating Interpretations of Policy Announcements (2024)

Abstract: Recent advances in language modeling have focused on (potentially multiple-choice) question answering, open-ended generation, or math and coding problems. We look at a more nuanced task: the interpretation of statements of political actors. To this end, we present a dataset of policy announcements and corresponding annotated interpretations, on the topic of US foreign policy relations with Russia in the years 1993 up to 2016. We analyze the performance of finetuning standard sequence-to-sequence models of varying sizes on predicting the annotated interpretations and compare them to few-shot prompted large language models. We find that 1) model size is not the main factor for success on this task, 2) finetuning smaller models provides both quantitatively and qualitatively superior results to in-context learning with large language models, but 3) large language models pick up the annotation format and approximate the category distribution with just a few in-context examples.

## introduction state-of-the-art language models are evaluated on multiple-choice question answering (e.g. mmlu; @xcite , math problems (e.g. gsm8k; @xcite , or coding (e.g. hu-maneval; @xcite . these benchmarks do not provide much insight for the use and analysis of such models in the humanities and social sciences. in this paper, we present a dataset on an important issue in the humanities and social sciences, namely interpretation. in this case, our concern is with how newspaper articles characterize policy announcements (press releases, q&a sessions, interviews, etc.). these interpretations are carefully annotated by political scientists to give them structure (through labeling spans of text with a category such as act or motive) and to provide additional background knowledge as comments. we then train language models on the task of generating the annotated interpretations when shown the announcement. we compare sequence-to-sequence models with large language models (llms), and find that model size is not indicative of task performance. we achieve better results by finetuning the comparatively much smaller sequence-to-sequence models than by few-shot prompting llms. our code, data and models are available on github foot_0 . ## related work language models have previously been used to interpret figurative language @xcite , contracts @xcite , and building regulations @xcite . we provide a novel dataset on interpreting policy announcements. although using large language models to perform interpretation seems to have become more popular recently, the analysis of policy statements has focused mainly on either monetary policy @xcite @xcite or legislative speech @xcite . regarding the latter, a dataset for sentiment analysis of political debates, parlvote (abercrombie and batista-navarro, 2020) has been created. other work has focused on analyzing speech acts in political debates @xcite . the policy announcements in this paper differ from political debates by being performed on behalf of a single actor and encompassing a wide range of issue areas; the interpretations of those announcements often presume background knowledge on the part of readers, which is challenging to capture. finally, using annotations to generate semistructured outputs with language models has been used in galactica @xcite to annotate paper citations and specific character sequences (dna, amino acids), and to interpret economic policy announcements by the federal reserve bank @xcite . we deem the format of the latter useful for our task as well and will employ it in the following section. ## dataset the dataset concerns the foreign policy relations of the united states of america with respect to russia in the years from 1993 up to 2016. a team of political scientists has curated announcements (press releases, q&a sessions, interviews, etc.) and corresponding interpretations (partial, complete, or multiple sentences of new york times articles). details of the dataset's creation can be found in appendix a. for annotation, we follow the guidelines of @xcite . we define the mandatory annotation categories of a standardized sentence to be act, actor, and reference, with the same meaning as in previous work, although in this case, the actor is almost always the united states. we add a mandatory annotation category russia link, which marks the connection of the announcement to russia. we import the optional categories (attribution, evidence, motive, scope) without any changes. comments, which make explicit newspaper readers' presumed background knowledge, are added in parentheses after text annotated as act or russia link. we convert the annotated interpretations into text-only format by inserting start and end markers @xcite ). an example from the training set can be seen in figure 1 , and statistics for the dataset are listed in ## experiments in our experiments, we compare different approaches to solve our proposed task. to generate the target interpretations, we compare finetuning sequence-to-sequence models with in-context learning with large language models. we select t5 @xcite , flan-t5 @xcite , @xcite as our sequenceto-sequence models for their track record on previous text generation tasks. as large language models, we choose llama-3.1 (8b) @xcite and gemma-2 (9b) @xcite . we decided to only use local language models for reasons of reproducibility. ## results the results in ## conclusion we presented a new dataset on generating interpretations for policy announcements concerning us foreign policy with respect to russia for the years 1993 to 2016. we evaluated common language models on this generation task and found that finetuned sequence-to-sequence models, specifically bart, outperformed few-shot prompted large language models. interesting directions for future work are investigating long-context methods that can access the entire announcement when writing the interpretation, and a more structured approach to generating the individual categories and their contents. we also hope to see whether models trained on this dataset can be successfully transferred to other tasks and domains.