A Corpus and Evaluation for Predicting Semi-Structured Human Annotations (2022)

Abstract: A wide variety of tasks have been framed as text-to-text tasks to allow processing by sequence-to-sequence models. We propose a new task of generating a semi-structured interpretation of a source document. The interpretation is semi-structured in that it contains mandatory and optional fields with free-text information. This structure is surfaced by human annotations, which we standardize and convert to text format. We then propose an evaluation technique that is generally applicable to any such semi-structured annotation, called equivalence classes evaluation. The evaluation technique is efficient and scalable; it creates a large number of evaluation instances from a comparably cheap clustering of the free-text information by domain experts. For our task, we release a dataset about the monetary policy of the Federal Reserve. On this corpus, our evaluation shows larger differences between pretrained models than standard text generation metrics.

## introduction general-purpose sequence-to-sequence models have achieved impressive results on conditional text generation @xcite , machine translation @xcite , and text summarization @xcite . this has lead to their application to ever more tasks; as long as the task can be formalized in a text-to-text format, it can be processed by these models @xcite . we apply sequence-to-sequence models in a different setting: documents interpreting other documents. this phenomenon is pervasive in our daily lives, be it a critic reviewing a play or book, a website presenting highlights of a travel guide, or, as in this paper, a journalist writing an article about an organization's press release. for social scientists, these reviews or articles present an interesting subject of study; they surface * correspondence to andreas.marfurt@idiap.ch. the author's interpretation of the original source document. with the tool of human annotations, domain experts can extract the core constituents and surface implicit information in these various interpretations to make them comparable. in this paper, we train models to learn this interpretation process (see figure 1 ). 1 we describe how human annotations can be standardized and converted into a text-only format to serve as semistructured targets in this interpretation prediction task. we introduce the fomc dataset, a corpus about the monetary policy of the federal reserve, the central bank of the united states of america. the dataset contains source documents of greatly varying length, containing policy announcements such as press releases or speeches. the target interpretations are short, and consist of selected sentences taken from new york times articles, which are then annotated by domain experts. we also devise a scalable evaluation technique for semi-structured outputs, which we call equivalence classes evaluation. domain experts cluster highlighted text spans from the human annotations into equivalence classes, signifying their semantic inter-changeability. a generative model is then probed with a prefix and either a true continuation from the data or a wrong continuation from a different equivalence class. if the model learned the process of interpretation well, it will give higher probability to the true continuation. from a single clustering, we can automatically generate a large number of evaluation instances by sampling negative text spans. we train transformer @xcite sequence-to-sequence models with varying levels of pretraining on the fomc dataset, and find that bart @xcite performs well on our equivalence classes evaluation and standard text generation evaluation metrics. our contributions are: 1) we introduce a new dataset on document interpretation, with semistructured annotations and documents on the monetary policy of the federal reserve. 2) we introduce a method to convert these annotations into text format and apply generative text models. 3) we devise a scalable technique to evaluate models on the task of generating semi-structured outputs, by efficiently utilizing domain experts' grouping of text spans into equivalence classes. 4) we perform an evaluation with our technique, and showcase its flexibility with an in-depth error analysis. ## semi-structured human annotations we consider a setting where a long text is interpreted in a few sentences. the interpretation may select an aspect of the source text to focus on, and it may be opinionated. each sentence is annotated by domain experts to surface and structure the important information. ## equivalence classes evaluation to evaluate a model on predicting the marked spans of individual annotation categories, we propose the equivalence classes evaluation as an efficient way of generating evaluation instances from domain experts' knowledge. ## experiments in our experiments, we use the fomc dataset described in section 2.4. ## related work to the best of our knowledge, our setting, task, and evaluation have not been studied in prior work. evaluation. the closest approach to our equivalence classes evaluation is the concept of verbalizers in pattern-exploiting training (pet) @xcite , the relation to which we already discussed in section 3.2. the biggest difference to our approach is that pet's verbalizers are limited to a small, bounded set of predefined single tokens or few-token spans, while our equivalence classes are unbounded, and their members are collected from the data without restrictions on length or content. other work has also tried to make human annotations more efficient, e.g. for importance judgements of sentences in multi-domain summarization @xcite , or multi-task information extraction @xcite . annie builds fact synsets to speed up open information extraction @xcite . aspect-oriented summarization. interpretations may focus on certain aspects in the source documents, making them somewhat similar to aspect-oriented summarization. aspectnews @xcite and space @xcite are two recent datasets with accompanying models. news summarization. since our interpretations are excerpts of new york times articles, news summarization is relevant to our work as well. this is a very active field of research, with multiple large-scale datasets (e.g. cnn/dm @xcite , xsum @xcite , among others). a lot of methods have been tried on these datasets. bart @xcite and pegasus @xcite have shown some of the best results for fine-tuned models. ## conclusion we have devised a method to convert semistructured human annotations into text format. we then introduced a task of predicting annotated interpretations of source documents that can be tackled with sequence-to-sequence models. we presented a human-annotated corpus about the monetary policy of the federal reserve. our equivalence classes evaluation is an efficient technique to create a large number of targeted evaluation instances from a comparably cheap clustering by domain experts. we use this technique to evaluate stateof-the-art generative models on our task, and find that it shows larger differences between pretrained models than standard text generation metrics. in further in-depth analyses, the equivalence classes evaluation tests the models for specific properties, such as how they handle negation, and detects why models struggle to correctly rank alternative text spans of certain human annotation categories.