Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts (2022)

Abstract: Anaphora resolution is an important task for information extraction across a range of languages, text genres, and domains, motivating the need for methods that do not require large annotated datasets. In-context learning has emerged as a promising approach, yet there are a number of challenges in applying in-context learning to resolve anaphora. For example, encoding a single in-context demonstration that consists of: an anaphor, a paragraph-length context, and a list of corresponding antecedents, requires conditioning a language model on a long sequence of tokens, limiting the number of demonstrations per prompt.In this paper, we present Mice (Mixtures of In-Context Experts), which we demonstrate is effective for few-shot anaphora resolution in scientific protocols. Given only a handful of training examples, Mice combines the predictions of hundreds of in-context experts, yielding a 30% increase in F1 score over a competitive prompt retrieval baseline. Furthermore, we show Mice can be used to train compact student models without sacrificing performance. As far as we are aware, this is the first work to present experimental results demonstrating the effectiveness of in-context learning on the task of few-shot anaphora resolution in scientific protocols.

## introduction prompting large language models (lms) with incontext demonstrations has enabled surprisingly effective few-shot learning @xcite . however, more complex linguistic annotations over paragraph-length inputs, such as anaphora and coreference, have proven challenging @xcite . prompting language models with demonstrations of anaphora and their corresponding antecedents requires encoding long sequences of tokens, limiting the number of demonstrations that can be used within a single prompt. furthermore, the performance of in-context learning has been shown to be sensitive to the choice of demonstrations @xcite and their ordering in the prompt @xcite . to address these challenges, this paper presents mixtures of in-context experts (mice). we demonstrate the effectiveness of mice on anaphora resolution in chemical synthesis protocols (see examples in figure 1 ). scientific protocols make an ideal testbed for few-shot anaphora resolution because they contain rich coreference and bridging links. furthermore, these linguistic structures are expressed very differently from well-studied domains with extensive annotated resources, such as newswire, and they are not easily amenable to annotation by non-expert crowd workers. mice works as follows. given an anaphor, such as "the mixture", it uses in-context learning to predict a list of substances contained in the mixture that are referenced earlier in the procedure, for example: "bromoacetyl bromide", "compound 54" and "water". with only a handful of training examples (e.g., 16 or 32), mice generates an ensemble of up to k d in-context experts, each of which consists of a prompt containing d demonstrations chosen from the k available training examples. for instance, if @xmath0, mice generates up to 256 in-context experts (prompts) to be ensembled. the experts' predictions are then combined in a mixture model, where mixture weights are computed by comparing embeddings of the input mention and available training examples. we find that, although some prompts and prompt orderings perform better than others, individual prompts act as local experts in different regions of the input space @xcite @xcite , and no single prompt works better than others on all inputs (see figure 2 ). furthermore, if the same antecedent is predicted by multiple in-context experts, this provides independent sources of evidence, increasing the probability the answer is correct @xcite . in extensive experiments, we show mice significantly improves the performance of figure 1 : resolving antecedents of "the mixture" in a chemical synthesis procedure using mice. given a small training set of 16 examples and a test input, we construct 256 prompts, each with two in-context demonstrations. the prompts are then fed into a pre-trained language model (e.g. gpt-j) to generate candidate antecedents. the probabilities of each candidate antecedent are computed and combined in a mixture of in-context experts using a similarity-based gating function. mice then selects the antecedents with the highest probabilities. in the figure, orange, blue, red denote the anaphor, the true antecedents, and incorrect antecedents, respectively. while mice consistently improves in-context anaphora resolution, inference is relatively expensive, due to the large number of prompts involved. to address this limitation, we show that fine-tuning bert-based models on data that is automatically labeled with mice yields further performance improvements, while also producing much more compact models @xcite . ## anaphora resolution in scientific protocols split-antecedent anaphors @xcite @xcite are plural mentions that refer to two or more antecedents in the previous discourse. for instance in the following text: "[alice] antecedent and @xcite antecedent went to the store. ## mixtures of in-context experts while in-context learning has achieved good performance when prompted with a few examples, the performance can vary significantly depending on different prompt design choices @xcite . furthermore, anaphora resolution requires paragraph-length contexts, limiting the number of in-context examples that can be encoded in a single prompt (figure 3 ). we address these challenges with mice. we show mice is an effective method for few-shot anaphora resolution in §5, and demonstrate that it can be used to automatically label data for fine-tuning more compact models, without sacrificing performance, in §4.1. figure 3 : distribution of the maximum number of incontext demonstrations of an anaphor, synthesis protocol, and corresponding antecedents that can be encoded in a single prompt. we compute the max number of demonstration tokens by subtracting the max sequence length (2048) by a fixed number for generated tokens (256) and the number of tokens for the test input. demonstrations are randomly sampled until the max number of tokens is reached. given the longer contexts needed to demonstrate anaphora resolution, a prompt can encode at most 8 demonstrations, much less than the 32 used in @xcite . in-context learning we formulate the task of anaphora resolution in synthetic protocols as follows. the input includes a document d and a query anaphor a. our goal is to identify a set of antecedents @xmath1, y 1 , ..., y m } that correspond to text spans in d. to tackle this problem via in-context learning, we frame it as a squadstyle extractive question answering task @xcite . specifically, as shown in figure 1 , each example (d, a) is formatted as the concatenation of document d and template question: "what does a contain?" an autoregressive language model then completes this sequence by generating y, with the antecedents separated by a special marker "|". following the typical approach to in-context learning, the prompt includes a few demonstrations in the prefix and ends with the test input. mixture of experts for a given test input @xmath2, a), we aim to find the antecedents y i with the highest probabilities p (y i |x). the notation y i denotes an antecedent from the union of antecedents generated by all prompts. mice computes p (y i |x) using a mixture of experts @xcite , treating the prompt, z, as a latent variable @xcite : in eq.1, p (z|x) represents the likelihood that prompt z is constructed given x, and p (y i |z, x) represents the probability that the lm predicts antecedent y i when prompted with z and x. similarity-based gating we compute p (z|x) by summing similarity scores s(x, u 1 ), ..., s(x, u d ) between x and the in-context demonstrations ## experimental setup we evaluate our approach on chemu-ref, a corpus of synthetic procedures from chemical patents that are annotated with coreference and anaphora @xcite . the chemu-ref annotations contain fine-grained chemistry-specific relations such as transformed, which indicates the mixture components undergo a chemical transformation, or work-up, which represents a combination of compounds to isolate or purify a reaction product. in this work, we focus on modeling the general structure of anaphora in scientific protocols and understanding which compounds are combined at each step of the procedure, which does not require making these more fine-grained distinctions. therefore, we pre-processed the data by collapsing all one-to-many chemu-ref relations into a single multiple-antecedent relation. we remove anaphors comprising compounds described with iupac nomenclature @xcite , for example "2,2,6,6-tetramethylpiperidine", while retaining nominal anaphors, such as "the mixture", "the solution" and "the reaction", which make up 90% of anaphors in the chemu-ref corpus. for train/dev/test splits, we use the original train split for training and development, and the original dev split for evaluation. we use f 1 as our evaluation metric. in particular, we compute the micro-f 1 between the predicted and gold sets of antecedents of the evaluation data. a predicted and a gold antecedent are considered the same if they are an exact match. ## results and analysis results on dev-256 are presented in results on the full test set ( ## related work ensembling prompts recent work has explored different strategies for ensembling lm predictions. @xcite and @xcite aggregate predictions from multiple prompts via either majority voting or weighting on development set performances. @xcite and @xcite ensemble a small number of prompt templates or soft prompts, then learn their weights from a large training set. @xcite combines soft prompts pre-trained on multiple source tasks with fine-tuned attention weights to generate prompts for a target task. in contrast to all the above work, mice generates a large number of prompts from a small number of training examples and combines their predictions using mixtures of experts with similarity-based gating. similar to our work, @xcite shows that ensembling the predictions of k one-shot prompts by multiplying their lm probabilities is better than concatenating all k examples as one prompt, which we use as a baseline (product). concurrently, @xcite demonstrates the advantage of ensembling k one-shot prompts via co-training the ensembling parameters using a smaller model. however, they train the ensembling parameters with an mlp layer and thus require all training examples to share the same label space, which is not applicable to anaphora resolution. importantly, both @xcite and @xcite only consider a small number of prompts, whereas we demonstrate the benefits of generating and combining hundreds of prompts as a mixtureof-experts. ## conclusion in this paper, we propose and demonstrate the effectiveness of mixtures of in-context experts for few-shot anaphora resolution in chemical synthesis protocols. mice generates a large number of in-context experts (prompts) from a few training examples, where each expert consists of randomly permuted demonstrations. predictions made by these experts are combined in a mixture model with a similarity-based gating function. our experiments show that mice significantly improves the performance of in-context learning for anaphora resolution in scientific protocols, using just a handful of training examples. we further demonstrate that knowledge distillation can dramatically reduce the costs of inference while maintaining performance, which increases mice's potential applicability in working systems.