Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains (2024)

Abstract: Although large language models (LLMs) exhibit remarkable capacity to leverage in-context demonstrations, it is still unclear to what extent they can learn new facts or concept definitions via prompts. To address this question, we examine the capacity of instruction-tuned LLMs to follow in-context concept annotation guidelines for zero-shot sentence labeling tasks. We design guidelines that present different types of factual and counterfactual concept definitions, which are used as prompts for zero-shot sentence classification tasks. Our results show that although concept definitions consistently help in task performance, only the larger models (with 70B parameters or more) have limited ability to work under counterfactual contexts. Importantly, only proprietary models such as GPT-3.5 can recognize nonsensical guidelines, which we hypothesize is due to more sophisticated alignment methods. Finally, we find that Falcon-180B-chat is outperformed by Llama-2-70B-chat is most cases, which indicates that increasing model scale does not guarantee better adherence to guidelines. Altogether, our simple evaluation method reveals significant gaps in concept understanding between the most capable open-source language models and the leading proprietary APIs.

## introduction large language models (llms) are known to distill knowledge from vast datasets during the pretraining phase @xcite . such knowledge can be queried via prompting, which allows the application of llms to several knowledgeintensive tasks in zero-shot and few-shot settings. in particular, recent work demonstrates promising applications of llms to reduce the cost of data annotation in several domains @xcite @xcite . 1 code and dataset are available at https://github.com/ thefonseca/concept-guidelines consider the following concepts: -background: a sentence that provides context, foundational knowledge, or relevant information about the research topic, existing theories, prior studies, or the broader scientific field in which the research is situated. classify the text below into one of the categories listed above. be concise and write only the category name. text: therefore, the phase transition can be classified as essentially driven by coulomb interactions. most data labeling efforts based on llms leverage in-context demonstrations to elicit the desired concepts. however, previous work suggests that language models cannot learn from in-context ground-truth labels but just leverage demonstrations to infer the task format and label space @xcite . in contrast, human annotators typically follow guidelines, which in addition to examples, include concept definitions @xcite that complement and modify the annotator's prior concept understanding to align with the labeling goals. ## concept classification with guidelines let s and c be random variables representing sentences (from the set of token sequences s) and corresponding latent concepts to be inferred (from the concept set c; e.g., whether the sentence conveys scientific background or methods). to specify the task, we introduce annotation guidelines g, which specify concept labels and their definitions. then, the concept annotation process for a sentence s given the guideline g is formalized as follows: the language model parameters θ capture the prior conceptual knowledge k acquired during pre-training and instruction-tuning. in addition, we hypothesize that θ encodes the conditional dependency between the guidelines g and the concepts c as the guidelines express relationships between concept labels and their task-specific definitions. by performing controlled interventions in the guidelines, we aim to measure how concept understanding is affected by the following factors: 1) the lexical information of concept labels and concept definitions; 2) the degree of factuality of concept definitions. in the next section, we present guidelines designed to capture such factors. ## experimental setup to experiment with different types of guidelines defined in section 2.1, we choose concepts c for which the language models have exposure via pretraining data. the first domain we explore relates to rhetorical roles in scientific articles (section 3.1), which is extensively covered in the literature @xcite . since the pre-training data for llms likely include various scientific concept classification datasets, we annotate a novel dataset of sentence-level financial concepts (section 3.2). in addition to controlling for label memorization, our financial annotation based on the integrated reporting framework @xcite covers concepts that are technical but arguably more accessible than scientific rhetoric. in sections 3.3 and 3.4, we detail the llm baselines used in the experiments and the classification task hyperparameters. ## conclusion by using factual and counterfactual concept guidelines for sentence classification, we demonstrate measurable gaps in concept understanding between leading open-source and proprietary instructiontuned models. while some level of counterfactual concept understanding emerges with scaling, opensource models cannot recognize nonsensical (outof-dictionary) guidelines, which the closed apis can address more consistently. one question to be addressed in future work would be to investigate potential correlations between the capacity of reasoning in counterfactual contexts and other common generation issues such as hallucination.