MAGNIFIC o: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations (2023)

Abstract: Humans possess a remarkable ability to assign novel interpretations to linguistic expressions, enabling them to learn new words and understand community-specific connotations. However, Large Language Models (LLMs) have a knowledge cutoff and are costly to finetune repeatedly. Therefore, it is crucial for LLMs to learn novel interpretations in-context. In this paper, we systematically analyse the ability of LLMs to acquire novel interpretations using in-context learning. To facilitate our study, we introduce MAGNIFICo, an evaluation suite implemented within a text-to-SQL semantic parsing framework that incorporates diverse tokens and prompt settings to simulate real-world complexity. Experimental results on MAGNIFICo demonstrate that LLMs exhibit a surprisingly robust capacity for comprehending novel interpretations from natural language descriptions as well as from discussions within long conversations. Nevertheless, our findings also highlight the need for further improvements, particularly when interpreting unfamiliar words or when composing multiple novel interpretations simultaneously in the same example. Additionally, our analysis uncovers the semantic predispositions in LLMs and reveals the impact of recency bias for information presented in long contexts.

## introduction humans can assign new interpretations to words or phrases in a language and consequently use them compositionally in utterances. for instance, the word 'zoom' is increasingly used to refer to a virtual calling service in the context of the covid-19 pandemic. similarly, as our society progresses, new words such as 'binge-watching' and 'selfie' keep getting coined frequently and become a part of our daily usage. moreover, in regular conversations, people might assign custom interpretations to words or phrases (e.g., see the interpretation of 'underpaid' in figure 2 ). the question of whether novel interpretation sql output "player who has received a yellow card" how many players have more than 1000 hours of training? with a yellow card base form plausible form foreign form adversarial form how many players have more than 1000 hours of training? giwzle how many players have more than 1000 hours of training? find the unique job ids from the employee's job history when is more than 3 and start date was after 1st january 1994. tenure select distinct job_id from job_history @xmath0. it is calculated by taking the difference between the end date and the start date. 'tenure' i think many of the instructors in the department are . underpaid who are the instructors in the computer science department? underpaid well, i consider someone to be if they . underpaid earn less than 100,000 really? how do you determine if someone is ? underpaid select name from instructor where @xmath1. sci.' and ; @xmath2, count(*) as t_prop from student group by major ) as t_tab t_tab.t_prop order by count(*) desc limit 2 order by limit 1 which advisor has number of students? runner-up select advisor from (select advisor , count(*) as t_prop from student group by advisor ) as t_tab t_tab.t_prop order by count(*) desc limit 2 order by limit 1 show the account id with number of transactions. runner-up select account_id from (select account_id , count(*) as t_prop from financial_transactions group by account_id ) as t_tab t_tab.t_prop order by count(*) desc limit 2 order by limit 1 natural language description few-shot examples dialogue in this work, we systematically analyse the ability of llms to in-context learn novel interpretations. we summarize our contributions below. ## magnifico we choose the text-to-sql task to test llms' ability to handle novel interpretations because of its relevance to real-world applications. moreover, contemporary llms already achieve good zero/fewshot in-context performance on the task. we create magnifico by modifying and re-tasking examples from an existing text-to-sql benchmark, spider @xcite . below, we describe the procedure in detail. ## experimental setup in this section, we will discuss the setup for our experiments on magnifico. foot_0 ## final remarks we studied the ability of llms to interpret new words and phrases in-context using their description or a few demonstrations. we also extended this study to a realistic scenario: understanding userdefined interpretations from long form dialogue. our results indicate that current llms can, to an extent, acquire novel interpretations from diverse forms of context. however, interpreting unfamiliar words or multiple novel words simultaneously still poses a significant challenge for existing llms. these tasks can serve as a measure to evaluate the compositional generalization abilities of future llms in practical settings. it is interesting to note that instruction finetuning leads to significant improvements in learning from descriptions across three different llm families. considering that instruction fine-tuning doesn't involve acquiring novel semantics, it could be useful to understand why it has this impact. in the past few years, several works @xcite showed that sequence models were limited in their ability to generalize to novel words on semantic parsing tasks based on a few examples in the training set. many specialised methods and approaches @xcite were designed to address this problem. it is therefore fascinating to see that contemporary general llms are able to generalize to novel words from not only processing a few examples in-context, but also from natural language descriptions and conversations. while a large part of the compositional generalization challenge still remains unsolved, we feel it is important to highlight this paradigm shift. we hope our work paves the way for further studies of practical setups that require llms to generalize compositionally.