Metric-Based In-context Learning: A Case Study in Text Simplification (2023)

Abstract: In-context learning (ICL) for large language models has proven to be a powerful approach for many natural language processing tasks. However, determining the best method to select examples for ICL is nontrivial as the results can vary greatly depending on the quality, quantity, and order of examples used. In this paper, we conduct a case study on text simplification (TS) to investigate how to select the best and most robust examples for ICL. We propose Metric-Based in-context Learning (MBL) method that utilizes commonly used TS metrics such as SARI, compression ratio, and BERT-Precision for selection. Through an extensive set of experiments with various-sized GPT models on standard TS benchmarks such as TurkCorpus and ASSET, we show that examples selected by the top SARI scores perform the best on larger models such as GPT-175B, while the compression ratio generally performs better on smaller models such as GPT-13B and GPT-6.7B. Furthermore, we demonstrate that MBL is generally robust to example orderings and out-of-domain test sets, and outperforms strong baselines and state-of-the-art finetuned language models. Finally, we show that the behavior of large GPT models can be implicitly controlled by the chosen metric. Our research provides a new framework for selecting examples in ICL, and demonstrates its effectiveness in text simplification tasks, breaking new ground for more accurate and efficient NLG systems.

## introduction text simplification (ts) is a crucial task in natural language processing, with the goal of converting complex text into simpler, easier-to-understand one. this is particularly important for individuals who struggle with comprehending complex languages, such as second language learners or individuals with cognitive impairments @xcite and disabilities like dyslexia @xcite and autism @xcite . for the aforementioned reasons, nlp community has shown great interest in the topic, introducing plenty of datasets (e.g., asset (alva-manchego et al., 2020)), models, and evaluation metrics (e.g., sari @xcite ). there have been numerous approaches to ts proposed in the literature, including non-neural or rule-based methods @xcite , machine translation approaches @xcite , and finetuning of large language models (sheang and saggion, 2021) on downstream task data. recently, it has been shown that large language models such as gpt-3, are capable of in-context learning (icl) @xcite )-an emerging ability to learn from in-context samples without modifying model parameters. 1 despite its strong ability, icl still mostly falls behind the performance of finetuning techniques @xcite . recent studies have shown that in-context learning is highly variable to a range of factors, such as the number of examples, quality of examples, and even the order of examples @xcite @xcite . to address these concerns, recent literature has proposed several techniques for selecting the most relevant examples for icl @xcite @xcite . the majority of them aim to retrieve a set of samples from the validation set that resembles the test set most by either training a separate retrieval model or utilizing an existing encoder to calculate similarities between pairs of sentences. however, adopting these techniques for text-generation tasks with multiple references is nontrivial, and the need to access to the full test set to pick examples from is not desirable, and may not always be possible in real-life scenarios. in order to address this problem, we propose a simple yet intuitive metric-based selection tech-nique, which we refer to as metric-based incontext learning (mbl), to perform efficient and robust in-context learning with large language models for text generation tasks. unlike previous icl techniques, mbl only requires access to the development set and uses more informed measures rather than requiring generating sentence embeddings or training separate specified retrieval models. furthermore, we perform an extensive set of experiments with gpt-3 models of various sizes (175b, 13b, and 6.7b) foot_1 , specifically focusing on their performance for ts. we investigate utilizing commonly used ts metrics (e.g., sari, compression ratio) for example selection and answer several research questions regarding their strengths and weaknesses on a variety of datasets and models. through our experiments, we show that metricbased selection can significantly improve the performance of large language models on ts. we also demonstrate that these results are generally robust to various orderings and perform well in out-ofdomain settings. this paper provides the following contributions: we release all generation outputs, baseline models and evaluation scripts publicly with https://github.com/nlp-ku/ metric-based-in-context-learning/ . ## metric-based in-context learning following the line of work for retrieving the best samples from development set @xcite , we introduce a simple and intuitive technique based on employing standard evaluation metrics for selecting the examples. task setup given the list of sentences @xmath0, r 1 , r 2 , ..., r n ], where c is the complex and r i is the simple reference sentence; our goal is to find the best k pairs, [c, r i ], such that the final text simplification performance on the test set is maximized. to do so, we go through each l in the development set and measure the distance between each c and r i according to a metric, m. finally, we pick the top k pairs and fill the prompt template with the samples: "complex sentence: {c}, simple sentence:{r i }". we initially considered a long list of task-specific as well as general generation metrics that contain the standard evaluation metrics for ts, namely as sari, bleu, fkgl; as well as a simple analysis metric: compression ratio, and a more recent textual similarity metric bertscore as suggested by @xcite . following the criticisms by @xcite and @xcite , we removed bleu and fkgl from the list of candidate metrics. compression ratio (cr) it is simply calculated by dividing the number of characters in c by the number of characters r i . we consider the pairs with higher compression ratios as more preferable candidates for ts. bertscore precision (bp) @xcite bertscore computes the cosine similarity between each token in the candidate, y, and reference, x, sentences. precision is calculated as: we discard pairs with a score of 1 since they would simply be duplicates. sari @xcite it is the defacto standard evaluation metric for ts. in general terms, it compares prediction against both the input and the reference sentences. it calculates a weighted average of f1 scores for three operations: addition, deletion, and keeping. precision and recalls for each operation are calculated based on n-gram overlaps between the prediction, input and reference sentences. to calculate the sari score for each c-r i pair, we denote r i as the prediction, c as the input, and [r 1 , ..., r i-1 , r i+1 , ..., r n ] as the reference sentences. hence, this measure can only be applied when there are multiple references. ## experimental setup to investigate the effects of metric-based selection techniques on ts, we perform a comprehensive set of experiments using various llms, sample sizes, and datasets; and compare against strong baseline and state-of-the-art models. following the criticism @xcite on using bleu @xcite , we use sari @xcite as our main evaluation metric. however, we also report bleu for two reasons: i) to be consistent with previous works (see §4.4) and ii) to gain more insights on how the chosen metric for mbl effects the results measured with different metrics. ## experiments and results we conduct a comprehensive set of experiments with the setup explained in §4. following the work by @xcite , we experiment with the k values as 1, 2, 4, 6, 8, 10, 15, 20 examples. we repeat the random baseline experiments three times for each k. we aim to answer the following research questions (rq): rq1: how do different metric-based selection techniques compare? ( §5.1) rq2: is metric-based sample selection robust to the order of the prompts? ( §5.2) rq3: how does metric-based icl compare to state-of-the-art text simplification methods? ( §5.3) rq4: does metric-based selection performance on one dataset transfer to other out-of-domain datasets? ( §5.4) ## qualitative analysis in this section, we perform a qualitative analysis of different model generated simplifications and metric top 2 examples compression ratio complex sentence they manifest with either neurological complications or with skin problems (or occasionally both). simple sentence they show either brain or skin problems (or both). complex sentence the psychological state of sympathy is closely linked with that of compassion, empathy and empathic concern. simple sentence sympathy is closely linked with compassion and empathy. ## conclusion and future work in conclusion, we propose a novel and robust method for selecting examples in the ts domain, evaluating its effectiveness on multiple well-known ts datasets and even on downstream tasks like cognitive simplification. our experiments demonstrate state-of-the-art results in the field of ts and cs, reaching scores of 44.86 on festability, 47.94 on asset and 43.46 on turkcorpus. we hope that future work will generalize our findings in other text generation tasks and other domains.