Demonstration Selection Strategies for Numerical Time Series Data-to-Text (2024)

Abstract: Demonstration selection, the process of selecting examples used in prompts, plays a critical role in in-context learning. This paper explores demonstration selection methods for data-to-text tasks that involve numerical time series data as inputs.Previously developed demonstration selection methods primarily focus on textual inputs, often relying on embedding similarities of textual tokens to select similar instances from an example bank. However, this approach may not be suitable for numerical time series data.To address this issue, we propose two novel selection methods: (1) sequence similarity-based selection using various similarity measures, and (2) task-specific knowledge-based selection.From our experiments on two benchmark datasets, we found that our proposed models significantly outperform baseline selections and often surpass fine-tuned models. We also found that scale-invariant similarity measures such as Pearson’s correlation work better than scale-variant measures such as Euclidean distance.Manual evaluation by human judges also confirms that our proposed methods outperform conventional methods.

## introduction this paper explores demonstration selection approaches in data-to-text tasks. in particular, we focus on tasks involving numerical time series input. two examples are depicted in figure 1 . the first example is drawn from market comment generation @xcite , while the second example is drawn from line graph-to-text generation in the chart-to-text dataset @xcite . for both tasks, graphs are represented as time series, i.e., as sequences of numbers, not images. in market comment generation, we generate concise textual explanations from fixed-length numerical time series, which represent price movements in the japanese stock market nikkei. in line graph-to-text generation, we generate explanations for varyinglength numerical time series in various domains. for both tasks, such time series are often hard to understand for non-experts, so language generation techniques are used to aid readers' comprehension. in-context learning, where a prompt is fed into large language models, has recently gained prominence across various language processing tasks. a pivotal step therein that significantly influences task performance is demonstration selection, the process of selecting instances to include in the prompt @xcite . given its significance, our focus is specifically on tasks with numerical time-series data as input-tasks that have not been extensively explored in existing studies. existing methods of demonstration selections for various language processing tasks are based on either supervised scoring of examples with a classifier/regressor or an unsupervised similarity measure. specifically, we focus on the latter, i.e., methods based on unsupervised similarity measures, because we aim to devise demonstration selection methods that are applicable to various domains without the additional cost of annotating training data. a prevalent approach for unsupervised selec-tion methods involves extracting the top-k examples that are most similar to the target test instance. such methods often use similarity metrics based on embedded tokens or surface similarity @xcite . however, numerical data-to-text proves challenging due to the absence of token embeddings foot_0 . to overcome the aforementioned challenge, we propose (1) using the similarity of numerical time series and (2) leveraging auxiliary data aligned with the input, such as timestamps or graph legends (e.g., "reported violent crime rate in the united states from 1990 to 2019," as shown in figure 1(b) ). first, we treat the input series as a vector or sequence and explore various vector similarity measures and different correlation functions. second, we suggest incorporating auxiliary information with the series, such as timestamps or graph legends. for example, a comment with a timestamp of 9:00am would often mention the opening of the market as in "nikkei opens at the price of 17,000 yen". thus, such taskspecific knowledge-i.e., the fact that timestamps can influence market comment generation-is useful for extracting similar examples. our experiments use two datasets: market comment generation foot_1 and single-line graphs in chart-totext dataset @xcite . from the results, we found that (1) models using our proposed methods outperform the baselines, (2) scale-invariant similarity measures, such as pearson's correlation, perform better, and (3) combining task-specific knowledge-aware methods with sequence similarity-based methods further enhances performance, often rivaling or surpassing strong fine-tuning-based models. manual evaluation by human judges corroborates these results in terms of correctness, i.e., whether generated comments mention correct price movements or numbers in graphs, against the input series. in summary, our contributions are threefold: (1) we present the first exploration of demonstration selection for data-to-text tasks involving numerical time series, (2) we introduce two distinct selection methods, and (3) we verify their effectiveness on two datasets through both automatic evaluation using various common metrics and man-ual evaluation by human judges. ## related work this paper connects two research fields: data-totext and demonstration selection in in-context learning. numerous studies have been conducted on datato-text. traditionally, studies have focused on various input types, including tables @xcite , graphs @xcite , sets of tuples @xcite and numerical time series data @xcite . while initial studies utilized rule-based approaches @xcite , but more recently, there has been a shift towards neural network-based models @xcite . among neural network-based approaches, in-context learning using large pretrained language models has been recognized as a promising direction @xcite . regarding the combination of data-to-text and demonstration selection, @xcite explored demonstration selection for tabular data, where the input is represented as a sequence of tokens. this paper presents a novel attempt to address demonstration selection in the context of numerical time series data. the existing studies on demonstration selection were conducted on tasks other than data-totext @xcite @xcite @xcite . as an exception, @xcite explored k-nearest neighbour-based approach for a data-to-text setting, i.e., the wikipedia table-to-text on the totto dataset @xcite . in contrast to this work, we focus on numerical data-to-text. existing studies take one of three approaches: token similarity-based @xcite , surface similarity-based @xcite , or learningbased approachs @xcite @xcite . these studies all use texts, while our focus is on numerical time series input. ## tasks we describe two different numerical data-to-text tasks: market comment generation and line graphto-text generation. ## demonstration selection approaches demonstration selection involves selecting shots from an example bank based on specific criteria. we present two distinct approaches for demonstration selection: 1) time-series similarity-based and 2) task-specific knowledge-based. additionally, we explore the combination of similarity-based methods with task-specific knowledge-aware approach. ## conclusion this paper explores demonstration selection in numerical time series data-to-text tasks. we explore various similarity measures for demonstration selection and also propose the task knowledge-aware approaches. we verified that our approaches outperform the random and embedding similarity-based baselines, which have been used in many existing studies. among the similarity measures, our experiments suggest that scale-invariant measures work better. we also found that the proposed methods with demonstration selection perform better than the method with fine-tuning, which had previously shown good performance as the de facto standard. as future work, we plan to enhance our demonstration selections for multiple time series and multi-modal settings.