Truth-Conditional Captions for Time Series Data (2021)

Abstract: In this paper, we explore the task of automatically generating natural language descriptions of salient patterns in a time series, such as stock prices of a company over a week. A model for this task should be able to extract high-level patterns such as presence of a peak or a dip. While typical contemporary neural models with attention mechanisms can generate fluent output descriptions for this task, they often generate factually incorrect descriptions. We propose a computational model with a truth-conditional architecture which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on *only* the chosen valid program (rather than the input time series) to generate the output text description. A program in our model is constructed from modules, which are small neural networks that are designed to capture numerical patterns and temporal information. The modules are shared across multiple programs, enabling compositionality as well as efficient learning of module parameters. The modules, as well as the composition of the modules, are unobserved in data, and we learn them in an end-to-end fashion with the only training signal coming from the accompanying natural language text descriptions. We find that the proposed model is able to generate high-precision captions even though we consider a small and simple space of module types.

## introduction there has been large interest in generating automatic text description @xcite of tabular data -for example, prior work has sought to generate biographies from tables of biographical information @xcite , and generating descriptions from structured meaning representations @xcite . however, in many of these tasks the main focus is on designing systems that are able to select entries from tabular or equivalent data during generation by using neural attention mechanisms. in many naturally occurring descriptions of tabular data, humans often refer to higherlevel patterns, for example in the description of stock index pricing over the week in fig. 1 , the speaker refers to how the stock price peaks towards the ending. some recent work has looked into setups which require non-trivial inference @xcite . however, they typically don't involve inference about numerical patterns in time series data. moreover, much recent prior work on identifying more complex patterns in data for captioning has relied on deep neural networks, often employing neural encoders and attention mechanisms. however, such approaches often fail to generate faithful responses and lack interpretability @xcite @xcite . we present a novel neural truth-conditional model for time series captioning, which learns to identify patterns which hold true for the input time series (figure 2 ). we first sample a latent program from the space of learned neural operators. each program produces a soft truth-value. then, with probability proportional to each program's truth-value, a language decoder generates a caption. thus, programs that yield low truth values, do not produce captions. critically, the decoder takes an encoding of the program itself, rather than the time series, in order to determine output text. overall, this approach allows for both: (a) precision in generated output through explicit truth conditioning, and explicit program structure as a representation of time series trends, and (b) diversity in caption generation through the sampling process. while some of the patterns in data are complex, they can be considered to have been constructed by composing simpler concepts such as slope (rate of change of value) or comparisons (between values at give points). as such, our programs are constructed by composing simpler operations/modules. such a modular design enables sharing of modules across multiple programs, leading to more data efficient learning of module parameters, and also providing better generalization to unseen compositions of modules. we consider a relatively simple space of three module types, using which our model is able to capture a significant fraction of the patterns present in data. the module types could be expanded in future to capture more complex patterns. our model treats the choice of composed computation graph of programs as a latent variable, learned using natural language descriptions as the only supervision. in this respect, our approach is related to neural module networks used in @xcite , which condition on a question to generate a program, which then operates on an image or other data to predict an answer. in our case, the constructed computation graph operates and identifies salient patterns in the source data directly, without being guided by an input question. our main contributions are as follows: we propose a novel method for time series captioning which first induces useful patterns via composing simpler modules, identifies the programs which hold true, and finally generates text describing the selected program. towards this end, we collect and release two datasets consisting of time series data with accompanying english language description of salient patterns. we observe that the proposed method is able to learn useful patterns, exhibits compositionality and interpretability, and generates outputs that are much more faithful to the input compared to strong traditional neural baselines. foot_0 ## truth-conditional natural language description our goal is to learn models for describing salient patterns in time series data. the main research challenge involved is to learn the types of patterns that humans find salient in time series data, using natural language descriptions as the only source of supervision during training. based on the novel dataset we collect (described in section 4 , we find that the patterns humans identify tend to describe increasing or decreasing trends, volatility, comparisons of start and end values, presence of peaks and dips. they also mention temporal location of patterns, such as 'at the beginning' of the time series. thus, our model should be able to learn patterns such as 'increase' or 'ends with higher value compared to start', and temporal aspects such as 'begin' or 'end'. one way to operationalize this process is through the lens of formal logic: e.g. an increasing trend at the beginning of a time series x can be represented trough the logic z: ∃ i s.t. increase(x i ) and begin(i) thereafter, if the program returns true on the input, one can condition on only the logical program z to generate output text that describes this pattern via a decoder, p(y|z). however, this still requires learning or defining modules for patterns and temporal location. inspired by neural module networks @xcite , we propose to use functions parameterized by neural networks (figure 2 ) as modules, incorporating inductive bias through architecture design. however, unlike past work, we condition only on an encoding of sampled programs that return true to generate output text. ## learning and inference the log probability of observing a natural language description y of the time series x under the model can be written as follows: inference network: the space of programs we currently employ is relatively small (about 20-60 number of programs), which makes it feasible to marginalize over the program space. however, any future work expanding the space of programs might run into feasibility issues when computing the exact likelihood. in such cases, we can perhaps resort to variational learning to optimize a lower bound to the likelihood by drawing samples from an inference network. additionally, use of inference networks can provide a useful inductive bias by using the observed text descriptions to guide the model learning. for example, words 'increase' and 'begin' in a caption could inform the inference network about a high chance of the presence of an increase pattern in the initial duration of the time series. we observe that training with inference networks results in models which can better capture the patterns in data. note that the inference network is used only for model training. at test time, we sample from the learned prior and decoder without regard to the inference network. we use amortized variational learning by introducing an inference network q γ , and train the model to maximize the following evidence lowerbound (elbo): we use a bilstm encoder to encode the caption y, followed by a classification layer to predict the approximate posterior q γ (z|y) over the programs. we also considered fine-tuning of a pre-trained bert model instead of bilstm, but did not observe any improvement in the model performance during the initial experiments. optimization: θ, φ and γ are learned through directly optimizing the elbo term. we compute the exact reconstruction and the kl-terms -the number of programs in our case is small enough to enable this exact computation (typically we consider 6-10 instances each of pattern and locate module). ## datasets we are interested in modeling numerical patterns and trends in time series data. however, there is a lack of existing data sources with time series data paired with natural language descriptions. some prior work on weather forecasting data (such as sumtime-mausam @xcite ) are typically small (only 1045 data instances), and are limited in the scope of patterns they encompass. totto dataset @xcite ) contains a small fraction of descriptions based on numerical reasoning and patterns -however, the main challenge is to find the correct value(s) by identifying the relevant row and column in a table. logic-nlg @xcite consists of 37k tables and corresponding natural language descriptions, some of which require comparisons of cells in a table. in contrast, we focus on trends and patterns in time series data. thus, we construct a new dataset where natural language descriptions are collected for naturally occurring stock price time series data (section 4.1). additionally, we collect natural language descriptions for a synthetically constructed set of time series to evaluate and analyse our models in a more controlled setup (section 4.2). ## conclusion we present a truth-conditional neural model for time series captioning. our model composes learned operations/modules to identify patterns which hold true for a given input. outputs from the proposed model demonstrate higher precision and diversity compared to various baselines. further, the proposed model (and some of the baselines) successfully generalize, to some extent, to multiple input sizes. we release two new datasets (in english) for the task of time series captioning. future work might expand to a broader set of module types to cover more numerical patterns.