Semi-Structured Object Sequence Encoders (2023)

Abstract: In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of key-value pairs over time and can present modeling challenges due to an ever-increasing sequence length. We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time; we then self-attend over these value-aware key representations to accomplish a downstream task. This allows us to operate on longer object sequences than existing methods. We introduce a novel shared-attention-head architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. Our experiments on multiple prediction tasks using real-world data demonstrate that our approach outperforms a unified network with hierarchical encoding, as well as other methods including a record-centric representation and a flattened representation of the sequence.

## introduction semi-structured object sequences comprise a significant portion of the myriad of data created daily. this data usually has a temporal aspect, with the data created sequentially and representing events happening in some order. more generally, the data is a sequence of structured objects, each represented by a set of key-value pairs that encode the attributes of the object (figure 1(a) ). examples include recordings of user interactions with websites, logs of machine activity, shopping decisions made by consumers, and many more (figure 1(b) ). the data is usually stored in semi-structured formats such as jsons, or tabular forms. in this paper, we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. if we think of the parallel to natural language data, we could treat each sentence of a text (figure 1(c )) akin to the set of key-value pairs at a particular time step. the challenge of sequence length: a trivial method of representing such sequences would be to flatten each structured object and view its constituents as individual words for tokenization in natural language (figure 1 (d) ). 2 however, this causes the sequence length to become extremely large (thousands of tokens) when operating on realword semi-structured sequences. for instance, in our study of semi-structured objects from userinteraction sessions on software from a large cloudbased service provider, we found these objects could contain 60 fields on average. the values of these fields include timestamps, identifiers, log messages, etc., with an average of 5 words each. a session length of 15 minutes results in 105 such session objects, amounting to nearly 31, 500 words which would further increase the sequence length after sub-words are created. record-centric representation: in contrast, to flattening structured objects, one could also create record-centric representations (figure 1(f) ) where the set of key-value representations at each time step are considered in sequence. as our experiments in section 3.3 demonstrate, such represen- ## modeling we now describe our approach for modeling keyvalue semi-structured object sequences. let a sequence of semi-structured objects be denoted as @xmath0, j 2 , j 3 , . . . , j n ] corresponding to n time steps. further, let the goal of our modeling is to learn a representation of a sequence of structured objects j ; and subsequently, learn f : embd(j ) → {1, 2, . . . , c} for an end task, such as a c-way classification task. we develop a modular two-part modeling strategy to represent a sequence of structured objects. temporal value modeling: let k be a key from the universe of all the keys k in the sequence. then, for each key k we encode the value-aware keyrepresentation v k , by considering the value of the key k at each timestamp, as a sequence. formally, v k is given by: we use the output embedding representation at the first position (corresponding to @xcite ) as the keyrepresentation. it is easy to see that this formulation allows us to accommodate the modeling of natural language text as in figure 1 (c ). for illustration, if the t extencoder is based on bert @xcite , eq. 2 reduces to the encoding scheme typically employed in bert for text paragraphs, where the [val_sep] corresponds to the [sep] token. ## related work our work is related to several areas of research. modeling tabular and timeseries data: as we explicitly model the value sequence for each key, the data object we work with is reminiscent of tabular data where each row is a time-step and each column comprises the values for a particular field. on the surface, the modeling of data may appear related, but the actual tasks and models developed for tasks on tabular data cannot be applied to semi-structured object sequences. this is because the work on modeling textual tabular data often involves developing specialized models focused on retrieving information from cells @xcite @xcite , multi-hop reasoning across information in different cells across parts of the table @xcite , combining information present in tables and unstructured text for information seeking tasks @xcite @xcite @xcite , etc. in addition, work on modeling time-series tabular data has focused on numerical data @xcite @xcite with purpose-built taskspecific architectures that cannot be easily adapted to other tasks @xcite . modeling temporal graph sequences and recommender systems: our approach is also related to a rich body of work on modeling temporal graphs and recommendation systems @xcite @xcite . temporal graph evolution problems involve constructing representations to enable tasks such as link prediction @xcite , item recommendation in user sessions @xcite , answering queries on graphs and sequences @xcite , classifying graph instances in a sequence, @xcite etc. our findings suggest that such approaches do not scale for long sequences for the tasks we experimented with. however, our recordcentric model baselines @xcite are similar in approach to these methods. parameter sharing in neural network models: deep neural networks are usually trained to tackle different tasks in isolation. networks that cater to multiple related tasks (multi-task neural networks) seek to improve generalization and process data efficiently through parameter sharing and joint learning. traditional hard-parameter sharing uses the same initial layers and splits the network into taskspecific branches at an ad hoc point @xcite . on the other hand, softparameter sharing shares features via a set of taskspecific networks @xcite . more recently adaptive sharing approaches have been proposed that decide what parameters to share across tasks to achieve the best performance @xcite . the parameter sharing utilized in this work is different from the aforementioned approaches, as we share some attention head weights between the two networks (as compared to shared layers), in a way that could improve overall end-task performance. ## discussion and conclusion in this paper, we have presented a two-part encoder to model structured object sequences. the choice of a key-centric representation enables us to encode larger objects as well as long sequences. our experiments show that by using the two-part tvm-ka architecture we are able to inject downstream task information into the temporal value modeler network to generate key representations that are more relevant. we additionally present a novel interleaving scheme to train our two-part encoder. we induce task bias into the model by sharing attention heads between the temporal value modeler and key aggregator components. our proposed approach outperforms both the baseline approaches that flatten structured object sequences and those based on record-centric representations.