On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to- SQL (2024)

Abstract: Structured data, prevalent in tables, databases, and knowledge graphs, poses a significant challenge in its representation. With the advent of large language models (LLMs), there has been a shift towards linearization-based methods, which process structured data as sequential token streams, diverging from approaches that explicitly model structure, often as a graph. Crucially, there remains a gap in our understanding of how these linearization-based methods handle structured data, which is inherently non-linear.This work investigates the linear handling of structured data in encoder-decoder language models, specifically T5. Our findings reveal the model’s ability to mimic human-designed processes such as schema linking and syntax prediction, indicating a deep, meaningful learning of structure beyond simple token sequencing. We also uncover insights into the model’s internal mechanisms, including the ego-centric nature of structure node encodings and the potential for model compression due to modality fusion redundancy. Overall, this work sheds light on the inner workings of linearization-based methods and could potentially provide guidance for future research.

## introduction motivation. natural language interfaces (nlis) to computer systems allow the use of everyday language to interact with computer systems, thus lowering technical barriers to advanced computing functionality. early systems such as shrdlu @xcite and lunar @xcite saw limited success due to the limited language processing capabilities of computer systems at the time. after years of steady progress, the strong language processing capabilities of large language models (llms) have led to renewed interest in nlis such as the widely used @xcite . systems such as chatgpt already serve as robust nlis. however, a critical challenge remains in applying the underlying models to specialized and personalized real-world scenarios. the challenge stems from the need for the model to handle "backend data" commonly stored in structured formats such as proprietary databases, knowledge graphs, or dialog states, including intents, slots, and values. we refer to this task as structured data representation (sdr) @xcite . in this study, our primary focus is on a representative sdr task: text-to-sql parsing @xcite . this task automatically maps natural language queries into sql commands, thus eliminating or at least reducing the need for programming knowledge. for such a system to be broadly applicable, it must be capable of adapting to new databases, encoding these databases together with user queries, and predicting the corresponding sql queries. the rise of linearization-based methods. recent approaches to text-to-sql parsing and other sdr tasks fall into two main categories: linearization-based and structure-based methods @xcite @xcite . structure-based approaches explicitly utilize the inherent structure in the data, often representing it with a graph @xcite @xcite . in contrast, linearization-based methods treat structured data as a token sequence, processed similarly to natural language sentences. the latter have gained traction due to their compatibility with llms, which have demonstrated impressive performance across various nlp benchmarks. open problems and our contributions. sdr tasks like text-to-sql remain a challenging problem for llms, as they are not completely "solved" by current models @xcite . towards shedding light on ways forward, our main contribution is a detailed exploration of the inner workings of a former state-of-the-art (sota) text-to-sql parser with t5 backbone @xcite . our analytical approaches include probing classifiers @xcite @xcite , as well as techniques that directly manipulate model intermediates, leveraging the recent causal tracing method @xcite . we find that despite their simplicity, linearizationbased methods can effectively represent structured data. specifically, we show that the prefix-tuned t5 model preserves low-level textual details and enhances understanding of node relationships in structured data. we also show the ego-centric nature of structure node encodings, which primarily contain information relevant to the node itself. additionally, we uncover a duplicative robustness in modality fusion, indicating potential avenues for model compression. our study also reveals the model's internal working pipeline, which aligns with human-designed processes like schema linking, syntax prediction, and node selection, suggesting meaningful learning rather than reliance on spurious correlations. the attention mechanism's role in modality fusion and the distinctive functionalities of different layer ranges in the decoder are also revealed. overall, our research contributes to our understanding of structured data representation in encoder-decoder llms. we opted not to analyze extremely large lms such as gpt-4, due to the high computational cost of our analytical methods and the opaque nature of their intermediate states. however, given the competitive performance of the model we study, coupled with its sequential input and autoregressive output which are analogous to llms, we believe that our findings are general and applicable to models in the same category. ## preliminaries the model we investigate in this work is t5-large, foot_0 with prefix-tuning on the spider dataset @xcite . the model is a standard encoder-decoder transformer architecture. 2 the encoder has two modules per layer: self-attention and linear multilayer perceptron (mlp). a decoder layer has three modules: self-attention, crossattention, and mlp. to work as a text-to-sql parser, the model takes the concatenation of user figure 1 : the input to the text-to-sql parser consists of the query in natural language text (blue), and the relevant structured data (red), other tokens (gray). "self-node," refers to the input tokens corresponding to the expected output node where a node refers to both column and table names, and "structure-context," represents all the structured input tokens excluding the self-node. the output is the predicted sql query @xcite . terminology. for clarity of exposition, we introduce the term "structure nodes," or simply "nodes," which collectively refer to both columns and tables. we refer to "layer ranges" in the encoder or decoder, grouping them into to low, middle, and high layers. both the encoder and decoder have 24 layers. in our discussion, "low layers" refer to layer 0-11, "middle" to 6-17 and "high" to 12-23. in addition, we mention "input sections," such as text, structure, and prefix. for a structure node, we also have the input section of "self-node," indicating the input tokens of the anticipated output node, and "structure-context," representing all the structured input tokens excluding the self-node. figure 1 illustrates the format of the input to the text-to-sql parser. ## research questions preliminary intuition open questions. the model's internal mechanics can be intuitively understood as follows: the encoder produces contextualized encodings by fusing both textual and structural input elements. these combined encodings then provide a comprehensive basis upon which the decoder constructs the sql. however, despite this preliminary understanding, certain aspects remain unclear. key questions that arise include: ## direct model manipulation besides the insights from the probing study, a remaining question is whether the model actually utilizes the information encoded in the representation. since probing experiments did not answer this question, we undertook an exploration of the internal mechanisms of the model by directly manipulating the model intermediates and observing the outcomes. we employ the idea of causal tracing @xcite , where specific intermediate information is corrupted or reinstated to analyze its influence on the ultimate prediction. we design our studies in a fine-grained way, computing prediction accuracy at the tokenlevel within a sql query and categorizing the outcomes based on token types. token types include columns, tables, table aliases, and syntax tokens (including keywords and operators). we focus on columns and syntax tokens, treating columns as being representative of structure node prediction. the results for rest of the token types are in the appendix. ## conclusion we conducted a comprehensive study on the internal behavior of an encoder-decoder language model, specifically t5, text-to-sql parser. through both probing and manipulation of internal states, we provide insight into various aspects such as the information transfer between encoder and decoder, the storage of crucial data within the model, the functions of attention mechanisms, the process of modality fusion, the internal processing pipeline of the model, and the intrinsic alignment of sql and natural language semantics. our findings can inform and guide future research in text-to-sql and related structured data representation tasks.