A na M eta: A Table Understanding Dataset of Field Metadata Knowledge Shared by Multi-dimensional Data Analysis Tasks (2023)

Abstract: Tabular data analysis is performed everyday across various domains. It requires an accurate understanding of field semantics to correctly operate on table fields and find common patterns in daily analysis. In this paper, we introduce the AnaMeta dataset, a collection of 467k tables with derived supervision labels for four types of commonly used field metadata: measure/dimension dichotomy, common field roles, semantic field type, and default aggregation function. We evaluate a wide range of models for inferring metadata as the benchmark. We also propose a multi-encoder framework, called KDF, which improves the metadata understanding capability of tabular models by incorporating distribution and knowledge information. Furthermore, we propose four interfaces for incorporating field metadata into downstream analysis tasks.

## introduction tabular data analysis is performed every day in popular tools such as excel, google sheets, and tableau @xcite for a wide range of domains including education, research, engineering, finance, hr, etc. to help non-expert users, various machine learning tasks are proposed to automate and accelerate the analysis process. for example, tableqa & text2sql @xcite , analysis & visualization recommendations @xcite , insights mining @xcite , etc. these analysis tasks require an accurate understanding of field semantics to correctly operate on table fields (or columns) and to further find common patterns in daily analysis. such analysis knowledge of field semantics is often shared across multiple tasks. in real-world applications, we also call it field metadata in contrast to the raw tabular input which does not directly provide this information. for example, measure / dimension dichotomy is one such metadata used in tableau @xcite and excel @xcite across diverse features. its definition is inspired by dimensional modeling in databases @xcite . it involves categorizing each field in a table as either measure or dimension. a measure (field) contains numerical measurement results on which calculations can be made, such as sum, count, average, minimum, and maximum. e.g., "price" and "discount" fields of beyond the simple dichotomy of measure / dimension, what are other types of useful field metadata? for each type of metadata, where do we find labels to evaluate or train models? to address these questions, in this paper we define the following four types of commonly used field metadata, and collect the anameta (analysis metadata) dataset incorporating tables from 3 diverse sources with derived supervision labels of the metadata: the collection of the anameta dataset is a complex process that requires a significant amount of data and supervision. due to the limitations of obtaining this information directly from raw tables, we have employed a multi-source approach to collecting anameta, which includes spreadsheet datasets (45,361 tables), public web table datasets (404,152 tables), and public synthetic datasets (17,995 tables). supervision is smartly collected from downstream tasks, manual labels through crowdsourcing, and existing information attached to tables( §3.2). after preprocessing, anameta contains 152,092 fields with measure/dimension, 149,197 with common field roles, 1,730,494 with semantic field type and 38,030 with aggregation. additionally, we have conducted quality inspection and further checks to ensure accuracy and reliability. based on the anameta dataset, in the rest of the paper, we first evaluate a wide range of models on inferring metadata in order to check how well they learned the knowledge of field semantics. in §6.1, pre-trained tabular (turl, tapas, tabbie) and large language (gpt-3/3.5 family) models are compared through additional classification heads and zero-/ few-shot prompts. semantic information captured by the pre-trained tabular models brings great gain to metadata tasks. to improve the metadata understanding capability of tabular models, in §4 we further propose a multi-encoder kdf (knowledge and distribution fusion) framework for transformer-based pretrained tabular models. knowledge fusion incorporates knowledge graph information (such as lined entities, column type, and properties). distribution fusion adds distribution information by calculating field data statistics. kdf outperforms the best baseline by 3.13% (see §6.2). this indicates that successful knowledge and distribution fusion brings a better representation of fields. finally, we demonstrate one general approach to leveraging metadata to enhance downstream analysis tasks -taking field metadata as an intermediate step and injecting the knowledge through different interfaces for downstream models. in order to incorporate metadata into downstream analysis tasks at various stages of the process, four interfaces are proposed in §5. these interfaces provide different forms of metadata, including field metadata classification results, column embeddings, metadata strings of the field, and field metadata tasks as pretraining objectives. in §6.3, we apply these interfaces on downstream tasks (tableqa and visualization recommendation) and observe that metadata knowledge helps downstream tasks when injected through appropriate interfaces. in summary, our main contributions are: • we collect a large anameta dataset with the definition and taxonomy of field metadata and smart supervision. the dataset and code will be open-sourced in https://github.com/ microsoft/anameta . ## metadata definition in this section, we delve deeper into the concept of metadata and its definitions. to facilitate understanding, we provide an example in figure 1 . when creating charts and pivot tables, individuals typically begin by analyzing common fields and assigning different roles to them, e.g., using the primary key -"product name" field as the x-axis in a bar chart. after selecting the fields, further analysis is required for each field. in the case of pivot tables, it is necessary to determine that the "original prices" field should be summed. we formulate the metadata tasks as machine learning classification tasks with the table as input. the specifics of this formulation can be found in §b.6. ## anameta dataset after defining the field metadata, the next challenge is to locate the supervision labels needed for training and evaluation. in this section, we will bring several sources together to prepare the labels. ## kdf framework as discussed in §1, to improve the metadata understanding capability of tabular models, we propose kdf (knowledge and distribution fusion) framework to infuse information. in this section, we provide a detailed description of metadata model architecture as depicted in figure 5 . ## downstream interfaces field metadata is formulated as classification problems, which can be effectively learned through the kdf framework. however, it remains a challenge to effectively incorporate this metadata into downstream applications. to address this challenge, we propose four interfaces including metadata ids, embeddings, sentences, and pre-training. these interfaces allow for the integration of metadata at different stages of the analysis process and facilitate its use in downstream applications. metadata ids interface provides downstream applications with the classification results of field metadata tasks. this interface can be used to provide more specific and targeted uses of field metadata. for example, ids can be used as rules to limit the scope of downstream searches. during quick-insights @xcite , different data mining strategies are applied based on whether a field is classified as a measure or dimension. additionally, ids can be used as tags in language models. metadata embeddings interface provides downstream applications with each column embeddings, which are augmented with metadata knowledge. these embeddings can represent columns in a continuous, dense, and low-dimensional vector space. thus they can be used in a variety of data analysis tasks to classify columns, generate analysis according to the column, and so on. additionally, they can be used as features in downstream machine learning models. ## experiments we conducted three parts of experiments to evaluate field metadata tasks on the anameta dataset and demonstrate how to incorporate analysis knowledge. first, a variety of models are evaluated on all metadata tasks as a benchmark. second, the kdf framework is evaluated on all metadata tasks finally, we evaluate the performance of downstream applications using four interfaces. all the experiments are run on linux machines with 448 gb memory, and 4 nvidia tesla v100 16g-memory gpus. ## conclusion in conclusion, this paper has presented the anameta dataset, a collection of tables with derived supervision labels for four types of commonly used field metadata. we have also proposed a multiencoder framework, called kdf, which incorporates distribution and knowledge information. additionally, we have proposed four interfaces for incorporating field metadata into downstream analysis tasks. through evaluations of a wide range of models, we have shown the importance of accurate metadata understanding for tabular data analysis and the effectiveness of the kdf framework and interfaces in improving the performance of downstream tasks.