Automating the Generation of a Functional Semantic Types Ontology with Foundational Models (2024)

Abstract: The rise of data science, the inherent dirtiness of data, and the proliferation of vast data providers have increased the value proposition of Semantic Types. Semantic Types are a way of encoding contextual information onto a data schema that informs the user about the definitional meaning of data, its broader context, and relationships to other types. We increasingly see a world where providing structure to this information, attached directly to data, will enable both people and systems to better understand the content of a dataset and the ability to efficiently automate data tasks such as validation, mapping/joins, and eventually machine learning. While ontological systems exist, they have not had widespread adoption due to challenges in mapping to operational datasets and lack of specificity of entity-types. Additionally, the validation checks associated with data are stored in code bases separate from the datasets that are distributed. In this paper, we address both challenges holistically by proposing a system that efficiently maps and encodes functional meaning on Semantic Types.

## introduction onboarding datasets at scale is human-intensive because recognizing semantics about how data was generated, its bounds or other idiosyncrasies is critical to how it is processed and eventually used. normalization requires understanding the form/representation as it is ingested, as well as the target form used by other datasets or people in an organization. much of the effort involved in semantic understanding can be automated when values are typed with something richer than the basic primitives (e.g. integer, float, or strings types). in fields like finance, medicine, or broadly-spanning ai systems, new data is constantly being added, and automation can defray some of the ingestion cost. the underlying pain-point in data onboarding stems from humans inconsistently naming tables and columns without knowing who may use them. even when using data from a high-quality source @xcite , the consumer often invests significant human capital to develop to normalize and map data to a canonical representation as there are many challenges associated with normalizing columnar tables at scale. in addition to handling null/not-a-number values, inconsistent data formats, or outof-distribution values, it is critical to understand the units of the values, which maybe implicit (e.g. dollars when giving the price of housing in the us), written in prose (e.g. revenue may be in millions of dollars as noted in the text of a 10k filing), or easily inferred (e.g. a summer temperature of 100 degrees is fahrenheit and not kelvin or celsius). additionally, to compare, join, or group values from columns in different tables, it may be necessary to recast values (e.g. convert a 12-hour am/pm time value to a 24-hour clock value or a country code to a country name). identifying these semantics is behind the reliance on humans during onboarding. we introduce functional semantic types (fst) to automatically annotate columnar data with semantic types and provide a library of functional attributes to normalize, validate, and cast real data values. fsts are placed in a synthetically generated ontology, which directly maps columnar datasets to their hierarchically organized fsts (this process is fsto-gen). the generation of fsts relies upon large language models (llms) @xcite to assign and generate a python class definition that contains semantic details and functional characteristics about the data. by their training breadth, llms offer a general solution for entity recognition @xcite , because they can leverage the distributional properties/real values of data, tabular metadata, and data dictionaries to construct more accurate type annotations. furthermore, llms have shown the ability to generate code, and therefore are capable of transforming semantic context into functional attributes. our work takes advantage of the natural hierarchical organization of tabular data. we refer to a collection of tables as a product. the information within a product is assumed to be initially constructed, maintained, and labeled by the same community of interest. as such, columns and tables, as well as context, are correlated. our system is likely to generate the same fst for multiple columns be-longing to the tables within a product. the automatic generation of semantic types saves human labor, and even more so when multiple columns are assigned the same type. the system verifies this by merging a subset of values from these columns and checking that they all pass the same validation test. at the final stage, we identify common fsts across different products. first semantic types with the same name are agglomerated. then all the uniquely named fsts are turned into a graph by finding semantically similar fsts in representation space and generating cross-type-cast functions to transfer data values between fsts. human verification shows that this first attempt identifies communities of semantically-identical entities. there have been many other efforts to automatically assign semantic types to columnar data, but with some major differences (see section 2). they assume the existence of an ontology or knowledge graph, do not build a bespoke ontology that is used for cross-type casting, and most of all they do not generate the functions that define the semantic type. hence, the contributions of this work are: ## related works the association of columnar tables with entities has been previously treated as a multi-class prediction problem over some user-defined distribution of entity types/properties. methods such as sherlock @xcite , sato @xcite , doduo @xcite , and tablegpt @xcite use 78 semantic types described by the t2dv2 gold standard @xcite which matches properties from the dbpedia ontology with column headers from the webtables corpus. autotype @xcite made predictions over 112 manually procured types that spanned different industries. however, the types used in these works are often too broad for industry-specific datasets (e.g. in finance, ebita is a commonplace term, but missing from dbpedia @xcite and wordnet @xcite knowledge graphs). we summarize the related works along several dimensions (although tablegpt @xcite and foundational models @xcite cover nearly all of these). table question and answer: table cell identification @xcite , semantic parsing @xcite , tabfact @xcite . row-to-row transformation: tde @xcite . entity matching between rows: ditto @xcite , deep entity matching @xcite , auto-em @xcite . schema matching between tables: valentine @xcite , smat @xcite . data imputation: datawig @xcite , eracer @xcite , imp @xcite , holoclean @xcite . a trend that spans most of these works is the success of llms as natural language processing engines for directly operating on real data values. archetype @xcite showed that llms are performant zero-shot annotators of semantic types across various domains. additionally, llms have shown success in code generation tasks @xcite , specifically, for data processing and ingestion coding @xcite . our work builds on these efforts but is unique in that it directly attaches any functional normalization during the entity recognition process (encapsulated in a fst), and hierarchically groups fsts to build an ontology that identifies semantically identical entities across products. additionally, unlike previous work in semantic type annotation, we do not specify the set of types beforehand; the aggregation of generated types across a universe of data becomes the type set (app. a.7). ## problem formulation our fst system is applied to a universe of data, hierarchically composed of tables and products. a product consists of at least one columnar table. columns, tables, and products all have labels. the tables within a product are assumed to have some informational similarity. in addition, our set of fsts can either be a subclass of datasetsemantictype (fig. 1 ), or of genericsemantictype (fig. 2 ). datasetsemantictype fsts correspond to the standard types of data: numeric (with/without units), boolean, strings, and categorical. examining the values in a column, as well as its relation to other columns, is all that is needed to make this type of assignment. in addition to a human-readable name, its definition includes descriptive characteristics about the type's semantics, domain, and example values, as well as a functional transformation from raw data to normalized, types. genericsemantictypes collate identicallynamed columnsemantictypes across products by applying a standardized normalization procedure that undergoes self-validation (see fig. 2 ). the fsts generated at the table and product levels are called t-fsts and p-fsts (each a subclass of datasetsemantictype ), while those generated at the universe level are called g-fsts (subclass of genericsemantictype ). goal: given a table, extract the underlying semantic entities (if any) per column and generate a datasetsemantictype with a descriptive class name, instance fields, and a cast() that transforms a single columnar value to the format dictated by the class. commonly named fsts across products are used to generate a subclass of a genericsemantictype that merges the semantics of all the input datasetsemantictype definitions, consolidates their transform logic into a single cast(), and evaluates the correctness with validate(). some g-fsts may be convertible, so cross_type_cast()'s transform their values. unlike traditional ontologies, the result is a synthetic ontology where the hierarchy between t-fst, p-fst, and g-fst represent increasing layers of transformational generality. edges between g-fsts represent a knowledge graph, but whose edges are not just semantic relationships, but functional transformations. ## fst ontology generation (fsto-gen) step 1: @xmath0, an llm is provided with a serialized format of the table (app. a.1.1) and identifies the subset of columns that correspond to semantic entities that might be more broadly applicable to other datasets. for each identified column, the llm generates the corresponding t-fst subclasses and a mapping from column name to subclass. abstract syntax trees are used to parse the output string and store any class definition and this mapping dictionary. in our experiments, the llm tends to create identical subclass names (but not always identical fields) for columns in the same product (app. a.1.2). ## experimental evaluation we evaluated the p-fsts, g-fsts, and cross_type_cast()'s on three data universes, two of which are freely available to the public (as well as all our code and prompts) with results shown here and code definitions in the appendix. . universe curation -our universes source from two open-source data providers, kaggle and harvard dataverse, as well as a commercial financial dataset, we refer to by a fictional name fdata. for kaggle, we selected the 707 most commonly used kaggle-datasets and extracted their associated tabular data files (files and datasets are represented as tables and products in our nomenclature). for harvard, we selected the top 484 most downloaded datasets and organized them similarly to kaggle, but harvard required additional preparation due to the large number of fully null columns (app. a.6). fdata consists of 428 tables containing a wide breadth of financial terminology. ## applications data normalization -data normalization is baked into the construction of our ontology. the cast() in t-fsts, super_cast() in g-fsts, and cross_type_cast()'s represent normalization layers for typing a columnar value. listing 1 shows the super_cast() generated for a g-fst for "income-level". across tables, income-level can be represented as a range or a specific amount, and the aggregation process results in a single allinclusive transformation covering both cases. data validation -validation of a semantic entity generally requires knowledge about the distributional properties of a column or worldly knowledge about the domain set of the entity. for accelerometer data, the llm asserted bounds between [-1, 1] (app. a.8.1). additionally, there is robustness in generating fault-tolerant t-fsts, such as the "precipitation" t-fst (app. a.8.2. precipitation data is usually numeric, but there were several occurrences of the letter 't'. leveraging worldly knowledge resulted in assigning 't' to a value of 0 as it generally refers to trace amounts of rainfall. finally, when dealing with entities with large domain sets, llms use external lookup tables, (e.g. the set of us states configured in pycountry) to assert the correctness of state names to their two-letter abbreviation (listing 2). an interesting avenue for future work is utilizing validation to perform better normalization. while we provide a summary report and a subset of values, these may not be enough to construct robust validation over an entire column, so an iterative approach that couples feedback from normalization may be used to improve validation. listing 2: the validate() of g-fst, "usstate," which uses pycountry to checks the validity of super_cast(). data fusion -data fusion is automatically unlocked with the graph construction of the product and general stages of fsto-gen to join tables on the common columnar entity. in general, any two tables can be merged using a common p-fst or g-fst ancestor, and the explicitness of the ancestorchild relationships in fsto-gen reduces the com-plexity of finding a semantic match. additionally, the construction of cross_type_cast()'s allow for non-trivial joins between tables. for example, the llm generated a cross_type_cast() to convert education level from string to integer enums by mapping their domains (app. a.8.9). data discovery -discovery is achieved by allowing practitioners to semantically search over the informational/functional properties of fsts or traverse the relationships in the synthetic ontology. for example, fsto-gen generated a fst for lifeexpectancy that spanned four products, two of which came from the "who-world-healthstatistics-2020" product where a column was named "first tooltip", and one from "barcelonadata-sets" product where columns associated with life-expectancy were named by time-ranges. it is doubtful that someone searching for data related to life-expectancy would have known that these products would be related, but automated contextualization with llms combined with the hierarchical composition of fsto-gen found these relations. ## conclusion and limitations fsto-gen is an llm-powered framework for automating the generation of g-fsts and their relations from columnar data. our synthetic ontology is hierarchical and functional, such that successive layers represent the transformation of semantically identical columns into a singular representation. these ontologies are useful in automating various data onboarding tasks: normalization, validation, fusion, and discovery. fsto-gen is less successful with domain-specific datasets, because llms are less familiar with terminology, leading to incorrectly scoped fsts. we are exploring fine-tuning @xcite and retrieval-augmented-generation @xcite to overcome this gap. however, even in its current state, fsto-gen shows promise in reducing the tedious job of data onboarding.