DC O M: A D EEP C OLUMN M APPER FOR S EMANTIC D ATA T YPE D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Detection of semantic data types is a very crucial task in data science for automated data cleaning, schema matching, data discovery, semantic data type normalization and sensitive data identiﬁcation.
Existing methods include regular expression-based or dictionary lookup-based methods that are not robust to dirty as well unseen data and are limited to a very less number of semantic data types to predict.
Existing
Machine Learning methods extract a large number of engineered features from data and build logistic regression, random forest or feedforward neural network for this purpose.
In this paper, we introduce DCoM, a collection of multi-input
NLP-based deep neural networks to detect semantic data types where instead of extracting a large number of features from the data, we feed the raw values of columns (or instances) to the model as texts.
We train DCoM on 686,765 data columns extracted from the VizNet corpus with 78 different semantic data types.
DCoM outperforms other contemporary results with a quite signiﬁcant margin on the same dataset achieving a support-weighted F1 score of 0.925.
I NTRODUCTION
Robust data preprocessing pipeline enables organizations to make data-driven business decisions.
Most unstructured and semi-structured data after initial pre-processing is available in tabular format for further processing.
These tabular datasets have to go through a process of data curation and quality check before consumption.
Companies apply standard data quality checks/rules on businesscritical columns to access and maintain the quality of data.
Additionally, some of these datasets may contain sensitive information e.g. Protected Health Information (PHI), Personally Identiﬁable
Information(PII), etc and need to be masked/de-identiﬁed i.e. identifying all columns which are
Social Security Numbers, First Names, telephone numbers, etc.
The ﬁrst step in this data curation and quality check process is column level semantic tagging/mapping.
There have been many attempts by Microsoft (2019), Ramnandan et al.
, Limaye et al.
, Puranik (2012), Hulsebos et al.
(2019), and Zhang et al.
to automate this process.
Traditionally, Semantic tagging is usually done using handcrafted rule-based systems e.g. tools from Google (2019), Microsoft (2019), etc.
In some cases, column descriptions are also available.
However, as the data grows exponentially, the cost associated with maintaining a rule-based system also increases.
The semantic data type tagging is still largely manual where data stewards manually scan through databases and map columns of interest.
Most ML-based approaches e.g. Zhang et al.
, Hulsebos et al.
and Pham et al.
(2016) use corresponding metadata, atomic datatypes, column description along with handcrafted features as input data for training the model.
We herein, propose DCoM - D eep
Co lumn M apper; a generic collection of Deep learning based
Semantic mappers to map the columns to semantic data types.
These models take the raw values of the columns (or instances) as inputs considering those as texts and build NLP-based deep learning architectures to predict the semantic type of the columns (or instances).
We also extractedengineered features and used those as auxiliary features to DCoM models.
We refrained ourselves extracting a large number of features because we wanted to make the DCoM models learn those features along with some interesting high-level features on their own, for better semantic data type detection.
As we are exploring NLP in this problem, we make the leverage of using advanced
NLP-based deep learning layers or models such as Bi-LSTM, BERT, etc.
R ELATED W ORK
Microsoft (2019) Power BI and Google (2019) Data Studio use some regular expression based patterns for dictionary looks ups of column headers and values to detect a limited number of semantic data types.
Venetis et al.
(2011) build a database of value-type mappings, then assign semantic types using a maximum likelihood estimator based on column values.
Syed et al.
(2010) use column values and headers to build a Wikitology query to map columns to semantic classes.
Ramnandan et al.
(2015) use heuristics to ﬁrst separate numerical and textual types, then describe those types using the Kolmogorov-Smirnov (K-S) test and Term Frequency-Inverse Document Frequency (TF-IDF), respectively.
Pham et al.
(2016) use slightly more features, including the Mann-Whitney test for numerical data and Jaccard similarity for textual data, to train logistic regression and random forest models.
Goel et al.
(2012) use conditional random ﬁelds to predict the semantic type of each value within a column, then combine these predictions into a prediction for the whole column.
Limaye et al.
(2010) use probabilistic graphical models to annotate values with entities, columns with types, and column pairs with relationships.
Puranik (2012) proposes a “specialist approach” combining the predictions of regular expressions, dictionaries, and machine learning models.
More recently, Yan &
He (2018) introduced a system that, given a search keyword and set of positive examples, synthesizes type detection logic from open source GitHub repositories.
Hulsebos et al.
(2019) built a multi-input deep neural network model for detecting 78 semantic data types, extracting 686, 765 data columns from VizNet corpus from Hu et al.
They extracted a total of 1,588 features for each column to train the model, thus resulting in a support-weighted F1 score of 0.89, exceeding that of machine learning baselines, dictionary and regular expression benchmarks, and the consensus of crowdsourced annotations.
Zhang et al.
(2019) introduced SATO, a hybrid machine learning model to automatically detect the semantic types of columns in tables, exploiting the signals from the context as well as the column values.
It combines a deep learning model trained on a large-scale table corpus with topic modelling and structured prediction to achieve support-weighted and macro average F1 scores of 0.925 and 0.735, respectively, exceeding the state-of-the-art performance by a signiﬁcant margin.
While in this paper we do not experiment with the context information of the column values, our work is mostly aligned to Hulsebos et al.
Keeping that in mind, we are planning to research and implement the context of the column values in tables in our DCoM models for our future works.
The paper is organized as follows: Section 3 describes the data used for DCoM models.
In Sectionwe present the details of data preparation and architecture for DCoM models.
Section 5 contains the training, evaluation and inference procedures of DCoM models while Section 6 discussed about the results of extensive experiments.
In Section 7, we talk about some known limitations from the data as well as application point of view and ﬁnally, in Section 8, we present the concluding remarks and some future directions.
D ATA
We have used the dataset prepared by Hulsebos et al.
and compared our model performance with them considering their results to be baselines.
This dataset contains 686,765 instances with 78 unique classes (or semantic data types).
It is divided into 60/20/20 training/validation/testing splits.
The instance count for classes varies from 584 (afﬁliate class) to 9088 (type class).
The count distribution for classes is shown in Figure 1.
To provide a more clear picture of the data a sample of the dataset is shown in Table 1.
[Table]
P ROPOSED M ETHOD
Prior approaches (Venetis et al., 2011; Limaye et al., 2010) to semantic type detection trained models, such as logistic regression, decision trees, feedforward neural network (Hulsebos et al., 2019), extracting various features from the data.
We, on the other hand, treated the data like natural language (text) and used the data itself as the input to the model.
We used a very small number of hand-engineered features unlike Hulsebos et al.
Therefore, our DCoM models have two inputs, the values of the instances as text or natural language and hand-engineered features.
We present two types of DCoM models based on the way we feed the text input to the model.
4. DC O M WITH S INGLE S EQUENCE I NPUT
In this subsection, we discuss how the values of each instance can be fed to the model as a single sequence input to the DCoM.
Considering the scenario, we can not simply pass the inputs separating values of each input with any separator token.
This gives the model wrong information about the sequence of the data resulting in faulty training and poor performance on the unseen data.
For example, if we consider the last example from Table 1 and create the input, Deletes the property <SEP>
Lets you edit the value of the property <SEP> Script execution will be stopped for the model, the model gets to learn that the value Lets you edit the value of the property has a relative position between Deletes the property and Script execution will be stopped , which is quite wrong because the values in an instance do not have any relative position between them.
To mitigate the problem we introduce permutations from mathematics.
With permutation, we order r items from the set of n i items.
Here n i items are all the values of an instance i , and r ∈ [1 , n ] .
Doing so, the above instance can be broken down into multiple subsets.
A sample of the subset is shown in Table 2.
If we feed all the new instances of the subset to the model, it does not learn any relative positional information of Lets you edit the value of the property with respect to other values in that instance unlike earlier.
Therefore, for the value Deletes the property , the model only learns the relative positions of the tokens e.g. Deletes, the, property, etc in a value, but not the relative position of the values in an instance.
This helps the model getting the actual information present in the data for accurate prediction on the output.
This permutation method also helps in augmenting new instances which help in training the data-hungry deep learning models with enriched data.
It is not feasible to generate all the possible permutations before the training because of the huge number of subsets.
Instead, during training, we sample r between 1 and n for each of the instances and generate one permutation for each instance.
More than one instance can be generated, but training the model for multiple epochs will result same for both cases.
[Table]
4. DC O M WITH M ULTIPLE S EQUENCES I NPUTS
Input preparation with this method is straightforward compared to the earlier method.
In this method, we also use permutations to generate new instances, but the value of r is ﬁxed during training and it is used as a hyper-parameter, where r ∈ [1 , ∞ ) .
Once we decide the value of r , r number of inputs are used as text inputs to the model.
Aggregation of embedding vectors of the inputs are performed once they are generated using the shared embedding weights.
If r > n , where n is the number of values of an instance, then this scenario can be handled in two ways.
The ﬁrst way is to pad r − n inputs and aggregate the embedding vectors only for the non-padded inputs.
Another way is while generating new instances use permutation with replacement to always sample r values out of n values.
Therefore, padding is not required for the latter method.
We tried both approaches and did not observe any signiﬁcant difference in the result.
We used 19 out of 27 global statistical features from Hulsebos et al.
(2019) as our engineered features for the DCoM models.
These features are normalized before feeding to the model.
The complete list of these features is shown in Table 6.
Once the text and engineered inputs are prepared, we attach LSTM/Transformer/BERT layers to the text input.
The output of the same is aggregated with the engineered inputs.
We use some feed-forward layer after that.
Finally, we use one softmax layer with 78 units to get the probability of each of the classes as output.
This is our generalized architecture design of DCoM models.
Extensive hyper-parameter tuning is performed to ﬁnalize the number of layers, number of units in a layer and many other hyper-parameters in the model.
This topic is discussed in detail in section 5.
To name the DCoM models, the type of the text input and the name of the deep learning layers are used as sufﬁxes with the name DCoM , e.g. DCoM model with single instance input and LSTM layers are named as DCoM-Single-LSTM .
The architecture design of DCoM models for both single sequence and multiple sequences inputs are shown in Figure 2.
T RAINING , E VALUATION AND I NFERENCE
We trained our model on the train dataset, validated it on the validation dataset and ﬁnally reported our results on the test dataset.
As class imbalance is present, like Hulsebos et al.
(2019), we also evaluated our model performance using the average F1-score, weighted by the number of columns per class in the test set (i.e., the support).
The DCoM models are trained in Tensorﬂow 2 (Abadi et al., 2015) and the hyperparameters are tuned using keras-tuner (O’Malley et al., 2019).
For inputs of DCoM models, we tried permutations with replacement as well as without replacement, but we did not observe any signiﬁcant difference in outputs.
The value of r for which we observed best performance for DCoM-Multi models is 45 .
For tokenizing text inputs, we experimented with character-based, word-based and BERT Wordpiece (Wu et al., 2016) tokenizers, and we found out BERT Wordpiece tokenizer stood out to be working better with respect to the other tokenizers because of the obvious reasons stated in the paper (Wu et al., 2016).
We compared our result with and without pre-trained embedding weights.
It is observed that, though in the ﬁnal output there is no considerable difference, training with pre-trained embedding weights take approximately 40% less amount of time to converge.
We experimented with the small, base and large variations of several BERT architectures and ﬁnalized DistilBERT-base (Sanh et al., 2019) and Electra-small (Clark et al., 2020) based on their performances.
We used
We used Adam (Kingma & Ba, 2014) optimizer with initial learning rate 10 − 4 .
Along with this, we implement a learning rate reducer with a factor of 0.5, which reduces the learning rate by 50% if the model performance does not improve on the validation dataset after consecutive 5 epochs.
Assigning class weights does not have much effect on the test results.
As this is a multi-class (78 classes) classiﬁcation problem, the DCoM models are trained with categorical-crossentropy loss and validated with accuracy and average F1 score metric.
we tried two approaches for inference on the test dataset.
In the ﬁrst approach, we performed single time inference on each of the instances of the test dataset.
While doing so, we set r to be the total number of values ( n i ) for each instance for DCoM-Single models to perform inference.
For
DCoM-Multi , as r values if preﬁxed, some values will be truncated for inputs with total values n > r .
The other approach is to generate k (where, k > 1 ) instances with permutation, sampling r values k times between 1 and n i , where n i is the total number of values for instance i .
Therefore, we get k class predictions for each of the instances and ﬁnally with majority voting we pick the prediction class for each instance.
We used k value to be 10 in our case and it is observed that for both
DCoM-Single and DCoM-Multi , we observed 0 .
2 − 0 .
5% improvement in the test average
F1 score, but this improvement comes with the price of increased inference time by approximately k times.
R ESULTS
We compared DCoM against Sherlock and other types of models shown by Hulsebos et al.
assuming those to be our baseline models.
Table 3 presents the comparison of results on the test dataset for each of the models.
Columns Engineered Features and k are speciﬁc to DCoM models.
Engineered Features says whether the 19 engineered features are used with the text inputs while training the model.
k column tells the number of times inference is performed on a single test instance.
It is discussed in detail in section 5.
Runtime column is the average time in seconds needed to extract features and generate a prediction for a single sample, and Size column reports the space required by the trained models in MB.
From the table, it is seen that several DCoM models outperform Sherlock (Hulsebos et al., 2019) in both F1 score and inference run time with signiﬁcant margins.
Hulsebos et al.
(2019) extracted various features e.g. global statistics, character-level distributions, word embeddings, paragraph vectors from the data and used those features to ﬁt a feedforward neural network model.
On the other hand, we treated the data as texts and feed those to more advanced NLPbased models.
This allows the DCoM models to extract and learn more useful features that Hulsebos et al.
(2019) were unable to extract using hand-engineering.
Along with this, hand-engineering takes a considerable amount of time to calculate the features which increase the inference time of Sherlock (Hulsebos et al., 2019) by 3 to 20 times with respect to DCoM models.
[Table]
6. P ERFORMANCE FOR I NDIVIDUAL T YPES
Following Hulsebos et al.
[Table] we also prepared Table 4 which displays the top and bottom ﬁve types, as measured by the F1 score achieved by the best performing DCoM model,
DCoM-Single-DistilBERT for single inference ( k = 1 ) for that type. .
High performing classes such as grades, industry, ISBN, etc. contain a ﬁnite set of valid values which help DCoM models to extract distinctive features with respect to the other classes.
To understand types for which
DCoM-Single-DistilBERT performs poorly, we include incorrectly predicted examples for the lowest precision type (Rank) and the lowest recall type (Ranking) in Table 5.
From the table, it is observed that purely numerical values or values appearing in multiple classes, cause a challenge for the DCoM models to correctly classify the semantic type of the data.
This issue is discussed in detail in section
[Table] [Table]
6. F EATURE I MPORTANCE
To calculate the feature importance of engineered features used in trained
DCoM-Single-DistilBERT model
, we extracted the learned feature weight matrix, W from the dense layer used after the engineered features input
[Table]
where W ∈ R 19 × D .
Here D is the number of units used in the dense layer.
W contains the weights/contribution of each of thefeatures on each of the D units
[Table]
thus forming a 19 × D array, which contains both positive and negative values based on the direction of contribution.
We take the absolute value of all the elements of W , as we are only interested in the amount of contribution of each of the engineered features, not the direction.
After that, we take the mean across the 19 features that result in a 19-dimensional array.
We normalized the array by dividing the maximum value of the array by each of the elements.
Table 6 enlists the importance of the engineered features in decreasing order.
[Table]
K NOWN L IMITATIONS
The major limitation of the process is with the pre-processed data prepared by Hulsebos et al.
(2019) used to train DCoM models.
The same values of instances are present in multiple classes, thus resulting in strong class overlapping among some classes.
It makes the models confusing during training to learn distinctive features for proper classiﬁcation.
Therefore, for some classes in the test dataset, the models perform very poorly.
Table 7 presents a sample of examples of class overlapping in the data prepared by Hulsebos et al
[Table]
It is to be noted that the proportion of this class overlap is considerably large in numbers with respect to the total number of instances.
Therefore, it affects
[Table]
the training as well as the model performance on the test dataset by a signiﬁcant margin.
Besides the class overlap, faulty or wrong values are present in some classes.
From the application point of view of semantic data type detection models, identifying and preparing a good dataset for training is very challenging as well as time-consuming.
Non-standardized column names are a major challenge large organizations face in doing semantic detection.
For example, social security numbers can be called as ssn, ssn_id, soc_sec_bnr etc.
Nonstandardized data columns also pose a major challenge in information ambiguity.
For example,
Gender column can have values Male, Female, unknown , or 0,1,2 .
Some organization can have mixed attributes which poses a major challenge in data security.
For example, use of PII data like Social security Number in Customer ID columns.
Some other challenges like corrupt/missing metadata also exist.
C ONCLUSION
DCoM presents a novel permutation-based method by which the instances can be fed to the deep learning models directly as natural language.
This takes the leverage of using more advanced NLPbased layers/models unlike feedforward neural networks in Sherlock (Hulsebos et al., 2019).
The permutation-based method also helps in generating a large number of new instances from the existing ones, helping DCoM models to boost their performance effectively, thus outperforming Sherlock (Hulsebos et al., 2019) and other recent works by quite a signiﬁcant margin in both F1 scores as well as inference time.
We also present an ensembling approach during inference time which improves the test average F1 score by 0 .
2 − 0 .
As mentioned earlier the next step of it to introduce the context of the column values to DCoM models while predicting the semantic types in relational tables