BERT-Sort: A Zero-shot MLM Semantic Encoder on Ordinal Features for AutoML
Abstract Data pre-processing is one of the key steps in creating machine learning pipelines for tabular data.
One of the common data pre-processing operations implemented in AutoML systems is to encode categorical features as numerical features.
Typically, this is implemented using a simple alphabetical sort on the categorical values, using functions such as OrdinalEncoder ,
LabelEncoder in Scikit-Learn and H2O .
However, often there exist semantic ordinal relationships among the categorical values, such as: quality level (i.e., [’very good’ > ’good’ > ’normal’> ’poor’]), or month (i.e., [’Jan’< ’Feb’ < ’Mar’]).
Such semantic relationships are not exploited by previous AutoML approaches.
In this paper, we introduce BERT-Sort, a novel approach to semantically encode ordinal categorical values via zero-shot Masked Language
Models (MLM) and apply it to AutoML for tabular data.
We created a new benchmark of 42 features from 10 public data sets for sorting categorical ordinal values for the first time, where BERT-Sort significantly improves semantic encoding of ordinal values in comparison to the existing approaches with 27% improvement.
We perform a comprehensive evaluation of BERT-Sort on different public MLMs, such as RoBERTa, XLM and DistilBERT.
We also compare the performance of raw data sets against encoded data sets through BERTSort in different AutoML platforms including AutoGluon, FLAML, H2O, and MLJAR to evaluate the proposed approach in an end-to-end scenario, where BERT-Sort achieved a performance close to a hard encoded feature. The artifacts of BERT-Sort is available at https://github.com/marscod/BERT-Sort .
1 Introduction
An Automated-Machine Learning (AutoML) platform aims to automate the process of feature engineering, data engineering, hyper-parameter optimization, training, prediction and deployment, where it minimizes human supervision in all stages (He et al., 2021).
Each data set may contain a variety of data types including ordinal values where the order of values is important.
Let
C 1 ≺ C 2 ≺ ... ≺ C 𝑛 denote a fixed set of arbitrary classes of C .
For instance, UCI Audiology data set (Porter, 2019) includes a feature of Air with a set of unique values of Normal ≺ Mild ≺ Moderate ≺ severe ≺ profound .
Although this field is not a target feature, the order of the values carries semantic meaning.
AutoML platforms encode each feature based on their types and content of values.
Often AutoMLs encode categorical features as an integer array (Pedregosa et al., 2011) function.
For instance, H2O AutoML (LeDell et al., 2018) and Scikit-Learn use categorical_encoding and OrdinalEncoder (SkL, 2021) functions, respectively, to transform categorical values to an integer array.
However, all categorical encoders rely on an alphabetical sort function such as Numpy
Sort (num, 2021) where the encoded values are based on the sorting results of categorical values alphabetically (Oliphant, 2006) (LeDell et al., 2018).
Such a simple method may fail to capture the semantic relationships between values.
For instance, the value ’profound’ in the Air feature of
UCI Audiology data set represents more serious level than the value ’severe’ but OrdinalEncoder returns the opposite result.
Similarly, alphabetically sorting values of num-of-cylinders feature in UCI Automobile Data Set returns [’eight’ ≺ ’five’ ≺ ’four’ ≺ ’six’ ≺ ’three’ ≺ ’twelve’ ≺ ’two’] as [ C 1 ≺ C 2 ≺ ... ≺ C 7 ] , respectively, where the ordinal values have been misplaced.
As a result, such © 2022 the incorrectly encoded values can pose more challenges to any machine-learning algorithms to predict the target value of price based on increase/decrease value of cylinders .
One hypothesis to be verified in this paper is that AutoML platforms incorrectly encoding ordinal categorical features might result in degraded performance.
To address this issue, we propose a novel approach BERT-Sort which utilizes pre-trained Masked Language Models (MLM) (Devlin et al., 2018) in a zero-sot setting to semantically sort and encode ordinal values.
The following are our main contributions in this study.
(i)
A zero-shot systematic sorting algorithm to sort ordinal values is introduced in Section 3; (ii) We compose a benchmark of 10 real-world data sets with 42 ordinal features for the first time, which is explained in Section 4 (detail in Appendix A); (iii) We conduct a comprehensive performance evaluation of benchmarks by i) comparing the results of BERT-Sort (with initialization on 4 different publicly available MLMs) and OrdinalEncoder (which is widely used in different AutoML platforms), and ii) evaluating between raw data set and encoded data set through BERT-Sort in 4 different AutoML platforms of AutoGluon, FLAML, H2O, MLJAR in Section 4.
2 Related Works
In the supervised approach, researchers and practitioners utilize existing limited data sets with ordinal values to develop a model where it can encode ordinal values and predict unseen ordinal values for encoding purposes.
However, the adaptation of a trained model from one domain to another is extremely limited.
Therefore, most related works in the supervised approach can be used in a set of selective domains/languages and it can be used in a form of training a model for a particular domain.
An early study by McCullagh (1980) and later other studies by Christensen (2015) and Harrell (2015), introduced a general class of regression models for ordinal values where it utilizes the ordinal values through various modes of stochastic ordering.
In another recent study by Lausser et al.
(2020), the authors developed an ordinal subcascades detection and encoding process, but the authors mentioned that their works have a limitation where analyzing suitable data representations may give a better answer for ordinal encoding.
Dahouda and Joe (2021) proposes a deep-learned embedding technique for categorical features encoding.
The proposed technique is a distributed representation for categorical features where each category is mapped to a distinct vector, and the properties of the vector are learned while training a neural network.
In our proposed approach, we utilize the semantic understanding of MLMs to overcome the supervision and the limitation of ordinal subcascade encoding and other similar approaches.
In addition to the aforementioned studies, categorical encoders have also been widely applied in AutoML platforms.
Auto-sklearn provides ordinal encoding or one-hot encoding as choices for categorical features (Feurer et al., 2020).
MLJAR (mlj, 2022) (Płońska and Płoński, 2021) converts categorical features into numeric with label encoder , one-hot encoder or target encoder , which is automatically selected based on feature cardinality and AutoML training stage (mlj, 2022).
As explained by LeDell et al.
(2018), H2O utilizes tree-based models (Gradient Boosting Machines,
Random Forests) to support group-splits on categorical variables, so categorical data can be handled natively.
However, as explained by Zhou and Hooker (2021), the default split-improvement method is biased towards increasing the importance of features with more potential splits especially when we are dealing with a large number of ordinal values.
H2O also uses categorical_encoding , which specifies the encoding scheme to use for handling categorical features.
In AutoGluon, each categorical feature is mapped to monotonically increasing integers (Aut, 2022a).
AutoKeras defines an argument categorical_encoding , which specifies whether to encode the categorical features to numerical features (Aut, 2022b).
However, these categorical encoders rely on purely alphabetically sort functions such as Numpy Sort (num, 2021) where it rearranges the values by alphabetically sorting values, then it assigns an integer value based on the index of sorted values (Oliphant, 2006)
(LeDell et al., 2018).
Although the method can encode the categorical data sets, it fails to encode ordinal values by misplacing the values in incorrect orders.
In this paper, we introduce an approach that takes advantage of ordinal values, their semantic definitions to define an approach that can be universally applied toward diverse domains/languages without any supervision.
3 BERT-Sort
The basic idea of BERT-Sort is to utilize the power of a pre-trained language model to recognize the order of a given sequence of values.
Language models learn from a large number of text corpus where a language model processes the context of sentences and paragraphs.
MaskedLanguage Modeling (MLM) is a self-supervised learning of text representations where it computes the probability distribution of a masked token from a given context.
A masked language model can be used across different down-stream tasks such as text generation.
Each feature of a given data set may contain categorical ordinal values.
The tokens of ranked ordinal values could be seen in many documents.
For instance, we may find a large number of documents that includes ’February’ appearing after ’January’ .
In NLP down-stream tasks, such as text generation, we may replace a subset of tokens of a given input string with a distinctive character of [MASK] and the objective is predicting the masked token.
We utilize this unmasking process to find the orders of a set of given values based on their probability of orders according to a pre-trained MLM.
3.1 Problem Formulation
Figure 1 shows the overall process of the proposed approach, BERT-Sort, where it aims to find the best order of ordinal values by computing the maximum probability of appearances of values in a set of possible orders.
First, BERT-Sort captures all categorical features ( F ) of a given data set, D .
Let A 𝑖 denotes all unique values of 𝑖 th feature ( F 𝑖 ).
If || A 𝑖 || ≤ 𝜑 , where 𝜑 is a threshold of the maximum number of unique values, BERT-Sort applies to A 𝑖 (a candidate for ordinal feature).
In Section 4, we extend this rule-based approach to automate the detection of ordinal values by applying BERT-Sort to detect if there is an ordinal relationship between values.
BERT-Sort generates 𝑁 different possible permutation cases (different orders of ordinal values) from A 𝑖 , which is denoted as C 𝑖,𝑗 where 𝑗 = [ 1 , .., 𝑁 ] .
C 𝑘 𝑖,𝑗 denotes 𝑘 th element of C 𝑖,𝑗 .
Since it is computationally expensive to generate all possible permutations for a large number of elements, we explain a sequential approach in Section 3.2 to process a fewer number of cases.
BERT-Sort produces 𝑘 different masked patterns for 𝑖 th feature, and 𝑗 th case as 𝑀 𝑘 𝑖,𝑗 where 𝑘 = 1 .. ||
A 𝑖 || .
In each iteration ( 𝑘 ) a single value of C 𝑘 𝑖,𝑗 is masked.
Let S 𝑘𝑖,𝑗 denotes a generated sentence by applying each masked pattern of 𝑀 𝑘𝑖,𝑗 to C 𝑘𝑖,𝑗 where it is masked 𝑘 th element of C 𝑖,𝑗 .
For instance, for a given three elements ( 𝑘 = [ 1 , 2 , 3 ] ) of A 𝑖 [ 𝐴, 𝐵,𝐶 ] , it applies 𝑀 11 , 1 = [ 1 , 0 , 0 ] , and it generates a sentence of S 11 , 1 = [CLS][MASK], B,C.[EOS].
In this step, different sentence structures can be generated (i.e., replacing the comma between the ordinal values with a blank space ).
We explain the performance of building different sentence structures in
Section 10.
Next, BERT-Sort performs a model inference on initiated MLM (i.e., RoBERTa) to unmask S 𝑘𝑖,𝑗 and MLM returns 𝜂 number of retrieved tokens, which is denoted by W 𝜂 .
if C 𝑘𝑖,𝑗 ∈ W 𝜂 , it indicates that MLM returns a probability for the sequence of 𝐶 𝑘𝑖,𝑗 with the context of S 𝑘𝑖,𝑗 and it is denoted by (cid:98) 𝑃 ( 𝐶 𝑘𝑖,𝑗 | 𝑆 𝑘𝑖,𝑗 ,𝜂 ) ; otherwise it returns 0.
Finally, it finds the average probability of all masked sentences of 𝑖 th feature for 𝑗 th case ( Θ 𝑖,𝑗 ) as follows.
[Equation]
Θ 𝑖,𝑗 denotes the score of appearing a sentence (which is constructed from ordinal values) for 𝑖 th feature, and 𝑗 th order (case).
Θ 𝑀 𝑖 = max
||
𝐴 || 𝑗 = 1 ( Θ 𝑖,𝑗 ) which is selecting a case with the highest score (the best order or sequences of input elements).
Figure 3 shows an example of score computation for 3 elements.
Intuitively, BERT-Sort finds the most likely correct order from all possible permutations with utilizing a pre-trained MLM.
Different MLMs can be applied under the same architecture across different domains (i.e., medical domain) and languages (i.e., Chinese).
We explain the boarder impact of BERT-Sort in Section 12.
3.2 Handling a large number of ordinal values
Although BERT-Sort semantically ranks values instead of non-semantic approaches (i.e., alphabetical sort), it is computationally intractable for BERT-Sort to directly support a large permutations of ordinal values.
BERT-Sort applies two approaches for handling a large number of ordinal values.
First, BERT-Sort uses a divide and conquer approach for sorting elements if there are any repeated words among ordinal values.
Let C 𝑖 =
[ 𝑊 𝑖 1 , ..,
𝑊 𝑖𝑛 ] denotes n-gram (Brown et al., 1992) composition of 𝑖 th ordinal value.
The following shows five steps for this process.
i)
It generates a set of groups where each group has a word (largest n-gram words) in common, and it denotes common words.
(i.e., define a group of [Lava Hot, Boiling Hot, Hot] ).
Note that in this step, if 𝑊 𝑖𝑚 of C 𝑖 is selected for a group, then [ 𝑊 𝑖 1 , ..,𝑊 𝑖𝑛 ] will be added into the group, and it will not repeat the process on other words of [ 𝑊 𝑖 1 ..
𝑊 𝑖𝑚 − 1 ,𝑊 𝑖𝑚 + 1 ..
𝑊 𝑖𝑛 ] ; ii) It selects a group leader word where it is most frequent largest n-gram word among the group values (i.e., ’Hot’ is selected as group leader in previous example); iii) It sorts elements within each group.
(i.e., [Hot ≺ Boiling Hot ≺ Lava Hot] ); iv) It sorts elements of group leaders and unique values (i.e., [Cold ≺ Hot] ); Finally, v) it replaces each sorted group leaders with their sorted values of the group (i.e., [Cold ≺ Hot ≺ Boiling Hot ≺
Lava Hot] ).
Second, BERT-Sort uses a sequential adding procedure by sorting 𝜁 number of elements, then adds the rest of elements sequentially.
It aims to avoid generating a large number of permutation cases when 𝜁 < ||
A || .
Figure 2 shows an example of sequential sorting values for 12 ordinal value elements (months abbreviations).
BERT-Sort uses a sequential approach in three steps as follows.
i)
It finds the best candidate for 𝜁 blank spots (in the figure, it is initiated with 𝜁 = 5 spots); ii) Once it finds the best candidates for 𝜁 blank spots as initial sorted elements (it finds [Jan ≺ Feb ≺ Apr ≺
Jun ≺ Sep] for 5 spots in the example)
, then it adds each remaining element to the initial sorted elements (in the second iteration, it finds the best position for "Oct" in initial sorted 5 elements);
Finally, iii) it repeats step (ii) until all elements are added to the final sorted values.
This process reduces the computation time from O ( 𝑛 ! )
where 𝑛 = ||
A || to i) a combination function for selecting 𝜁 ordinal values from 𝑛 number of values with O ( 𝑛 𝜁 ) , and ii) a sequential adding function with
O ( 𝑛 2 ) where the whole procedure can be completed in O ( 𝑛 𝜁
+ 𝑛 2 ) s.t.
<
𝜁 < 𝑛 2 .
BERT-Sort aims to have a better accuracy with a higher probability score with lengthy contents (larger 𝜁 have better accuracy, see Section 11).
For instance, Θ 𝑀 with 𝑈 ([ 𝑀𝐴𝑆𝐾 ]) =
𝐽𝑎𝑛 where 𝑈 ( 𝑐 ) =
𝑡 returns the probability of a given masked context of 𝑐 if 𝑡 token exist in the list and considering 𝑐 = "
[CLS][MASK], Feb, Mar, Apr.[EOS]" is higher than 𝑈 ([ 𝑀𝐴𝑆𝐾 ])
=
𝐽𝑎𝑛 where 𝑐 = "[CLS][MASK], Feb.[EOS]" .
Therefore, we recommend to keep maximum possible of 𝜁 elements per available computation resources.
For instance, 𝜁 =
if ||
A || ≤ 12, and 𝜁 =
if 12 <
||
A || ≤ 20.
Algorithm 1 shows the details of BERT-Sort procedures.
4 Experiments
Experimental Setup.
We evaluate the proposed BERT-Sort approach under two cases: i) the performance of BERT-Sort in detecting and encoding categorical ordinal features; ii) the effectiveness of encoding applied to categorical ordinal features by BERT-Sort before input to various AutoMLs.
We use 10 different publicly available real-world data sets because it includes categorical ordinal values.
We annotate and generate 42 different categorical features (as explained in Appendix A), where we compare the performance of Scikit-Learn OrdinalEncoder as a baseline against BERT-Sort encoder.
We initiate BERT-Sort algorithm with 4 different popular and publicly available MLMs:
DistilBERT, RoBERTa, XLM-RoBERTa and BERT-base-uncased ( 𝑀 1 .. 4 respectively) in a zero-shot setting.
The inference on MLM can be optimized on CPU to reach 2ms per case (Philipp Schmid, 2022) and further detail explained in Section 8.
We compare the results of different MLMs and recommend the best MLM to researchers and practitioners.
The details of MLM, configurations and
BERT-Sort hyper-parameters with a link to reproduction are presented in Section 7 and Section 8.
Like other encoders, such as OrdinalEncoder , LabelEncoder , we use BERT-Sort to rank ordinal values semantically (Altınel and Ganiz, 2018), then assign an integer for each element per their orders.
Since the orders of the values are principal factors in either ascending or descending, we do not distinguish between two ranks.
However, we expect that BERT-Sort returns ordinal values ranked in ascending order (i.,e.
, ’low’ to ’high’ or ’Jan’ to ’Dec’ )
because most documents (which has been used to train MLMs) are written in ascending format.
For instance, we may find many documents in Wikipedia that indicate ’Jan, Feb’ and fewer document that include ’Feb, Jan’ .
9: return
⊲
10:11: procedure BERT_Compose
Sort
⊲ Sort input array of A through MLM 12: if || A
|| >
𝜁 then 13:
C ← 𝐶𝑜𝑚𝑏𝑖𝑛𝑒 ( A 𝜁 )
⊲ initial combination case generator length of 𝜁 14:
I ← 𝐵𝐸𝑅𝑇
_ 𝐵𝑎𝑠𝑒 _ 𝑆𝑜𝑟𝑡 ( C )
⊲ sort initiated sequences 15:
S ← I
⊲ sort initiated sequences 16: for E in A − I do ⊲ sequential adding the rest of elements 17:
C ← 𝑆𝑒𝑞 ( S , E )
⊲ generate new sequential cases by adding E in all 𝑝 positions of S ( 𝑝
[ 1 , ..., || S || + 1 ] ) 18:
S ← 𝐵𝐸𝑅𝑇 _ 𝐵𝑎𝑠𝑒
𝑆𝑜𝑟𝑡 ( C )
⊲ sort sequential cases 19: else 20:
C ← 𝑃𝑒𝑟𝑚𝑢𝑡𝑒 ( A ,𝜁 )
⊲ generate all new cases with all permutations of A 21:
S ← 𝐵𝐸𝑅𝑇 _ 𝐵𝑎𝑠𝑒 _ 𝑆𝑜𝑟𝑡 ( C )
⊲ sort all possible permutation of A 22: return S 23:24: procedure BERT_Base_Sort( C )
⊲ Sort input cases of C through MLM 25: Init MLM(Model)
⊲ initialize a MLM model (i.e., Model="RoBERTa") 26: for C in C do ⊲ process each case 27: for range do 28:
M 𝑖,𝑗 ← 𝑀𝑎𝑠𝑘 ( C 𝑖,𝑗 )
⊲ generate mask pattern 29: Θ 𝑖 ← Θ 𝑖 + [ (cid:99) P 𝑖 ( M 𝑖,𝑗 | 𝑆 𝑖 ,𝜂 ) ]
⊲ collect probability of each unmasked pattern 30: Θ 𝑖 = (cid:205) ( (cid:99) P 𝑖 )
⊲ calculate score of each case 31: if Θ 𝑖 > Θ 𝑚 then : 32:
C 𝑚 =
C 𝑖 33: Θ 𝑚 =
Θ 𝑖 ⊲ keep the best case with highest score return C 𝑚
[Table]
4.1 Evaluation of Detecting Ordinal Features
We define Ordinal Value Detection function to apply BERT-Sort effectively, where it checks whether a set of unique values of a given feature corresponds to a semantic order.
We select the ordinal values with length of 3 to 20, and remove numerical values in string format as part of pre-processing stage.
Then, we shuffle input unique values, and generate 𝑚 number of sample cases.
If 𝐴𝑣𝑔 ( Θ 1 ..𝑚 ) > 1 𝑒 − 4,
BERT-Sort consider that there is an ordinal relationship between values.
In our experiment, we use benchmark that includes 10 data sets with 212 features.
In this experiment, 𝑚 = 3 and 𝜂 = 20 ,in BERT_Base_Sort() and initiate MLM on RoBERTa model.
As shown in Table 1, the ordinal value detection function predicts ordinal values with 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 0 .
875, 𝑅𝑒𝑐𝑎𝑙𝑙 = 1 and 𝐹 1 = 0 .
As shown in Table 1 and Table 2-row#5, BERT-Sort detects 6 false-positive features out of 212 features due to possibility of generating a probability score for a set of given values in zero-shot environment.
[Table] [Table]
Even though BERT-Sort may encode those false-positive detected features, the encoded values may not cause negative impacts to down-stream machine learning tasks
[Table]
4.2 Evaluation of Encoding Ordinal Values
In addition to classical evaluation of classification problem, such as Accuracy (Mosley, 2013)(Urbanowicz and Moore, 2015)
[Table]
we proposed a new metric based on ordinal value error rate where it calculates the ordinal distance error between the ground truths and predictions as follows
[Table]
[Equation]
Let ∥ A ∥ denotes the length of input array (unique values of a given feature, F ), | L 𝑖 − (cid:98) L 𝑖 | is the absolute distance of 𝑖 -th element in predicted order ( (cid:98) L 𝑖 ) from the actual position ( L 𝑖 ) in ground truth data set, and 𝑂𝑟𝑑 𝐴𝑐𝑐 represents the Ordinal Accuracy .
For instance, as shown in Table 3, row#1, both values of Jan and Feb have distance (error rate) of one to their actual positions and thus 𝑂𝑟𝑑 𝐴𝑐𝑐 = 0 .
On the other hand, the accuracy for this example is 0.5 because two of four elements are incorrect.
In contrast, another example shown in row #2 also has the same accuracy equal to 0.5.
However, the second example has worse Ordinal Accuracy where 𝑂𝑟𝑑 𝐴𝑐𝑐 = 0 .
75 in compared to the first one.
As shown row #2 has a longer distance (higher error rate) for both Jan and Mar to their ground truth positions.
Although we evaluate the benchmark on both evaluation metrics, we recommend 𝑂𝑟𝑑 𝐴𝑐𝑐 for ordinal evaluation.
Table 4 shows the evaluation results on 10 different data sets for 42 distinctive features of annotated ground truth of ordinal values where 𝜁
= 4, in our benchmark 𝐴𝑣𝑔 (|| A ||)
= 4.
𝑂𝑟𝑑 𝐴𝑐𝑐 corresponds to ordinal accuracy and 𝐴𝑐𝑐 corresponds to classical accuracy metric.
As part of the pre-processing stage, we consider a set of special categories where it can be removed from unique values due to general information or non-ordinal values, such as: "?", Null string, Null value, "unknown", "unmeasured", etc.
We remove special categories from both BERT-Sort and
OrdinalEncoder to have an unbiased comparison.
As shown in this table, BERT-Sort improves the performance in terms of both 𝑂𝑟𝑑 𝐴𝑐𝑐 and 𝐴𝑐𝑐 on all four different initiated MLMs.
BERT-Sort with 𝑀 2 initialization (RoBERTa) achieves the best performance with significant improvements of 27% and 55% against the baseline based on 𝑂𝑟𝑑 𝐴𝑐𝑐 and 𝐴𝑐𝑐 metrics, respectively.
4.3 AutoML Evaluation.
We use 4 AutoML platforms including AutoGluon, FLAML, H2O, and MLJAR to evaluate the effectiveness of encoded categorical ordinal features by using BERT-Sort to the ultimate machine learning performance. We evaluate 5 different versions of 10 data sets, where each method encodes
[Table]
F0.
0.20 F0.
0.33 F0.
F1.
F0. F0.
F 151.
F1.
F 171.
1.00 F 181.
F 190.
F0. F0.
F 221.
F 230.
F1.
F0.
F1.
F0. F0.
F0. F0.
F0. F0.
F 330.
0.60 F 341.
F 350.
F0.
F1. F1.
F0.
0.50 F0.
F0.
F 421.
#champions         5 w.r.t.
OrdinalEncoder
[Table] [Table]
only ordinal features and leave the rest of features as-is.
The following shows 5 different methods which transform the original data set to different encoded features as inputs to 4 AutoML platforms.
Original.
It refers to the original data set without any changes (our baseline).
Encoded BERT.
This method encodes the ordinal value by utilizing Sentence-BERT (Reimers and
Gurevych, 2019) to generate a high-dimension continuous vector representation with a size 238.
Then, we apply Scikit-Learn PCA (Szlam et al., 2014), a linear dimensionality reduction method based on Singular Value Decomposition(Maćkiewicz and Ratajczak, 1993), to transform the highdimension vector into a low-dimension (a single value) that represents the ordinal feature.
Human Annotation.
We manually assign the orders (i.e. integers) to ordinal values based on their semantic meanings.
The annotated values are considered as ground-truth.
OrdinalEncoder.
where it uses Scikit-Learn OrdinalEncoder to encode the ordinal values.
BERT-Sort. which encodes ordinal value based on BERT-Sort approach.
We split each input data set (each version of data set) into 75% which is sent to each AutoML platform, and 25% which is used to test the trained model.
The experiments have been completed with 4 different seeds of [108, 180, 234, 309] to split data sets for training and testing (see Sectionfor detail).
We limit the execution time of AutoML platforms to 5 minutes.
Note that the data sets in our evaluation include all different features including numerical, text, etc.
In this evaluation, we use F1 metric and accuracy evaluation.
F1 metric has more restriction in compared to accuracy metric because it aggregates both Recall and Precision by considering the concept of harmonic mean (Grandini et al., 2020)
(Takahashi et al., 2021).
We use F1-macro where it is computes the average of F1 score of each class with weighting depends on the average parameter.
All AutoML platforms failed on two data sets (regressions task) of UCI_Coil_1999_Competition where it requires 7 features predictions (7 of Algae frequency distributions must be determined), and UCI-Automobile which required additional pre-processing step.
Although AutoGluon and H2O generate at least a model for these data sets but returns a negative score value which indicates that the model fits data poorly.
We decided to remove these two data set to avoid bias evaluation across different
AutoMLs.
All AutoML platforms successfully generated at least a model (success) for data sets with classification task except H2O where it is failed on two data sets of UCI Bank and Kaggle
Cat-in-the-Dat-ii data sets due to time limitation.
Table 5 shows the overall performance of all data sets across 4 AutoML platforms.
The performance results in this table indicates that human annotation of ordinal values have the best performance and BERT-Sort was able to produce results with F1 score and Accuracy score close to manually annotated version of data sets that has highest performance.
In addition, manually transformed ordinal values (human annotated) shows that a correct encoding can improve ultimate end-to-end performance of machine-learning models.
Table 6 shows the details of our experiments per AutoML platform.
Although nominal values and other features (such as text, and numerical) may contributed to model prediction, BERT-Sort improves the performance of 3 AutoML platforms.
The fine-grained evaluation of this table per
AutoML platform per Data Set per Encoded Method is shown in Table 11 and Table 12. [Table]
5 Discussions
As shown in Table [Table] encoding ordinal values by BERT-Sort with initialization of any MLM indicates
BERT-Sort is able to achieve the state-of-the-art semantic encoding performance on categorical features.
On the other hand [Table] these results show that widely used categorical encoding functions such as OrdinalEncoder will lead to diminish the semantic order of ordinal values .
Interestingly, BERT-Sort can go beyond expectation by sorting categorical data where it is too complex for data scientists to rank elements manually.
For instance, by initiating BERT-Sort on BioClinical BERT (Alsentzer et al., 2019) it can sort the severity of cancer as [Melanoma ≻
Leukemia ≻
Cancer] without any supervision (a supervised approach used by Lausser et al.
(2020) to generate this order).
More details about the broader impact of BERT-Sort usages across different domains, languages (i.e., Chinese, Japanese and Spanish) and semantic image sorting are explained in Section 12.
Note that evaluated data sets contains many other non-ordinal features including nominal, text, etc.
Therefore, the evaluation results reflects only the generalization of BERT-Sort.
6 Conclusion
In this paper, we introduced BERT-Sort, a novel automated approach to detect and encode categorical ordinal features.
We introduced a benchmark that includes 10 different public data sets withdifferent ordinal features.
We conducted an extensive evaluation on the benchmark where BERTSort is initialized on four popular MLMs of DistilBERT, RoBERTa, XLM and BERT-base.
BERT-Sort significantly improves the performance of the state-of-the-art categorical encoders (i.e., Scikit-Learn
OrdinalEncoder) by 22%.
We also evaluated the effectiveness of the encoded features by BERT-Sort on 4 AutoML platforms.
We compared the performances of encoded data sets via BERT-Sort against 4 different versions the original data sets.
The evaluation results show that the trained model based on BERT-Sort encoder achieved a performance close to a hard encoded feature by a data scientist.
7 Appendix A: Experiment Setup
Configuration Environment.
Our BERT-Sort encoder process is completed on a machine with Ubuntu, an Intel(R) Xeon(R)
Gold 5120 CPU @
2.20GHz (56 cores) with 128 GB RAM and 2 TB disk.
AutoML experiments are completed on a machine with 4 Intel(R)
Xeon(R)
Silver 4114 CPU @
2.20GHz (total 40 cores) with 512 GB RAM and 2 TB disk.
All experiments are developed in Python with version ’3.8’.
About 17 hours CPU computation is required to complete all experiments except reported results in Figure 9 that required 50 hours.
Benchmark Data Sets.
We use 10 publicly available data sets with 42 distinct features as listed in Table 7.
We select these data sets because i) each data set has at least a categorical ordinal value; ii) data sets are publicly available.
Artifacts.
In addition to provided link to the resources in Table 7, we provide the following artifacts that allow researchers and practitioners to reproduce the results.
i) a copy of raw data sets; ii ) annotated categorical features (Human Annotation), which is used to evaluate BERT-Sort as the results shown in Table 4; and iii) encoded raw data sets through BERT-Sort with each model ( 𝑀 1 .. 4 ).
The artifacts can be found at: https://github.com/marscod/BERT-Sort/blob/main/README.md .
In the repository, each folder in outputs/MLM contains a configuration file as ’config.json’ with a set of keys of [’model’, ’mask’, ’separator’, ’eta’, ’lower’, ’target_files’, ’ground_truth’, ’default_grouping’, ’default_zeta’, ’preprocess’] .
The key of ’target_files’ represent task information such as data set filename, a URL reference, type of task (classification or regression for AutoML evaluation), type of evaluation metric (F1 or RMSE).
The key of ’ground_truth’ is a dictionary where the keys are representing the feature name (if any) or feature index, and the values are a list of ranked ordinal values.
As explained in Section 4.2, the values (such as "?", and null values) are appended in the beginning of the list, and null values have been ignored for evaluation process.
8 Appendix B: BERT-Sort and MLMs hyper-parameters
We use four publicly available MLMs to evaluate the benchmark data sets based on their ranked ordinal values as listed in Table 8.
Since the majority of models outperformed in compared to
OrdinalEncoder, we may use DistilBERT which is 60% faster (Sanh et al., 2019).
In addition, the inference on MLM can be optimized on CPU to reach 2ms per case (Philipp Schmid, 2022).
In addition, we initiate BERT-Sort on diverse MLMs to demonstrate the broader impact of the proposed approach across different domains and languages (see Appendix E - Section 12).
The list of MLMs and the parameters are listed in Table 9.
9 Appendix C: Evaluation Results Detail 9.1 Ordinal Encoding Evaluation
Figure 3 shows an example score computation for a given unique array of A 1 =
[ 𝐴, 𝐵,𝐶 ] .
This figure shows the average computation score for the first case with A = 0 .
Figure 7 shows a heat map plot of 4 initialized MLM for evaluation results of Table 4.
In this figure,
X-axis represents (from left to right)
𝑂𝑟𝑑 𝐴𝑐𝑐 of BERT-Sort, 𝑂𝑟𝑑 𝐴𝑐𝑐 of OrdinalEncoder, Differences of 𝑂𝑟𝑑 𝐴𝑐𝑐 metric, 𝐴𝑐𝑐 metric of BERT-Sort, 𝐴𝑐𝑐 metric of OrdinalEncoder, and Differences of 𝐴𝑐𝑐 metric, respectively.
Y-axis represents the data set name:feature name or feature index.
9.2 AutoML Evaluations
First, we use Scikit-Learn train_test_split function 1 to split the given input data set of the benchmark into 75%/25%.
We use the following random seeds in all experiences: [108, 180, 234, 309] .
Second, as explained in Section 4, we use 4 different AutoML platforms which are publicly available to train a model on training data sets and evaluate each AutoML platform on test data set.
The configuration and version of each tool is shown in Table 10.
[Table]
[Table] [Table]
[Figure]
’time
[Figure]
’metric’
[Figure]
’f1’
[Figure]
’r2’
[Figure]
’f
[Figure]
’max
[Figure]
_secs’
[Figure] [Figure]
(classification problem)
[Figure]
presets:’best_quality’
[Figure]
time_limit
[Figure]
In addition to evaluation results in Table 6 (with 4 seeds and 5 minute time limitation), we conducted extensive experiences with 1, 4 and 5 different randomized seeds to split each data set with a time limitation of 30 minutes.
The results are concluded in Figure 9.
In overall, the average
F1 performance of all AutoML platforms on raw data sets is 0.346 versus F1 score of 0.377 on encoded data sets via BERT-Sort.
Since there might be many features (i.e., numerical and text features beside ordinal features) affect the overall performance of an end-to-end scenario for evaluating the encoded data sets, we may use data sets that only rely on categorical features.
As such example, we use UCI Car Evaluation data set where all features are encoded through i)
OrdinalEncoder, ii) BERT-Sort to produce two versions of the original raw data set.
Then, we apply 11 different machine-learning algorithms to both data sets.
In this experiment, we use CatBoostClassifier from CatBoost 7 package with version ’1.0.5’ and other ML algorithms from Scikit-Learn package with version ’1.0.0’.
Figure 10 shows the results of this evaluation.
This pure comparison shows that BERT-Sort encoder outperformed on all algorithms with an average F1 score of 0.897 in comparison to OrdinalEncoder with an average
F1 score of 0.532.
9.3 Fine-grained AutoML Evaluation Results
Table 11 and Table 12 show the fine-grained evaluation results of Table 6 with F1 metric and Accuracy metric, respectively.
Note that a comparison between values in this fine-grained evaluation results may not show a clear affect of semantic ordinal values since each experiment rely on many factors such as characteristics of data set (i.e., the number of features, importance of ordinal features),
AutoML hyper-parameters, AutoML approach to train a model, etc.
10 Appendix D: MLM Input Structure
As discussed in Section 4, once a case ( C 𝑖 ), a set of ranked ordinal values, and its mask pattern ( P 𝑖,𝑘 ) have been generated, BERT-Sort produces an input similar to a sentence structure where it consists of the ordinal value and masked element.
The generated structure tokenized and submitted to the initialized MLM for unmasking process.
BERT-Sort may generate different sentences based on different separators for the unique values of [Jan, Feb, Mar] , such as "," or blank space " " or combination of both (", ") to separate values.
Similarly, "." can be added to the end of sentence or can
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
MLJAR
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
be dropped.
For example, BERT-Sort can generate "Jan, [MASK], Mar." for a separator of ", " or "Jan [MASK] Mar." for a separator of " "(blank space).
We conduct an extensive empirical study to find the best pattern to construct different inputs to MLM.
As shown previously (Table 4) BERT-Sort with initialization of RoBERTa is outperformed in compared to other MLMs.
We construct different input structures only on RoBERTa.
Table 13 shows the results of this empirical study where it shows the total number of champions based on 𝑂𝑟𝑑 𝐴𝑐𝑐 metric and 𝐴𝑐𝑐 metric for 42 distinct features in our benchmark data sets.
The results show that adding a comma between values and adding ".
at the end construct best practice with 𝑂𝑟𝑑 𝐴𝑐𝑐 = 0 .
27 and 𝐴𝑐𝑐 = 0 .
55 improvement over OrdinalEncoder.
[Table]
11 Appendix E: BERT-Sort Acceleration Parameters
In Section 3.2, we introduce two scaling approaches for handling BERT-Sort with large number of ordinal values for saving computation time.
In an ideal case, the native BERT-Sort can generate the best results without two scaling approaches because it finds the best ordinal values by checking all different possibilities.
Although most of the ordinal features do not have more than 15 values (Table 7 indicates that the average number of ordinal values in our benchmark is 4), the native
BERT-Sort has O ( 𝑛 ! ) computation which makes the algorithm impossible to be executed for a large number of ordinal values of a feature.
The first approach of BERT-Sort acceleration is grouping where it can be easily disabled as a hyper-parameter.
In the second acceleration, we recommend selecting maximum possible value for 𝜁 (based on available resources, i.e. CPU cores) where it disables or postpone scaling algorithms as much as possible.
As one example, Figure 11 shows
BERT-Sort results on a feature with randomized values of 12 months abbreviations ( [’Jan’, ’Feb’, ..., ’Dec’] ) with 𝜁 parameter in range of 2 to 5, where 𝜁 ≥ 5 sorts all elements correctly.
Scaling algorithms may reduce the performance of BERT-Sort, but it helps to reduce computation time.
An alternative approach for scaling BERT-Sort is parallel implementations because each case does not rely on other cases, and all BERT-Sort agents may generate the score of their cases at the same time.
12 Appendix F: Multilingual and Multi-domain Sorting
In this section, we demonstrate the broader impact of the zero-shot of BERT-Sort across different domains (e.g., medical), different languages of ordinal values (i.e., English, Spanish, Japanese and
Chinese).
BERT-Sort approach finds the semantic orders of the ordinal values based the highest probability (top 1) of cases as explained in Section 3.
Therefore, BERT-Sort easily can be extended by initializing the algorithm on different pre-trained MLMs to sort elements across different domains and languages.
Table 14 showcases how easily the same algorithm applied toward multilingual and diverse domains.
For instance, the first row compares the output of BERT-Sort against OrdinalEncoder with top score of BERT-Sort of 0.961205.
Different automated approaches can be used for switching between pre-trained MLMs in
BERT-Sort.
We may automate selecting pre-trained MLM process by using a domain or language classification (i.e., detecting English language vs Spanish) to select the best model for a specific set of ordinal values.
We may also use the score to easily switch between MLMs in BERT-Sort.
For instance, BERT-Sort which is initiated by English RoBERTa returns Θ 𝑀 = 0 on a set of Spanish ordinal values or on a specific domain (e.g., Θ 𝑀 = 0 in row#6 where it is using generic English
MLM, RoBERTa).
However, row#7 shows the same input on BioClinicalBERT(Alsentzer et al., 2019) where BERT-Sort ranks values based on cancer’s severity correctly.
The automated process of switching between languages can be applied toward different domains.
For example, BERT-Sort returns Θ 𝑀 = 0 score on a pre-trained English RoBERTa (Liu et al., 2019b) for input values of [ Leukemia, Cancer, Melanoma ] but after switching the model to BioClinicalBERT(Alsentzer et al., 2019) BERT-Sorts returns correctly sorted elements where Θ 𝑀 > 0.
Row#8 to 10 show the output of
BERT-Sort for given input in Japanese, Spanish, and Chinese languages where the same algorithm was able to sort all values accurately ( 𝑂𝑟𝑑 𝐴𝑐𝑐 = 1 . 0) by initializing BERT-Sort on different MLMs.
The red color text highlights misplaced ordinal values in OrdinalEncoder.
Furthermore, BERT-Sort can be applied in multi modal environment.
As such example, a set of raw images can be annotated through CLIP (Radford et al., 2021) and the unique values of annotated images can be sorted semantically without user’s annotation through BERT-Sort as shown in
Figure 12.
As another example
[Table]
BERT-Sort is capable of sorting any categorical information such as number words (i
[Table]
e., ["One","Two", "Four"] ), that might be captured from an OCR process
[Table]
[Table] [Table]
[Figure]
13 Appendix G: AutoML Failures
Assumptions.
Our evaluation is based on fitting 75% of data sets into different AutoML platforms and predicting 25% of the same test data set on all AutoMLs.
Therefore, we did not consider any update on any specific AutoML platform beside using the same hyper-parameters such as evaluation metric and etc.
The same configuration allows us to have a fair comparison between different
AutoML platforms according to given input data sets.
Therefore, we assume that any small fixes beside system configuration are out of scope in our evaluation.
In addition, AutoML platforms are evaluated based on our split test data set which has been used to test all AutoML platforms.
We also use prediction function of each AutoML platform that possibly handle missing data, unknown values and etc.
We list the most frequent exception errors as follows.
We observed several issues where an AutoML platform was not able to produce any model (failure cases).
We explain the detail of errors for each AutoML platform as follows.
13.1 AutoGluon
AutoGluon failed to generate a model when it raises error of " ValueError(’AutoGluon did not successfully train any models’) ".
The following shows different failure reasons to produce a model.
ValueError:
[Table]
AutoGluon could not set parameters automatically for multi-class classification and binary classification.
It is required to be set based on given input data set.
This error has been raise in data sets Audiology (seed=180), Pittsburgh_Bridges (seed=180) data sets.
[Table]
AutoGluon failed to generate model when it is rely on ‘fold_fitting_strategy.py‘.
This issue raised in "audiology (seed:108)" raise
[Table]
This is a known issue in Dask-on-Ray 9 despite we are using the latest version of Dask.
[Table]
It is an issue when there is inconsistency between the total number of outputs and the total number of classes.
Observed this issue in cat-in-the-dat-ii , bank , car_eval , uci-automobile data sets.
13.2 FLAML
The
[Table]
the
[Table]
Training data normalization is required.
13.3 H2O
[Table]
Training data normalization is required.
13.4 MLJAR
[Table]
encoding because no parameters were generated.
It is raised when there is missing load of already trained models after training restore (a known issued 10 , despite using the latest version.
14 Reproducibility Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1 and Section(b)
Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
The study does not have any direct potential negative effect.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
https://automl.cc/ethics-accessibility/ [Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions are described in the paper as well as the detail in Appendix section.
(b) Did you include complete proofs of all theoretical results?
[Yes] Explained in Section 3 and 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental material or as a url)?
[Yes] Added all required hyper-parameters, seeds, link to data sets, link to publicly available MLMs; see Section 4, Appendix A to C, Algorithm 1, and GitHub repository files for processed data sets.
(b) Did you include the raw results of running the given instructions on the given code and data?
[Yes] Configurations of each experiment saved as a JSON file that includes all parameters, raw data sets, encoded data sets and it is available in GitHub Repository.
We also provide dumped pickle files of intermediate steps (.PKL) that the detail of sorted ordinal values.
(c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?
[Yes] We added all detailed as JSON and CSV file in GitHub repository (the source of figures).
(d) Did you ensure sufficient code quality such that your code can be safely executed and the code is properly documented?
[Yes] We explained the detail in Algorithm 1, Appendix A-C; we believe that based on all publicly available resources (as cited), the instructions and explained algorithm, readers can reproduce our results.
We also added raw/BERT-Sort encoded data sets that submitted to AutoML tools among seeds and the evaluation results (e.g., see outputsout_roberta ) in GitHub Repository.
(e) Did you specify all the training details (e.g., data splits, pre-processing, search spaces, fixed hyperparameter settings, and how they were chosen)?
[Yes] All details are explained in Section 4 and Appendix section
and it is linked to GitHub Repository.
For instance, dumped pickle file include the pre-processed ordinal values.
(f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and hyperparameters for that code?
[Yes] We strictly follow a fair comparison to compare our proposed approach against others; see Section 4.
As an example, we explain the pre-process that is added to both BERT-Sort encoder and OrdinalEncoder.
(in a real-world scenario those pre-process is not applied to OrdinalEncoder and BERT-Sort’s performance improvement might be significantly higher).
(g)
Did you run ablation studies to assess the impact of different components of your approach?
[Yes] See Appendix D as an example where we consider different types of MLMs and different input formats.
(h)
Did you use the same evaluation protocol for the methods being compared?
[Yes] We ensured to report a fair comparison as explained in Section 3 and 4 (i.e., pre-process step is conducted on both OrdinalEncoder and BERT-Sort encoder; however, we expect that those pre-process step is not apply to OrdinalEncoder in a real-world scenario that increases the performance improvement of BERT-Sort.
(i)
Did you compare performance over time?
[Yes] See Section 4.3.
(j) Did you perform multiple runs of your experiments and report random seeds?
[Yes] The random seeds for all figures are reported in GitHub repository.
(k) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The detail of all results are reported in GitHub repository that includes the ranges of outputs (all outputs of different seeds and input data sets).
(l) Did you use tabular or surrogate benchmarks for in-depth evaluations?
[Yes] All detailed results and summary reports are available in GitHub Repository as pickle file, JSON and CSV files.
(m) Did you include the total amount of compute and the type of resources used (e.g., type of gpus, internal cluster, or cloud provider)?
[Yes] See Section 4.3 and Appendix A where we performed the experiment only on a CPU.
(n) Did you report how you tuned hyperparameters, and what time and resources this required (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and also hyperparameters of your own method)?
[Yes] All hyper-parameters are reported in Section 4 and Appendix section as well as listed details in each experiment folder( outputs ) in GitHub Repository.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Added reference and link to the resources (MLMs and data sets); See Appendix Section (b) Did you mention the license of the assets?
[No] the licence of all data sets and pre-trained MLMs are available in provided link as explained in Appendix section.
(c) Did you include any new assets either in the supplemental material or as a url?
[N/A]
We did not develop a model or a data set, we use only publicly available resources and provide the link to those resources.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We used only publicly available resources and we expect that the consent has been acquired by the provider, if it is applied.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] Not applicable, we do not believe that the external resources contains personally identifiable information or offensive content since has been widely evaluated/used by ML community.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use any crowdsourcing or conduct any research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable?
[N/A]
We did not use any crowdsourcing or conduct any research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We did not use any crowdsourcing or conduct any research with human subjects.