A UTOMATICALLY L EARNING F EATURE C ROSSING
FROM M ODEL
I NTERPRETATION
FOR T ABULAR D
ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Automatically feature generation is a major topic of automated machine learning.
Among various feature generation approaches, feature crossing, which takes cross-product of sparse features, is a promising way to effectively capture the interactions among categorical features in tabular data.
Previous works on feature crossing try to search in the set of all the possible cross feature ﬁelds.
This is obviously not efﬁcient when the size of original feature ﬁelds is large.
Meanwhile, some deep learning-based methods combines deep neural networks and various interaction components.
However, due to the existing of Deep Neural Networks (DNN), only a few cross features can be explicitly generated by the interaction components.
Recently, piece-wise interpretation of DNN has been widely studied, and the piece-wise interpretations are usually inconsistent in different samples.
Inspired by this, we give a deﬁnition of interpretation inconsistency in DNN, and propose a novel method called CrossGO, which selects useful cross features according to the interpretation inconsistency.
The whole process of learning feature crossing can be done via simply training a DNN model and a logistic regression (LR) model.
CrossGO can generate compact candidate set of cross feature ﬁelds, and promote the efﬁciency of searching.
Extensive experiments have been conducted on several real-world datasets.
Cross features generated by CrossGO can empower a simple LR model achieving approximate or even better performances comparing with complex DNN models.
I NTRODUCTION
Recently, automated machine learning (AutoML) has been widely studied, and is proven a practical and promising machine learning technique (Yao et al., 2018).
AutoML aims to provide a easy way to apply machine learning technique, and automate the procedure of machine learning partly or thoroughly.
This reduces labour on data preprocessing, feature engineering, model selection, hyperparameter tuning, model training and performance evaluation.
Among different components of machine learning, the quality of features plays an extremely important role (Liu & Motoda, 1998; Domingos, 2012).
Accordingly, to improve the performances of machine learning tasks, various automatic feature generation methods have been proposed (Chapelle et al., 2015; Katz et al., 2016; Zhang et al., 2016;
Qu et al., 2016; Juan et al., 2016; Blondel et al., 2016; Guo et al., 2017; Lian et al., 2018; Luo et al., 2019).
Among these feature generation methods, feature crossing, which takes cross-product of sparse features, is a promising way to capture the interactions among categorical features and is widely used to enhance the performance of machine learning on tabular data in real-world applications (Chapelle et al., 2015; Cheng et al., 2016;
Luo et al., 2019).
Previous works on feature crossing mostly try to search in the set of all the possible cross features (Rosales et al., 2012; Chapelle et al., 2015; Katz et al., 2016).
Based on these methods, AutoCross (Luo et al., 2019) employs engineering optimization tricks, and accelerates the searching process.
However, search-based methods are not efﬁcient when the size of original feature ﬁelds is large.
Meanwhile, some deep learning-based methods combine deep neural networks and various interaction components (Qu et al., 2016; Cheng et al., 2016; Wang et al., 2017; Guo et al., 2017; Lian et al., 2018).
However, due to the existing of Deep Neural Networks (DNN), which can capture variety of interactions between original features, only a few cross features can be explicitly generated by the interaction components.
As mentioned above, DNN is a powerful method for capturing various feature interactions in its hidden layers (Guo et al., 2017).
This means, DNN can generate features implicitly, but can not provide interpretable cross features explicitly.
Recently, the interpreration of deep models has drawn great attention in academia, and mostly focus on piece-wise interpretation, which means assigning a piece of local interpretation for each sample (Bastani et al., 2017; Alvarez-Melis & Jaakkola, 2018;
Chu et al., 2018).
This can be done via the gradient backpropagation from the prediction layer to the feature layer.
Usually, local interpretations of a speciﬁc feature are inconsistent in different samples.
And large interpretation inconsistency indicates that, the corresponding feature has interacted with other feature ﬁelds in the hidden layers of DNN.
Therefore, the interpretations of DNN can help us ﬁnd useful cross features.
Accordingly, in this paper, we propose a novel method called CrossGO, which can directly and automatically learn useful cross feature ﬁelds from the interpretation of DNN.
(1) First, we train a common sparse DNN with the whole training set.
(2) Then, for each sample in the validation set, we calculate the gradients of the output predictions with respect to the input features, i.e., local interpretations.
We also calculate the global interpretation of a speciﬁc feature as the average value of all the corresponding local interpretations.
For a speciﬁc feature in a speciﬁc sample, if the corresponding local interpretation is far from the corresponding global interpretation, i.e., interpretation inconsistency is large, we regard this feature interacted with others by DNN in the sample.
Thus, we obtain a set of feasible features that works for feature crossing in each sample.
(3) Then, we employ a greedy algorithm to generate a global candidate set containing both second-order and high-order cross feature ﬁelds.
(4) With the the candidate set, we can train a simple LR model, therefore rank and select feature ﬁelds according to their contribution measured in the validation set.
So far, we can obtain the ﬁnal set of cross feature ﬁelds.
We have conducted experiments on several real-world datasets.
The size of candidate set of cross feature ﬁelds is constrained to be 2 N , where N is the number of original feature ﬁelds.
To be noted, 2 N is extremely small compared to the whole set of second-order and high-order cross feature ﬁelds, especially when N is large.
And a simple LR model empowered by the ﬁnal set of cross feature ﬁelds achieves approximate or even better performances comparing with DNN on all datasets.
This whole process of learning feature crossing can be done via simply training a DNN model and a LR model.
This shows that, our proposed CrossGO method can automatically generate a compact candidate set of cross feature ﬁelds, and efﬁciently ﬁnd enough useful cross features.
M OTIVATION
In this section, we ﬁrst summarize the advantages of feature crossing and the drawbacks of conventional methods.
Then we give a brief analysis about why and how we learn feature crossing from the interpretation of DNN.
2. F EATURE C ROSSING
According to the deﬁnition in previous works (Cheng et al., 2016; Luo et al., 2019), we can conduct feature crossing as
[Equation]
(1) where ⊗ denotes cross-product, g x f 1 ,i ,x f 2 ,j ,...,x fN,m is the corresponding generated cross feature, and x f,i is a binary feature associated with ﬁeld f , e.g., feature ”gender=male” associated with ﬁeld ”gender”.
Via feature crossing, the cross feature of a male English-speaking student can be denoted as (”gender=male” and ”language=English” and ”job=student”).
Moreover, considering feature discretization has been proven useful to improve capability of numerical features (Liu et al., 2002;
Kotsiantis & Kanellopoulos, 2006; Chapelle et al., 2015), we can conduct feature discretization on numerical feature ﬁelds to generate associated binary features.
Via learning feature crossing, several advantages can be achieved: (1) new features can be generated; (2) cross features, instead of latent embeddings in deep neural networks, are highly interpretable (Luo et al., 2019); (3) simple linear machine learning methods can take use of cross features to achieve approximate or even better performances comparing with complex nonlinear methods (Luo et al., 2019); (4) it is ﬂexible and suits for large-scale online tasks (Cheng et al., 2016).
First attempts on feature generation mainly focus on generating second-order features (Rosales et al., 2012; Cheng et al., 2014; Chapelle et al., 2015; Juan et al., 2016; Katz et al., 2016).
In (Chapelle et al., 2015), the authors try to generate and select second-order cross features according to conditional mutual information.
The main problem of conditional mutual information is that, once the mutual information of an original feature is high, the generated cross features containing it will also have high conditional mutual information.
ExploreKit (Katz et al., 2016) presents a framework for generating candidate features and ranking them.
The proposed ranking and selecting methods are mainly based on the corresponding performances in classiﬁers, which is not efﬁcient when the feature set is large.
Moreover, these conventional methods fail to capture powerful high-order features.
Nowadays, deep learning-based prediction methods have shown their effectiveness.
Among these methods, some try to generate and represent features implicitly or explicitly via designing various interaction components.
The Wide & Deep model (Cheng et al., 2016) directly learns parameters of manually designed cross features in the wide component.
The Product-based Neural Network (PNN) (Qu et al., 2016) applies inner-product or outer-product to capture second-order features.
The Deep & Cross Network (DCN) (Wang et al., 2017) designs a incremental cross structure to capture second-order as well as high-order features.
Factorization Machine (FM) (Rendle, 2010; 2012; Juan et al., 2016) is a suitable and success way to capture second-order feature interactions, as well as high-order feature interactions (Blondel et al., 2016).
Thus, FM has been extended to deep architecture, e.g., DeepFM (Guo et al., 2017) and xDeepFM (Lian et al., 2018).
As we know, these deep learning-based methods usually consists of DNN.
And DNN is able to capture variety of feature interactions in its hidden layers, while not able to generate interpretable cross features.
Thus, only a few cross features can be generated by the interaction components in deep learning-based methods.
Another practical method for feature generation is to employ explicit search strategies to ﬁnd useful features (Fan et al., 2010; Kanter & Veeramachaneni, 2015; Katz et al., 2016; Luo et al., 2019).
As we can imagine, in search-based methods, the candidate set of cross feature ﬁelds is inevitable to be extremely large, thus the searching efﬁciency is usually low.
To solve this problem, AutoCross (Luo et al., 2019) presents a framework to search in the large candidate set more efﬁciently, which is a greedy and approximate alternative: (1) AutoCross iteratively searches in a set of cross feature ﬁelds, where the set is initialized as all the second-order feature ﬁelds, and the selected cross feature ﬁeld is greedily used to generate new high-order cross feature ﬁelds in the next iteration; (2) AutoCross uniformly divides the dataset and validate the contribution of a cross feature ﬁeld on a batch of data, which will face the problem of data lacking and random resulting when the candidate set is large.
The authors apply the generated cross features, containing both second- and high-order cross features, in a LR model, and it achieves approximate or even better performances comparing with the complex DNN model in 10 real-world datasets.
However, AutoCross still searches for useful cross feature ﬁelds in a large candidate set, whose size shows exponential relation with the number of the original feature ﬁelds.
For example, when the size of original feature ﬁelds is 10 , the number of second-order cross feature ﬁelds is 45 .
And when the size of original feature ﬁelds becomes 100 , 200 , 500 and 1000 , the number of second-order cross feature ﬁelds becomes 4950 , 19900 ,and 499500 respectively.
When the candidate set is large, the searching efﬁciency will still be low, and data for verifying the contribution of a candidate cross feature ﬁeld will be too little to produce reliable results.
This strongly constrains the applicability of AutoCross in real-world applications.
2. L EARNING FROM THE I NTERPRETATION OF DNN
To solve above problems of existing methods, we need to automatically generate a compact and accurate candidate set, so that the search procedure can be performed efﬁciently.
The widely used
DNN model has shown to be capable of capturing various feature interactions in its hidden layers (Guo et al., 2017).
And various works has done to conduct piece-wise interpretation of DNN, which means a DNN model can be regarded as a combination of an inﬁnite number of linear classiﬁers (Bastani et al., 2017; Alvarez-Melis & Jaakkola, 2018; Chu et al., 2018).
Via the gradient backpropagation from the prediction layer to the feature layer, for a speciﬁc feature x f,i with the feature ﬁeld f and a speciﬁc sample k in the dataset, we have [Equation]
[Table]
where ˆ y k denotes the prediction made by DNN for the sample k
[Table]
and w f,i,k is the local weights computed via gradient backpropagation.
Deﬁnition 2.1 (Local Interpretation)
Given a speciﬁc feature x f,i and a speciﬁc sample k , the corresponding local interpretation is
[Equation]
) where e f,i denotes the corresponding feature embedding in sparse DNN.
Usually, local interpretations of a speciﬁc feature are inconsistent in different samples in the dataset.
Deﬁnition 2.2 (Global Interpretation)
Given a speciﬁc feature x f,i , the corresponding global interpretation is
[Equation]
where ¯ w f is the averaged local weights of features associated with the feature ﬁeld f in all samples, named as global weights.
Deﬁnition 2.3 (Interpretation Inconsistency)
Given a speciﬁc feature x f,i and a speciﬁc sample k , the corresponding interpretation inconsistency is
[Equation]
When a speciﬁc feature has interacted with other features in the hidden layers of DNN, the corresponding local interpretations will be very inconsistent among different samples in a dataset, i.e., interpretation inconsistency is large.
Thus, the interpretation inconsistency in DNN is able to lead us to generate compact and accurate candidate sets of cross feature ﬁelds.
Accordingly, we can automatically learn feature crossing based on Assumption 2.1.
Assumption 2.1 Once the interpretation inconsistency of features in a speciﬁc feature ﬁeld exceeds a threshold, the corresponding feature ﬁeld works for generating cross features.
To verify Assumption 2.1, we conduct empirical experiments on four toy datasets.
The four datasets characterize four different logical operations: AND, OR, XNOR and XOR.
And we have two input feature ﬁelds, where α ∈ { 0 , 1 } and β ∈ { 0 , 1 } .
Thus, for each toy dataset, we have four different samples, and the output lies in { 0 , 1 } .
As we know, the logical operations AND and OR are easy and linearly separable, therefore no cross features are needed.
In contrast, the logical operations
XNOR and XOR are not linearly separable, therefore second-order cross feature ﬁeld consisting of α and β should be generated for promoting linear classiﬁers.
We train a two-layer DNN model on the four toy datasets until convergence, where the Area Under Curve (AUC) evaluation becomes 1 . 0 .
On all datasets, the gradients from the prediction layer to the feature layer is computed, and the interpretation inconsistency is obtained, as shown in Table 1.
It is clear that, interpretation inconsistency values on AND and OR are extremely small, while those on XNOR and XOR are relatively large.
This observation clearly shows that, α and β should be crossed on XNOR and
XOR, while should not on AND and OR.
This is clearly consistent with common sense, and strongly proves Assumption 2.1.
According to the values in Table 1, 0 .
01 might be a proper threshold for ﬁltering feature ﬁelds to construct compact and accurate candidate sets of cross feature ﬁelds.
So far, we can conclude that, it is proper to learn feature crossing from the interpretation inconsistency computed in DNN.
Global
Interpretability
Local
Interpretability
Gradients ≥
DNN
LR
Feasible Features Interpretation Inconsistency s 1 ss 3 ssf 1 f 2 f 3 f 4 fs 1 ss 3 ssf 1 f 2 f 3 f 4 f{f 1 ,f 3 },{f 1 ,f 5 },{f 2 ,f 3 },{f 3 ,f 4 },{f 3 ,f 5 },{f 4 ,f 5 } {f 1 ,f 3 ,f 4 },{f 1 ,f 3 ,f 5 },{f 3 ,f 4 ,f 5 } {f 1 ,f 3 ,f 4 ,f 5 }
Filter
Candidate Set Cross
[Figure]
Fields
Consistency
Analysis
Search {f 1 ,f 3 },{f 1 ,f 5 },{f 2 ,f 3 } {f 1 ,f 3 ,f 4 }
Greedy Generation
[Figure]
C ROSS GO
In this section, we formally propose the CrossGO model.
In general, CrossGO consists of two steps: (1) generating a compact and accurate candidate set of cross feature ﬁelds; (2) searching in the candidate set for the ﬁnal cross feature ﬁelds.
We use { f 1 , f 2 , ..., f n } to denote the cross feature ﬁeld generated by crossing f 1 , f 2 , ..., f n .
Figure 1 provides an overview of the proposed CrossGO approach, which will be described in detail below.
3. C ANDIDATE S ET G ENERATION
Firstly, as we rely on the piece-wise interpretation of DNN to generate the compact and accurate candidate set of cross features, we need to train a DNN model.
The input is the original categorical features and categorical features are ﬁeld-wise high dimensional vectors.
Thus, we use embedding layer (Zhang et al., 2016) to transform the input features into low dimensional dense representations.
Then the dense representations are passed through some linear transformation and nonlinear activation to obtain the predictions of samples, where we use relu as the activation for hidden layers, and sigmoid as the activation for the output layer to support binary classiﬁcation tasks.
Based on the trained DNN model, according to Assumption 2.1, in every validation sample k , we ﬁrst compute the interpretation inconsistency d f,i,k of each feature x f,i , where d f,i,k is deﬁned in Deﬁnition 2.3.
Thus, we obtain a matrix D , where D kf is the interpretation inconsistency of f-th feature in the k-th sample.
Then we conduct element-wise ﬁlter on matrix D with a threshold η which is the threshold in Assumption 2.1.
And η is 0.01 for default.
The ﬁlter operation can be formulated as
[Equation]
where D (cid:48) is a binary feasible feature matrix.
According to Assumption 2.1, for each feasible feature
D (cid:48) kf = 1 , the corresponding feature ﬁeld f can be used to generate candidate cross feature ﬁelds.
Lastly, we perform greedy generation to obtain the candidate set of cross features ﬁelds.
Pseudocode of greedy generation is given in Algorithm 3.1.
Here, we introduce three new hyper-parameters: δ , ε and γ .
δ is the maximum order of cross features, and δ is 4 for default, where higher-oder cross features are rarely useful.
ε denotes the threshold of the occurrence ratio of cross feature ﬁelds on validation set, and ε is 0 .
01 for default.
γ is the number of candidate cross feature ﬁelds returned by greedy generation, and γ is 2 N for default.
To be noted, in general, 2 N is extremely small comparing with the size of the whole set of 2nd-order, 3th-order and 4th-order cross feature ﬁelds.
For example, N is 100, the size of the whole set is 4087875 ( 4950 + 161700 + 3921225 ).
Require: feasible feature matrix D (cid:48) , set of original feature ﬁelds F = { f 1 , f 2 , ..., f N } , maximum order δ of cross features, threshold ε of the occurrence ratio of cross feature ﬁelds on validation set, number γ of candidate cross feature ﬁelds, the size of validation set is M valid .
Ensure: compact and accurate candidate set S .
1: S = {} , S (cid:48)
= {} , set of ﬁrst-order features S 1 = {{ f 1 } , { f 2 } , ..., { f N }} , current order o = 2 , count of occurrences count c = 0 for every cross feature ﬁeld c ; 2: for o ≤ δ do 3:
S o = {} , P = {} ; 4: for sample k in validation set do 5: set of feasible feature ﬁelds Q k = { f | D (cid:48) kf
= 1 } ; 6: for f in F , c in S o − 1 do 7: generate cross feature ﬁeld p = { f } ∪ c ; 8: if p ⊂ Q k then 9: count p = count p + 1 ; 10: p → P ; 11: end if 12: end for 13: end for 14: for p in P do 15: if count p /M valid ≥ ε then 16: p → S o ; 17: end if 18: end for 19:
S (cid:48)
=
S (cid:48) ∪ S o ; 20: o = o + 1 ; 21: end for 22: select top γ cross feature ﬁelds with largest counts from S (cid:48) , and assign them to S ; 23: return S .
3. S EARCHING F OR C ROSS F EATURE F IELDS
After obtaining the candidate set of cross feature ﬁelds S = { c 1 , c 2 , ..., c 2 N } , where c i is a speciﬁc cross feature ﬁeld, we can search for the ﬁnal useful cross feature ﬁelds according to their contribution measured in the validation set.
We ﬁrst need to train a LR model.
The input of LR model consists of both original features and candidate cross features.
We use S as the schema to process training data to generate training cross features, which are denoted as X crosstrain .
The shape of X crosstrain is [ M train , 2 N ] .
We also denote the original features of training set as X originaltrain , and the shape of it is [ M train , N ] .
Thus, the number of model’s input feature ﬁelds is 3 N .
Therefore, the time cost of training this LR model is in the same order of magnitude with directly training a LR model with only original features.
Based on the trained LR model with model weights W , we conduct our searching process.
Here, we use X originalvalid and X crossvalid to denote the original features and the cross features in the validation set respectively.
Pseudocode of searching process is given in Algorithm 3.2.
It is important to note that the step 4-10 of Algorithm 3.2 can be conveniently paralleled via multi-threading or multiprocess implementation.
Moreover, the inference process is much faster than the training process, and therefore the time cost of searching can be neglected.
This searching process is fast and effective.
The reason why we can use such a easy searching strategy is, CrossGO can generate a compact and accurate candidate set of cross feature ﬁelds according to the interpretation inconsistency in DNN.
Require: candidate set S
= { c 1 , c 2 , ..., c 2 N } , original features X originalvalid with N feature ﬁelds and M valid samples, candidate cross features X crossvalid with
N cross feature ﬁelds and M valid samples, labels Y valid with M valid samples, model weights W of LR trained with both original and cross features on training set, sigmoid function σ () and AUC computing function compute auc () .
Ensure: ﬁnal set of cross feature ﬁelds S ∗ .
1: S ∗ = {} ; 2: b ( − 1) =
X originalvalid W (cid:62) [0 : N − 1] ; 3: AUC ( − 1) =
compute auc ( Y valid , σ ( b ( − 1))) ; 4: for i in [0 , 2 N − 1] do 5: for j in [0 , 2 N − 1] do 6: if c j not in S ∗ then 7: b ( j ) =
b ( − 1) + X crossvalid
[: , j ] W [ N + j ] ; 8:
AUC ( j ) = compute auc
( Y valid , σ ( b ( j ))) ; 9: end if 10: end for 11: k = argmax j
AUC ( j ) ; 12: if k ≥
then 13: c k
→ S ∗ ; 14: b ( − 1)
=
b ( k ) ; 15:
AUC ( − 1) =
AUC ( k ) ; 16: else 17: break; 18: end if 19: end for 20: return S ∗ . [Table]
E XPERIMENTS
In this section, we empirically test our proposed crossGO model on 8 datasets for thorough comparisons.
We ﬁrst describe the datasets and settings of the experiments, then report and analyze the experimental results.
4. D ATASETS
We evaluate the proposed crossGO model on six benchmark and two anonymous datasets.
The two anonymous datasets are provided with sanitization and named as Anon1 and Anon2.
The statistic of these datasets are shown in Table 2.
All the tasks are for binary classiﬁcation.
Employee, Adult and
Criteo are the same datasets as in (Luo et al., 2019).
For other datasets except for the former three and Movielens, we use the ﬁrst 70% data as the training data, while the rest 30% as testing.
And we cut out the last 20% of the training data as validation set.
[Table] [Table] For Movielens, we regard reviews as behaviors, and sort the reviews from one user by time.
For user u , we split u ’s behaviors into three parts: the ﬁrst 50% is for training, following 20% is for validation, while the rest is for testing.
Meanwhile, we transform Movielens into a binary classiﬁcation data.
Speciﬁcally, original user rating of the movies is continuous value ranging fromto 5.
We label the samples with rating of 4 and 5 to be positive and the samples with rating ofand 2 to be negative.
We extract a series of statistical features based on each user’s historical behavior.
Features include user age, user gender, user occupation, is cate, recent k cate mean score and recent k cate cnt, where k ∈ { 10 , 30 , 50 } and cate ∈ { action, adventure, animation, children’s, comedy, crime, documentary, drama, fantasy, ﬁlm-noir, horror, musical, mystery, romance, sci-ﬁ, thriller, war, western } .
The recent k cate cnt is the count of each category in movies that each user has reviewed in recent k times.
Accordingly, the recent k cate mean score is the average score of each category in movies that each user has reviewed in recent k times.
In summary, in terms of whether or not the number of original feature ﬁelds is bigger than 100 , we test our approach on two types of datasets:
• Narrow Datasets : Employee, Adult and Criteo; • Wide Datasets : Movielens, Allstate, Prudential, Anon1 and Anon2.
4. E XPERIMENTAL S ET - UP
As our goal is to make a simple LR model empowered by generated cross features to achieve approximate or even better performances comparing with DNN, we compare against the strong baselines (LR, CMI+LR and DNN) and the state-of-the-art feature crossing approach (AutoCross+LR) as in (Luo et al., 2019).
We simplify AutoCross+LR and CMI+LR to AutoCross and CMI.
Accordingly, our approach is named as CrossGO.
We use the same discretization method as in (Luo et al., 2019) for preprocessing data.
For the DNN model, which we rely on to compute the interpretation inconsistency of features: learning rate is 0.001, the dimensionality of embedding vector is 16, network structure is 64-32.
For the LR model, which we rely on to search for cross features: learning rate ∈ [0 . 0001 , 0 . 001] .
For all models, we use
Adam with weight decay as zero as optimizer, the Area-Under-Curve (AUC) as our experimental metric and the same early-stopping strategy.
Speciﬁcally, if the validation AUC dose not increase in three epochs, the training process will be stopped.
4. P ERFORMANCE A NALYSIS
Results of our experiments are summarized in Table 3, Table 4, Table 6 and Figure 2.
According to Table 3 and Table 4, all models except CMI signiﬁcantly beat LR model on all datasets.
CMI is inferior to LR on some datasets, which shows that adding new cross features does not guarantee the improvement of performance.
The set generated by CMI may contain many redundant and over-ﬁtting cross features, which leads to poor performance on some datasets. [Table] [Table]
Obviously, our proposed CrossGO model performs better than DNN on all datasets.
That is to say, a simple LR model empowered by our generated cross features always achieves better performance comparing with DNN.
Moreover, Figure 2 demonstrates that we only need a small portion of our generated cross feature ﬁelds to make LR model achieving approximate DNN performance.
And as continuing adding our generated cross feature ﬁelds, the performances of LR with CrossGO are increasing.
Thus, user can decide the number of generated cross feature ﬁelds to add according to their expectation of testing performance.
To be noted that as mentioned in section 3, the number of generated cross feature ﬁelds has little impact on the time cost of CrossGO.
In summary, these results strongly veriﬁes the effectiveness, efﬁciency and ﬂexibility of our approach.
Meanwhile, our proposed approach outperforms AutoCross on all datasets.
On the narrow datasets, the results in Table 3 demonstrate that AutoCross is inferior to our proposed CrossGO model.
On the wide datasets, the candidate set of cross features is inevitable to be extremely large, and samples for evaluating each candidate cross feature ﬁeld are too little.
Thus, AutoCross performs poorly on wide datasets, and can not achieve similar perfromances as DNN.
Our porposed CrossGO model can signiﬁcantly outperform AutoCross on wide datasets. Moreover, according to Table 5, CrossGO is more efﬁcient than AutoCross, especially on wide datasets.
And wider the dataset, larger our advantage.
4. I NTERPRETABILITY OF C ROSS GO
As CrossGO can explicitly generate cross features, CrossGO has good interpretability.
In order to make the interpretability of CrossGO comprehensible, we choose a representative case from dataset MovieLens in the process of testing.
The ﬁve most important cross feature ﬁelds added to the ﬁnal set are { is horror, recent 50 action cnt } , { is action, recent 50 crime mean rate } , { is horror, recent 50 thriller mean rate } , { is children’s, recent 30 fantasy cnt } and { is action, recent 30 crime mean rate } .
Obviously, these ﬁve cross feature ﬁelds are complete comprehensible to human.
And from a logical point of view, they are highly likely to make data more linear-separable.
For example, { is horror, recent 50 action cnt } implies that for predicting current rating, it is a good choice to consider both whether current ﬁlm is horror and whether current user often watches action ﬁlms.
And { is action, recent 50 crime mean rate } implies that it is also a good choice to consider both whether current ﬁlm is action and whether current user likes crime ﬁlms.
In practice, adding these cross features signiﬁcantly improves the performance of LR model.
This demonstrates that our proposed CrossGO model is interpretable and effective.
C ONCLUSION
In this paper, we deﬁne the interpretation inconsistency in DNN for the ﬁrst time.
According to the interpretation inconsistency, we propose a novel CrossGO method.
CrossGO can generate a compact candidate set of cross feature ﬁelds, with extremely small amount compared to the whole set of second-order and high-order cross feature ﬁelds in a dataset.
Based on the corresponding performances in the validation set, useful cross feature ﬁelds can be directly ranked and selected from our compact candidate set. The whole process of learning feature crossing can be done via simply training a DNN model and a LR model. Extensive experiments have been conducted on several real-world datasets. Cross features generated by CrossGO can empower a simple LR model
14 &URVV)HDWXUH)LHOGV 92.
$8& &URVV*2 '1 (a) Testing performances on Adult w.r.t.
the number of our generated cross features.
14 &URVV)HDWXUH)LHOGV 87.
$8& &URVV*2 '1 (
[Figure]
performances on Employee w
[Figure]
num
[Figure]
of our generated cross features.
achieving approximate or even better performances comparing with complex DNN models, as well as the state-of-the-art feature crossing method, i e., AutoCross, which greedily explores the whole set of cross feature ﬁelds.
Moreover, cross features generated by CrossGO have great interpretability