Tabular Data Binary Classification: All Tips and Tricks from 5 Kaggle Competitions
Tags: machinelearning, datascience, python
Published: 2020-07-22 09:37:19

*This article was originally written by [Shahul Es](https://www.linkedin.com/in/shahules/) and posted on the [Neptune blog](https://neptune.ai/blog/tabular-data-binary-classification-tips-and-tricks-from-5-kaggle-competitions?utm_source=devto&utm_medium=crosspost&utm_campaign=blog-tabular-data-binary-classification-tips-and-tricks-from-5-kaggle-competitions).*

---

In this article, I will discuss some great tips and tricks to improve the performance of your structured data binary classification model. These tricks are obtained from solutions of some of Kaggle’s top tabular data competitions. Without much lag, let’s begin.

These are the five competitions that I have gone through to create this article:

* [Home credit default risk](https://www.kaggle.com/c/home-credit-default-risk/)
* [Santander Customer Transaction Prediction](https://www.kaggle.com/c/santander-customer-transaction-prediction/notebooks)
* [VSB Power Line Fault Detection](https://www.kaggle.com/c/vsb-power-line-fault-detection/overview/evaluation)
* [Microsoft Malware Prediction](https://www.kaggle.com/c/microsoft-malware-prediction/overview)
* [IEEE-CIS Fraud Detection](https://www.kaggle.com/c/ieee-fraud-detection/overview/evaluation/)

# Dealing with larger datasets
One issue you might face in any machine learning competition is the size of your data set. If the size of your data is large, that is 3GB + for kaggle kernels and more basic laptops you could find it difficult to load and process with limited resources. Here is the link to some of the articles and kernels that I have found useful in such situations.

* Faster [data loading with pandas](https://www.kaggle.com/c/home-credit-default-risk/discussion/59575).
* Data compression techniques to [reduce the size of data by 70%](https://www.kaggle.com/nickycan/compress-70-of-dataset).
* Optimize the memory by [reducing the size of some attributes](https://www.kaggle.com/shrutimechlearn/large-data-loading-trick-with-ms-malware-data). 
* Use open-source libraries such as [Dask to read and manipulate the data](https://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask), it performs parallel computing and saves up memory space. 
* Use [cudf](https://github.com/rapidsai/cudf).
* Convert data to [parquet](https://arrow.apache.org/docs/python/parquet.html) format.
* Converting data to [feather](https://medium.com/@snehotosh.banerjee/feather-a-fast-on-disk-format-for-r-and-python-data-frames-de33d0516b03) format.
* Reducing memory usage for [optimizing RAM](https://www.kaggle.com/mjbahmani/reducing-memory-size-for-ieee).

# Data exploration
Data exploration always helps to better understand the data and gain insights from it. Before starting to develop machine learning models, top competitors always read/do a lot of exploratory data analysis for the data. This helps in feature engineering and cleaning of the data.

* EDA for Microsoft [malware detection](https://www.kaggle.com/youhanlee/my-eda-i-want-to-see-all).
* Time Series [EDA for malware detection](https://www.kaggle.com/cdeotte/time-split-validation-malware-0-68).
* Complete [EDA for home credit loan prediction](https://www.kaggle.com/codename007/home-credit-complete-eda-feature-importance).
* Complete [EDA for Santader prediction](https://www.kaggle.com/gpreda/santander-eda-and-prediction).
* EDA for [VSB Power Line Fault Detection](https://www.kaggle.com/go1dfish/basic-eda).

# Data preparation
After data exploration, the first thing to do is to use those insights to prepare the data. To tackle issues like class imbalance, encoding categorical data, etc. Let’s see the methods used to do it.

* Methods to [tackle class imbalance](https://www.kaggle.com/shahules/tackling-class-imbalance).
* Data augmentation by [Synthetic Minority Oversampling Technique](https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/).
* [Fast inplace shuffle for augmentation](https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/).
* Finding [synthetic samples in the dataset](https://www.kaggle.com/yag320/list-of-fake-samples-and-public-private-lb-split).
* [Signal denoising](https://www.kaggle.com/jackvial/dwt-signal-denoising) used in signal processing competitions.
* Finding [patterns of missing data](https://www.kaggle.com/jpmiller/patterns-of-missing-data).
* Methods to handle [missing data](https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779).
* An overview of various [encoding techniques for categorical data](https://www.kaggle.com/shahules/an-overview-of-encoding-techniques).
* Building [model to predict missing values](https://www.kaggle.com/c/home-credit-default-risk/discussion/64598).
* Random [shuffling of data](https://www.kaggle.com/brandenkmurray/randomly-shuffled-data-also-work) to create new synthetic training set.

# Feature engineering
Next, you can check the most popular feature and feature engineering techniques used in these top kaggle competitions. The feature engineering part varies from problem to problem depending on the domain.

* Target [encoding cross validation](https://medium.com/@pouryaayria/k-fold-target-encoding-dfe9a594874b) for better encoding.
* Entity embedding to [handle categories](https://www.kaggle.com/abhishek/entity-embeddings-to-handle-categories). 
* Encoding [cyclic features for deep learning](https://www.kaggle.com/avanwyk/encoding-cyclical-features-for-deep-learning).
* Manual [feature engineering methods](https://www.kaggle.com/willkoehrsen/introduction-to-manual-feature-engineering).
* Automated feature engineering techniques [using featuretools](https://www.kaggle.com/willkoehrsen/automated-feature-engineering-basics).
* Top hard crafted features used in [microsoft malware detection](https://www.kaggle.com/sanderf/7th-place-solution-microsoft-malware-prediction).
* Denoising NN for [feature extraction](https://towardsdatascience.com/applied-deep-learning-part-3-autoencoders-1c083af4d798).
* Feature engineering [using RAPIDS framework](https://www.kaggle.com/cdeotte/rapids-feature-engineering-fraud-0-96/).
* Things to remember while processing [features using LGBM](https://www.kaggle.com/c/ieee-fraud-detection/discussion/108575).
* [Lag features and moving averages](https://www.kaggle.com/c/home-credit-default-risk/discussion/64593).
* [Principal component analysis](https://medium.com/machine-learning-researcher/dimensionality-reduction-pca-and-lda-6be91734f567) for dimensionality reduction.
* LDA for [dimensionality reduction](https://medium.com/machine-learning-researcher/dimensionality-reduction-pca-and-lda-6be91734f567).
* Best hand crafted LGBM features for [microsoft malware detection](https://www.kaggle.com/c/microsoft-malware-prediction/discussion/85157).
* Generating [frequency features](https://www.kaggle.com/philippsinger/frequency-features-without-test-data-information).
* Dropping variables with [different train and test distribution](https://www.kaggle.com/bogorodvo/lightgbm-baseline-model-using-sparse-matrix).
* [Aggregate time series features](https://www.kaggle.com/c/home-credit-default-risk/discussion/64593) for home credit competition.
* [Time Series features](https://www.kaggle.com/c/home-credit-default-risk/discussion/64593) used in home credit default risk.
* Scale,Standardize and [normalize with sklearn](https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02).
* Handcrafted features for [Home default risk competition](https://www.kaggle.com/c/home-credit-default-risk/discussion/57750).
* Handcrafted  [features used in Santander Transaction Prediction.](https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/89070)

# Feature selection 
After generating many features from your data, you need to decide which all features to use in your model to get the maximum performance out of your model. This step also includes identifying the impact each feature is having on your model. Let’s see some of the most popular feature selection methods.

* Six ways to do [features selection using sklearn](https://www.kaggle.com/sz8416/6-ways-for-feature-selection).
* [Permutation feature importance](https://www.kaggle.com/c/ieee-fraud-detection/discussion/107877#latest-635386).
* [Adversarial feature validation](https://www.kaggle.com/tunguz/adversarial-ieee/).
* Feature selection using [null importance](https://www.kaggle.com/ogrellier/feature-selection-with-null-importances).
* Tree explainer using [SHAP](https://github.com/slundberg/shap).
* DeepNN explainer using [SHAP](https://github.com/slundberg/shap).

# Modeling
After handcrafting and selecting your features, you should choose the right Machine learning algorithm to make your prediction. These are the collection of some of the most used ML models in structured data classification challenges.

* [Random forest classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).
* XGBoost : [Gradient boosted decision trees](https://xgboost.readthedocs.io/en/latest/).
* [LightGBM](https://lightgbm.readthedocs.io/en/latest/) for distributed and faster training.
* [CatBoost](https://catboost.ai/docs/concepts/about.html) to handle categorical data.
* [Naive bayes](https://www.kaggle.com/cdeotte/modified-naive-bayes-santander-0-899) classifier.
* [Gaussian naive bayes](https://www.kaggle.com/blackblitz/gaussian-naive-bayes) model.
* [LGBM + CNN  model](https://www.kaggle.com/nawidsayed/lightgbm-and-cnn-3rd-place-solution/notebook) used in 3rd place solution of Santander Customer Transaction Prediction
* [Knowledge distillation](https://www.kaggle.com/mathormad/knowledge-distillation-with-nn-rankgauss) in Neural Network.
* [Follow the regularized leader](https://medium.com/@dhirajreddy13/factorization-machines-and-follow-the-regression-leader-for-dummies-7657652dce69) method.
* Comparison between [LGB boosting methods](https://www.kaggle.com/c/home-credit-default-risk/discussion/60921) (goss, gbdt and dart).
* [NN + focal loss experiment](https://www.kaggle.com/abazdyrev/keras-nn-focal-loss-experiments).
* Keras [NN with timeseries splitter](https://www.kaggle.com/ryches/keras-nn-starter-w-time-series-split).
* 5th place [NN architecture with code](https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/88929) for Santander Transaction prediction.

# Hyperparameter tuning
* LGBM [hyperparameter tuning](https://www.kaggle.com/mlisovyi/lightgbm-hyperparameter-optimisation-lb-0-761) methods.
* Automated [model tuning](https://www.kaggle.com/willkoehrsen/automated-model-tuning) methods.
* Parametre tuning with [hyper plot](https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt).
* [Bayesian optimization](http://krasserm.github.io/2018/03/21/bayesian-optimization/) for hyperparameter tuning.
* [Gpyopt Hyperparameter Optimisation](https://www.kaggle.com/nicapotato/gpyopt-hyperparameter-optimisation-gpu-lgbm).

# Evaluation
Choosing a suitable validation strategy is very important to avoid huge shake-ups or poor performance of the model in the private test set. 

The traditional 80:20 split wouldn’t work for many cases. Cross-validation works in most cases over the traditional single train-validation split to estimate the model performance. 

There are different variations of KFold cross-validation such as group k-fold that should be chosen accordingly.

* [K-fold cross-validation](https://machinelearningmastery.com/k-fold-cross-validation/).
* [Stratified KFold cross-validation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html).
* [Group KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html)
*  [Adversarial validation](https://neptune.ai/blog/tabular-data-binary-classification-tips-and-tricks-from-5-kaggle-competitions) to check if train and test distributions are similar or not.
* [Time Series split](https://www.kaggle.com/cdeotte/time-split-validation-malware-0-68) validation.
* Extensive [time series splitter](https://www.kaggle.com/mpearmain/extended-timeseriessplitter).

*Note:*

*There are various metrics that you can use to evaluate the performance of your tabular models. A bunch of useful [classification metrics are listed and explained here](https://neptune.ai/blog/evaluation-metrics-binary-classification?utm_source=hackernoon&utm_medium=crosspost&utm_campaign=blog-tabular-data-binary-classification-tips-and-tricks-from-5-kaggle-competitions).*

# Other training tricks
* [GPU acceleration](https://www.kaggle.com/vinhnguyen/gpu-acceleration-for-lightgbm) for LGBM.
* [Use the GPU efficiently](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/89498).
* [Free keras memory](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/89498).
* [Save and load models](https://machinelearningmastery.com/save-load-keras-deep-learning-models/) to save runtime and memory.

# Ensemble
If you’re in the competing environment one won’t get to the top of the leaderboard without ensembling. Selecting the appropriate ensembling/stacking method is very important to get the maximum performance out of your models. 

Let’s see some of the popular ensembling techniques used in kaggle competitions:

* [Weighted average ensemble](https://machinelearningmastery.com/weighted-average-ensemble-for-deep-learning-neural-networks/).
* [Stacked generalization ensemble](https://machinelearningmastery.com/stacking-ensemble-for-deep-learning-neural-networks/).
* [Out of folds predictions](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52224).
* [Blending with linear regression](https://www.kaggle.com/suicaokhoailang/blending-with-linear-regression-0-688-lb).
* Use [optuna](https://github.com/optuna/optuna) to determine blending weights.
* [Power average ensemble](https://medium.com/data-design/reaching-the-depths-of-power-geometric-ensembling-when-targeting-the-auc-metric-2f356ea3250e).
* [Power 3.5 blending strategy](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/100661).
* [Blending diverse models](https://www.kaggle.com/c/microsoft-malware-prediction/discussion/80368#478088).
* Different [stacking approaches](https://www.kaggle.com/stocks/stacking-higher-and-higher).
* [AUC weight optimization](https://www.kaggle.com/lucaskg/20th-solution-part-2-auc-weight-optimization).
* [Geometric mean](https://www.kaggle.com/paulorzp/gmean-of-low-correlation-lb-0-952x) for low correlation predictions.
* Weighted [rank average](https://www.kaggle.com/shaz13/magic-of-weighted-average-rank-0-80/input).

#Final thoughts
In this article, you saw many popular and effective ways to improve the performance of your tabular data binary classification model. Hopefully, you will find them useful in your projects

---

*This article was originally written by [Shahul Es](https://www.linkedin.com/in/shahules/) and posted on the [Neptune blog](https://neptune.ai/blog/tabular-data-binary-classification-tips-and-tricks-from-5-kaggle-competitions?utm_source=devto&utm_medium=crosspost&utm_campaign=blog-tabular-data-binary-classification-tips-and-tricks-from-5-kaggle-competitions), where you can find more in-depth articles for machine learning practitioners.*