Repository: shap/shap

Issue #1061: shap.TreeExplainer fails when CatBoost has a categorical feature.
Created: 2020-02-21T15:09:00Z
Shapley isn't working for some reason and I'm getting incredibly obscure errors. Even though CatBoost is throwing the error it must be because of shap as the CatBoost models still work when called directly. I think the problem is because shap calls get_feature_importance() with a deprecated argument.
The error is
Traceback (most recent call last):
  File "C:/Users/antho/.PyCharm2019.3/config/scratches/scratch_5.py", line 17, in <module>
    shap_values = explainer.shap_values(x)
  File "C:\Users\antho\Miniconda3\lib\site-packages\shap\explainers\tree.py", line 247, in shap_values
    phi = self.model.original_model.get_feature_importance(data=X, fstr_type='ShapValues')
  File "C:\Users\antho\Miniconda3\lib\site-packages\catboost\core.py", line 2322, in get_feature_importance
    fstr, feature_names = self._calc_fstr(type, data, thread_count, verbose, shap_mode)
  File "C:\Users\antho\Miniconda3\lib\site-packages\catboost\core.py", line 1287, in _calc_fstr
    return self._object._calc_fstr(type.name, pool, thread_count, verbose, shap_mode)
  File "_catboost.pyx", line 4084, in _catboost._CatBoost._calc_fstr
  File "_catboost.pyx", line 4114, in _catboost._CatBoost._calc_fstr
_catboost.CatBoostError: c:/goagent/pipelines/buildmaster/catboost.git/catboost/libs/data/model_dataset_compatibility.cpp:47: Feature CHAS is Categorical in model but marked different in the dataset

Process finished with exit code 1
it's probably due to calling a deprecated argument fstr_type='ShapValues'...
File "C:\Users\antho\Miniconda3\lib\site-packages\shap\explainers\tree.py", line 247, in shap_values
    phi = self.model.original_model.get_feature_importance(data=X, fstr_type='ShapValues')
The code required to get this error
import shap
import pandas as pd
from catboost import CatBoostRegressor
from sklearn.datasets import load_boston

bunch = load_boston()
x, y = load_boston(True)
x = pd.DataFrame(x, columns=bunch.feature_names)
x['CHAS'] = x['CHAS'].astype(str)
print(x)

model = CatBoostRegressor(1000, cat_features=['CHAS'], verbose=False)
model.fit(x, y)

if __name__ == '__main__':
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(x)

Duplicate issue #1305: Cant produce dependece plots for CatBoost model with categorical data
Created: 2020-07-09T14:41:41Z
I've trained CatBoost on Income Adult UCI data set. It has several categoricals. I leave categorical unconverted, because CatBoost do the job. When trying to create a dependence plot I got the error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-100-c416e23e6fdd> in <module>
----> 1 shap.dependence_plot("workclass", shap_values, X_test, interaction_index=None)

~/sandbox/mljar-examples/Income_classification/venv4/lib/python3.7/site-packages/shap/plots/dependence.py in dependence_plot(ind, shap_values, features, feature_names, display_features, interaction_index, color, axis_color, cmap, dot_size, x_jitter, alpha, title, xmin, xmax, ax, show)
    154     oinds = np.arange(shap_values.shape[0]) # we randomize the ordering so plotting overlaps are not related to data ordering
    155     np.random.shuffle(oinds)
--> 156     xv = features[oinds, ind].astype(np.float64)
    157     xd = display_features[oinds, ind]
    158     s = shap_values[oinds, ind]

ValueError: could not convert string to float: 'Private'
CatBoost version: 0.23.1
shap version 0.35.0
The error is only for categorical features, for numerical features plots are produced.
The code to reproduce:
import numpy as np
import pandas as pd
import catboost
import shap
from sklearn.model_selection import train_test_split

df = pd.read_csv(
    "https://raw.githubusercontent.com/pplonski/datasets-for-start/master/adult/data.csv",
    skipinitialspace=True,
)

X = df[df.columns[:-1]]
y = df["income"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True, stratify=y, random_state=1)
# fill missing values
X_train = X_train.fillna(X_train.mode().iloc[0])
X_test = X_test.fillna(X_train.mode().iloc[0])

model = catboost.CatBoostClassifier(verbose=False)
model.fit(X_train, y_train, cat_features=['workclass', 'education', 
       'marital-status', 'occupation', 'relationship', 'race', 'sex',
       'native-country'])

# SHAP
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# works on numerical feature
shap.dependence_plot("age", shap_values, X_test, interaction_index=None)

# procudes the error
shap.dependence_plot("workclass", shap_values, X_test, interaction_index=None)

Closing comment (2020-07-09T14:42:57Z): Related to #1061 and mljar/mljar-supervised#114