Repository: EpistasisLab/tpot

Issue #739: TPOTClassifier.set_params doesn't follow scikit-learn estimator API
Created: 2018-08-09T14:01:13Z
Typically, calling old_estimator.set_params(new_param=foo) will copy all the other parameters from the old estimator, and only update new_param. Currently, TPOT will __init__ with the kwargs passed to new_param, and use the default or everything else.
Context of the issue
See http://scikit-learn.org/stable/developers/contributing.html#estimators and http://scikit-learn.org/stable/developers/contributing.html#get-params-and-set-params.
Process to reproduce the issue
clf = TPOTClassifier(random_state=0)
clf.set_params(n_jobs=-1).random_state
# output is None, the default random_state for TPOTClassifier
Expected result
>>> clf.set_params(n_jobs=-1).random_state
0

Possible fix
Hopefully just delete the custom set_params :) That may not be possible though, as it looks like there's some parameter handling logic in the __init__. That logic would need to be either moved to fit (and transform) or a helper method.

Duplicate issue #748: TPOTClassifier parameter scoring bug
Created: 2018-08-27T23:54:15Z
I set scoring to "roc_auc" among others parameters:
tpot = TPOTClassifier(scoring='roc_auc', max_time_mins=120, verbosity=3, random_state=RANDOM_STATE, n_jobs=-1)
print(tpot)

but when i print the object configuration, the scoring is None:
30 operators have been imported by TPOT.
TPOTClassifier(config_dict={'sklearn.naive_bayes.GaussianNB': {}, 'sklearn.naive_bayes.BernoulliNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.naive_bayes.MultinomialNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.tree.DecisionT....3 , 0.35, 0.4 , 0.45, 0.5 , 0.55,
0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95, 1.  ])}}}},
crossover_rate=0.1, cv=5, disable_update_check=False,
early_stop=None, generations=1000000, max_eval_time_mins=5,
max_time_mins=120, memory=None, mutation_rate=0.9, n_jobs=4,
offspring_size=100, periodic_checkpoint_folder=None,
population_size=100, random_state=42, scoring=None, subsample=1.0,
verbosity=3, warm_start=False)
i've also tried with others estimators from: https://epistasislab.github.io/tpot/api/ such as: 'balanced_accuracy', 'f1' and the result is still None.
Context of the issue
I'm running the code in:

conda version 4.4.8
python v 3.6.5
tpot v 0.9.3
sklearn v 0.19.1

I had also replicated the process in google colab (python 3) and the same issue is happening.
Process to reproduce the issue

User creates TPOT instance and change the parameter "scoring" from default
User print TPOT object

Expected result
To change the object default configuration. In the previous example:
30 operators have been imported by TPOT.
TPOTClassifier(config_dict={'sklearn.naive_bayes.GaussianNB': {}, 'sklearn.naive_bayes.BernoulliNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.naive_bayes.MultinomialNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.tree.DecisionT....3 , 0.35, 0.4 , 0.45, 0.5 , 0.55,
0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95, 1.  ])}}}},
crossover_rate=0.1, cv=5, disable_update_check=False,
early_stop=None, generations=1000000, max_eval_time_mins=5,
max_time_mins=120, memory=None, mutation_rate=0.9, n_jobs=4,
offspring_size=100, periodic_checkpoint_folder=None,
population_size=100, random_state=42, scoring=roc_auc, subsample=1.0,
verbosity=3, warm_start=False)

Closing comment (2018-08-28T01:26:00Z): Thank you for reporting this issue. TPOT internally didn't use self.scoring but uses self.scoring_function generated by _setup_scoring_function and scoring parameter in pipeline evaluation, so I think TPOT uses the right scoring function as expected during optimization. But, indeed, this is a API bug related to #739. I think I fixed this API bug in PR #740 and we will fixed this issue in the next version of TPOT.