Repository: scikit-learn/scikit-learn
File: examples/ensemble/plot_stack_predictors.py (Python)

Code before:

plt.suptitle("Prediction errors of base versus stacked predictors", y=1)
plt.tight_layout()
plt.subplots_adjust(top=0.9)
plt.show()

# %%
# Even if the scores overlap considerably after cross-validation, the predictions
# from the stacked regressor are slightly better.
#
# Once fitted, we can inspect the coefficients (or meta-weights) of the trained
# `final_estimator_` (as long as it is a linear model). They reveal how much the
# individual estimators contribute to the the stacked regressor:

stacking_regressor.fit(X, y)
stacking_regressor.final_estimator_.coef_

# %%
# We see that in this case, the HGBT model is somewhat dominant, with the spline
# ridge also contributing meaningfully. The plain linear model does not add
# useful signal once those two are in place; with
# :class:`~sklearn.linear_model.RidgeCV` as the `final_estimator`, it is not
# dropped but gets a tiny negative weight that acts as a small correction to
# counter its residual bias.
#
# If we switch to :class:`~sklearn.linear_model.LassoCV` as the
# `final_estimator`, that small, unhelpful contribution is set exactly to zero,
# yielding a simpler blend of the spline ridge and HGBT models.

from sklearn.linear_model import LassoCV

stacking_regressor = StackingRegressor(estimators=estimators, final_estimator=LassoCV())
stacking_regressor.fit(X, y)
stacking_regressor.final_estimator_.coef_

# %%
# How to mimic SuperLearner with scikit-learn
# ###########################################
#
# The `SuperLearner` [Polley2010]_ is a stacking strategy implemented as `an R
# package <https://cran.r-project.org/web/packages/SuperLearner/index.html>`_, but
# not available off-the-shelf in Python. It is closely related to the
# :class:`~sklearn.ensemble.StackingRegressor`, as both train the meta-model on
# out-of-fold predictions from the base estimators.
#
# The key difference is that `SuperLearner` estimates a convex set of
# meta-weights (non-negative and summing to 1) and omits an intercept; by
# contrast, :class:`~sklearn.ensemble.StackingRegressor` uses an unconstrained
# meta-learner with an intercept by default (and can optionally include raw
# features via passthrough).
#

Review comment (suggestion):

```suggestion
# If we use :class:`~sklearn.linear_model.LassoCV` as the
```

Code after:
plt.suptitle("Prediction Errors of Base versus Stacked Predictors", y=1)
plt.tight_layout()
plt.subplots_adjust(top=0.9)
plt.show()

# %%
# Even if the scores overlap considerably after cross-validation, the predictions
# from the stacked regressor are slightly better.
#
# Once fitted, we can inspect the coefficients (or meta-weights) of the trained
# `final_estimator_` (as long as it is a linear model). They reveal how much the
# individual estimators contribute to the the stacked regressor:

stacking_regressor.fit(X, y)
stacking_regressor.final_estimator_.coef_

# %%
# We see that in this case, the HGBT model dominates, with the spline
# ridge also contributing meaningfully. The plain linear model does not add
# useful signal once those two are included; with
# :class:`~sklearn.linear_model.RidgeCV` as the `final_estimator`, it is not
# dropped, but receives a small negative weight to correct its residual bias.
#
# If we use :class:`~sklearn.linear_model.LassoCV` as the
# `final_estimator`, that small, unhelpful contribution is set exactly to zero,
# yielding a simpler blend of the spline ridge and HGBT models.

from sklearn.linear_model import LassoCV

stacking_regressor = StackingRegressor(estimators=estimators, final_estimator=LassoCV())
stacking_regressor.fit(X, y)
stacking_regressor.final_estimator_.coef_

# %%
# How to mimic SuperLearner with scikit-learn
# ###########################################
#
# The `SuperLearner` [Polley2010]_ is a stacking strategy implemented as `an R
# package <https://cran.r-project.org/web/packages/SuperLearner/index.html>`_, but
# not available off-the-shelf in Python. It is closely related to the
# :class:`~sklearn.ensemble.StackingRegressor`, as both train the meta-model on
# out-of-fold predictions from the base estimators.
#
# The key difference is that `SuperLearner` estimates a convex set of
# meta-weights (non-negative and summing to 1) and omits an intercept; by
# contrast, :class:`~sklearn.ensemble.StackingRegressor` uses an unconstrained
# meta-learner with an intercept by default (and can optionally include raw
# features via passthrough).
#
# Without an intercept, the meta-weights are directly interpretable as
# fractional contributions to the final prediction.