Sklearn 集成学习sklearn.ensemble提供 Bagging、Boosting、Voting、Stacking 等集成方法。 Bagging 类方法1.RandomForestClassifier— 随机森林分类器 ⭐fromsklearn.ensembleimportRandomForestClassifier modelRandomForestClassifier(n_estimators100,# 树的数量criteriongini,# gini 或 entropy 或 log_lossmax_depthNone,# 树的最大深度min_samples_split2,# 内部节点再划分所需最小样本数min_samples_leaf1,# 叶节点最少样本数min_weight_fraction_leaf0.0,max_featuressqrt,# 每棵树随机选择的特征数# sqrt, log2, None, int, float(比例)max_leaf_nodesNone,# 最大叶节点数min_impurity_decrease0.0,bootstrapTrue,# 是否 bootstrap 采样oob_scoreFalse,# 是否计算袋外分数n_jobs-1,random_state42,verbose0,warm_startFalse,class_weightNone,# None, balanced, balanced_subsample, dictccp_alpha0.0,# 最小代价复杂度剪枝参数max_samplesNone# bootstrap 样本数None全部)model.fit(X,y)# 核心属性print(model.feature_importances_)# 特征重要性 ⭐print(model.oob_score_)# 袋外分数oob_scoreTrueprint(model.estimators_)# 所有决策树列表print(model.classes_)# 类别print(model.n_classes_)# 类别数print(model.n_features_in_)# 特征数# 预测y_predmodel.predict(X)y_probmodel.predict_proba(X)y_log_probamodel.predict_log_proba(X)# 应用到新数据model.apply(X)# 返回每个样本每棵树的叶节点索引2.RandomForestRegressor— 随机森林回归器 ⭐fromsklearn.ensembleimportRandomForestRegressor modelRandomForestRegressor(n_estimators100,criterionsquared_error,# squared_error,absolute_error,friedman_mse,poissonmax_depthNone,min_samples_split2,min_samples_leaf1,max_features1.0,# 回归建议用 sqrt 或 None(n_features)bootstrapTrue,oob_scoreFalse,n_jobs-1,random_state42,max_samplesNone)model.fit(X,y)print(model.feature_importances_)# 特征重要性print(model.oob_prediction_)# 袋外预测oob_scoreTrue3.ExtraTreesClassifier/ExtraTreesRegressor— 极端随机树与随机森林的区别分裂阈值完全随机而不是选择最优。fromsklearn.ensembleimportExtraTreesClassifier,ExtraTreesRegressor modelExtraTreesClassifier(n_estimators100,criteriongini,max_featuressqrt,bootstrapFalse,# 默认不用 bootstrapoob_scoreFalse,# 默认不开n_jobs-1,random_state42)model.fit(X,y)4.BaggingClassifier/BaggingRegressor— 通用 Bagging ⭐fromsklearn.ensembleimportBaggingClassifier,BaggingRegressor modelBaggingClassifier(estimatorNone,# NoneDecisionTreeClassifier, 或传入任意估计器n_estimators10,max_samples1.0,# 每轮采样的样本比例max_features1.0,# 每轮采样的特征比例bootstrapTrue,# True有放回, False无放回(pasting)bootstrap_featuresFalse,oob_scoreFalse,warm_startFalse,n_jobs-1,random_state42)# 使用 SVM 作为基学习器fromsklearn.svmimportSVC bagging_svmBaggingClassifier(estimatorSVC(),n_estimators10,max_samples0.5,max_features0.5) Boosting 类方法1.GradientBoostingClassifier— 梯度提升分类器 ⭐fromsklearn.ensembleimportGradientBoostingClassifier modelGradientBoostingClassifier(losslog_loss,# log_loss 或 exponentiallearning_rate0.1,# 学习率步长n_estimators100,# 提升阶段数树的数量subsample1.0,# 每棵树的样本比例stochastic GBcriterionfriedman_mse,min_samples_split2,min_samples_leaf1,min_weight_fraction_leaf0.0,max_depth3,# 树的深度较大的值会增加交互min_impurity_decrease0.0,initNone,# 初始估计器或 zerorandom_state42,max_featuresNone,# 特征子采样verbose0,max_leaf_nodesNone,warm_startFalse,validation_fraction0.1,# 早停验证集比例n_iter_no_changeNone,# 早停容忍轮数tol1e-4,ccp_alpha0.0)model.fit(X,y)# 核心属性print(model.feature_importances_)# 特征重要性print(model.train_score_)# 每轮训练分数print(model.oob_improvement_)# 袋外改善subsample 1 时print(model.estimators_)# 每棵树的列表lenn_estimators# 分阶段预测y_pred_stagedlist(model.staged_predict(X_test))y_prob_stagedlist(model.staged_predict_proba(X_test))2.GradientBoostingRegressor— 梯度提升回归器 ⭐fromsklearn.ensembleimportGradientBoostingRegressor modelGradientBoostingRegressor(losssquared_error,# squared_error,absolute_error,huber,quantilelearning_rate0.1,n_estimators100,subsample1.0,criterionfriedman_mse,max_depth3,alpha0.9,# huber 和 quantile 损失的参数random_state42)model.fit(X,y)# 分阶段预测y_pred_iterablemodel.staged_predict(X)# 生成器3.AdaBoostClassifier/AdaBoostRegressor— AdaBoostfromsklearn.ensembleimportAdaBoostClassifier,AdaBoostRegressor modelAdaBoostClassifier(estimatorNone,# NoneDecisionTreeClassifier(max_depth1)n_estimators50,learning_rate1.0,algorithmSAMME.R,# SAMME 或 SAMME.R需概率支持random_state42)model.fit(X,y)print(model.estimator_weights_)# 每轮权重print(model.estimator_errors_)# 每轮误差print(model.feature_importances_)4.HistGradientBoostingClassifier— 直方图梯度提升 ⭐基于直方图的快速实现原生支持缺失值和类别特征。fromsklearn.ensembleimportHistGradientBoostingClassifier modelHistGradientBoostingClassifier(losslog_loss,# log_loss 或 autolearning_rate0.1,max_iter100,# 等效于 n_estimatorsmax_leaf_nodes31,max_depthNone,min_samples_leaf20,l2_regularization0.0,max_bins255,# 连续特征的分箱数categorical_featuresNone,# 类别特征的索引列表monotonic_cstNone,# 单调约束interaction_cstNone,# 交互约束warm_startFalse,early_stoppingauto,# True, False, autoscoringloss,validation_fraction0.1,n_iter_no_change10,# 早停容忍轮数tol1e-7,verbose0,random_state42,class_weightNone)model.fit(X,y)print(model.feature_importances_)print(model.n_iter_)# 实际迭代数早停后5.HistGradientBoostingRegressor— 直方图梯度提升回归器fromsklearn.ensembleimportHistGradientBoostingRegressor modelHistGradientBoostingRegressor(losssquared_error,quantile0.5,# quantile 损失时使用learning_rate0.1,max_iter100,max_leaf_nodes31,min_samples_leaf20,l2_regularization0.0,max_bins255,monotonic_cstNone,early_stoppingauto,random_state42)model.fit(X,y)6.VotingClassifier/VotingRegressor— 投票集成 ⭐fromsklearn.ensembleimportVotingClassifierfromsklearn.linear_modelimportLogisticRegressionfromsklearn.svmimportSVCfromsklearn.treeimportDecisionTreeClassifier votingVotingClassifier(estimators[(lr,LogisticRegression()),(svc,SVC(probabilityTrue)),# soft voting 需要 predict_proba(dt,DecisionTreeClassifier())],votingsoft,# hard(多数投票) 或 soft(概率加权)weights[1,2,1],# 各模型权重soft voting 时n_jobs-1,flatten_transformTrue# transform 方法)voting.fit(X_train,y_train)y_predvoting.predict(X_test)# 查看各模型的类别概率probsvoting.transform(X_test)# 属性print(voting.named_estimators_)# 模型字典print(voting.estimators_)# 模型列表回归版本:fromsklearn.ensembleimportVotingRegressorfromsklearn.linear_modelimportLinearRegressionfromsklearn.ensembleimportRandomForestRegressor voting_regVotingRegressor(estimators[(lr,LinearRegression()),(rf,RandomForestRegressor())],weights[1,2])voting_reg.fit(X_train,y_train)7.StackingClassifier/StackingRegressor— 堆叠集成 ⭐fromsklearn.ensembleimportStackingClassifierfromsklearn.linear_modelimportLogisticRegressionfromsklearn.svmimportSVCfromsklearn.treeimportDecisionTreeClassifier stackingStackingClassifier(estimators[(svc,SVC(probabilityTrue)),(dt,DecisionTreeClassifier()),],final_estimatorLogisticRegression(),# 元学习器cv5,# 交叉验证折叠数生成第一层预测stack_methodauto,# auto,predict_proba,decision_function,predictn_jobs-1,passthroughFalse,# 是否将原始 X 也传给元学习器verbose0)stacking.fit(X_train,y_train)y_predstacking.predict(X_test)# 属性print(stacking.final_estimator_)# 元学习器print(stacking.named_estimators_)# 基学习器print(stacking.stack_method_)# 实际使用的 stack_method回归版本:fromsklearn.ensembleimportStackingRegressorfromsklearn.linear_modelimportRidge stacking_regStackingRegressor(estimators[(rf,RandomForestRegressor()),(gb,GradientBoostingRegressor()),],final_estimatorRidge(),cv5,passthroughTrue)8.IsolationForest— 孤立森林异常检测fromsklearn.ensembleimportIsolationForest modelIsolationForest(n_estimators100,max_samplesauto,# 每棵树的样本数contaminationauto,# 异常比例float或 automax_features1.0,bootstrapFalse,n_jobs-1,random_state42,verbose0)model.fit(X)# 预测: 1正常, -1异常y_predmodel.predict(X)# 异常分数越低越异常scoresmodel.decision_function(X)print(model.offset_)# 决策偏移量# 异常分数转为概率scoresmodel.score_samples(X)# 负分数越高越正常 特征重要性所有树模型都支持特征重要性importpandasaspdimportnumpyasnpimportmatplotlib.pyplotaspltfromsklearn.ensembleimportRandomForestClassifier modelRandomForestClassifier(n_estimators100,random_state42)model.fit(X,y)# 获取特征重要性importancesmodel.feature_importances_ indicesnp.argsort(importances)[::-1]# 可视化plt.figure(figsize(10,6))plt.bar(range(X.shape[1]),importances[indices])plt.xticks(range(X.shape[1]),[fFeature{i}foriinindices],rotation90)plt.title(Feature Importances)plt.tight_layout()plt.show()基于置换的特征重要性模型无关:fromsklearn.inspectionimportpermutation_importance resultpermutation_importance(model,X_test,y_test,n_repeats10,random_state42,n_jobs-1)print(result.importances_mean)print(result.importances_std) 实践选择指南场景推荐模型通用分类/回归基线RandomForestClassifier/RandomForestRegressor追求准确率GradientBoostingClassifier/HistGradientBoostingClassifier大数据集 快速HistGradientBoostingClassifier多模型融合VotingClassifier/StackingClassifier自定义基学习器BaggingClassifier异常检测IsolationForest特征含缺失值HistGradientBoostingClassifier原生支持类别特征HistGradientBoostingClassifiercategorical_features[[sklearn-总览|← 返回总览]] Python-sklearn-集成学习
Sklearn 集成学习sklearn.ensemble提供 Bagging、Boosting、Voting、Stacking 等集成方法。 Bagging 类方法1.RandomForestClassifier— 随机森林分类器 ⭐fromsklearn.ensembleimportRandomForestClassifier modelRandomForestClassifier(n_estimators100,# 树的数量criteriongini,# gini 或 entropy 或 log_lossmax_depthNone,# 树的最大深度min_samples_split2,# 内部节点再划分所需最小样本数min_samples_leaf1,# 叶节点最少样本数min_weight_fraction_leaf0.0,max_featuressqrt,# 每棵树随机选择的特征数# sqrt, log2, None, int, float(比例)max_leaf_nodesNone,# 最大叶节点数min_impurity_decrease0.0,bootstrapTrue,# 是否 bootstrap 采样oob_scoreFalse,# 是否计算袋外分数n_jobs-1,random_state42,verbose0,warm_startFalse,class_weightNone,# None, balanced, balanced_subsample, dictccp_alpha0.0,# 最小代价复杂度剪枝参数max_samplesNone# bootstrap 样本数None全部)model.fit(X,y)# 核心属性print(model.feature_importances_)# 特征重要性 ⭐print(model.oob_score_)# 袋外分数oob_scoreTrueprint(model.estimators_)# 所有决策树列表print(model.classes_)# 类别print(model.n_classes_)# 类别数print(model.n_features_in_)# 特征数# 预测y_predmodel.predict(X)y_probmodel.predict_proba(X)y_log_probamodel.predict_log_proba(X)# 应用到新数据model.apply(X)# 返回每个样本每棵树的叶节点索引2.RandomForestRegressor— 随机森林回归器 ⭐fromsklearn.ensembleimportRandomForestRegressor modelRandomForestRegressor(n_estimators100,criterionsquared_error,# squared_error,absolute_error,friedman_mse,poissonmax_depthNone,min_samples_split2,min_samples_leaf1,max_features1.0,# 回归建议用 sqrt 或 None(n_features)bootstrapTrue,oob_scoreFalse,n_jobs-1,random_state42,max_samplesNone)model.fit(X,y)print(model.feature_importances_)# 特征重要性print(model.oob_prediction_)# 袋外预测oob_scoreTrue3.ExtraTreesClassifier/ExtraTreesRegressor— 极端随机树与随机森林的区别分裂阈值完全随机而不是选择最优。fromsklearn.ensembleimportExtraTreesClassifier,ExtraTreesRegressor modelExtraTreesClassifier(n_estimators100,criteriongini,max_featuressqrt,bootstrapFalse,# 默认不用 bootstrapoob_scoreFalse,# 默认不开n_jobs-1,random_state42)model.fit(X,y)4.BaggingClassifier/BaggingRegressor— 通用 Bagging ⭐fromsklearn.ensembleimportBaggingClassifier,BaggingRegressor modelBaggingClassifier(estimatorNone,# NoneDecisionTreeClassifier, 或传入任意估计器n_estimators10,max_samples1.0,# 每轮采样的样本比例max_features1.0,# 每轮采样的特征比例bootstrapTrue,# True有放回, False无放回(pasting)bootstrap_featuresFalse,oob_scoreFalse,warm_startFalse,n_jobs-1,random_state42)# 使用 SVM 作为基学习器fromsklearn.svmimportSVC bagging_svmBaggingClassifier(estimatorSVC(),n_estimators10,max_samples0.5,max_features0.5) Boosting 类方法1.GradientBoostingClassifier— 梯度提升分类器 ⭐fromsklearn.ensembleimportGradientBoostingClassifier modelGradientBoostingClassifier(losslog_loss,# log_loss 或 exponentiallearning_rate0.1,# 学习率步长n_estimators100,# 提升阶段数树的数量subsample1.0,# 每棵树的样本比例stochastic GBcriterionfriedman_mse,min_samples_split2,min_samples_leaf1,min_weight_fraction_leaf0.0,max_depth3,# 树的深度较大的值会增加交互min_impurity_decrease0.0,initNone,# 初始估计器或 zerorandom_state42,max_featuresNone,# 特征子采样verbose0,max_leaf_nodesNone,warm_startFalse,validation_fraction0.1,# 早停验证集比例n_iter_no_changeNone,# 早停容忍轮数tol1e-4,ccp_alpha0.0)model.fit(X,y)# 核心属性print(model.feature_importances_)# 特征重要性print(model.train_score_)# 每轮训练分数print(model.oob_improvement_)# 袋外改善subsample 1 时print(model.estimators_)# 每棵树的列表lenn_estimators# 分阶段预测y_pred_stagedlist(model.staged_predict(X_test))y_prob_stagedlist(model.staged_predict_proba(X_test))2.GradientBoostingRegressor— 梯度提升回归器 ⭐fromsklearn.ensembleimportGradientBoostingRegressor modelGradientBoostingRegressor(losssquared_error,# squared_error,absolute_error,huber,quantilelearning_rate0.1,n_estimators100,subsample1.0,criterionfriedman_mse,max_depth3,alpha0.9,# huber 和 quantile 损失的参数random_state42)model.fit(X,y)# 分阶段预测y_pred_iterablemodel.staged_predict(X)# 生成器3.AdaBoostClassifier/AdaBoostRegressor— AdaBoostfromsklearn.ensembleimportAdaBoostClassifier,AdaBoostRegressor modelAdaBoostClassifier(estimatorNone,# NoneDecisionTreeClassifier(max_depth1)n_estimators50,learning_rate1.0,algorithmSAMME.R,# SAMME 或 SAMME.R需概率支持random_state42)model.fit(X,y)print(model.estimator_weights_)# 每轮权重print(model.estimator_errors_)# 每轮误差print(model.feature_importances_)4.HistGradientBoostingClassifier— 直方图梯度提升 ⭐基于直方图的快速实现原生支持缺失值和类别特征。fromsklearn.ensembleimportHistGradientBoostingClassifier modelHistGradientBoostingClassifier(losslog_loss,# log_loss 或 autolearning_rate0.1,max_iter100,# 等效于 n_estimatorsmax_leaf_nodes31,max_depthNone,min_samples_leaf20,l2_regularization0.0,max_bins255,# 连续特征的分箱数categorical_featuresNone,# 类别特征的索引列表monotonic_cstNone,# 单调约束interaction_cstNone,# 交互约束warm_startFalse,early_stoppingauto,# True, False, autoscoringloss,validation_fraction0.1,n_iter_no_change10,# 早停容忍轮数tol1e-7,verbose0,random_state42,class_weightNone)model.fit(X,y)print(model.feature_importances_)print(model.n_iter_)# 实际迭代数早停后5.HistGradientBoostingRegressor— 直方图梯度提升回归器fromsklearn.ensembleimportHistGradientBoostingRegressor modelHistGradientBoostingRegressor(losssquared_error,quantile0.5,# quantile 损失时使用learning_rate0.1,max_iter100,max_leaf_nodes31,min_samples_leaf20,l2_regularization0.0,max_bins255,monotonic_cstNone,early_stoppingauto,random_state42)model.fit(X,y)6.VotingClassifier/VotingRegressor— 投票集成 ⭐fromsklearn.ensembleimportVotingClassifierfromsklearn.linear_modelimportLogisticRegressionfromsklearn.svmimportSVCfromsklearn.treeimportDecisionTreeClassifier votingVotingClassifier(estimators[(lr,LogisticRegression()),(svc,SVC(probabilityTrue)),# soft voting 需要 predict_proba(dt,DecisionTreeClassifier())],votingsoft,# hard(多数投票) 或 soft(概率加权)weights[1,2,1],# 各模型权重soft voting 时n_jobs-1,flatten_transformTrue# transform 方法)voting.fit(X_train,y_train)y_predvoting.predict(X_test)# 查看各模型的类别概率probsvoting.transform(X_test)# 属性print(voting.named_estimators_)# 模型字典print(voting.estimators_)# 模型列表回归版本:fromsklearn.ensembleimportVotingRegressorfromsklearn.linear_modelimportLinearRegressionfromsklearn.ensembleimportRandomForestRegressor voting_regVotingRegressor(estimators[(lr,LinearRegression()),(rf,RandomForestRegressor())],weights[1,2])voting_reg.fit(X_train,y_train)7.StackingClassifier/StackingRegressor— 堆叠集成 ⭐fromsklearn.ensembleimportStackingClassifierfromsklearn.linear_modelimportLogisticRegressionfromsklearn.svmimportSVCfromsklearn.treeimportDecisionTreeClassifier stackingStackingClassifier(estimators[(svc,SVC(probabilityTrue)),(dt,DecisionTreeClassifier()),],final_estimatorLogisticRegression(),# 元学习器cv5,# 交叉验证折叠数生成第一层预测stack_methodauto,# auto,predict_proba,decision_function,predictn_jobs-1,passthroughFalse,# 是否将原始 X 也传给元学习器verbose0)stacking.fit(X_train,y_train)y_predstacking.predict(X_test)# 属性print(stacking.final_estimator_)# 元学习器print(stacking.named_estimators_)# 基学习器print(stacking.stack_method_)# 实际使用的 stack_method回归版本:fromsklearn.ensembleimportStackingRegressorfromsklearn.linear_modelimportRidge stacking_regStackingRegressor(estimators[(rf,RandomForestRegressor()),(gb,GradientBoostingRegressor()),],final_estimatorRidge(),cv5,passthroughTrue)8.IsolationForest— 孤立森林异常检测fromsklearn.ensembleimportIsolationForest modelIsolationForest(n_estimators100,max_samplesauto,# 每棵树的样本数contaminationauto,# 异常比例float或 automax_features1.0,bootstrapFalse,n_jobs-1,random_state42,verbose0)model.fit(X)# 预测: 1正常, -1异常y_predmodel.predict(X)# 异常分数越低越异常scoresmodel.decision_function(X)print(model.offset_)# 决策偏移量# 异常分数转为概率scoresmodel.score_samples(X)# 负分数越高越正常 特征重要性所有树模型都支持特征重要性importpandasaspdimportnumpyasnpimportmatplotlib.pyplotaspltfromsklearn.ensembleimportRandomForestClassifier modelRandomForestClassifier(n_estimators100,random_state42)model.fit(X,y)# 获取特征重要性importancesmodel.feature_importances_ indicesnp.argsort(importances)[::-1]# 可视化plt.figure(figsize(10,6))plt.bar(range(X.shape[1]),importances[indices])plt.xticks(range(X.shape[1]),[fFeature{i}foriinindices],rotation90)plt.title(Feature Importances)plt.tight_layout()plt.show()基于置换的特征重要性模型无关:fromsklearn.inspectionimportpermutation_importance resultpermutation_importance(model,X_test,y_test,n_repeats10,random_state42,n_jobs-1)print(result.importances_mean)print(result.importances_std) 实践选择指南场景推荐模型通用分类/回归基线RandomForestClassifier/RandomForestRegressor追求准确率GradientBoostingClassifier/HistGradientBoostingClassifier大数据集 快速HistGradientBoostingClassifier多模型融合VotingClassifier/StackingClassifier自定义基学习器BaggingClassifier异常检测IsolationForest特征含缺失值HistGradientBoostingClassifier原生支持类别特征HistGradientBoostingClassifiercategorical_features[[sklearn-总览|← 返回总览]]