{"id":20729,"date":"2021-11-12T18:59:37","date_gmt":"2021-11-12T13:29:37","guid":{"rendered":"http:\/\/ismiletechnologies.com\/?p=20729"},"modified":"2022-12-10T01:58:20","modified_gmt":"2022-12-09T20:28:20","slug":"ensemble-models-and-use-of-feature-importance-in-tree-based-models","status":"publish","type":"post","link":"https:\/\/ismiletechnologies.com\/en_us\/dataops\/ensemble-models-and-use-of-feature-importance-in-tree-based-models\/","title":{"rendered":"Ensemble models and use of feature importance in tree-based models"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t

Ensemble models <\/h2>\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Ensemble models are machine learning<\/span><\/a> models that use more than one predictor to predict. A group of predictors forms an ensemble. In general, ensemble models perform better than using a single predictor. There are three ensemble models, bagging, boosting, and blending.\u00a0\u00a0<\/span><\/span>\u00a0<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t\t

Random forests <\/h2>\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The performance of a single decision tree will be limited, so instead of relying on one tree, a better approach is to aggregate the predictions of multiple trees. On average, the aggregation will perform better than a single predictor. We call a tree-based model that aggregates the predictions of multiple trees a random forest.\u00a0<\/span>\u00a0<\/span><\/p>

There is a need for a diverse collection of trees for a random forest to be effective, so there should be variations in the chosen thresholds for splitting and the number of nodes and branches as there will be no point aggregating the predicted results if all the trees are nearly identical and produce the same result. To achieve a diverse set of trees, we need to: Train each tree in the forest using a different training set and only consider a subset of features when deciding how to split the nodes.\u00a0<\/span>\u00a0<\/span><\/p>

Ideally, we would generate a new training set for each tree for the first point. However, often it’s too difficult or expensive to collect more data; we have to make do with what we have. Bootstrapping is a general statistical technique to generate new data sets with a single set by random sampling with a replacement that allows a data point to be sampled more than once. <\/span>\u00a0<\/span><\/p>


Typically, when training the standard decision tree model, the algorithm will consider all features in deciding the node split. Assuming only a subset of your features ensures that your trees do not resemble each other. If the algorithm had considered all features, a dominant feature would be continuously chosen for node splits.\u00a0<\/span>\u00a0<\/span><\/p>

Random forest<\/span><\/a> is also referred to as bagging ensemble models because it is based on bootstrapping and aggregating the results. <\/span>\u00a0<\/span><\/p>

The hyperparameters available for random forests include decision trees with some additions.\u00a0<\/span>\u00a0<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t\t\t\t\t\t