Trees, forests, discriminant analysis, k-means

Decision trees, linear discriminant analysis and k-means

New model › type "Decision tree" (rpart): a quantitative Y variable (regression tree) or a class variable (classification tree), and X variables by their raw values, so that the thresholds read in their units; missing X values are handled by surrogate variables.

General tab: at least 20 observations per leaf, but at most a sixth of the observations, so that the tree can split (a node is split only if it holds three times as many), at most 5 levels, cp = 0.001, 5 surrogates, pruning by the 1-SE rule (the smallest tree within one standard error of the minimum error) or at the minimum error. The pruning comes from an internal cross-validation by the cross-validation groups (for sites followed over several years, choose the site), and Q2 from an external cross-validation: each group is predicted by a tree entirely rebuilt without it. Views: Tree (a click on a node selects its observations), Pruning, Leaf rules, Variable importance, First-split stability (200 bootstrap draws of the groups: the share of the draws where each variable makes the first split, and its median threshold), Observed / predicted and Residuals, or Classification for a classification tree. Test set and Predict: a new observation follows the tree, surrogates included.

Type "Linear discriminant analysis (LDA)" (MASS::lda): a class variable and X variables by their raw values; prior probabilities proportional to the class sizes or equal. Each observation goes to the class of largest posterior probability; cross-validated, each group is classified by the model fitted without it. Views: Discriminant scores, Discriminant coefficients (structure correlations, coefficients in the variables' units or for variables of standard deviation 1), Classification, Posterior probabilities. Test set and Predict.

Type "k-means": X variables scaled as the Variables tab says (UV by default), k clusters (Hartigan and Wong algorithm, 25 starts, seed 1), numbered from the largest to the smallest; an observation with a missing value is left out. Views: Clusters (on the first two principal components), Silhouettes, Choice of k (2 to 10: within-cluster sum of squares and mean silhouette), Cluster centres. "Clusters → variable" adds the clusters to the project as a qualitative variable (Edit › Undo removes it); Predict assigns a new observation to the cluster of the nearest centre.

For all three: Methods text with its references, Export, PDF or Word report, Export to R (a script or an R Markdown / Quarto report that redoes the analysis with rpart, MASS or cluster, which come with R). Compare models shows a regression tree's Q2 and RMSEcv beside the other models.

Random forests and boosted trees

New model › type "Random forest" (the ranger package, Wright and Ziegler 2017): a quantitative Y variable (regression forest) or a class variable (classification forest), and X variables by their raw values. Each tree is grown on a bootstrap draw of the observations (with replacement), each split chosen among mtry randomly drawn variables; the forest predicts the mean of its trees, or the class most of them vote for (Breiman 2001). General tab: 500 trees, ranger's mtry and node size by default (the square root of the number of X rounded down; nodes of 5 observations for a regression, 1 for a classification), the share drawn, the permutation importance unscaled or divided by its standard error. Seed 1 and a single thread: the same forest every time. ranger takes no missing values: an observation with a missing X value is left out (the summary counts them). The out-of-bag error: each observation predicted by the trees whose draw doesn't hold it (MSE and R², or the share misclassified); the cross-validation: each cross-validation group (the site) predicted by a forest rebuilt without it (Q2 and RMSEcv, or the share correctly classified). Views: Variable importance (by permutation, how much the out-of-bag error rises when the variable's values are permuted; or by impurity), Partial dependence (computed on request, Friedman 2001: the forest's mean prediction when a variable takes each value of a grid, the other X unchanged; all the variables side by side, or one, the observed values under the curve; for a classification, the share of a class's votes), Error against the number of trees, Observed / predicted (out of bag, cross-validated or fitted) or Classification (with ROC curves from the votes). Test set and Predict.

Type "Boosted trees (BRT)": gradient boosting of regression trees (Friedman 2001, 2002), as the gbm package computes it and as Elith, Leathwick and Hastie (2008) describe it for ecologists, written in R on rpart. A quantitative Y: a measurement (normal distribution), a presence-absence 0/1 (binomial) or counts (Poisson), chosen as a GLM's. The trees are added one by one, each fitted to what the model still misses (the gradient of the deviance) on a draw without replacement of a share of the observations, its leaves re-estimated for the deviance, and added with a small weight (the learning rate). General tab: at most 2000 trees, a learning rate of 0.01, 3 splits per tree (the interaction depth: 1 gives an additive model), at least 10 observations per leaf, draws of 50%. The number of trees is the one where the cross-validated deviance is smallest (each cross-validation group predicted after each tree by the boosting redone without it): the more groups and trees, the longer the fit. Observations with a missing X value are left out. Views: Relative influence (the error reductions of each variable's splits, in %), Partial dependence (on the link scale, logit or log, or on the response's), Error against the number of trees (the deviance of the fit and the cross-validated one, the number of trees retained), Observed / predicted, and Classification (at a probability of 0.5, with the ROC curve) for a 0/1 response. Test set and Predict.

For both: Methods text with its references, Export, PDF or Word report. Compare models shows the Q2 and RMSEcv of a regression forest, or of boosted trees of a measurement, beside the other models. Export to R: boosted trees with R and rpart alone (the boosting of Nuée's engine copied into the script: with the same seed, the same draws and the same trees, which are gbm's); a random forest with R and the ranger package, the one exception to "no package": Nuée depends on it and installs it with itself (elsewhere, install.packages("ranger")), and the script loads it with the same seed, one thread and the same settings, so its forest is Nuée's, tree for tree, with the same version of ranger. The test set and the partial dependence, when it was computed, are in it too.