Statistical tests

Two classes (A, the reference, and B): Welch's t test (the default, unequal variances), Student's t or Mann–Whitney (on ranks). Effect: log2 of the ratio of the means B / A (when both means are positive) or Cohen's d. Volcano plot: the effect against -log10(p).

Two classes paired by a key ("Pair by": a variable or an observation ID, the year for two sectors followed over the same years): for each variable, the rows of A and B that share a key, and have a value, make the pairs, and the differences B − A go through the paired t test (the default) or Wilcoxon's signed-rank test. The table gives each variable's number of pairs and the means of A and B over those pairs; the effect is log2 of the ratio of those means (when both are positive) or Cohen's d_z (the mean difference over the standard deviation of the differences). A variable with fewer than three pairs is not tested. A key may appear only once per class: one repeated within A or B is refused, as in "Two variables".

All classes: one-way ANOVA or Kruskal–Wallis; the variables ranked by p.

Two qualitative variables: the contingency table and Pearson's chi-square test of independence, with Cramér's V. In red and in blue, the cells more or less frequent than if the two variables were independent (adjusted residual beyond ±1.96). When the expected counts are too small for the chi-square test (one below 1, or more than 20 % below 5: Cochran's rule), Fisher's exact test is recommended; for a 2 × 2 table it is always given, with the odds ratio. Beyond 2 × 2 and 200 observations, its p value is simulated (100,000 tables).

A measurement and a qualitative variable: the measurement compared between the groups, in box plots. Two groups: Welch's t test and Mann–Whitney (exact below 50 values on each side with no ties; otherwise by the normal approximation with continuity correction: the same calculation on every R version). More: one-way ANOVA, Welch's ANOVA and Kruskal–Wallis, each with its pairwise comparisons (Tukey, Games–Howell, Dunn with Holm's correction; up to 50 groups): a click on a test shows them. The rank test is recommended when normality is doubtful in a group (Shapiro–Wilk p < 0.05, or fewer than three observations), Welch's ANOVA when the variances differ (Brown–Forsythe p < 0.05).

A measurement and a qualitative variable, paired by a third variable or ID ("Pair by": the year, for two sectors of a site followed over the same years): the data stay in long format, one row per sector and year. Two levels A and B are compared on the keys both have: the differences B − A by the paired t-test (mean difference, its 95 % interval, Cohen's d_z) and by Wilcoxon's signed-rank test, the rank test recommended when the normality of the differences is doubtful (Shapiro–Wilk p < 0.05; not tested beyond 5,000 pairs, nor with constant differences, which leave Wilcoxon alone). Wilcoxon's p is exact below 50 pairs when no difference is zero or tied; otherwise the zero differences are dropped and p comes from the normal approximation with continuity correction, the same on every R version. At least three pairs are needed, and a key may appear only once per level. The rows left without a partner, and those without a value, are counted and listed; the plot joins each pair. With three levels or more, the table below can also list every pair of levels, each on its own shared keys, with Holm's correction over the pairs tested, for the test chosen above (a click on a test): when all the levels share the same keys, those of pairwise.t.test() and pairwise.wilcox.test() with paired = TRUE. A pair not tested (fewer than three shared keys, a key twice in a level, the same value in every pair) says why, and a click on a pair compares it; when A and B can't be compared, the table stays, under their reason. "Pairs of levels within each" (a qualitative variable or an ID: the site) keeps to the pairs of levels of the same group, the sectors of each site compared two by two. Holm's correction covers the pairs tested, all groups together; the raw p reads one site at a time. A level belongs to the group its rows carry: one with rows in several groups is refused, and those without a group are listed. Above the pairs comes the omnibus test, for both paired tests: Friedman's test on the keys where every level has a value (complete blocks; the keys left out for a missing level are counted), its χ², degrees of freedom and p, with Kendall's W = χ² / (n (k − 1)) as effect size (0: no agreement from one key to the next, 1: the same ranking of the levels at every key), as a sentence. Within each group, one Friedman test per group, on its own levels and complete keys, its p corrected by Holm over the groups tested; a group with fewer than three levels or fewer than two complete keys gets no test, and the reason is given.

Two measurements: Pearson's and Spearman's correlations; Spearman is recommended when normality is doubtful for either. Paired by "the observation (same row)", two measurements of the same rows (before and after, say) are compared instead: the differences Variable 2 − Variable 1, on the observations where both are known, by the same two paired tests.