In Harmoni, you can calculate statistical significant differences between the results for one group and a reference group (SIG), or you can perform multiple reference significance testing (M-SIG). This article explains the multi-reference significance tests.
In this article
- Multi-Reference Statistical Testing
- Multi-Reference Testing on Proportions
- Multi-Reference Testing on Measures
1. Multi-Reference Statistical Testing
Multiple testing refers to any instance that involves the simultaneous testing of several hypotheses.
The multiple reference significance test (M-SIG) makes several simultaneous pairwise comparisons and therefore needs to adjust for the inflation in false-positive rates (known as the family-wise error rate). Harmoni automatically corrects the family-wise error rate using a technique known as the Holm method.
Standard SIG testing, are simple comparisons between two proportions and do not apply any multiple comparisons correction.
Therefore you may notice some discrepancies when comparing the two techniques.
2. Multi-Reference Testing on Proportions
When doing multiple comparison significance testing on proportions (percentages), Harmoni performs a pairwise proportions test.
A pairwise proportions test is the equivalent of a chi-square test between the proportions in each pair of cells in the row (or column).
The process of conducting multiple statistical tests introduces the multiple testing problem, which must be corrected. Essentially, we must correct for the fact that as we make more and more comparisons, we increase the chances that we’ll find false positives, i.e., detect a significant difference when one does not exist.
- When the confidence level of 95% is set, we are saying that, if in reality, there is no difference in the values, we will accept a probability of 1 in 20 (or 0.05) of getting a false positive, i.e., concluding a difference exists when it does not.
- When we do two tests, the chance of getting a false positive from either test (if there are no actual differences) increases to (1 - 0.95*0.95) = 0.0975.
- As we perform more and more tests, this chance increases. This is known as the family-wise error rate.
Many techniques have been developed to prevent the inflation of false-positive rates that occur with multiple statistical tests. By default, when Harmoni performs multiple comparison significance testing on proportions, it applies the Holm method to correct the family-wise error rate.
The Holm method is a commonly applied technique that is uniformly more powerful than some others such as the Bonferroni correction. (Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.)
If a new variable is added to the analysis, for example, a new time period is added with a new update, multi-ref significant differences may change in previously saved analyses (stories). Because a new variable has been added to the calculation, different results may occur when all the comparisons between the variables are recalculated.
3. Multi-Reference Testing on Measures
For multiple comparison significance testing on measures (real-valued numeric variables), Harmoni applies Tukey's HSD test. Tukey's HSD tests all pairwise comparisons among values and is based on the studentized range distribution.
Tukey's test is based on a formula very similar to that of the t-test. Tukey's test is essentially a t-test, except that it corrects for family-wise error rate.
As with proportions, when multiple comparisons are made, the probability of making a type I error (false positive) increases. Tukey's test corrects for that and is thus more suitable for multiple comparisons than performing several t-tests.
As with comparisons made in significance testing on proportions, multi-ref significant differences may change when new measures are added to the analysis.
Where to from here?
Learn more about statistics.