By default, Harmoni calculates single reference and multi-reference significance testing with an overlap correction factor. Non-overlapping significance is available via a personalization key.
In this article
- Overlap Significance Testing
- Comparisons Between Overlapping Groups and with Total
- Verifying Results
- Effective Base
1. Overlap Significance Testing
Default statistical testing in Harmoni does not assume independence of the comparison groups as the assumption is made that it is common for groups to overlap. Instead it considers respondents who can be in both the test cell and the reference cell and adjusts the result accordingly.
This is achieved in Harmoni by using modified t-tests adding an overlap correction factor to the significance testing. The tests are used for both discrete and continuous variables and is applied in both multi-reference and single-reference significance testing.
Harmoni displays the statistical method used in the information bar:
Learn more about the information bar.
Statistical testing in Harmoni can also be carried out using non-overlapping groups. If this is your preferred method, please contact support@infotools.com.
2. Comparisons Between Overlapping Groups and with Total
In some cases, the overlap respondent count between the reference cell and the test cell may be 100%. This is a particular issue with all "compare with Total" tests (i.e., Total is set as the Reference) because the Total usually includes the test group. So, it can be better to compare, for example, Males with Females rather than Males with Total because the Males and Total groups have a high overlap. However, to account for the overlap, the default test setting adds an overlap correction factor.
A similar situation arises with rolling averages in time series data. If, for example, rolling 3-month averages are calculated, then when testing for significant differences, the reference and any test cells can be compared as the overlap correction factor accounts for any overlap.
The effect of overlapping groups is small if the test group is small compared to the reference group or Total group.
3. Verifying Results
Verifying Overlap Significance Difference results is not possible in Harmoni directly. If you need assistance, please contact Support@infotools.com.
4. Effective Base
In statistics, effective base is used as a safeguard against making statistical conclusions from a sample that has been drastically adjusted using weights to match target values. Using the effective base is considered a more conservative approach, but it provides accurate statistical results for weighted data.
The effective base indicates how much statistical power is lost by weighting. The closer the effective base is to the unweighted base, the better the weighting is.
Effective base = (sum of weighted base) squared divided by the sum of the squared weighted base.
The effective base is the default base for calculating significant differences. This can be changed via a personalization keyKeys that can be applied at the user, site (all users), or company (all users) level.. Please contact support@infotools.com for more information.
Learn more about effective base.
Where to from here?
Learn more about statistical significance in Harmoni: