In Harmoni, options in the modify menu allow for statistical testing to be applied to tables and charts. The modify menu is available to Creators and Explorers. Viewers can also interact with these features when they have access to dashboards, or stories where the analysis can be zoomed.
Statistical tests are used in hypothesis testing and can determine whether a predictor variable has a statistically significant relationship with an outcome variable. Tests can also estimate the difference between two or more groups.
Significance testing is available for record/respondent level data and is not available in aggregate data.
In this article
1. Statistical Significance Testing (SIG)
When using the SIG option, Harmoni calculates and displays the statistically significant differences between the results for one group and a reference group with an overlap correction factor.
- For an analysis without a time axis, the default reference is the total column.
- With a time axis included, the default reference is the previous period.
- You can adjust the reference as you need.
To learn more about the types of significant tests used in Harmoni, please review the Statistical Significance Tests article.
The SIG option is available in the modify menu when no items are selected in the analysis. (If you select a row, column or any other part of a table, the SIG option is not available.)
- SIG can be set as a default and may automatically turn on. If not, manually click the SIG option.
- When SIG testing is turned on, it does not display until your analysis is a crosstab or has a time axis as a filter.
- When you intentionally switch SIG on, it will remain on for your session, or until you reset the analysis. Clearing the analysis does not remove SIG.
SIG testing on as a default
The personalization key Keys that can be applied at the user, site (all users), or company (all users) level. Contact support@infotools.com for more information. for significance difference testing can be set to ON (True) as a default.
When significance difference testing is on by default, SIG testing displays when you create an analysis with a cross tab or with a single axis if you add a time axis as a filter.
Green and Red arrows
In Harmoni, significant differences are represented using green and red arrows.
- Green - significantly higher compared to the reference.
- Red - significantly lower compared to the reference.
- The confidence level is displayed on the info bar.
When you see a red arrow pointing down in a cell, it means that the number in the cell is significantly lower than the reference number it is being compared to. And of course, the reference number is therefore significantly higher. Often, but not always, lower in the cell can mean worse, for example, brand usage, hence the red color approach.
2. Reference for Testing (REF)
You can select a specific item (i.e. column or row) as the reference for testing. Once selected, the reference is fixed to that item; if you flip or reorder your analysis, the reference will remain fixed.
REF appears in the modify menu in the Selection section when you select a row or column or when you add a time axis to the analysis. Click REF to select the reference for testing or adjust the periodicity.
- The selected row or column becomes the reference for significant difference calculations.
- When a time-based axis is a part of your analysis you have the ability to set significance testing to be based on a rolling period. The axis must be marked as a date variable. See Previous period referencing below.
- The comparison reference is displayed on the info bar.
- The default reference is the total column
- Select a column and click REF
- You can move the column or flip the table and the reference remains fixed
- Select the reference column and click REF to deselect the reference
Significance testing is also available within and across nested groups. Learn more about significance testing in nested analyses.
Previous Period Referencing (Time Axis)
When a time-based axis is a part of your analysis, Harmoni automatically uses it as the reference. Open the i-bar to see the reference displayed. The significance testing is set to be based on a rolling period. This could be the previous period or even the previous 4th period.
The offset determines how far away from the test cell the cell to be compared with is, for example, an offset of 1 is the cell next on the left of the cell being compared.
Using the drop-down menu for the REF icon allows you to access the reference period. Enter the offset number in the text box.
In this example, you can see that there are 25 Quarters (1-25) and the offset is 8.
- 1 = minimum available offset
- 25 = maximum available offset
- 8 = selected offset
Note, if the offset number sits too close to the start of the range, the minimum available offset may not display.
Time Axis in the Across Dimension
In the example below, the time axis, Quarter, is in the across dimension. The default offset is 1, the Previous Quarter. By accessing the REF drop-down menu, this is changed to 2 so that the reference is 2nd Previous Quarter. At the bottom of the analysis, open the i-bar to display the reference information.
- With Quarter in the across dimension, the reference is Previous Quarter
- Change the offset to 2 and the reference becomes 2nd Previous Quarter
Time Axis as a Filter
In the example below, Year is dragged in to the filter drop zone and the reference changes to Previous Year. When a time axis is dragged to the filter, Harmoni defaults to using the time axis as the reference for significant differences. The offset defaults to 1 and can be changed as demonstrated above.
In this example, the offset is 1 and each cell displayed for the most recent Year (2025) is compared to the previous Year (2024). When you change the filter to 2023, each cell is then compared to 2022 and so on.
- Drag the time period to the filter
- The i-bar updates the reference
- Use the filter to view each year and the comparison to the previous year updates
Include or Exclude Hidden Items
When a time-based axis is a part of your analysis (generally columns/across), hidden items are included in the reference set. You may choose to exclude by unticking the option "Include Hidden Items" from the REF drop box.
This is a useful option when, for example, there are missing time periods in your survey, but you still want to compare vs. the most recent period.
Analysis with a Single Time Axis
When you first add a time axis to your analysis, the reference defaults to the Total for significance testing. As soon as you add a second item to create a crosstab, the reference changes to the previous time period.
Analysis with Multiple Time Axes
When you add more than one axis in different dimensions in an analysis, you are able to choose which axis to use as a reference. By accessing the drop-down menu under the REF icon, you can select the required dimension.
In the following example, the time axis for Quarter is added to the across dimension, while Year is added to the filter. Using the REF icon, the default reference of Quarter is changed to Year.
- Drag a time axis to the across
- Drag a time axis to the filter
- In this example, we can use Suppress to remove empty cells
- Change the REF via the drop-down menu from Quarter to Year
Where to from here?
Learn more about statistical tests in Harmoni