Tags:
• Creator
• Explorer
• Viewer

In this article

1. Multi-Reference Significance Overview
2. Reference Letter Colors and Cases
3. Multi-Reference Significance Testing Pop-Up
4. Multi-Reference Significance Testing Options
   1. Multi Reference Across or Down
   2. Multi Reference on Filter
   3. Include Hidden Items in Sig Test Calculations
   4. Include Total in Sig Test Calculations
      • Total rows with Measures/Averages
   5. Reference Within a Group
   6. Show Color and High/Low Significant differences
   7. Multi-Reference Significance on Charts
5. Multi-Reference Significance Export to Excel and PPT

1. Multi-Reference Significance Overview

Multiple testing refers to any instance that involves the simultaneous testing of several hypotheses. In Harmoni, multiple reference significant differences (M-SIG) are represented using 52 letters (26 upper case and 26 lower case) of the English alphabet.

• The reference column/row is labeled with a blue letter
• M-SIG is indicated in each cell

Harmoni typically applies a default confidence level of 95% for determining significant differences. This default setting can be changed via a personalization key. As a user, you can override this via the modify menu. Learn more here.

2. Multi-Reference Letter Color and Cases

Each column (or row) is labeled with a blue reference letter. The first 26 columns are labeled with upper case letters A - Z and the second 26 columns are labelled with lower case a - z. If you have more than 52 columns in your table, Harmoni still tests these columns, but the results are not displayed in the table. The pop-up box displays the significance for any columns beyond 52.

The significance results in each cell are shown using:

• Green letters: The cell's value is significantly higher than the reference cell
• Red letters: The cell's value is significantly lower than the reference cell

Below you can see an enlarged view of the example analysis table above. Here we can see that amongst those who visited a Geothermal Park, 44% were from the United Kingdom. Reviewing the lettering below this percentage, you can see that the letter A is colored red, indicating that this result (44%) is significantly lower than UK visitors going to a Beach (46%). You can also see that the letter C is colored green, indicating that significantly more UK visitors are going to Geothermal Parks when compared to a National Park.

Harmoni typically applies a default confidence level of 95% for determining significant differences. This default setting can be changed via a personalization key. As a user, you can override this via the modify menu. Learn more here.

3. Multi-Reference Significance Testing Pop-Up

When calculating multiple reference significant differences, you can select any relevant cell in the analysis to display a hover pop-up menu in which you will find the following information:

• The axis/element name of the reference cell, the letter assigned to it, and the value of the selected cell.
• Items that are significantly higher than or lower than the value of the cell you have selected.
• Confidence level at which the tests have been performed.

To display the pop-up, hover over the cell and click the three dots on the left side. 

Harmoni tests all columns/rows in the table. If there are more than 52 columns/row in the table, the popup displays all significant differences, so you may see items without a reference letter.

• With M-SIG turn on you can see the colored lettering in each cell
• Click the 3 dots within the cell, to the left of the percentage
• The pop-up displays the significant differences

4. Multi-Reference Sig Testing Options

a. Multi Reference Across or Down

Multiple comparison significance testing can compare cells on the across (columns) or on the down (rows).

You can control the direction by clicking the drop-down arrow* under the M-SIG icon and selecting the  Across or Down option.

*M-SIG needs to be turned on for the drop-down arrow to appear.

• By default, the Across option is selected
• Blue reference lettering displays for each column
• Select Down to switch to row comparisons
• Blue reference letters display for each row

b. Multi Reference on Filter

Multiple comparison significance testing can compare cells based on a filter item.

In an analysis where a filter variable is included, you can select the filter option from the drop-down menu. 

• Elements in the filter axis are assigned the ABC reference letters.
• When more than one filter is included in the analysis, the first axis in the filter is assigned the reference letters.
• You can choose a different filter axis using the select menu and choosing the M-SIG option next to the axis name.

• Select Filter in the M-SIG drop-down menu
• The filter in the interactive tile and the Selector show the reference letters
• If you have multiple filters, change the reference by clicking M-SIG next to the filter label in the Selector

c. Include Hidden Items in Sig Test Calculations

When doing multiple comparison significance testing, items you hide in the analysis are not included in the reference set. You may choose to include by ticking the option "Include Hidden Items" from the multi-ref drop box.

• If you hide a column, the reference lettering adjusts to the displayed columns
• You can Include Hidden Items from the drop-down menu
• Open the pop-up, you can see significance results including hidden items
• The blue reference lettering skips a letter, indicating the hidden column

d. Include Total in Sig Test Calculations

When doing multiple comparison significance testing, you may choose to include Totals by ticking the option Include Total from the multi-ref drop box.

This makes the Total column or row in the dimension chosen, the first reference for M-SIG and it is assigned the letter “A”. 

• By default, the M-SIG test does not include totals
• Select Include Total from the drop-down menu
• The letter A reference starts from the total column

Total rows with Measures/Averages

When creating an analysis with measures, Harmoni does not automatically include M-SIG testing for the averages in the total row. You need to tick the Include Total option to add M-SIG to the total row.

You can select a measure or measure group, or an axis with values and with Include Total selected, the significant difference letters are added. Note that single ref statistical differences (SIG) are automatically applied to the total row.

• The down axis displays a rating scale with an average in the total row
• SIG displays significant differences in the total row while M-SIG does not
• Select Include Total from the drop-down menu to add M-SIG into the total row

e. Reference Within a Group

When analyzing data with multiple categories or groups, you can compare values within each group or across different groups.

• Reference within Group: By default, Harmoni compares values within the same group, using the first item labeled A as the reference. Each group starts its reference set from A.
• Comparing Across Groups: Uncheck the Reference within a Group option to compare across groups. All items are labeled A to Z regardless of the group.

• With two axes across, the references sets are within each group
• Uncheck Reference within a Group in the drop-down menu
• The M-SIG test then compares significance across different groups

f. Show Color and High/Low Significance

When doing multiple comparison significance testing, you may choose to exclude the red and green coloring. Unticking the option "Show SigDiff Color" from the multi-ref drop-box removes the coloring.

Because color indicates the significance and with coloring off there is no way to tell which is higher or lower significance, when the coloring is turned off, only one significance option is available to display at one time. You can select either the higher or lower significance.

• Uncheck Show SigDiff Color to remove coloring
• Select either higher or lower significance
• The pop-up retains the coloring

5. Multi-Ref Significance on Time Axes

When variables are flagged as a time axis, Harmoni adjusts to consider the latest time periods. Instead of starting from the left (for columns) or top (for rows), the testing starts from the right (for columns) or the bottom (for rows). Although the reference lettering still starts with A, when you have more than 52 time periods, you can see that the first variables in your table (to the left or top) do not have an indicator label (A, B, C, etc.) as the testing starts to the right or bottom.

In this example, a time axis with 56 columns displays in the across drop zone and so the first four columns are not indicated with a letter. Harmoni still includes these columns in the testing and the variables are included in the popup.

• With the time axis in the across drop zone, the testing starts from the right
• The first four columns do not have a reference label
• The analysis is flipped and the reference labels are switched to down
• The testing start from the bottom and so the first four rows do not have a reference label

Grouped Axes

When grouping time axes and standard axes together, Harmoni treats each axis as per the axis type.

• A time axis and standard axis are grouped in the across drop zone using Add.
• The time axis testing starts at the right and the standard axis starts at the left.

Nested axes

When nesting, axes nested at the bottom dictate the ordering of the reference labels.

• When the time axis is nested over the standard axis, the standard axis controls the direction of the testing.
• When the standard axis is nested over the time axis, the time axis controls the direction of the testing.

Where to from here?

Learn more about significance testing.

• Multi-Reference Statistical Significance Tests
• Statistical Significance Tests
• Statistical Significance in Harmoni (SIG)