Tags:
• Creator
• Explorer
• Viewer

In this article

1. Multi-Reference Significance Overview
2. Multi-Reference Significance Testing Pop-Up
3. Reference Letter Colors and Cases
   1. Options when cell is higher
   2. Options when cell is lower
   3. No color applied
4. Multi-Reference Significance Testing Options
   1. Multi Reference Across or Down
   2. Multi Reference on Filter
   3. Include Hidden Items in Sig Test Calculations
   4. Include Total in Sig Test Calculations
      • Total Rows with Measures/Averages
   5. Reference Within a Group
   6. Show Color
   7. Show High/Low Significant differences
5. Multi-Reference Significance on Time Axes

1. Multi-Reference Significance Overview

Multiple testing refers to any instance that involves the simultaneous testing of several hypotheses. In Harmoni, multiple reference significant differences (M-SIG) are represented using the 26 letters of the English alphabet.

• The reference column/row is labeled with a blue letter; A - Z
• M-SIG is indicated in each cell

Harmoni typically applies a default confidence level of 95% for determining significant differences. This default setting can be changed via a personalization key. As a user, you can override this via the modify menu. Learn more here.

2. Multi-Ref Significance Testing Pop-Up

When calculating multiple reference significant differences, you can select any relevant cell in the analysis to display a hover pop-up menu in which you will find the following information:

• The axis/element name of the reference cell, the letter assigned to it, and the value of the selected cell.
• Items that are significantly higher than or lower than the value of the cell you have selected.
• Confidence level at which the tests have been performed.

To display the pop-up, hover over the cell and click the three dots on the left side. 

• In the Modify menu, select M-SIG
• Click the 3 dots within the cell, to the left of the percentage
• The pop-up displays the significant differences

3. Multi-Reference Letter Color and Cases

Harmoni has a default setting that is automatically applied. Option 1 (described below) is the default setting, but there are five additional options (Options 2-6) that can be set using personalization keys.Contact support@infotools.com for more information.

Let's look at all the possible options using this example where:

• Cell A  (Australia) -  20% of visitors from Australia are aged 45-54 years old.
• Cell B (United Kingdom) - 13% of visitors from UK are aged 45-54 years old.
• Cell E (China) - 27% of visitors from China are aged 45-54 years old.

a. Options when reference cell is higher - "Color Green when I am higher than others"

These options use a green font when the reference cell is significantly higher than other cells.

b. Options when reference cell is lower - "Color Green when others are higher"

These options use a green font when the reference cell is significantly lower than other cells.

c. No font color applied 

4. Multi-Reference Sig Testing Options

a. Multi Reference Across or Down

Multiple comparison significance testing can compare cells on the across (columns) or on the down (rows).

You can control the direction by clicking the drop-down arrow* under the M-SIG icon and selecting the  Across or Down option.

*M-SIG needs to be turned on for the drop-down arrow to appear.

• By default, the Across option is selected
• Blue reference lettering displays for each column
• Select Down to switch to row comparisons
• Blue reference letters display for each row

b. Multi Reference on Filter

Multiple comparison significance testing can compare cells based on a filter item.

In an analysis where a filter variable is included, you can select the filter option from the drop-down menu. 

• Elements in the filter axis are assigned the ABC reference letters.
• When more than one filter is included in the analysis, the first axis in the filter is assigned the reference letters.
• You can choose a different filter axis using the select menu and choosing the M-SIG option next to the axis name.

• Select Filter in the M-SIG drop-down menu
• The filter in the interactive tile and the Selector show the reference letters
• If you have multiple filters, change the reference by clicking M-SIG next to the filter label in the Selector

c. Include Hidden Items in Sig Test Calculations

When doing multiple comparison significance testing, items you hide in the analysis are not included in the reference set. You may choose to include by ticking the option "Include Hidden Items" from the multi-ref drop box.

• If you hide a column, the reference lettering adjusts to the displayed columns
• You can Include Hidden Items from the drop-down menu
• Open the pop-up, you can see significance results including hidden items
• The blue reference lettering skips a letter, indicating the hidden column

d. Include Total in Sig Test Calculations

When doing multiple comparison significance testing, you may choose to include Totals by ticking the option Include Total from the multi-ref drop box.

This makes the Total column or row in the dimension chosen, the first reference for M-SIG and it is assigned the letter “A”. 

• By default, the M-SIG test does not include totals
• Select Include Total from the drop-down menu
• The letter A reference starts from the total column

Total Rows with Measures/Averages

When creating an analysis with measures, Harmoni does not automatically include M-SIG testing for the averages in the total row. You need to tick the Include Total option to add M-SIG to the total row.

You can select a measure or measure group, or an axis with values and with Include Total selected, the significant difference letters are added. Note that single ref statistical differences (SIG) are automatically applied to the total row.

• The down axis displays a rating scale with an average in the total row
• SIG displays significant differences in the total row while M-SIG does not
• Select Include Total from the drop-down menu to add M-SIG into the total row

e. Reference Within a Group

When analyzing data with multiple categories or groups, you can compare values within each group or across different groups.

• Reference within Group: By default, Harmoni compares values within the same group, using the first item labeled A as the reference. Each group starts its reference set from A.
• Comparing Across Groups: Uncheck the Reference within a Group option to compare across groups. All items are labeled A to Z regardless of the group.

• With two axes across, the references sets are within each group
• Uncheck Reference within a Group in the drop-down menu
• The M-SIG test then compares significance across different groups

f. Show Color 

When doing multiple comparison significance testing, you may choose to exclude the red and green coloring. Unticking the option "Show SigDiff Color" from the multi-ref drop box removes the coloring.

The upper and lower case still serves as a good indicator of significant differences and also the information pop-up can give more context on the results.

• Uncheck Show SigDiff Color to remove coloring
• The pop-up retains the coloring

g. Show High/Low Significant Differences

When doing multiple comparison significance testing, you can choose to show only the cells that are significantly higher or only those that are lower.

Use the drop-down menu to select the required options:

• Show Higher Significance
• Show Lower Significance

• Tick the check box to add or remove the Higher/Lower significance option

5. Multi-Ref Significance on Time Axes

When variables are flagged as a time axis, Harmoni adjusts to consider the latest time periods. Instead of starting from the left (for columns) or top (for rows), the testing starts from the right (for columns) or the bottom (for rows). Although the reference lettering still starts with A, when you have more than 26 time periods, you can see that the first variables in your table (to the left or top) do not have an indicator label (A, B, C, etc.) as the testing starts to the right or bottom.

In this example, a time axis with 30 columns displays in the across drop zone and so the first four columns are not indicated with a letter. Harmoni still includes these columns in the testing and the variables are included in the popup.

• With the time axis in the across drop zone, the testing starts from the right
• The first four columns do not have a reference label
• The analysis is flipped and the reference labels are switched to down
• The testing start from the bottom and so the first four rows do not have a reference label

Grouped Axes

When grouping time axes and standard axes together, Harmoni treats each axis as per the axis type.

• A time axis and standard axis are grouped in the across drop zone using Add.
• The time axis testing starts at the right and the standard axis starts at the left.

Nested axes

When nesting, axes nested at the bottom dictate the ordering of the reference labels.

• When the time axis is nested over the standard axis, the standard axis controls the direction of the testing.
• When the standard axis is nested over the time axis, the time axis controls the direction of the testing.

Where to from here?

Learn more about significance testing.

• Multi-Reference Statistical Significance Tests
• Statistical Significance in Harmoni (SIG)
• Multi-Ref Significance on Charts and Exports