How to Change Significance Levels for Column Comparisons

Introduction

By default, the Column Comparisons in Q use:

• Uppercase letters for results that are significant at the 0.001 level (99.9%) and lower-case at the 0.05 level (95%). That is, uppercase results are more significant.
• The False Discovery Rate Correction when determining whether or not to show any letters (but not in determining which are lowercase and which are uppercase).

This is different to the norm within the industry, which is to test at the 0.05 and 0.1 levels (i.e., 95% and 90% confidence levels). If Q's settings are left at their default levels, the results reported in Q will tend to be more robust than those that are computed in many other programs, but fewer results will be reported as being significant.

Method

Changing the significance levels

Settings governing how Q tests are set in the Statistical Assumptions menu. These settings can be changed for an entire project or a specific table. To change the settings for a selected table, select the table in the Report tree, then go to Edit > Table Options > Statistical Assumptions. Or right-click the table and select Table Options. 

To change the settings for the entire project, go to Edit > Project Options > Customize > Statistical Assumptions.

For example, to change Q so that lowercase letters are shown at the 90% level of confidence and UPPERCASE at the 95% level:

• Set the Overall significance level (top-left) to 0.10 (i.e., the 90% level of confidence).
• Use the Significance levels and appearance grid to control the level of significance associated with different column letters. That is, copy one of the uppercase Column letters fields (i.e., A,B,C,...,) and paste it over the top of each of the lower-case fields (i.e., a,b,c,...,) for 0.005, 0.01 and 0.05.
• Change both Multiple comparison correction settings to None.
  • Note: In Q 5.14 and onward, these two settings are in the Statistical Assumptions > Exception Tests and Statistical Assumptions > Column Comparisons tabs within the Statistical Assumptions menu.

Using These Changes in Future Projects

Use Project Templates to automatically apply these changes in future projects.

Next

How to Modify Significance Tests

How to Show Statistical Significance in Q

How to Interpret Column Comparisons

How to Apply, Save or Build a Project Template