How to Interpret "Inconsistent" Statistical Testing Results

Introduction

Q contains multiple different systems for computing and displaying statistical testing results. Each of these systems is doing different things and, consequently, their results can seem to be inconsistent. However, unless the user has modified default settings, in general the seeming inconsistencies are in misinterpretations of outputs rather than genuine inconsistencies.

Methods

Cell comparisons (arrows and font color) versus column comparisons

In the table below, both Column comparisons and Cell comparisons (also called Exception Tests) are shown (this has been done by selecting Font colors in the Show significance menu, and then right-clicking on the table and selecting Statistics - Cells > Column Comparisons).

In the first row, no letters are shown, and instead the "-" symbol is shown, which indicates that no columns have been compared. This is because, by default, columns are only compared if they are in the same span. That is, in this table, the Non-workers columns are compared with each other and the Work columns are compared with each other (see also How to Specify Columns to be Compared). However, the font color, which represents the Cell comparisons (also called Exception Tests), shows that in June, the percentage for If I wanted to eat fried chicken, I would go to KFC is higher among the Work segment than the Non-workers segment, as the cell comparisons do not take the spans into account.

The second row of the table more starkly illustrates the difference between the column comparisons and the cell comparisons. The font colors show us that the Non-Workers, in general, find the side orders more appealing than the Workers. The column comparisons show that among the Non-Workers, the June score was significantly higher than the April score.

Planned comparisons (alpha) versus Cell and Column comparisons

In the second row of the table below, the only significant cell comparisons are for May (i.e., these are the only comparisons in color). But, if you select the April cell and press , the result is shown as being Significant, as is shown below. On face value, this appears to be inconsistent.

The reason for the difference is that by default in Q, Cell and Column comparisons have multiple comparison corrections automatically applied to them, whereas planned tests, like the one conducted with the button do not.

If you look at the output shown below, you will see that this seeming inconsistency is also described in the sentence immediately below where it is marked as significant. This difference in behavior is by design.

Note also that when selecting a single cell and pressing Q will show the results of a Cell comparison (also called Exception Test), whereas if you select a pair of cells Q will show a Column comparison.