How to Investigate a Sample Size or NET that is Too Small

This article describes how to determine why the Sample Size (Base n, Column n, Row n) or NET on a table is smaller than expected. Such as the following table showing a Top 2 Box score across different brands:

Requirements

A table with a NET or Sample Size statistic that is smaller than expected.

Knowledge of how NETs and Sample Sizes are computed with missing data. NETs and Sample Sizes are calculated including only common respondents across all the categories being included. To understand the numbers in the table above, adding sample size statistics or Missing n to the table (as seen below) will help you identify what is causing the small number.

In the above table, 9 respondents have missing data for the Pepsi category, thus there are 591 respondents who have data across all categories making the Base n of the table 591. The NET on the table is 96% (567/591) - meaning out of the 591 respondents that answered all categories, 96% of them gave a Top 2 Box score for at least one of the categories. As Column n is the number of people who selected the category in the column (in this instance it is a Top 2 Box score), Column n is the same as the NET of the table at 567.

Increase Sample Size

Method - Uncheck Missing Data

If you believe the missing data is in error and want to include those 9 respondents with missing data for Pepsi in the numbers, right click on the table and select Values to change the Value Attributes. You will need to uncheck Missing Data on all values to include all respondents in the base, similar to below:

Method - Show the Maximum Sample Size

Q's approach is based on the principle of conservativism - it minimizes the chance that a user of the research will rely upon an unreliable finding. Any approach which puts a higher sample size than what is used by Q runs the risk that users of the research will believe the data to be more robust than is the case. Given this, there is a custom rule you can use, see How to Show Maximum Column Sample Size in a Table to show the maximum Column n at the bottom of the table. For tables where this is not applicable, you can modify the JavaScript code of the rule to use Base n instead, see How to Customize a Rule .

Method - Revise Weights

If your table is being weighted, you should reference Population statistics to see the sample sizes used in the figures and testing on the table. If these numbers appear to be incorrect, review the targets and construction of your weight variable to determine if it needs to be revised. Do note, that if a respondent is given a 0 weight, they will also be removed from the unweighted n statistics as well.

Method - Revise Data

If you have investigated the above options, and you still believe your sample size is too small. You can right click on the variables in question on the Variables and Questions tab and Export Variables to Excel to manually review the raw data in Q. If the sample sizes in Excel match that of Q, you will need to go back to your data provider to investigate.

Make NET = 100%

Method - Rebase to the NET

Rebasing the table to the NET will force the NET to be 100%, see How to Rebase Questions. After running the automation, a new version of the question will be created where respondents not included in the NET will have missing data.

Method - Add a None of These Option

Adding a None of These option will force the NET to be 100%, see How to Add a "None of These" to a Pick Any Question. This will create a new question with a None of These category for respondents who weren't included in the NET originally and thus force the NET to be 100%.