How to Replicate Survey Reporter Significance Tests

Survey Reporter is an analytics software that has been around for a long time. Many customers and their clients who want to transition to more modern software may want to replicate, exactly, the significance testing for legacy analyses

Method - Configure Stat Testing

1. Make the following changes in Statistical Assumptions:
   • Change Show significance to Compare columns.
   • Change both Multiple comparison correction settings to None.
   • Set the Minimal sample size for testing to its level in Survey Reporter (which defaults to 30).
   • Change the Proportions setting to Survey Reporter Proportions.
   • Change the Means setting to Survey Reporter Means.
   • Change Weights and significance to Kish's approximation.
   • Uncheck Recycle column letters.
   • Replace the No test symbol with **.
   • In the Significance levels and appearance grid, copy the one uppercase Column letters fields (i.e., A,B,C,...,) and paste it over the top of each of the lowercase fields (i.e., a,b,c,...,). Note that this will remove Q's default of using uppercase and lowercase to denote different levels of significance. However, it will not precisely replicate Survey Reporter's use of column letters as Q assigns a letter to NET fields but Survey Reporter does not. Two ways of resolving this are to:
     • Remove the NET categories from within Spans.
     • Manually change the contents of Column letters to exactly replace Survey Reporter if so required, but this will need to be redone for each question in the Brown Drop-down Menu.
     • Where a Pick Any question is in the Brown Drop-down Menu or analyzing a Number - Grid, Pick Any - Grid or Pick One - Multi SUMMARY set Overlaps to Dependent.

Method - To Look into Differences in Results

1. Move the NET categories to be at the beginning top/left of the tables (and sections within a table where Spans are used).
2. Ensure that the base used in statistical comparisons is equivalent (e.g., on a Pick Any - Grid Survey Reporter's base is the total number of responses, whereas Q's is the total number of respondents.
3. Ensure that the rounding is being computed in a way consistent with Q (Q rounds up; by default Survey Reporter rounds to even).
4. Check that the percentages and means being compared in the statistical tests are the same. Where they differ this likely relates to the base. Sometimes Filters can be used to get Q to replicate Survey Reporter's percentages (however, keep in mind that this is something that is only done to aid in comparison rather than being a desirable approach to compute percentages).
5. Ensure that any filters are equivalent (including Survey Reporter's Interview Filter settings).

Next

How to Replicate Quantum Significance Tests