- Tables with hundreds of rows and/or columns will take longer to load as the code needs to run through each row and column every time it loads.
- Applying a rule multiple times to a table to perform a specific task on a different row or column (instead of including this within the same loop) will result in inefficient code that could cause performance issues when loading the table.
- Rules that rely on temporary tables being generated within the code may also cause slow-running tables, especially when used on a large table, a large data set or both.
Significance Testing in Rules
One example is the addition of rows or columns to the table. Since the significance results are computed prior to the addition of these rows or columns, the new cells can have no significance results.
- Changes the number of cells in the table
- Changes the value of a statistic in any of the cells
- Applying filters or weights to some cells and not others
Modifying Significance Tests via Rules
There can be situations where it may be appropriate to override Q's in-built significance tests:
- There is a need to modify some aspects of the setting that cannot be modified using Edit > Project Options > Customize > Statistical Assumptions or by modifying how the data is set up (e.g., by changing the Question Type or modifying the table).
- To replicate results from another program (see Results Are Different to those from Another Program).
- Because there are properties of the data that Q is not able to discern.
- To overcome limitations of Q's tests.
In general, however, we do not recommend modifying tests. The general experience of our support team is that more often than not, modifications of testing performed by users seem to be likely to reduce the validity of the users' analyses.
Care needs to be undertaken if modifying Q's tests using rules. Some particular aspects that need to be addressed are:
- That the test that is to be used is actually better. That is, the automatic tests in Q are designed to take the following factors into account, and if over-riding these tests it is important to verify that these issues are still being addressed appropriately:
- Multiple Comparisons (Post Hoc Testing).
- Statistical power.
- Edge cases (e.g., variables with zero variance).
- Numerical precision.
- That users do not inadvertently apply the modifications to the wrong data (e.g., if applying a rule to a specific table, care needs to be taken to avoid inadvertently copying that rule and applying it to other rules).
- Avoiding inconsistencies. For example:
- If you modify the way that tests are conducted on tables, this will not have any impact on other areas where Q conducts testing automatically.
- Different implications from tests shown on the main section of the table versus those shown in Statistics - Right and Statistics - Below.