Articles in this section

See more

How to Troubleshoot Weights

Liz
  • Updated
Follow

This article walks through steps on what to look into when you have an issue with a weight in your document. This could be an error or simply the weight is not weighting targets as you expect. Work through the steps below to uncover what is going on.

1. Gather the details about the issue

2. Find what is causing the issue

3. Figure out a workaround or a solution

4. Contact Support

 

Requirements

 

1. Gather the details about the issue

You will use these details and information to help in the troubleshooting steps below.

    1. If the weight was created in Q, do you see any error messages in the bottom right of the weighting tool? We have an article that discusses particular weight errors and how to troubleshoot them here: Common Errors in Weights.
    2. Apply the weight to a table of each adjustment variable used to build the weight to see whether the target values are what you expect. Note any issues.
    3. What documentation and help information is available about the tool(s) or issue? Consult How to Configure a Weight from Variable(s) and How To Apply Different Weighting Structures To Sub-Samples. These are two useful documents that walk through what the various fields can do and common techniques for creating complex weights. You can search the Q Help Center,  Q wiki, and Q blog/website for any other relevant material.
    4. If comparing a weighted table from Q to one created outside of Q, you will need to gather the underlying information about the data used to create both tables. This includes the weighted and unweighted values for: ns and the relevant Sample Sizes (Base n, Column n, Row n). Do note, Q rounds to 13 significant digits (not decimals) and “away from zero”.
    5. If comparing respondent-level weight variable values created in Q vs those created outside, know that these likely will not match due to various algorithms and tools used to create respondent-level weighting values. This is expected.

2. Find what is causing the issue

What aspects of the issue do you have control over (i.e. structure of the table, order of rules, user permissions, etc.) that you can tweak and test to flag as the root cause of the problem? Once you figure out what is causing the issue, then you can go to step 3. Figure out a workaround or a solution.

  1. Confirm your current inputs and process match the documentation (if there is any available).
    1. Confirm the weight variable has Tags > W highlighted on the Variables and Questions tab.
    2. Confirm any tables in question have the correct weight variable selected in the Weight dropdown.
    3. Confirm any settings in the weighting tool are used appropriately according to the documentation. Three fields to double check specifically are what it means to use: Recompute weights for, Making weights sum to, and Design Weights. For more information see: How to Configure a Weight from Variable(s)
    4. Confirm the structure of the adjustment variables are single Pick One questions and any recompute variables are Pick One or Pick Any questions.
    5. Check that the categories in the Value Attributes are what you expect to be using. For example, when using a variable to recompute the weighting, all respondents in categories - even respondents with missing data for that variable - will be recomputed. Remember you can merge together categories within the tool by dragging and dropping or using the steps here: How to Make Merged Categories Appear in Targets when Weighting.
    6. If using a combined weight variable (a weight created using R code or JS code to stitch together weight values from individual weight variables as-in here), ensure the correct groups are being used in the R conditional statements and those are paired with the appropriate sub-sample or individual weight variable.  You can hover over the sub-sample variable name to see the categories used to confirm. Keep in mind if you want to leave any respondent unweighted in a combined weight variable, they will be given a weight of 1. To exclude respondents from the final weight you will give them a value of 0. Ensure the last line of code is the final variable name, this will return the full lists of weights for the variable. 
    7. If applying a weight to a custom R Output, remember that weights only can be applied to raw data values - not tables referenced in the code. Weights should be applied to input tables referenced by the code on the table directly. If using raw data in the R Output, you can adjust it by multiplying it by the variable QPopulationWeight (which will be the respondent-level weight values of the weight selected in the Weight dropdown on the R Output), see How to Perform Mathematical Calculations Using R for more info.
  2. Test aspects you have control over to see what triggers the issue.
    1. If an issue with values shown on a table using rules. Remove all rules from the table to see if that fixes the issue. If it does, add back the rules one at a time to see which one triggers the problem.
    2. If using a combined weight variable (see explanation in 2.1.f above), apply the individual weights to the adjustment variable tables to confirm that they are working as expected. If they aren't, troubleshoot potential issues with the weighting tool restarting with 1 above for confirming inputs and processes. Otherwise, if the individual weights are fine, use How to Troubleshoot R Code to figure out why the individual weights aren't getting put together correctly in the R code of the combined variable.
    3. If using a weight created outside Q, compare the unweighted ns and sample sizes (Base n, Column n, Row n) of the tables that aren't matching to confirm the data used to create the tables is the same. Then, apply the weight and compare the weighted statistics to identify a particular category that is off. Note that you should look at the Population and population sample size statistics. Then compare the individual weight values for respondents in that category to confirm the values of their weight in Q vs the outside program.
    4. For weights created inside Q, try changing the field Recompute weights for (if using), Making weights sum to, removing any maximums and minimums, and changing the algorithm back to the default Raking algorithm, if you have modified any of those fields.
    5. If step d doesn't fix anything, rebuild the weight from scratch, using just the first adjustment variable, and any supplementary settings (Recompute weights for and Making weights sum to, maximum and minimum). Create a table of the adjustment variable (crossed by the recompute variable if used) and apply the weight to confirm that the targets are as expected. If not, then start changing supplementary settings back to their default one at a time until you get what you expect.
    6. If things look fine after step e, continue adding adjustment variables and checking the adjustment variables tables to confirm their targets one at a time. Eventually adding an adjustment variable will trigger the issue.
    7. Once you find the adjustment variable that creates the issue, create crosstabs of it and the other adjustment variables to see if any cells have very small sample sizes. This can affect how the raking algorithm converges and may be seen in the rounding of aggregated numbers in summary tables. 
  3. Use what you've learned and confirmed to identify what causes the trigger to create the issue.
    1. Are any of the adjustment variables modified or different in a way that may affect the weight?
    2. Do you need to reorder or remove any rules applied to the table? Do read the technical documentation of the relevant rules to ensure they are appropriate to apply or note any caveats regarding the weights.
    3. Are the settings appropriate given the limitations of your data? If dealing with small sample sizes for the intersection of some targets (found in f above) having min and max restrictions on weighting values may cause the algorithm to not converge. 

3. Figure out a workaround or a solution

If the steps above don't lead to an obvious solution, use what you've learned to work toward a solution.

  1. If dealing with small sample sizes, can you merge categories or remove any adjustment variables to allow the weighting algorithm to converge? Can you weight a portion of your data differently to account for the small sample? (see How To Apply Different Weighting Structures To Sub-Samples)
  2. Is the setting or issue that caused the issue required or can you work without it?

4. Contact Support

If you've worked through these steps and haven't found a solution to the issue or need help in making the solution, please contact our support team at support@q-researchsoftware.com or by clicking on File > Share > Send to Q Support (Encrypted) from within your specific project in Q. Please provide the details of what you learned in the steps above.

 

Next

How to Investigate a Sample Size or NET that is Too Small

Related articles

Comments

0 comments

Article is closed for comments.