Sample Matters (to Tracking Programs)

As Insights professionals who manage tracking programs, at some point we’ll be asked “Are those true changes in behaviors or perceptions?” That is when we want to be confident that changes in tracking results reflect shifts in the market, not sample inconsistencies! Start your project on the right path, and stay on track, by following these best practices for sample management.

  1. When the population is large, sample selection should be random; this is true even when the population is customers. Sometimes companies attempt to survey all customers, which can result in more surveys than needed, a lower response rate (due to repeated attempts), and less representative findings due to increased non-response.
  2. For an online general population sample, ENGINE’s Census Balancer® tracks the demographics of respondents coming into the survey so that a representative audience is sampled for the survey screener. By doing so, any demographic differences found after any additional screener criteria is reflective of the market.
  3. For B2B studies, quotas are preferred to ensure that the sample mirrors the target audience. Weighting should be reserved for adjusting slight skews that arise from the inability to reach certain segments of the audience in proper proportions.
  4. For transaction-based surveys, sample procedures depend on frequency of the transaction. It is not necessary to get feedback on all re-occurring transactions (e.g., restaurant orders), but it is typically good to get feedback on all of the less common ones (e.g., a claim). If it is a call transfer or website intercept, immediately after a transaction, it’s not possible to control how frequently a person is invited to participate; however, if contact is not immediate, it is possible to control how often a person is contacted.
  5. When partnering with online sample providers it is best to remain consistent throughout a tracking period when trending is involved. To boost response rates, consider offering an incentive, extending the deadline, sending reminders, or attempting to reach via multiple modes.

When making changes, a parallel test is strongly recommended for one or more waves to determine the impact on results. If changes are drastic, the best approach might be to draw a line in the sand and not compare back to less representative samples. However, in less severe circumstances, calibration can be done on some key measures of the past research to align with current results.

The steps to align results from different samples or methods can differ based on the change (e.g, mode, reminders, new panel, etc.). Weighting is one option that is applicable in certain circumstances.

Taking appropriate actions to sample the population correctly will help to ensure changes in results are based on consumer or business opinions not sample miss-steps. ENGINE has the expertise to guide sampling decisions, and other methodology considerations, so you can rest easy your tracking research is measuring what is intended.

Stay tuned or contact us for our future blog post that covers best practices in weighting!

Written by Sheilah Wagner, Research Director at ENGINE Insights.

Share on linkedin
Share on twitter
Share on facebook


Mother's Day

ENGINE Insights Mother’s Day Survey Reveals Americans’ Shopping and Visitation Plans in The Age of COVID-19

This Sunday, May 10th is Mother’s Day, and it will be an unparalleled one as Americans across the nation are faced with COVID-19 and what that means for one of our most celebrated and beloved holidays. The ENGINE Insights team conducted a survey to discover just how the coronavirus will be impacting all of us this Mother’s Day – from how and where we are shopping to whether we plan to visit our Mothers in-person (and the precautions we will be taking) to virtual plans.

COVID 19 Personal Finance


The COVID-19 pandemic can be described as sudden, tragic, and an utter upheaval of the way that we live. And at the same time, it can be viewed as a great equalizer; with essentially every American impacted to a certain degree. In this new world that we live in household savings are down, unemployment is (way) up and for those still employed income has weakened with dampened future expectations.

Covid-19 consumer


These Online CARAVAN® omnibus surveys were conducted March 13-15, March 20-22, March 27-29, April 3-5, April 10-12, April 17-19, April 24-26, and May 1-3, 2020. Approximately 1000 adults selected from opt-in panels were surveyed each time. The results are also weighted to U.S. Census data to be demographically representative. 


Start typing and press Enter to search