Perhaps one of the most unsettling experiences for corporate researchers is when the results of a study are called into question based on conflicting data from another report.
When decisions are made that have multi-year, strategic consequences, the impact of mistakes can be alarming (and costly). In this type of environment, the pressure is sky-high and the stress of reporting accuracy is equally magnified.
Given the hyperabundance of information that is available, it is inevitable that data from different sources at some point will contradict or, at minimum, seem to tell inconsistent stories.
When this happens, push back on the urge to panic. Instead, ask clear questions and move systematically through a process of interrogating the research with a goal of gaining a deeper understanding of the data.
What the team needs most in these moments is a well-analyzed response. It is in these times of high-stakes tension that our expertise is invaluable.
When we help our clients navigate these types of challenges, here are a few of the steps we advise with recommended questions to get to the other side.
Review the raw data.
An important step is to go back to the source data, if you have direct access to it. A thorough quality check can often illuminate issues that were over looked. Then ask:
- Are there any errors or discrepancies in the data itself?
- Has the data set been checked for duplicate records, missing values, units of analysis, etc.?
Compare methodologies.
Closely consider timeframes, fielding strategies, contextual considerations, and sample groups. Even minor variations can account for differences, especially if one or both of the studies has a smaller sample size. This step also can clarify whether two reports are contradictory or just highlight the issue from completely different angles. Then ask:
- How do the methodologies compare from study to study?
- Were any differences in approach substantive enough to guarantee alternate perspectives?
Evaluate the questionnaires.
Take a look at the entirety of each questionnaire. Spend some time considering the implications of any structural differences and what impact those differences could have had on outcomes. Then ask:
- Were there differences in question phrasing that could explain outcome disparities?
- Were the response choices that were provided the same?
- Was the question order rotated or fixed?
- Were there any questions asked earlier in the survey that might have biased respondent answers to questions asked later?
Reconsider the analytics.
Deep expertise can make all the difference if an analytics error is at the root of an inconsistency, so have a third-party expert thoroughly review any statistical analyses. Sometimes an extra set of eyes will spot errors that are hiding in plain sight. Then ask:
- Is a variable miscategorized?
- Are there flaws in the algorithm?
Communicate the findings.
Once the answer has been discovered, don’t overlook the importance of building a communication plan to report the findings. Work with your team to clearly articulate the steps that were taken to evaluate the differences between studies and transparently share the results.
We always recommend taking a proactive approach that both elevates the value of the team’s critical thinking and demonstrates true partnership in reaching a resolution. It is one of our core values at Quantum Leap Insights to Do The Right Thing, which means being there for our clients every step of the way, especially when challenges emerge.