Quantitative Research Methods

Quantitative Research Methods

Quantitative research answers "how many" and "how much" questions with statistical reliability. It tests hypotheses generated by qualitative research, measures market size, validates message effectiveness, and tracks changes over time. Understanding quantitative methods enables you to design studies, interpret data, and avoid common statistical traps.

Survey Design Principles

A well-designed survey produces reliable, valid data. A poorly designed survey produces data that systematically misrepresents reality.

Wording principles:

• Neutral language: "How often do you use product X?" not "How often do you enjoy product X?"
• Avoid double-barreled questions: "Is this product affordable and high quality?" — which are you rating?
• Avoid leading questions: "Don't you think this feature is useful?" implies the correct answer
• Use mutually exclusive, exhaustive response options: gaps or overlaps in scales produce unreliable data
• Specific rather than abstract: "In the last 7 days, how many times did you..." rather than "How often do you..."

Common biases in survey responses:

• Acquiescence bias: respondents tend to agree with statements regardless of content. Use balanced statements (some favorable, some unfavorable).
• Social desirability bias: respondents answer in ways they think are viewed favorably. Use indirect questions or behavioral measures instead of attitude questions for sensitive topics.
• Halo effect: overall impression of a brand biases ratings of specific attributes. Randomize attribute order; consider separate questionnaires for overall and attribute ratings.
• Primacy and recency effects: items listed first or last receive higher ratings. Randomize item order.

Likert scales: 5-point or 7-point scales from "Strongly Disagree" to "Strongly Agree." 5-point scales are easier; 7-point scales provide more discrimination. Always use the same direction (1=low, 5=high) throughout.

Conjoint Analysis

Conjoint analysis reveals how consumers make trade-offs between product attributes. Instead of asking "how important is price to you?" (which produces acquiescence and rationalization), it presents choices between full product profiles.

Example: choosing between:

• Option A: $15/month, 5GB storage, no support
• Option B: $25/month, 20GB storage, email support
• Option C: $40/month, unlimited storage, 24/7 support

By observing many such choices, the analysis calculates the utility consumers assign to each attribute level. This reveals:

• Which attributes matter most (relative importance)
• The price consumers will pay for each feature upgrade
• The optimal product configuration for a target segment

Types of conjoint:

• Full-profile conjoint: evaluate complete product descriptions. Realistic but cognitively demanding.
• Discrete Choice Modeling (DCE): choose between sets of profiles. Most closely mirrors real purchase decisions.
• Adaptive Conjoint Analysis (ACA): adapts questions based on prior responses; more efficient for many attributes.

A/B Testing

An A/B test randomly assigns users to two (or more) variants and measures a defined outcome. The key to causal inference is randomization — it ensures the only systematic difference between groups is the variant.

The hypothesis structure:

• Null hypothesis (H0): the variant has no effect on the outcome
• Alternative hypothesis (H1): the variant has an effect
• Significance level (α): the probability of rejecting H0 when it is true (typically 5%)
• Statistical power (1-β): the probability of detecting a true effect (typically 80-90%)

Sample size: determined by effect size (how large a difference you want to detect), significance level, and statistical power. Smaller effects require larger samples. A 1% conversion rate change needs far more observations than a 20% change.

Common A/B testing mistakes:

• Peeking: checking results before the predetermined sample size is reached and stopping when significant. This massively inflates false positive rates. Commit to a sample size before starting.
• Testing too many things at once: multi-variable changes make it impossible to attribute effects to specific changes.
• Ignoring practical significance: a statistically significant 0.01% improvement is not worth implementing. Statistical significance ≠ practical significance.
• Not checking for sample ratio mismatch: if one variant receives significantly more traffic than designed, the randomization is broken.

Net Promoter Score

NPS asks: "On a scale of 0-10, how likely are you to recommend [brand/product] to a friend or colleague?"

Promoters (9-10) - Detractors (0-6) = NPS. Passives (7-8) are excluded.

NPS utility: simple, fast, industry-benchmarkable, correlates with growth in some industries. The follow-up qualitative question ("What's the main reason for your score?") is often more valuable than the score itself.

NPS limitations:

• Not a statistically optimal scale (ordinal data treated as continuous)
• The same NPS can mask very different distributions (100% passives = NPS 0; 50% promoters + 50% detractors = NPS 0)
• Industry benchmarks vary widely — NPS 30 is excellent in insurance, poor in consumer electronics
• Doesn't directly measure what drives promoter vs. detractor status

MaxDiff

MaxDiff asks: "From this set of attributes, which is most important to you? Which is least important?"

By rotating sets and observing patterns, MaxDiff produces a ratio-scale ranking of attribute importance.

Advantage over rating scales: forces discrimination. On a Likert scale, respondents tend to rate everything as "very important." MaxDiff forces prioritization — you must choose, revealing true relative importance.

Application: identifying the 3-5 features most critical for your product roadmap; determining which messages to prioritize in advertising; understanding segment differences in what matters most.

Segmentation Analysis

Quantitative segmentation reveals distinct consumer groups with different needs, attitudes, or behaviors:

Cluster analysis: statistical technique grouping consumers by similarity on measured variables (attitudes, behaviors, demographics). Identifies natural groupings in the data.

Latent Class Analysis (LCA): identifies unobserved subgroups in a population; assumes consumers in the same class share similar response patterns; useful for attitudinal and behavioral segmentation.

Segment validation: a good segment is measurable, substantial (large enough to target profitably), accessible (you can reach them), differentiable (members respond differently than non-members), and actionable (you can design a distinct offering or message for them).