Hypothesis Testing, Statistical Power, and Sample Size

Summary of the Previous Session

In the previous session, we explored the use of online platforms like MTurk in experimental research, emphasizing their potential to enhance participant diversity, lower costs, and increase scalability. We discussed key challenges such as response noise, attentiveness, and learning effects, alongside mitigation strategies to address these issues. Additionally, we examined the role of randomization, external validity, and ecological validity in designing robust experiments, with a focus on balancing realism and control across laboratory, artefactual, framed, and natural field experiments. Lastly, we introduced a hands-on experiment design exercise using Qualtrics.

Building on these foundations, this session focuses on sample size and statistical power—key elements in designing robust experiments that are both credible and replicable. Understanding how to determine an appropriate sample size and ensure sufficient power is essential for overcoming challenges related to variability and ensuring reliable results.

Overview of Hypothesis Testing, Type I and Type II Errors, and p-value

Hypothesis Testing Framework

• Null Hypothesis (H$_0$): Assumes no effect or no difference (e.g., "The intervention has no impact").

• Alternative Hypothesis (H$_1$): Suggests there is an effect or difference (e.g., "The intervention has a significant impact").

Type I and II Errors

Type I and Type II errors are critical concepts in statistical analysis and hypothesis testing. Understanding these errors is essential for designing robust studies and interpreting results accurately.

• Type I Error (False Positive): Occurs when the null hypothesis (H$_0$) is incorrectly rejected, suggesting an effect or difference that does not exist. In other words, it occurs when H$_0$ is rejected even though it is true. The probability of this error is denoted by the significance level, $\alpha$.

• Type II Error (False Negative): Occurs when the null hypothesis is incorrectly not rejected, failing to detect a real effect or difference. In other words, it occurs when H$_0$ is not rejected even though H$_1$ is true. The probability of this error is denoted by $\beta$.

Simple Life Example: Home Pregnancy Test

• Hypotheses:

  • H$_0$: The person is not pregnant.

  • H$_1$: The person is pregnant.

• Type I Error: The test indicates pregnancy (reject H$_0$) when the person is not pregnant. This can lead to unnecessary emotional and logistical preparations.

• Type II Error: The test fails to detect pregnancy (fail to reject H$_0$) when the person is actually pregnant, causing missed opportunities for preparation.

The Concept of the p-value

Definition: The p-value represents the probability of obtaining a test statistic as extreme as the observed one, assuming H$_0$ is true.

Common Misconception: The p-value does not indicate the probability that H$_0$ is true but rather the compatibility of the data with H$_0$.

A small p-value indicates that the observed data would be highly unlikely if the null hypothesis ($H_0$) were true. However, this does not mean that the probability of $H_0$ being true, given the data, is low. Here’s why:

A. Direction of the Conditional

• $P(\text{data} \mid H_0)$: The p-value represents the probability of obtaining data as extreme or more extreme than the observed data, assuming that $H_0$ is true.

• $P(H_0 \mid \text{data})$: This is the probability that the null hypothesis is true, given the observed data—which is what is often mistakenly inferred from the p-value.

The key distinction is that the p-value provides $P(\text{data} \mid H_0)$, not $P(H_0 \mid \text{data})$. These are fundamentally different, and confusing them is a logical error known as the transposed conditional or prosecutor’s fallacy.

Suppose that in a murder trial, the prosecution presents DNA evidence found at the crime scene and argues:

"The probability of this DNA match occurring by chance if the defendant were innocent is only 1 in 1,000,000. Therefore, the probability that the defendant is innocent is only 1 in 1,000,000."

This reasoning is incorrect because even though the chance of a false match for any one person is very small (1 in 1,000,000), when you apply it to a large population (10 million people), you end up with multiple innocent people who match the DNA. This dramatically reduces the probability that the matched person is actually guilty.

B. Lack of Prior Information

In frequentist statistics, the p-value is calculated without incorporating any prior belief or probability about $H_0$. It does not account for how plausible $H_0$ was before the data were observed.

To estimate $P(H_0 \mid \text{data})$, one typically needs a Bayesian approach, which involves prior probabilities and uses Bayes’ theorem to update beliefs based on the data:

$$P(H_0 \mid \text{data}) = \frac{P(\text{data} \mid H_0) P(H_0)}{P(\text{data})}$$

where:

• $P(H_0)$ is the prior probability of $H_0$, representing its plausibility before observing the data.

• $P(\text{data} \mid H_0)$ is the likelihood of the observed data assuming $H_0$ is true (this is what the p-value measures).

• $P(\text{data})$ is the overall probability of the data under all possible hypotheses.

Unlike the frequentist approach, Bayesian inference allows us to compute $P(H_0 \mid \text{data})$, which directly answers how probable $H_0$ is given the data. However, it requires specifying a prior, which can be subjective.

p-value is Not Enough

A high p-value in hypothesis testing can arise from two key scenarios:

1. The Null Hypothesis is True: In this case, the observed data aligns well with the assumption of no effect or no difference. The high p-value correctly reflects that there is insufficient evidence to reject the null hypothesis (H$_0$). For instance, if a new drug is genuinely no better than the placebo, the p-value will likely be high, and the study correctly fails to reject H$_0$.

2. The Study Lacked the Power to Detect a Real Effect: Even if the alternative hypothesis (H$_1$) is true, the study may fail to detect the effect if it is underpowered. An underpowered study can result from:

   • Small Sample Size: Insufficient data reduces the study’s ability to distinguish between the null and alternative hypotheses.

   • Large Variability in Data: High variability in measurements obscures the effect, making it harder to achieve statistical significance.

   • Small Effect Size: If the actual difference or effect is subtle, a study with inadequate power might not detect it.

   This situation leads to a Type II error, where we fail to reject H$_0$ despite H$_1$ being true.

Statistical Power

Definition and Meaning

Statistical Power is the probability that a test will correctly reject the null hypothesis (H$_0$) when the alternative hypothesis (H$_1$) is true. Mathematically, it is expressed as:

$$\text{Power} = 1 - \beta$$

where $\beta$ is the probability of making a Type II error.

Importance of Statistical Power

• Minimizing Errors: High power reduces the likelihood of Type II errors, ensuring that real effects are not overlooked.

• Efficient Resource Use: Adequate power prevents wasted resources on inconclusive studies.

• Reliable Conclusions: Ensures that detected effects are meaningful and not due to chance.

Factors Affecting Statistical Power

Several factors influence the power of a study:

• Sample Size: Larger samples reduce variability and increase power.

• Effect Size: Detecting larger differences or effects is easier and requires less power.

• Significance Level ($\alpha$): Lowering $\alpha$ (e.g., from 0.05 to 0.01) reduces Type I errors but requires more power to detect an effect.

• Variability in Data: Lower variability increases the study’s ability to detect effects.

Simple Example: Flashlight Testing

Imagine you are testing whether a flashlight works, which corresponds to testing the null and alternative hypotheses:

• Null Hypothesis (H$_0$): The flashlight does not work.

• Alternative Hypothesis (H$_1$): The flashlight works.

The role of the battery in this test is analogous to the concept of statistical power:

• High Power (Fully Charged Battery): A fully charged battery ensures the flashlight will emit light if it works. In this case:

  • If H$_1$ (the flashlight works) is true, the high power ensures that the flashlight lights up, and we correctly reject H$_0$.

  • If H$_0$ (the flashlight does not work) is true, the high power still allows us to correctly conclude that the flashlight is faulty.

• Low Power (Weak Battery): A weak or nearly depleted battery reduces the chance of detecting whether the flashlight works, even if H$_1$ is true. In this case:

  • If H$_1$ is true (the flashlight works), the weak battery might fail to light the flashlight, leading to a Type II error (failing to reject H$_0$ when H$_1$ is true).

  • If H$_0$ is true (the flashlight does not work), the weak battery might not affect the outcome, as the flashlight would not light up regardless of the battery’s state.

Takeaway: The "battery" (statistical power) in this analogy determines how well the test can detect the truth of the alternative hypothesis. A high-power study (like using a fully charged battery) ensures that if the flashlight works (H$_1$), we will detect it. A low-power study (like using a weak battery) risks failing to detect the truth of H$_1$, leading to incorrect conclusions.

Business Example: Product Launch Decision

• Scenario: A company tests whether a new product improves customer satisfaction.

• High Power Study: A sufficiently large sample size and robust design reveal that the product significantly increases satisfaction.

• Low Power Study: An underpowered study might fail to detect the improvement, leading the company to wrongly assume the product has no impact.

Ensuring Adequate Power

Researchers can enhance power by:

• Increasing the sample size.

• Reducing variability through better measurement techniques.

• Choosing a suitable significance level and effect size for the study context.

Sample Size and Power Calculations

Definition

Sample size refers to the minimum number of observations required to achieve reliable and valid conclusions in a study. Determining the appropriate sample size is critical for:

• Avoiding wasted resources (overestimating sample size).

• Preventing underpowered studies (leading to Type II errors).

• Ensuring statistical validity (by reducing the liklihood of type I and type II errors).

Key Parameters in Sample Size Calculations

1. Research Question: Use frameworks like PICO (Population, Intervention, Comparison, Outcome) to formulate research questions.

   • Example (Medical): Testing the efficacy of a new diabetes drug versus a placebo on reducing HbA1c levels on a group of Medicare enrolees.

   • Example (Business): Evaluating the effect of a new loyalty program on customer retention rates.

2. Null and Alternative Hypotheses:

   • H$_0$: No significant difference or effect.

   • H$_1$: Significant difference or effect.

3. Significance Level ($\alpha$):

   • The probability of a Type I error.

   • Common values: 0.05 (5%), 0.01 (1%).

4. Power ($1 - \beta$):

   • The probability of correctly rejecting H$_0$ when H$_1$ is true.

   • Common target: 80% or 90%.

5. Effect Size (ES):

   • Magnitude of the difference between groups.

   • Small ES requires larger sample sizes.

   • Example: Cohen’s $d$, Risk Difference (RD) (see Table 3.1 for details).

   Table 3.1. Comparison of Cohen’s $d$ and Risk Difference (RD).

   Aspect	Cohen’s $d$	Risk Difference (RD)
   Definition	Standardized mean difference between two groups, measuring the magnitude of the effect.	Absolute difference in proportions of an outcome between two groups.
   Mathematical Formula	$d = \frac{\bar{X}_1 - \bar{X}_2}{s}$, where $\bar{X}_1$ and $\bar{X}_2$ are group means, and $s$ is the pooled standard deviation.	$RD = p_1 - p_2$, where $p_1$ and $p_2$ are the proportions of an outcome in the intervention and control groups.
   Type of Outcome	Continuous (e.g., test scores, blood pressure).	Binary (e.g., disease presence/absence, treatment success/failure).
   Unit	Standardized (in units of standard deviation).	Absolute difference (e.g., percentage or proportion).
   Interpretation	$d = 0.2$: Small, $d = 0.5$: Medium, $d = 0.8$: Large effect size.	$RD > 0$: Increased risk in intervention group. $RD < 0$: Decreased risk. $RD = 0$: No difference.
   Example Context	Comparing average test scores or blood pressure between two groups.	Comparing infection rates or treatment success between intervention and control groups.

6. Variability (Standard Deviation): Variability refers to the spread or dispersion of data within a population, often measured by the standard deviation ($\sigma$). It reflects how much individual data points differ from the population mean.

7. Margin of Error ($d$): The margin of error represents the maximum allowable difference between the true population parameter (e.g., mean or proportion) and the estimate obtained from the sample. It defines the level of precision desired in the results. For example, $d = 5\%$ means the estimate will be within $\pm 5\%$ of the true value with the specified confidence level.

   Relationship Between Variability and Margin of Error ($d$):

   • Variability and margin of error are inversely related:

     • Higher Variability: Increases the required sample size to achieve a smaller margin of error, as it makes the data less precise.

     • Lower Variability: Reduces the sample size needed to maintain precision, allowing for a tighter margin of error.

   • For a fixed sample size, greater variability results in a wider margin of error, reducing the precision of the estimate.

8. Dropout Rate:

   • Adjust sample size for anticipated losses (e.g., 10–20%).

General Formula for Sample Size Calculation

• Categorical Outcomes (Proportions/Prevalence): $$n = \frac{(Z_{1-\alpha/2} + Z_{1-\beta})^2 \cdot \big[p_1(1-p_1) + p_2(1-p_2)\big]}{d^2}$$ Where:

  • $n$: Required sample size per group.

  • $Z_{1-\alpha/2}$: Z-value corresponding to the desired confidence level (e.g., 1.96 for 95% confidence).

  • $Z_{1-\beta}$: Z-value corresponding to the desired power (e.g., 0.84 for 80% power).

  • $p_1$: Proportion of success in the first group (e.g., control group).

  • $p_2$: Proportion of success in the second group (e.g., intervention group).

  • $d$: Minimum detectable difference between proportions ($|p_1 - p_2|$).

• Continuous Outcomes (Means): $$n = \frac{(Z_{1-\alpha/2} + Z_{1-\beta})^2 \cdot \sigma^2}{d^2}$$ Where:

  • $n$: Required sample size per group.

  • $Z_{1-\alpha/2}$: Z-value corresponding to the desired confidence level (e.g., 1.96 for 95% confidence).

  • $Z_{1-\beta}$: Z-value corresponding to the desired power (e.g., 0.84 for 80% power).

  • $\sigma$: Standard deviation of the population (a measure of variability in the data).

  • $d$: Minimum detectable difference or effect size (the margin of error or precision desired).

  Note that this formula is very similar to the one given in Columb et al. (2016).

  $$n > \frac{2K\sigma^2}{d^2}$$ Where:

  • $K$ is a constant determined by the desired confidence level ($\alpha$) and power ($1-\beta$).

  • Typically, for $\alpha = 0.05$ (two-tailed) and $\beta = 0.20$ (80% power), $K = 7.9$.

  • This form is useful for quick approximations.

  Simplified Formula: $$n > \frac{16}{(\text{standardized difference})^2}$$ Where:

  • $\text{standardized difference} = \frac{d}{\sigma}$.

  • If $d = \sigma$, then $n = 16$ per group.

  • This formula assumes $\alpha = 0.05$ (two-tailed) and $1 - \beta = 0.80$.

Examples

Categorical Outcome – Free Shipping and Purchase Rate

Scenario:

A business wants to test whether offering free shipping increases the proportion of customers who make a purchase compared to no free shipping. The goal is to calculate the required sample size to detect a meaningful difference in purchase rates between the two groups.

Inputs:

• Proportion in the control group ($p_1$): $p_1 = 0.1$ (10% of customers make a purchase without free shipping).

• Proportion in the intervention group ($p_2$): $p_2 = 0.15$ (15% of customers are expected to make a purchase with free shipping).

• Margin of error ($d$): $d = |p_2 - p_1| = 0.05$ (minimum detectable difference in purchase rates).

• Confidence level ($1-\alpha$): 95% ($Z_{1-\alpha/2} = 1.96$).

• Power ($1-\beta$): 80% ($Z_{1-\beta} = 0.84$).

Formula:

$$n = \frac{(Z_{1-\alpha/2} + Z_{1-\beta})^2 \cdot \big[p_1(1-p_1) + p_2(1-p_2)\big]}{d^2}$$

Calculation:

$$n = \frac{(2.8)^2 \cdot 0.2175}{0.0025} = \frac{7.84 \cdot 0.2175}{0.0025} = \frac{1.7052}{0.0025} \approx 683$$

Adjustment for Dropout:

Assume a 10% dropout rate: $$n^* = \frac{n}{1 - \text{Dropout Rate}} = \frac{683}{0.9} \approx 759$$

Conclusion:

The study requires 759 participants per group (1518 total) to detect a 5% increase in purchase rates due to free shipping, with 80% power and a 95% confidence level, accounting for a 10% dropout rate.

Continuous Outcome – Blood Pressure Reduction

Scenario:

A clinical trial tests whether a new drug lowers systolic blood pressure (SBP) compared to a placebo. The required sample size needs to be calculated to detect a meaningful reduction in SBP.

Inputs:

• Expected difference in means ($d$): $d = 10$ mmHg (the drug is hypothesized to reduce SBP by 10 mmHg).

• Standard deviation ($\sigma$): $\sigma = 15$ mmHg (based on prior studies).

• Confidence level ($1-\alpha$): 95% ($Z_{1-\alpha/2} = 1.96$).

• Power ($1-\beta$): 80% ($Z_{1-\beta} = 0.84$).

Formula:

$$n = \frac{(Z_{1-\alpha/2} + Z_{1-\beta})^2 \cdot \sigma^2}{d^2}$$

Calculation:

$$n = \frac{(1.96 + 0.84)^2 \cdot (15)^2}{(10)^2} \approx 18$$

Adjustment for Dropout: Assume a 10% dropout rate: $$n^* = \frac{n}{1 - \text{Dropout Rate}} = \frac{18}{0.9} = 20$$

Conclusion:

The study requires 20 participants per group (40 total) to detect a 10 mmHg reduction in systolic blood pressure with 80% power and a 95% confidence level, accounting for a 10% dropout rate.

Calculate the Requeired Sample Size in Gu et al. 2022

(Zhuojun Gu, Ravi Bapna, Jason Chan, and Alok Gupta. Measuring the impact of crowdsourcing features on mobile app user engagement and retention: A randomized field experiment. Management Science, 68(2):1297–1329, 2022)

This paper examines how crowdsourcing features in mobile apps impact user engagement and retention. The experiment involved four randomized groups of app users:

• Control Group: No crowdsourcing features enabled.

• Submission-Only Group: Users could submit content but not access it.

• Access-Only Group: Users could view community-generated content but not contribute.

• Full-Crowdsourcing Group: Users could both submit and access content.

Each group consisted of approximately 750 users, totaling 3,056 participants.

Sample Size Calculations For User Engagement (Continuous Outcome)

Inputs:

• Significance level ($1-\alpha$): 95% ($Z_{1-\alpha/2} = 1.96$).

• Power ($1-\beta$): 90% ($Z_{1-\beta} = 1.28$).

• Effect size ($d$): 0.12.

• Standard deviation ($\sigma$): Assume $\sigma = 1$ (This assumption works under the context of effect sizes that are expressed in standardized units, like Cohen’s $d$, where $\sigma$ is typically normalized to 1. ).

Formula:

$$n = \frac{(Z_{1-\alpha/2} + Z_{1-\beta})^2 \cdot \sigma^2}{d^2}$$

Calculation:

$$n = \frac{(1.96 + 1.28)^2 \cdot 1^2}{(0.12)^2} \approx 729 \text{ participants per group.}$$

Ethical Considerations in Determination of Sample Size

While we have to consider theoretical and statistical justifications to determine sample sizes, in practice, we may deviate from those calculations. Also, more is not always better when it comes to number of participants.

Underpowered Studies:

• An underpowered study has a high probability of Type II errors, meaning it may fail to detect a true effect.

• In clinical research, this could mean that a potentially life-saving treatment is dismissed due to lack of statistical significance.

• Ethical concerns arise because participants are exposed to risks and burdens without generating meaningful scientific knowledge.

• Example: A cancer drug trial with inadequate sample size may conclude "no significant benefit," even if a larger study would have found an effect.

Overpowered Studies:

• Increasing sample size reduces Type II errors, but excessive sample sizes raise concerns about unnecessary resource use and ethical considerations.

• In clinical trials, enrolling more participants than needed exposes them to potential risks without justification.

• Overpowered studies can detect statistically significant but clinically meaningless differences, leading to overinterpretation.

• Example: A study finds a new drug reduces blood pressure by 0.5 mmHg with $p < 0.001$. While statistically significant, the effect size is too small to be clinically meaningful.

Balancing Feasibility Constraints and Statistical Rigor

Researchers face budget, time, and recruitment challenges, making it impractical to always aim for high statistical power. Strategies for balance include:

• Using pilot studies to estimate variance and optimize sample size.

• Considering adaptive designs where sample size can be adjusted mid-study.

• Leveraging meta-analysis to aggregate findings from multiple smaller studies.

• Pre-registering effect sizes to prevent unnecessary increases in sample size.

When Would a Researcher Choose Lower Power (e.g., $<80\%$)?

• Rare diseases or hard-to-recruit populations (e.g., genetic disorders, high-risk trials) Recruiting a large sample is infeasible, so researchers may accept lower power (e.g., 50-70%) to gain at least some evidence rather than none.

• Exploratory or early-phase studies Small sample sizes are acceptable when the goal is hypothesis generation rather than definitive conclusions.

• When effect sizes are expected to be large If prior evidence strongly suggests a large effect, researchers may accept lower power because even a small study is likely to detect it.

Required readings