Key Differences: Statistical vs. Clinical Significance

1. Statistical Significance
   • Tells us if an observed effect is likely real and not due to chance.
   • Determined by the p-value (e.g., p < 0.05 means less than a 5% chance the result is random).
   • Example: A study finds Drug A lowers blood pressure (BP) by 10 mmHg with p = 0.03. This means there’s a 97% chance the reduction is real.
2. Clinical Significance
   • Focuses on whether the effect actually improves patient health.
   • A result can be statistically significant but not clinically meaningful.
   • Example: A drug that lowers BP by 1 mmHg with p = 0.001 is statistically significant, but a 1 mmHg reduction won’t meaningfully impact patient health.

---

P-Value (Probability Value)

• Definition: Measures how likely the observed result happened by chance.
• Low p-value (<0.05): Unlikely the result is random → suggests a real effect.
• Example:
  • A study finds that a new diabetes drug reduces HbA1c by 1.5% with p = 0.02. This suggests the effect is unlikely due to chance.
  • A p-value of 0.03 means there is a 3% chance that the observed result happened by random chance, implying 97% confidence that the effect is real.

---

Confidence Interval (CI)

• Definition: A range showing where the true effect likely lies, indicating precision.
• Narrow CI: More confidence in the estimate.
• Wide CI: Greater uncertainty.
• Example: A drug reduces cholesterol by 20 mg/dL with a CI of 18 to 22 mg/dL → precise estimate.
  • If CI were 10 to 30 mg/dL, the effect is uncertain and less reliable.

---

Example: Evaluating a New Antihypertensive Drug (Drug A)

Study Design

• 200 patients with high BP: 100 get Drug A, 100 get a placebo.
• Outcome measures: BP reduction and proportion reaching target BP (<140/90 mmHg).

Results:

1. BP Reduction:
   • Drug A: Mean reduction = 15 mmHg
   • Placebo: Mean reduction = 7 mmHg
   • p = 0.02 → Statistically significant (unlikely due to chance).
2. Proportion of Patients Reaching Target BP:
   • Drug A Group: 60% (60/100 patients)
   • Placebo Group: 40% (40/100 patients)
   • Chi-square test p-value = 0.02 → Statistically significant.

Is the Effect Clinically Meaningful?

• Relative Risk (RR): Patients on Drug A are 1.5 times more likely to reach target BP.
• Absolute Risk Reduction (ARR): 20% more patients reach target BP with Drug A.
• Number Needed to Treat (NNT): 5 patients need to be treated for 1 extra patient to benefit.

Conclusion

• Statistically significant (p = 0.02).
• Clinically significant (BP drop of 15 mmHg + NNT of 5 shows real patient benefit).
• Overall: Drug A is both statistically and clinically meaningful.

---

Key Takeaways

Term	Meaning	Example
Statistical Significance	Effect is real, not random.	A BP drug lowers BP by 10 mmHg with p = 0.02.
Clinical Significance	Effect is meaningful for patients.	A BP drug lowers BP by 15 mmHg, which improves heart health.
P-Value	Probability the result is due to chance.	p = 0.03 means a 3% chance it’s random.
Confidence Interval (CI)	Range of the likely true effect.	BP reduction 10 mmHg (CI: 8 to 12) → precise estimate.

Example:

Understanding the Results of Drug A Study

---

1. Descriptive Statistics: Understanding the BP Reduction Data

What was measured?

• How much systolic blood pressure (SBP) was reduced on average in each group.

Group	Mean SBP Reduction	Median SBP Reduction
Drug A	15 mmHg	14 mmHg
Placebo	7 mmHg	7 mmHg

• Mean vs. Median:
  • The mean is the average reduction in BP.
  • The median is the middle value when all reductions are arranged in order.
  • Since the mean (15 mmHg) and median (14 mmHg) are close, this suggests the data is symmetrically distributed (i.e., not skewed by extreme values).

---

2. Statistical Significance: Is the Difference Real or Just Random?

What was measured?

• Proportion of patients who reached the target BP (<140/90 mmHg).

Group	Reached Target BP	% of Patients
Drug A	60/100 patients	60%
Placebo	40/100 patients	40%

How was significance tested?

• Chi-square test was used to compare the percentages.
• p-value = 0.02 → Since 0.02 < 0.05, this result is statistically significant, meaning it’s unlikely to be due to random chance.

Real-World Meaning: If we repeated this study many times, we would expect to see a similar benefit 98% of the time (since p = 0.02, meaning only a 2% chance this result was random).

---

3. Clinical Significance: Is the Effect Meaningful for Patients?

Now, let’s quantify how beneficial Drug A is compared to placebo.

Step 1: Calculate the Baseline and Treatment Event Rates

• Control Event Rate (CER) = % of placebo patients who reached target BP
  • CER = 40% (40 out of 100 patients).
• Experimental Event Rate (EER) = % of Drug A patients who reached target BP
  • EER = 60% (60 out of 100 patients).

Step 2: Compare Drug A’s Effect to Placebo

Metric	Formula	Calculation	Interpretation
Relative Risk (RR)	EER / CER	0.60 / 0.40 = 1.5	Patients on Drug A are 1.5 times more likely to reach target BP compared to placebo.
Relative Risk Reduction (RRR)	(CER – EER) / CER	(0.40 – 0.60) / 0.40 = 50%	Drug A reduces the risk of not reaching target BP by 50%.
Absolute Risk Reduction (ARR)	EER – CER	0.60 – 0.40 = 20%	20% more patients achieve target BP with Drug A.
Number Needed to Treat (NNT)	1 / ARR	1 / 0.20 = 5	5 patients need to be treated with Drug A for one extra patient to reach target BP.

---

4. Final Interpretation: What Do These Numbers Mean?

Is the Effect Statistically Meaningful?

✅ Yes. The p-value (0.02) shows the improvement is unlikely due to chance.

Is the Effect Clinically Meaningful?

✅ Yes.

• A BP drop of 15 mmHg is a significant reduction that can help lower stroke and heart attack risk.
• 20% more patients reach the target BP with Drug A than placebo.
• Only 5 patients need treatment for one extra person to benefit (NNT = 5 is considered a good result in clinical practice).

---

Key Takeaways

Concept	Meaning	Real-World Example
Descriptive Statistics	Drug A lowers BP more than placebo	Mean BP drop: 15 mmHg (Drug A) vs. 7 mmHg (Placebo)
Statistical Significance	p = 0.02 → Difference is unlikely due to chance	If we repeat this study 100 times, we’d see this benefit 98 times.
Relative Risk (RR)	Drug A patients 1.5x more likely to reach target BP	A 50% higher chance of reaching BP goal.
Absolute Risk Reduction (ARR)	20% more patients reach target BP with Drug A	20 extra people per 100 benefit from the drug.
Number Needed to Treat (NNT)	Treat 5 people for 1 extra to benefit	A strong clinical effect (lower NNT = better).

Final Conclusion

• Drug A is effective at lowering BP.
• Both statistically and clinically significant.
• Real-world benefit: 5 patients need treatment for 1 to reach the target BP who otherwise wouldn’t.