Statistics / Unit 9: Hypothesis Testing Basics
9.6 Statistical vs Practical Significance
Notes
Statistical significance means the result is unlikely to be due to chance alone. Practical significance asks whether the size of the effect matters in real life.
- With a very large sample, even a tiny difference can have a very small p-value.
- With a small sample, a large and important difference may fail to be statistically significant.
- A p-value says nothing about the size of the effect. Look at the estimated difference or an interval too.
- Example: A drug lowers blood pressure by 0.2 points on average with $p < 0.001$ in a study of 100,000 patients. This is statistically significant but too small to matter.
Practice
A study of 100,000 patients finds a drug lowers blood pressure by an average of 0.2 points with $p < 0.001$. Which is the best conclusion?
A small p-value with a huge sample can detect a tiny effect.
A small study of 12 students finds a new tutoring program raised scores by 10 points on average, but $p = 0.12$. A reasonable response is:
A large but noisy result from a small sample is not convincing, but it can justify more data.
Which statement about a very small p-value is correct?
The p-value measures evidence against $H_0$, not the size of the effect.
Ready for more? Take the Unit 9 test.