Demystifying the technical definition of a P-value
Breaking down the most misunderstood concept in statistics
The P-value is arguably the most misused and misunderstood concept in all of statistics, so I will use this article to explain what it means from a technical perspective. (In a later article, I will discuss how best to interpret a P-value and explain it to a non-technical audience.)
Here is the technical definition: In the context of hypothesis testing, the P-value is the probability of the test statistic being at least as extreme as the observed value, assuming that the null hypothesis is true.
This is quite a mouthful, so let’s break this definition down into its key elements.
The P-value makes sense only in the context of a hypothesis test, so make sure that you identify the 4 basic elements of your hypothesis test.
The test statistic is the random variable of interest, and it has a certain probability distribution. For example, if you are comparing two population means, I encourage you to use Welch’s t-test. Its test statistic is a t-statistic, which has a t-distribution. Let’s call this random variable T.
This distribution is based on the null hypothesis (not the alternative hypothesis). For the null hypothesis in a Welch’s t-test, we commonly assume that the two groups have no difference in their population means.
The observed value of the test statistic is what you calculate from your data. It is a known value, not a random variable. Let’s call this observed value t₀.
The P-value is a probability. Thus, it can range from 0 to 1.
To find the P-value for your hypothesis test, visualize the probability density function (PDF) of T. Find t₀ and all values that fall beyond it. (For a 2-sided t-test, you need to do this for both tails of the distribution, as shown in the image below.) The P-value is the area under the PDF for those values.
This is the only technical definition of the P-value that you should use. I have seen many other incorrect definitions of the P-value; I will elaborate on them in a future article.