An example of a hypothesis test in statistics

Applying the 4 basic steps to a Welch's t-test

A hypothesis test can be very confusing - especially to a newcomer in statistics. I recently wrote about the 4 basic elements of any hypothesis test to make this easier to understand. Let’s apply them to an example in the intersection of retail, grocery, and consumer packaged goods (CPG); this is where I currently work as a statistician.

Suppose that a retailer wants to compare an old display (which I denote as “x”) and a new display (which I denote as “y”) for selling its cereal. It shows both displays at 100 stores, and it sells the brand of cereal boxes with different barcodes to track which display each box came from.

• Which display sells more cereal: the old display or the new display?

• If there is a difference, is it statistically significant¹?

To answer this question, let’s use an independent, two-sample t-test. I will apply the 4 basic elements of a hypothesis test to our example.

1. What is the null hypothesis, H₀?

• There is no difference in the sales between the old display and the new display.

\(H_0: \mu_x = \mu_y\)

2. What is the alternative hypothesis, Hₐ?

• There is a difference in the sales between the old display and the new display.

\(H_a: \mu_x \neq \mu_y\)

3. What is the test statistic?

Let’s use an independent two-sided Welch's t-test, because it can account for the possibility that the variances in the sales are different between the old display and the new display.

Here is the test statistic.

\(t = \frac{\bar{X} - \bar{Y}}{\sqrt{\frac{s_x^2}{n_x} + \frac{s_y^2}{n_y}}}\)

The distribution of this test statistic depends on the degrees of freedom, which has the following approximation.

\(\nu \approx \frac{\left( \frac{s_x^2}{n_x} + \frac{s_y^2}{n_y} \right)^2}{\frac{\left( s_x^2/n_x \right)^2}{n_x-1} + \frac{\left( s_y^2/n_y \right)^2}{n_y-1}}\)

4. What is the rejection rule?

Let’s use a significance level (α) of 5%. Thus, if the P-value is less than 0.05, I will reject the null hypothesis.

The P-value is a tricky concept to understand, let alone explain to a non-technical audience. I will devote several articles to this topic in the future; please stay tuned.

---

Astute readers will notice that I have written nothing about whether the new display is actually causing any change in sales; this is because I cannot establish causation in my example. For simplicity, I did not discuss other variables that may have contributed to the sales or affected the two displays; these variables are called confounding variables (or confounders). To establish causation, I must control for the confounders; this is another topic in the realms of experimental design and causal inference, and I will consider writing about them in the future.

1. Statistical significance is different from practical significance. A statistically significant difference may be too small to matter for the relevant business problem. This is why an effect size with a 95% confidence interval is more informative than a simple P-value. I will elaborate on this issue in a future article.