Basic hypothesis testing. Null vs alternative hypothesis. Type 1, Type 2 errors, and p-values.

Here we’ll go over the fundamental concepts of hypothesis testing.  Generally we want to test two hypotheses.  Let’s say we have two web pages, and we assume that the click-through-rates do not vary across time or users (iid for each web page across users and time).  We want to compare the click-through-rates p_1 and p_2.

We have two hypotheses,

  • H_0: the null hypothesis represents a standard assumption.  In this case it could be that p_1=p_2.  That is, the two pages have the same click-through rate.
  • H_1: the alternative hypothesis.  This is that they are different p_1\neq p_2.
We have two types of errors.
  • Type 1 error: we reject the null hypothesis when it is true.  That is, we conclude that the click-through-rates are different, even when they aren’t.
  • Type 2 error: the alternative hypothesis is true, but we fail to reject the null hypothesis.
An important concept is the p-value, which is the probability of observing at least as extreme a result as what we observe, given that the null hypothesis is true.  We select a threshold for the p-value and use it to decide whether to reject the null hypothesis or not.  Next time, we will describe how to actually do the test, and decide whether or not to reject the null hypothesis.

Leave a Reply

Your email address will not be published. Required fields are marked *