Here we’ll go over the fundamental concepts of hypothesis testing. Generally we want to test two hypotheses. Let’s say we have two web pages, and we assume that the click-through-rates do not vary across time or users (iid for each web page across users and time). We want to compare the click-through-rates and .
We have two hypotheses,
- : the null hypothesis represents a standard assumption. In this case it could be that . That is, the two pages have the same click-through rate.
- : the alternative hypothesis. This is that they are different .
- Type 1 error: we reject the null hypothesis when it is true. That is, we conclude that the click-through-rates are different, even when they aren’t.
- Type 2 error: the alternative hypothesis is true, but we fail to reject the null hypothesis.