Our previous Analytical Idea entitled What Size Sample Do I Need? focused on how to size samples to answer questions about a population quantity, such as an average or a proportion. We typically use surveys to collect such sample information.
This article discusses sample sizes when our questions centre on comparisons. We usually answer these sorts of questions using data collected in experiments, studies or trials, which typically involve two or more sample groups. A common approach is to use a formal hypothesis testing framework, which is a convenient statistical model for how the result of an experiment should be interpreted. Under this approach, we assume there is a basic hypothesis (the null hypothesis, or H0) and accept this hypothesis unless the experiment shows enough evidence to the contrary. The null hypothesis is therefore set to the opposite of what we are trying to prove true. For example, if we expect reading program A is better than reading program B, a null hypothesis might be “H0: Reading scores from program A are no better than reading scores from program B”.
We design and conduct an experiment to test this hypothesis. If our experiment has enough power, we hope it will gather enough evidence to let us reject the null hypothesis and hence conclude that reading program A is superior.