What Size Sample Do I Need?

At Data Analysis Australia, one of the most common queries we receive from current and prospective clients is "what size sample do I need?". A common misconception is that 400 is the magic number. However, it is not always this easy - one size does not fit all applications. In fact 400 is rarely the right answer. Not surprisingly the answer depends upon the details of the question and understanding the question is the best starting point.

Survey results are often used to find an answer to a question or to help make an informed decision. Sometimes this is expressed in terms of estimating a number such as the proportion of shoppers who might buy a product, the proportion of customers who are satisfied with a service, the average turnover of companies or the gold grade in a deposit. Major decisions can be based on such survey estimates and clearly reliable decisions need reliable estimates.

However any results that come from a survey will be subject to some degree of error. This error can be separated into two types:

Sampling error. This type of error is caused by surveying only some of the population rather than surveying all of the population. If you repeat the survey, but randomly choose a different group of units to include in the sample, you would expect to receive a slightly different answer simply by virtue of surveying these different units. Both are equally valid answers but both have a degree of uncertainty.
There is a well developed statistical theory that helps us understand this type of error. The theory is used when setting the sample size and choosing the sampling design. The error can often be readily quantified (often before the survey) which helps when choosing the most appropriate sample size and sampling design.
Non-sampling error. This refers to all other sources of error. Examples include leading questions, communication error, ambiguous questions, data entry errors, poorly defined populations, non-response and deliberately false answers.
Usually this type of error can't be quantified, but steps can be taken to minimise its effects. Having a good and clear questionnaire is the first step. It is good practice to have questionnaires tested before the survey begins, so that these sources of error can be identified and fixed.

It is important to consider both types of error when designing a survey. Any benefit achieved from reducing the size of one type of error can very easily be wasted if the other type of error is larger.

Terminology

Before proceeding too far, we need to define a few terms.

Population.

This is the entire group about which answers are to be obtained. It is important to realise that populations are not restricted to people. For example, populations can also refer to businesses, clubs, households, mine deposits or whatever else is of interest to the survey. The population of interest needs to be clearly defined before the survey begins and sometimes it is quite difficult to define accurately. Examples of populations are:

All people in the Perth Metropolitan Area;
All small businesses in Australia;
All employees of an organisation;
All doctors surgeries in Western Australia's South West;
All tennis clubs in the Southern Suburbs; or
A gold deposit that is about to be mined.

Sample.

The subset of units in the population who are actually surveyed. This is often but not always a random selection. To demonstrate, the figure to the right shows a population of 9 units (people). The sample consists of 3 units (people) as shown by the red circles.

Standard error.

A measure of the error that results from surveying a subset of the population rather than the entire population.

Questions to be Asked

A number of questions need to be asked (and answered) before a suitable sample size can be determined. These include:

What level of accuracy is required? In general, the higher the level of accuracy required, the larger the sample size should be. However, smart sample designs can often be used to reduce the sample size without reducing the accuracy. Also, the changes in sample size that are required to achieve a change in accuracy are often not proportional.

Are estimates for subgroups also required? In many surveys, specific interest lies in subgroups as well as in the overall population. This is often because one survey may cover many issues, or comparisons between different subpopulations (subgroups) may be of interest. For example, in a survey of people some answers may be required broken down by gender and some questions such as number of pregnancies may only be relevant to one gender. The overall sample size needs to be large enough to ensure that an adequate level of accuracy for these subgroups can also be achieved. Usually, less accurate results are required for subgroups.

What resources are available? The sample size and design obviously need to fit within the available resources. Obviously, the more time and money that is available to conduct a survey, the more accurate the results you would expect to achieve. In some cases, unrealistic expectations are placed on the survey, and these ideals may need to be relaxed. In other cases, it may suggest that the available budget is not sufficient and the question may arise of obtaining extra resources or dropping the survey.

What method of data collection? Self-completion questionnaires are often cheaper to implement than personal interviews (either face to face or telephone), but their response rates tend to be lower and results cannot be obtained as quickly. Personal interviews also usually result in higher quality survey data with fewer missing or inconsistent responses, as the interviewer can provide clarification when needed. Self-completion questionnaires therefore need to have a very good design and production in order to be answered to a high standard. This is an upfront cost that may be acceptable for a large survey but not for a small one. A trade-off often needs to be made between the data collection method and sample size that can be afforded.

Should everyone be surveyed? For some surveys, it is more appropriate to survey the entire population, rather than surveying only a sample. In some surveys, to achieve the desired level of accuracy the sample size required is so close to the entire population that it makes more sense to simply survey everyone. Surveying the entire population essentially removes the sampling error component and is often useful for staff or customer satisfaction surveys and has the added benefit of everyone feeling that they are listened to.

How does the population size affect the sample size? Contrary to popular opinion, the population size plays a relatively small role in determining the required sample size, particularly if only a small percentage of the population is being surveyed. For example, to conduct an opinion poll in Western Australia, approximately the same size sample needs to be selected as for a poll in New South Wales, despite New South Wales having a much larger population.

How variable are the responses expected or known to be? The level of variability between responses has a large impact on the sample sizes required. The less variable the responses are, the smaller sample size that is required to achieve the same level of accuracy. For example, there is a large degree of variability between household income but a relatively small amount of variation between the number of jobs that an employed person has. Hence an income survey would need a larger sample size to achieve the same accuracy as a survey on the number of jobs.

Is the burden being placed on respondents too high? Respondent burden is a big issue in today's society, with many surveys being conducted for many different purposes. If people or businesses get surveyed too frequently, they are less likely to take the survey seriously. This means that the sample size should not be larger than necessary to obtain the accuracy needed.

Sample Design

So, determining the "correct sample size" is not a simple task. In fact, a large part of determining the sample size is not simply "how many should we sample", but how cleverly the sample is chosen. A "smarter" sample design can give more accurate estimates with a smaller sample size.

In general, the more complex a survey that is being conducted the more effective a smart design can be. It is often more cost effective to spend additional resources in designing the sampling methodology than simply sampling more units. Techniques such as systematic sampling, stratified sampling, cluster sampling, multi-stage or multi-phase sampling can all be used to improve the sample.

Stratified sampling is one of the most common types of survey design. This involves separating the population into distinct groups and then choosing a sample size for each group (for example, males/females, states of Australia or divisions of a company).

There are two main benefits of a stratified sample:

Stratified sampling ensures that an adequate number of respondents are gained for each subgroup of interest. This also helps to ensure that a representative sample is achieved.
To maximise the benefit achieved from using a stratified sample, the distinct groups should be chosen so that units within the same group are as similar to each other as possible. This allows the sample to be allocated to the groups so that the more variable groups get a higher proportion of the sample. This leads to a better estimate at the overall level and also at the subgroup level.

A good example is for surveys relating to business activity. In many cases, a few very large companies have a big effect on the overall value. However, it is also important to get a good estimate of the combined value of the smaller companies. In such cases, groups (strata) can be formed according to the company size with only a percentage of the smaller companies being surveyed, and all of the larger companies being surveyed. All companies with 0 to 9 employees might be grouped, and 10% of them surveyed, 15% of companies with 10 to 20 employees could be surveyed and all companies with more than 20 employees could be surveyed. Although this means that the responses need to be weighted using statistical techniques to provide meaningful estimates at the overall level, this method provides superior estimates. This sampling design also helps to reduce burden for small companies that often have the greatest difficulty in responding.

What if the "right sample size" is not affordable?

Sometimes, the ideal sample size and design just doesn't fit into the budgeted time and/or money constraints. In these cases, a trade-off decision typically needs to be made between the competing priorities of the survey. Some options include:

Accepting a slightly lower level of accuracy, particularly for subgroups of the population;

Diverting resources from another component of the research - for example by reducing the length of the questionnaire;

Changing the implementation methodology to a cheaper alternative, perhaps replacing interviews with a mail back questionnaire;

Redefining the survey aims. Consideration should be given as to whether the right questions are being asked in the right way, or whether unrealistic goals are being set with regards to the practical importance of the accuracy of the survey results; or

Reassessing the budget. If none of the above trade-offs can be made it could be because the budget is just not reasonable or realistic to meet the requirements of the survey. If the purpose of the survey is so important that these trade-offs can't be made then extra resources need to be supplied.

So, when is a sample size of 400 appropriate?

A sample size of 400 units can be appropriate for some surveys but only with the right assumptions and accuracy requirements. But these are often not met. In fact the usual argument for it is based upon:

Asking a question with a yes/no response;

Expecting approximately a 50/50 split in the responses; and

Needing an accuracy of ±5% with 95% confidence.

When are all these assumptions met? Probably rarely. This means that many surveys are carried out with sample sizes that are either unnecessarily large, leading to unnecessary cost, or giving insufficient accuracy to make proper decisions.

Data Analysis Australia has both the statistical expertise on these issues and the practical experience in conducting high quality surveys. Our consultants understand the process from the first steps of formulating the questions that the survey must answer through to the analysis of results, providing professional judgement on what is best for each situation.