Finding out that a data collection or survey is wrong after it has been run can be an expensive lesson, particularly if that survey is long-term, needs to deliver highly robust and accurate figures or uses new or untested techniques. In these circumstances, going through a test phase of the survey is usually a good investment and forms part of a quality design process.
Pilot surveys test the methodology, sample, questionnaire and all other aspects of the collection under real conditions in a scaled down format. A true pilot evaluates all components of a survey and provides an opportunity to try out different ways of collecting information to assess whether questions are interpreted and responded to as expected, confirm that the sample process gets a representative collection of respondents or achieves acceptable response rates, and generally to see how the survey process itself will go. A pilot validates a process for retention in the main survey or provides justification for changing a methodology to optimise the data collection.
Once a pilot has been conducted, it must go through a rigorous evaluation and reflection phase. This part is often overlooked or not given adequate time. However to rush it through or miss it entirely may mean that a less than ideal survey is conducted. Response rates, question responses, feedback from respondents, budgets and timeframes are all considered as part of the evaluation. The outcome of a pilot survey is a validated and tested survey methodology.
A pilot survey can be thought of as one cycle through a standard quality improvement process and has many similarities to the famous Shewhart Cycle used in Total Quality Management. Of course, when a survey runs for an extended period additional cycles are used to further improve the design.