The Value of Pilots
Pilot, n. To guide or conduct, as through unknown places, intricate affairs, etc. Experimental; denoting investigation on a small scale designed to assess the practicability of a major commitment, as a pilot film. The Macquarie Dictionary, Revised Edition.
Finding out that a data collection or survey is wrong after it has been run can be an expensive lesson, particularly if that survey is long-term, needs to deliver highly robust and accurate figures or uses new or untested techniques. In these circumstances, going through a test phase of the survey is usually a good investment and forms part of a quality design process.
Pilot surveys test the methodology, sample, questionnaire and all other aspects of the collection under real conditions in a scaled down format. A true pilot evaluates all components of a survey and provides an opportunity to try out different ways of collecting information to assess whether questions are interpreted and responded to as expected, confirm that the sample process gets a representative collection of respondents or achieves acceptable response rates, and generally to see how the survey process itself will go. A pilot validates a process for retention in the main survey or provides justification for changing a methodology to optimise the data collection.
Once a pilot has been conducted, it must go through a rigorous evaluation and reflection phase. This part is often overlooked or not given adequate time. However to rush it through or miss it entirely may mean that a less than ideal survey is conducted. Response rates, question responses, feedback from respondents, budgets and timeframes are all considered as part of the evaluation. The outcome of a pilot survey is a validated and tested survey methodology.
A pilot survey can be thought of as one cycle through a standard quality improvement process and has many similarities to the famous Shewhart Cycle used in Total Quality Management. Of course, when a survey runs for an extended period additional cycles are used to further improve the design.
PARTS Pilot
An example of good design practice using a pilot survey is the Perth and Travel Regions Survey (PARTS). This collection is designed to be a long-term, continual survey that is able to measure change in day-to-day travel behaviour, and will involve nearly 10,000 Perth and regional households over a number of years. The data collected in PARTS will be used by policy planners, transport modellers and decision makers to inform and guide their activities. Therefore, the data needs to be trustworthy, representative and have high quality at a detailed level.
While travel surveys have been run in Perth and other major Australian cities in some form or another in the past, the challenge of this collection was to perfect the survey for current Perth conditions.
The PARTS pilot survey was designed to test a number of particular features of the survey and the initial design parameters:
- Understanding response rates in suburbs where responses were considered easy or difficult to get. For this reason the pilot selected certain suburbs where particular problems were likely to arise.
- Determining the cost-effective and resource-effective way of getting survey packs to the household and back to Data Analysis Australia again, and understanding how the delivery and collection process affects data quality. The mail methodology was very dependent on timely reminders being posted to sample households and the effectiveness of these had to be measured.
- Validating the questionnaire content and layout by finding out what information can be provided and in what format. The review of responses and interviews with respondents identified questions that worked well and others that needed change.
- Checking the sample frame for accuracy and completeness. The sample frame - a list of all land blocks in Perth - was chosen because it was likely to be complete with the exception of some flats but it also contains many non-residential properties.
It became evident through the pilot survey that the mailout/mailback process could not cope with problems in the sample frame. The sample frame could not be readily changed since it provided the best calibration for the whole survey. Hence the mailout/mailback process needed changing. The pilot also identified some small but necessary changes to the questionnaire.
An adjunct to the pilot was then run to test a new methodology involving personal visits to sampled households to deliver and collect the survey packs. The visit by trained survey staff could clarify any problems in the sample frame, as well as explaining the survey to the household. The pilot adjunct assessed the feasibility, costs and benefits of this alternate collection methodology and found it to be superior to the original postal process. The main survey was then able to start with a validated collection methodology.
December 2002