Data Analysis Australia
STRATEGIC INFORMATION CONSULTANTS
Copyright © 2012
Data Analysis Australia
The Value of Pilots
Finding out that a data collection or survey is wrong after it has been run can be an expensive lesson, particularly if that survey is long-term, needs to deliver highly robust and accurate figures or uses new or untested techniques. In these circumstances, going through a test phase of the survey is usually a good investment and forms part of a quality design process.
Pilot surveys test the methodology, sample, questionnaire and all other aspects of the collection under real conditions in a scaled down format. A true pilot evaluates all components of a survey and provides an opportunity to try out different ways of collecting information to assess whether questions are interpreted and responded to as expected, confirm that the sample process gets a representative collection of respondents or achieves acceptable response rates, and generally to see how the survey process itself will go. A pilot validates a process for retention in the main survey or provides justification for changing a methodology to optimise the data collection.
Once a pilot has been conducted, it must go through a rigorous evaluation and reflection phase. This part is often overlooked or not given adequate time. However to rush it through or miss it entirely may mean that a less than ideal survey is conducted. Response rates, question responses, feedback from respondents, budgets and timeframes are all considered as part of the evaluation. The outcome of a pilot survey is a validated and tested survey methodology.
A pilot survey can be thought of as one cycle through a standard quality improvement process and has many similarities to the famous Shewhart Cycle used in Total Quality Management. Of course, when a survey runs for an extended period additional cycles are used to further improve the design.
An example of good design practice using a pilot survey is the Perth and Travel Regions Survey (PARTS). This collection is designed to be a long-term, continual survey that is able to measure change in day-to-day travel behaviour, and will involve nearly 10,000 Perth and regional households over a number of years. The data collected in PARTS will be used by policy planners, transport modellers and decision makers to inform and guide their activities. Therefore, the data needs to be trustworthy, representative and have high quality at a detailed level.
While travel surveys have been run in Perth and other major Australian cities in some form or another in the past, the challenge of this collection was to perfect the survey for current Perth conditions.
The PARTS pilot survey was designed to test a number of particular features of the survey and the initial design parameters:
It became evident through the pilot survey that the mailout/mailback process could not cope with problems in the sample frame. The sample frame could not be readily changed since it provided the best calibration for the whole survey. Hence the mailout/mailback process needed changing. The pilot also identified some small but necessary changes to the questionnaire.
An adjunct to the pilot was then run to test a new methodology involving personal visits to sampled households to deliver and collect the survey packs. The visit by trained survey staff could clarify any problems in the sample frame, as well as explaining the survey to the household. The pilot adjunct assessed the feasibility, costs and benefits of this alternate collection methodology and found it to be superior to the original postal process. The main survey was then able to start with a validated collection methodology.