Copyright © 2016
Data Analysis Australia

Survey Design and Analysis

Surveys are a key tool in collecting information about a "population", whether that population is a set of people, households, businesses or even pieces of equipment.  Most surveys use information from a subset or sample of the population, which if done right will lead to an understanding of the whole targeted population.  Such sample surveys require proper statistical input in their design and analysis.

 

Surveys can be a major investment, with results often playing an important role in strategic planning decisions.  Therefore, it is critical to invest in a good design, good implementation and the best analysis.  Data Analysis Australia's statisticians and survey researchers have years of expertise, specialising in ensuring surveys meet the individual needs of our clients.

 

Data Analysis Australia consultants can advise on all aspects of survey design, implementation and reporting:

  • The implementation of a survey can use a mixture of collection techniques or modes, including telephone interviews, face-to-face interviews, self completion forms and on-line questionnaires.  Each method has its particular strengths, weaknesses and costs and for each survey Data Analysis Australia can guide the client so the right choice is made, and the best results achieved. 
  • A well designed questionnaire is vital to ensure that the information collected is accurate, unbiased and, most importantly, gathers the information needed to address research issues.  Analysis of past survey results can sometimes provide valuable information for a new questionnaire, eliminating ineffective questions and directing resources to areas of more value.
  • Closely linked to the collection method is the issue of sampling - how is the sample that must represent the population chosen?  Good sample design can minimise costs and increase the statistical power of the analysis to provide better results.
  • Following on from sampling is the application of survey weighting to enable inferences to be made about the whole population.  This weighting can incorporate differing sampling probabilities as well as benchmarking to known population totals.
  • The analysis stage may simply be a case of generating tables of results or it can involve more sophisticated analytical techniques.  The planned analysis should be kept in mind whilst developing the questionnaire, to ensure that the information necessary to undertake the analysis will be obtained.  If you have data that has already been collected, we can usually devise an analysis plan to extract useful information from your data, even if it isn't exactly what was anticipated when the data was originally collected.
  • Interpreting the results from the analysis usually requires a formal assessment of the accuracy and significance of key results to ensure decisions are not made based on random variation.  Data Analysis Australia provides interpretations and recommendations based on sound statistical principles and our consultants are well versed with the statistical theory and practice in determining the sampling error for a survey.

Examples of project experience where Data Analysis Australia has completed surveys for clients are listed below. 

Large Surveys 

Between 2002 and 2005 Data Analysis Australia carried out the Perth and Regions Travel Survey (PARTS) for a consortium of Western Australian government departments.  This was the most comprehensive survey of the movements and activities of Perth residents for decades and involved sampling over 11,000 households, collecting for each a snapshot of a single day's travel for all the people in the household.  The detail was unmatched - it even itemised how far people walked to the bus.

 

The National Preschool Census has been conducted by Data Analysis Australia almost annually since 1993.  This survey initially targeted issues of Indigenous education and participation but has expanded to be the major information collection to inform national policy on preschools.  It aims to be a Census and regularly achieves response rates in excess of 98%.  A unique feature is the merging of survey data with data collected through some of the State and Territory education systems.

Customer Satisfaction Surveys 

Data Analysis Australia has been involved with customer satisfaction projects for many government and non-government organisations including the Insurance Commission of Western Australia, Department of Housing and Works, Legal Aid WA and the City of Subiaco, including many others that meet the Auditor General's requirements of achieving a minimum of 5% error at the 95% confidence level.

 

The Insurance Commission of Western Australia has contracted Data Analysis Australia since 1995 for a number of these surveys including two organisation wide customer satisfaction surveys in 1995 and 1996, and the RiskCover Division Customer Satisfaction Survey annually from 1997 to 2009.  The methodology developed for these surveys combines quantitative expertise with qualitative research in order to investigate customer satisfaction.  This allows for the development of Division-wide strategic recommendations, whilst also giving specific feedback on customer satisfaction within each key area of service.

 

From 2002 to 2006, Data Analysis Australia conducted annual client satisfaction surveys for Legal Aid Western Australia.  The surveys alternated between family law and criminal law clients on an annual basis.  Telephone interviews with clients were conducted under the strictest confidence, since the environment in which respondents find themselves to be clients of Legal Aid are highly sensitive.  Data Analysis Australia also produced an index of satisfaction by combining the responses to a number of questions and by weighting the contribution of various surveyed client groups.

Online Surveys

Data Analysis Australia has designed and hosted many web-based surveys.  Through our in-house software developed using the python language, we have experience in implementing web surveys ranging from the very simple to the extremely complicated.  Data Analysis Australia has a thorough understanding of the advantages of the technology that can be exploited, as well as potential pitfalls to the results if not implemented correctly.  Some of our web-based surveys have been supplemented by other forms of data collection (such as paper based questionnaires) to reduce biases that can arise if web-based data collection is used alone.

 

Data Analysis Australia's IT professionals develop the online surveys, so there is a great deal of flexibility in how the survey is set up allowing it to be fully tailored to suit the requirements of the client.  The following features can be applied:

  • A modern and professional appearance, which can include individual colour schemes and clients' logos.
  • Full security, ensuring the survey is only accessible to the intended participants and that duplicate responses are not recorded.
  • Potential participants of the survey can be informed on how the information will be used, the benefits of completing the survey, how confidentiality will be maintained, and who will have access to the raw data, tables and reports.
  • Processes can be put in place to automatically save data at the end of each page thus ensuring data is not lost. 
  • Warnings can be generated to alert respondents, as they progress through the questionnaire. 
  • Automated checks for unusual or inconsistent responses can be implemented. 
  • Data Analysis Australia can set up a secure webpage so the client can view the number of responses.

Sampling, Weighting and Standard Errors

Data Analysis Australia has provided assistance in designing sampling strategies for numerous clients where sufficient sample was required to allow analysis at an aggregated and disaggregated level. One such example related to a survey of 100 schools across Australia. The sample was designed to ensure a reasonable representation of primary and secondary schools in each state, and a mixture of government, catholic and independent schools, whilst maintaining a minimum of two schools per stratum to enable standard errors to be calculated.  Another example was a survey to measure average speed,  fuel consumption and tyre life of Western Australian school buses, whereby the sample was designed to enable coverage and analysis of a range of bus types, locations across Western Australia and times of year.  Survey weights and standard errors were calculated taking the sampling methodology into account.

 

The Student Outcomes Surveys (SOS) are a key tool in the ongoing monitoring and improvement in the Vocational Education Training (VET) sector across Australia.  Thousands of students were surveyed with a high degree of stratification in the sampling and estimation stages to ensure representation across different institutions, fields of study, age and gender.  The high level of stratification and resulting low numbers of respondents in some strata required a complex weighting process to be implemented.  Data Analysis Australia provided expert advice on how the weighting processes could be improved in future years, as well as ensuring consistency with previous years' outputs.  Data Analysis Australia also calculated the standard errors and confidence intervals for the key tables, ensuring that the effects of the stratification and weighting processes were properly taken into account.

Other Applications

  • Development of performance measures and indicators (KPIs) for both government and non-government organisations.
  • Segmentation analysis, using techniques such as cluster analysis, factor analysis, principal component analysis, decision trees and discriminant analysis.
  • Price choice modelling, using techniques such as conjoint analysis and perceptual mapping.

 

For more information about any of our Areas of Expertise please contact one of our consultants, email us at daa(at)daa.com.au or call 08 9468 2533.