Surveys, the Internet and Data Entry

At Data Analysis Australia we feel that our ability to provide innovative and unique IT solutions in our projects is one of the many factors that separates us from other consulting firms. The ability to combine statistical, IT and business aspects has been a feature since the company was founded in 1988. One aspect of this has been the use of Internet based survey technology.

Data Analysis Australia installed its first Internet gateway in 1995 and since then has developed expertise in Internet technology. One reason for having this facility in-house was to ensure the confidentiality of client data but the other was to remain at the leading edge of emerging technologies. This has enabled us to implement new methods and use the latest IT developments.

Internet surveys use forms viewed through a standard browser to enter answers that are then submitted back on the server. The forms can be predefined or they can be generated "on the fly" to fit the context. This is often used in surveys where the answers to some questions may determine which questions are offered next. The submitted answers can also be checked, either before being sent back or after they have been received by the server. Once checked, the answers are stored in a database.

Data Analysis Australia has carried out such surveys on issues including employee satisfaction, magazine readership and community concerns. An example of a survey form login page can be found at here.

The Internet based survey has a number of advantages including the immediate collection of results and substantial cost savings. At Data Analysis Australia it has also had a role of complementing other survey methods - the greater choice of response methods provided in a survey the higher the response rate. This can significantly reduce sampling biases in surveys.

A key part of this advantage lies in Data Analysis Australia's use of the Linux operating system, a modern version of the Unix operating system around which the Internet was designed. Linux makes it possible to integrate Internet technologies with other applications such as databases. It also embodies the concept of software tools; small but flexible programs that can be used together. Linux also makes available a number of modern languages that offer good support for Web based applications. At Data Analysis Australia we use Perl and Python extensively.

An Application to Data Entry

An example is provided by the Perth and Regions Travel Survey (PARTS) being carried out over four years for the Department for Planning and Infrastructure. Household travel surveys are perhaps the most demanding of all surveys - every member of each household surveyed records all movements for a specified day, including activities, locations, modes of transport and times, even down to where and when they got off a bus. The demands of accuracy and reliability are enormous.

At the beginning, the need for a specialised data entry system was identified. It needed features and flexibility not offered in commercially available packages. Data Analysis Australia drew upon its expertise in travel surveys, geographical information systems and, perhaps surprisingly, Internet surveys in developing a solution.

In a survey the server and the browser may be far apart but there is no reason why the same system cannot be used locally. This is exactly how Data Analysis Australia implemented the PARTS data entry. A dedicated server for the project supported up to five data entry computers, all running Linux. The Mozilla browser was chosen since it had a more complete implementation of JavaScript that gave it an advantage over others. (This is one aspect that the data entry system differed from a survey - the data entry system could make use of all these browser features while in a survey there is no control over what browser is used, forcing the design to only use the subset of features that work on all common browsers.)

The system is largely programmed in the Python language. This provides an efficient environment for both managing the temporary data objects and for dynamically creating the web pages. In fact the survey data is maintained as "serialised Python objects" until it has passed through all the checking stages.

The first advantage of this approach lies in the ability to incorporate a system of complex logic checks. These logic checks occur both at the browser and back at the server. The latter can be quite complex, drawing upon large databases where necessary. The checks ensure that the data entered lies within a range of acceptable responses and that the data is consistent with data previously entered. When errors are detected, warnings are immediately issued.

For example, an individual says on their survey form that they drove a car. If the information previously entered indicated that the individual does not have a licence the system will warn that the data is not compatible. The data is further checked and if necessary the survey respondent is contacted to verify the facts.

The system also utilises an advanced spell check program. The program uses a custom dictionary that contains only street names in the study region. It also takes into account common typing errors (e.g. transposition of characters) that may have occurred during the data entry process. This was easy to implement with the standard software tools under Linux.

As the system utilises the Postgress database program, the system is multi-user. This enables the system to be easily incorporated into a web-based survey. It will allow the same logic checks to be used when a respondent is entering information into a web survey, i.e. if the information entered did not make sense the respondent would be alerted and would be able to change the response if necessary.

Data Analysis Australia was one of the first consulting firms to offer Internet solutions to clients conducting surveys and we continue to lead the field in this area. Our IT department has expertise in the design and programming of software systems for statistical based applications, design and implementation of web surveys and database systems design and management. 

By offering this capability through our IT department rather than by utilising externally built software packages and technology we are able to offer solutions that are adapted to each client's specific needs.

February 2004

Back to Top