IT and the Classroom
In 2001, the Department of Education in Western Australia introduced their Education to Community (e2c) vision, aiming at delivering "a world-class education system, bringing Western Australian education communities together online". By promoting the use of information and communication technologies (ICTs) in the classroom, the e2c program is designed to improve learning opportunities for students in government schools.
Data Analysis Australia was appointed as the sole provider of survey and statistical support services to e2c programs. This recognises the place that independent and objective evaluation has in the effective development and implementation of programs and services. Under this arrangement, Data Analysis Australia has been assisting the ICT section of the Department to evaluate, measure, analyse and interpret programs, trials and services. Having independent statisticians involved in the evaluation of programs has delivered unique benefits to the Department.
Integrating ICTS with Traditional Classroom Resources
The concept of computers in classrooms is not new, going back more than twenty years. However their effectiveness has often been limited by the actual difficulty of maintaining computers and peripherals in the classroom, and the costs in keeping them up-to-date. What is the best method of implementing computers in schools remains an open question.
This has led to the Department trying a new approach. The Application and Service Provider (ASP) Trial was established to test an approach based on tighter integration, more powerful networking and a client-server model. It also included modern peripherals such as scanners, data projectors and digital cameras. The trial maintained a curriculum perspective, where the technology is just one of the resources alongside more traditional ones, so that teachers and students can choose whichever method most improves learning opportunities.
Data Analysis Australia became involved in the very early stages of the evaluation of the ASP Trial and worked closely with the ASP Team to design a robust, justifiable and practical evaluation methodology framework to measure outcomes against the trial objectives. Teachers, students and staff at the trial schools have participated in surveys run by Data Analysis Australia and other activities managed by the Department to record their perceptions, development and capabilities. A critical component designed by Data Analysis Australia is a survey of schools outside of the trial to adjust or control for any general changes in education and ICTs affecting all schools.
Most of the trial was completed in December 2002 and currently Data Analysis Australia is engaged in further analysis to present an overall evaluation, drawing together the many components so far studied.
Improving Access to ICTS for Teachers
Another major e2c program is the Notebook For Teachers (NFT) Program. Teachers enrol in the voluntary program to lease a Notebook Personal Computer complete with software, network access, training, insurance and support. The program provides opportunities for teachers to access ICTs and encourage them to use these as part of their professional development.
Data Analysis Australia designed an evaluation methodology consisting of two surveys completed by teachers - one at the time they sign on to the program and a second survey some time after having the Notebook laptop. The surveys capture baseline measures in competencies, use and capabilities of ICTs and changes in these over time. Demographic information on teachers was also collected to characterise the teachers who find this program most beneficial. The stratified sampling design developed for the evaluation reflected the rollout of Notebook offers to schools, enabling a close match between when an offer is taken up and the baseline survey completed, while also providing a sample of schools covering all socio-economic and locational categories.
Help Desk and ICT Support Services
Data Analysis Australia has also conducted client satisfaction surveys as part of evaluating the performance of Local Area Networks (LAN) and Wide Area Networks (WAN) Support Services and Application Support Services. The surveys were designed and implemented in line with the contractual arrangements between the service providers and the Department. Here the customers were Department staff, including teachers and Head and District Office staff. Here, independence and impartiality was critical in measuring performance against the contract in a manner acceptable to both parties. Statistically valid and robust techniques gave both parties confidence in the results. Aside from fulfilling contractual obligations, feedback from the surveys was also valuable information that could be used by the contractors to improve service provision.
The involvement of Data Analysis Australia in the evaluation of these programs has benefited the users, program developers and the education system. The role as independent evaluators is not to pour cold water on innovative programs, but to ask how they can be made better. The application of robust design carried out professionally, objectively and impartially, delivers that evaluation in the most cost effective manner.
March 2003