Skip to main content

Managing Big Data

In many situations the problem is not a lack of data but rather an apparent excess of data.  Even in this situation, the question still remains "what does it all mean"?  Interpreting large volumes of data and finding the information gems is the challenge.  At Data Analysis Australia we have the IT and statistical capability to store and process such data sets, to provide useful and valuable information to the client.

Data Analysis Australia approaches this from a statistical viewpoint, but one that is informed by modern computer science so that the two are incorporated to fully disseminate the data, providing the client with real insight into "what the data means".  This gives several advantages over approaches that are essentially computer technology driven:

  • Statistics provide the most effective methods of understanding relationships in data.  While these are often standard to statisticians, their application to large datasets requires special algorithms and an understanding of how the data is stored.
  • Statistics provide measures of significance for what is found.  This is critical since data mining methods may throw up many chance patterns and it is important to discard those that are not real.
  • Statistics provide a methodology for exploration using subsets of the data, often saving enormous amounts of computer time that might be prohibitive on many operational systems.

Systems at Data Analysis Australia are optimised for handling large datasets - we have capacity to handle databases measured in hundreds of gigabytes and our high bandwidth network means that computation is almost unlimited.  A range of software tools is available so that the best can be chosen for each task or step in an analysis.

Data Analysis Australia is frequently consulted by organisations that need to make the most effective use of their data.  Since our focus is on information content and what information is required to drive decisions, our expertise complements that of more traditional IT companies.

Industry Case Studies

Financial Literacy in Australia Segmentation Analysis
Horizon Power - Profiles of Power
Identifying Cycles in a Chemical Plant