Longitudinal Data – Is it Worth the Weight?

In our earlier article on measuring change, we discussed longitudinal designs involving two surveys with the same participants.  However, longitudinal surveys need not be limited to only two surveys and may continue for many “waves”.  For example, the Household Income and Labour Dynamics of Australia (HILDA) Survey has been conducted on an annual basis since 2001 and is still ongoing.

Of course, while longitudinal surveys try to include the same sample every time, there are always some who drop out – sometimes permanently, but sometimes only for one or two waves.  As with many one-off or “cross-sectional” surveys, longitudinal survey data often includes weights that can be applied when using the results to make inferences about the population as a whole.  The weights effectively scale the sample up to the population.  Since the number of respondents will vary from wave to wave, a given respondent is likely to have different weights assigned to their responses for different waves.  In addition, there may be multiple sets of weights, with consideration of the analysis being undertaken necessary to determine which set of weights to use.

So why use weights anyway?

Weights are designed to ensure results are representative of the group about which inferences are being made.  They are an indication of how many people each respondent represents.  Imagine a population of 100 people who have just become pet owners, with 50 owning a cat and the other 50 owning a dog.  If 10 cat owners and 20 dog owners were surveyed, it could be said that each surveyed cat owner represented 5 people, but each surveyed dog owner only represented 2.5 people.  In this way the results from the 30 respondents can be made representative of the original population.  In this example, a weight of 5 is assigned to the cat owning respondent and a weight of 2.5 is assigned to each dog owning respondent.  If the survey asked whether people bought cat food or dog food, the unweighted results would probably indicate that 20 out of 30 pet owners buy dog food, but by applying weights we would estimate that 50 out of 100 pet owners buy dog food. 

For  simplicity, in our example we are only weighting according to the type  of pet owned, but things are rarely this simple in practice and weights  frequently take into account sampling probabilities and key demographic  variables such as age, sex and geographic location.

What’s different about weights for longitudinal surveys? 

As mentioned above, the same respondent may have different weights for each wave of a survey.  The weights for each wave should be designed to make the overall responses more representative of the population of interest at the time represented by that wave.  If Wave 1 was collected in 2011, the Wave 1 responses would be weighted to reflect the 2011 population.  Wave 3 weights would apply only to Wave 3 respondents and would relate to the 2013 population (assuming the survey was conducted on an annual basis).  This can complicate things if we want to analyse changes between waves, which is after all the key reason for conducting a longitudinal survey.  The obvious question is “should we use weights from the earliest wave of interest, weights from the latest wave of interest, or some combination of both?”.

The  general answer is that weights are calculated as if the sample from the  later wave was the whole sample at the time of the first wave of  interest, and are calculated for the population at the time of that  first wave.

The good news is that longitudinal survey datasets that are made available for research often include a set of longitudinal weights covering each pair of waves that could be compared.  For example, data from both HILDA and the Longitudinal Survey of Australian Children (LSAC, also known as Growing Up in Australia) can be obtained for research purposes and both datasets include cross-sectional and longitudinal weights.  So, when undertaking cross-sectional analysis (i.e. analysing the data for a single wave only), the cross-sectional weights are used and when comparing between waves the longitudinal weights are used.

Do I really need to use weights?

Not always!  For some types of analysis it may not be necessary, and may even be inappropriate, to use weights at all.

In  general, weights should be used when making inferences about the  population, but if the aim is simply to examine the properties of the  respondents themselves, then weights can often be ignored. If you want  to understand the properties of respondents and also make inferences  about the population, it is likely that a combination of weighted and  unweighted analyses will be required.

Tabulations and Cross-Tabulations 

There is no single “right” answer regarding the use of weights, as it will depend on the ultimate aims of your research.  However, often weights will be necessary if you wish to expand your results to say something about the population as a whole, as discussed in our example above of new pet owners.  Continuing with that example, let’s imagine we obtained the respondent numbers shown in Table 1 over four survey waves.

It can be seen that the total number of records declines with each wave due to respondent attrition, but the attrition rates are not equal for our two types of pet owners.  Therefore, any unweighted analysis is likely to give misleading results and it is more appropriate to undertake a weighted analysis. 

Perhaps we want to know how many pet owners buy a particular brand of pet food, CuddlyPets, which caters for both cats and dogs.  Having asked whether owners purchased this brand we might sum over the weights for each wave and find weighted counts as shown in Table 2

One immediate and interesting observation is that the totals are now constant across waves.  This is because the weighting has used the initial population of 100 as the reference and all weights have been calculated to sum to that population, as shown in Table 3.

The weighted results in Table 2 show that purchasing rates for CuddlyPets are increasing slightly over time, but they don’t show us how this has occurred.  Are the Wave 4 buyers of CuddlyPets the original buyers plus some new ones, or are they nearly all new purchasers?  A longitudinal analysis can help answer this, by comparing changes at the individual level.  

In principle, it is possible for a respondent to follow many paths, potentially changing their purchasing behaviour wave by wave (in fact there are 16 possibilities across the four waves).  Here, we will consider just two waves, 1 and 4.  To see the changes that take place between these waves, we consider at an individual level whether people purchased the brand.  Each respondent to Wave 4 may have one of four purchasing behaviour combinations:

  • They purchased CuddlyPets in both Wave 1 and Wave 4;
  • They purchased CuddlyPets in Wave 1, but not in Wave 4;
  • They didn’t purchase CuddlyPets in Wave 1, but did in Wave 4; or
  • They didn’t purchase CuddlyPets in either Wave 1 or Wave 4.

We may find the weighted results shown in Table 4 when comparing each pet owner’s Wave 1 and Wave 4 responses.  These results show that the overall increase in purchasing of CuddlyPets has largely been driven by new purchasers.  Fewer than half of the Wave 1 buyers continued to buy CuddlyPets in Wave 4, but two thirds of pet owners who didn’t buy the brand in Wave 1 did buy it in Wave 4.  This could be bad news for the CuddlyPets brand – if they are not retaining their customers, the increase in purchasing cannot be maintained.  Fortunately, by conducting a longitudinal survey, and not just accepting the results of the cross-sectional analysis at face value, they should be able to drill down even further to understand what is driving customer behaviour and hence develop strategies to retain their current customers and recover those who have been lost.

Regression Analysis

Generally it is recommended that unweighted data is used for regression analyses.  While in some contexts weighted regression is appropriate, the weights serve a different purpose to survey weights.  In regression a high weight is associated with observations for which the dependent variable is known to be more accurate, whereas the sampling weights provided with survey data tend to be highest where there are relatively low sample counts and hence relatively high uncertainty in the response patterns.

This  highlights the two different uses of the term “weight” in statistical  analysis. In survey work, where tabulation is common, the weights are  best thought of as “expansion factors”, scaling from the sample to the  population. In regression, weights represent the precision of the  observations which is a very different concept.

Perhaps our survey has also recorded information on the number of different brands of pet food that owners buy and we have observed a decrease in the average over time.  To understand why some pet owners start buying fewer different brands over time we may decide to perform a regression analysis.  For each survey respondent, we could calculate the difference between the number of brands purchased in Wave 4 and the number purchased in Wave 1 and use this as our response or dependent variable.  We would probably want to include the type of pet as one of the explanatory variables and if we used the survey weights as regression weights, we’d be saying that we were much more certain of the results from the cat owners than those of the dog owners.  This is because the cat owners get weights that are about double that of the dog owners for each wave, but in reality the uncertainty for each respondent is likely to be very similar.

Multivariate Analysis

Multivariate analysis techniques (such as principal components analysis, cluster analysis, discriminant analysis) should generally be undertaken using unweighted data.  A common area where such techniques are used is segmentation analysis, where the research aim is to identify groups of respondents who exhibit similar behaviours and/or attitudes.  However, once the initial analyses have been undertaken and segments identified using unweighted data, it is common practice to then make inferences regarding the percentage of the population that each segment may represent and, for this purpose, it would be necessary to use weighted data.

 

December 2016