Data Analysis Australia
STRATEGIC INFORMATION CONSULTANTS
Copyright © 2012
Data Analysis Australia
"Can I Have Confidence In Political Polls?"
Modern political polling began in 1936, with two polls attempting to predict the outcome of the American presidential election. The Literary Digest conducted its poll by sending out 10 million post cards asking people how they would vote. They received almost 2.3 million back and said that Alfred Landon was leading Franklin Roosevelt by 57-43 per cent. In contrast, market researcher George Gallup employed a much smaller sample of only 5,000, but because he ensured that it was representative of the American voting public, he predicted Roosevelt to win by a landslide. In the event, Roosevelt won 60% and Landon just 37%. The Literary Digest lost credibility and subsequently merged with Time magazine in 1938.
Confidence in political polls is a belief that the results will be close to the actual election outcome. However because the election results are unknown at the time of polls, the basis for this confidence comes from understanding the procedures. These procedures should be based upon sound statistical principles. The two most relevant principles are those of representivity and precision. The Literary Digest thought it was precise because of its enormous sample size, but it failed on being representative.
Representivity can be understood by analogy with the way a chef can judge a large vat of soup by tasting just one spoonful. Providing that the soup has been well stirred, so that the spoonful is properly "representative", one spoonful is sufficient. Polls operate on the same principle: achieving representative samples is broadly akin to stirring the soup. If the vat is unstirred, a chef could taste a large amount from the top, and still obtain a misleading view if some of the ingredients have sunk to the bottom. Just as the trick in checking soup is to stir well, rather than to drink lots, so the essence of a poll is to secure a representative sample, rather than a vast one.
The Literary Digest chose its sample from its own subscriber lists, augmented by lists of telephone subscribers and car owners. Today these might seem to be lists that would include almost everyone, but in 1936 they were all indicators of wealth. In an election that pitted the dispossessed poor from the Depression against established classes, this was a fatal bias.
There are many practical aspects to achieving representivity including the sample method by which persons are chosen to take part in the poll. It is not possible to include everyone in the sample, but it is possible to give every voter a chance of being in the sample. Randomness is a key tool here. Statisticians use procedures to randomly select whilst giving every voter a known probability of being in the sample.
Sometimes this is augmented by setting quotas, whereby the sample is constrained to match the population in various ways. In political polls, quotas might be set so that the sample matches the population in gender, age groups and geography. In general this will improve the sample, particularly for small sample sizes. However, setting quotas is no substitute for having a poor random selection procedure - it cannot make poor methods good.
A random selection procedure is easy if the polling organisation has a list of voters and their contact details. This is rarely the case. More commonly telephone interviewing is used, with randomly dialled telephone numbers. Most but not all voters are contactable in this way, but there are a number of potential errors introduced. It is not sensible to interview the person who answers the phone, as that would give a bias towards females because research that they are more likely to answer the phone compared to males. Instead a randomly selected voter within the household should be interviewed. Polling companies differ in the way they do this random selection of a voter.
Selecting the sample for a poll does not guarantee that those voters will take part in the poll. Ideally every effort is made to obtain a response since there is a risk that non-responders will be different from those who respond. If the telephone is not answered, the same number should be tried later again and again. If the selected person in the household is not there at the time of the call, a time to call back should be arranged. The survey should be carried out over several days and several times of day, to allow for people who have various work and social patterns. This is not always cheap and not easy to fit within the pressures of delivering a timely result, so many pollsters cut corners on this. Different ways of cutting corners leads to different biases, and explains why companies may differ in their results.
While samples using these methods should provide a broad approximation to the voting public, there are all kinds of reasons why they might contain slightly too many of some groups and slightly too few of others. A well-designed poll will ask the respondents not only about their voting preference, but also about themselves such as their age and gender. This information is then used to compare the sample with, for example, census data. The raw numbers from the poll are then adjusted slightly, up or down, to match the profile of the public. If, for example, a poll finds that, when its survey work is complete, it has 100 members of a particular demographic group, but it should have 110 of them (in a poll of, say, 1,000 or 2,000), then it will "weight" the answers of the group so that each of those 100 respondents counts as 1.1 respondents. The responses from over represented groups would similarly be weighted down. This way the percentages reflect the population as a whole.
Many polls fall far short of these standards. The poorest use self-selecting sampling where voters choose to take part in the poll. Phone-in polls conducted by television programmes are typical of this group. This method leads to the most inaccurate results because voters who feel passionately about the subject of the poll tend to participate and therefore are not representative of the public. These polls also do not tend to collect the kind of extra information required to make some judgement about the nature of the sample.
The sample size is important in achieving precision, but as shown by the Literary Digest, cannot overcome biases in the methods. In Australia, sample sizes typically vary between 400 and 3,000. At the lower part of the range a poll is likely to be accurate to within 5% - which is useless in a finely balanced election or when considering marginal seats. A sample size of 2,500 reduces this to around 2% - much better but still limited when looking at trends.
No matter what mode the poll is conducted in - telephone, face-to-face or the Internet - the poll needs to capture the votes that people are most likely to cast in the election. This requires an understanding of the election process and this should be matched as much as possible. For instance, a survey should ideally mirror the actual voting ballot paper used. Except in polls for specific electorates, this is not always easy. In an Australian Federal election with 150 electorates, this would require 150 different forms and knowing which electorate each person interviewed is in. This is no easy task and too early before the election, the candidates are not yet known. Most polls use a simplified version, for example asking which party they will vote for as the first preference.
The preferential system used in Australia means that a poll must go beyond the first preference vote, especially when it is not for one of the two major parties. This can sometimes be asked as a separate question, such as "Which of the two major parties would receive your higher preference?" Different polling organisations address this issue in different ways, leading to some of the differences in their results.
There can also be effects in polls related to the format of the poll. People sometimes give different answers depending on whether they are asked questions in person, by an interviewer, or impersonally in self-completion surveys. Voting is a very individual affair, with confidentiality a key component, but most polls use telephone interviewing.
It is possible to improve political polls through careful techniques. One that is sometimes used is to also ask voters how they voted in the last election. The poll is then directly measuring change. Since the last election result is known precisely, measuring change from this base can be particularly accurate. It is not straightforward when considering more than two parties in Australia's preferential voting system, and this perhaps explains why few pollsters use it. However, when it can be used the improvement in precision can be enormous, making it much more cost effective than increasing the sample size.
A second method is to smooth out or average across several polls. Potentially this combines the sample sizes of each to give greater precision. It does have a significant drawback in that if the electorate is changing rapidly then these changes will be averaged out. This means that a balance is required between sufficient averaging to give precision but not so much that the trends are lost.
The public appears to have a great appetite for political polls and they often seem to drive public policy. They will not go away, so it is important that they are of the best quality possible. Ideally the users of polls would be able to recognise the differences between the good ones and the poor ones. This means asking for details on how they are conducted.
1 Sourced 29/10/2007 from http://www.britishpollingcouncil.org/questions.html