OrientationObservationIn-depth interviewsDocument analysis and semiologyConversation and discourse analysisSecondary Data
SurveysExperimentsEthicsResearch outcomes
Conclusion
8.4.3.4 Statistical significance Statistical significance is the measure of the probability that deviation in a sample from the population parameter (mean, proprotion, standard deviation) is likely to be the result of sampling error.
It is thus delaing with the same issues as confidence limits but just framing the difference as a probabilistic outcome rather than constructing the confidence limits and seeing whether a specified value lies within the confidence interval.
For example, Table 8.4.2.2.1 indicates that older people in the sample are more likely to know about Clause 28 than younger people. This looks to be quite marked, with twice the percentage of over-eighteens as under-eighteens having heard of the clause. However, once again we are dealing with a sample and need to consider sampling error. The example reveals a difference in knowledge about Clause 28 for each age group. However, is that a 'real' difference in the population or just a difference created by sample variation?
In short, what is the probability that such a difference could be the result of taking a random sample from a population where in fact all the three age groups have the same knowledge about Clause 28?
We can work this out by using a statistical significance test. A significance test operates in the same sort of way as confidence limits. It tells us what the probability is that any observed difference within a sample could be the result of sampling error.
All significance tests examine a null hypothesis. A null hypothesis is simply a statement that says there is no difference between two samples (or sub-samples) or that there is no relation between two variables. The significance test then works out the probability that this null hypothesis is true. The resercher decides on a cut-off point called the significance level. In the social sciences this is usually 5% (p=0.05) (which is equivalent to 95% confidence limits). See Section 8.4.3.5 for more on hypothesis testing.
If the probability that results from the test is less than 5% then we reject the null hypothesis. This means that we reject the idea that the difference is due to sampling error. In other words, if the test probability is less than 0.05 then the we are at least 95% confident that the difference in the sample reflects a difference in the population.
A widely used significance test for crosstabulated data is the chi-square test. It is applicable to all size tables (provided that there are sufficient numbers in each cell) and is fairly easy to work out and understand (although it is not always the best test in all the circumstances where it is used).
However, it is important to know how to interpret a chi-square result. The chi-square test operates by comparing the actual frequencies in the different cells of the table with the frequencies that would be expected if the null hypothesis were true.
Taking Table 8.4.2.2.1 as an example, the null hypothesis in this case would be that age makes no difference to knowledge about Clause 28. What this would mean in practice is that we would expect the same percentage of 'No' answers for each age group. There are very nearly twice as many 'No' answers overall as 'Yes' answers, so we would expect twice as many of each age group to say 'No' rather than 'Yes' if the null hypothesis that there was no difference was true.
Table 8.4.3.4.1 Approximate expected values given the null hypothesis of no difference for Table 8.3.12.6 V1 'Had you heard about Clause 28?' by V30 'Age in years'
Count
Expected frequencies
16 or 17
(1)
18
(2)
19 or 20
(3)
Row total
V1
Yes (1)
Actual
Expected
12
18
23
21
15
11
50
No (2)
Actual
Expected
42
46
40
32
17
21
99
Column
Total
54
63
32
149
In practice, to calculate chi-square you need to work out the expected frequencies exactly, (see Section 8.4.3.6.2.1).
The observed and expected frequencies are then compared. A simple visual check in this case suggests that there is a difference between the observed frequencies and the expected ones, with fewer younger people having heard of Clause 28 than expected. But is this difference statistically significant or just the result of sampling error.
8.4.3.4.1 Chi-square test
The chi-square test procedure provides the probability that the difference is the result of sampling error. The test procedure produces a value for chi-square. This can then be converted into the probability of the differences in the sample being the result of sampling error.
If you are working it out by hand (see Section 8.4.3.6.2.1), there are chi-square tables that provide the conversion from the statistic to a probability figure depending on sample size and the number of cells in the table (degrees of freedom, as it is known). A computer program a probability will provide both the value of chi-square and the corresponding probability. In the example, (Table 8.4.2.2.1 and Table 8.4.3.4.1) the calculated chi-square was 5.90 and this gives a probability (or significance value) of p=0.052.
What does that mean? It tells us that there is only a 5.2% chance that the observed sample came from a population in which there was no difference between age groups. This means, conversely, that there was a 94.8% chance that the differences in the sample reflect real differences in the population. Is this good enough? The answer to that depends on convention.
In the social sciences, as was noted above, it is conventional to take 5% as the cut-off point for all significance tests. If there is less than a 5% chance that the observed difference in the sample is due to sampling error then the difference is said to be statistically significantly different (the null hypothesis of no change is rejected). This means that you can generalise the difference in the sample to the population from which the sample is taken.
In the example, it means that although the sample shows a difference, this is not significant because probability is just over 5%. So it is possible that the differences in the sample are caused by random sampling error at the conventional 5% significance level.
Activity 8.4.3.4.1
Do males and females have significantly different views as to whether homosexuality should be part of sex education in schools? (compare V7 with V31) (See CASE STUDY Data File and Coding Frame.)
Note that there are a large number of significance tests that suit different situations. What this Section has concentrated on is the meaning of significance tests and how to make sense of the statistics that you might come across, either in books or generated by computer programs.
To sum up, in general a statistical significance test will give you a value for the statistic and a probability (or significance value), which you may have to look up unless you are using a computer program. If the significance value is less than 0.05 (5%) then the sample difference is significant at the 5% significance level, that is, the null hypothesis is rejected and the differences in the sample are not due to random sampling error but represent a real difference in the population.
So significance tests and confidence limits are statistical procedures used to make generalisations about the population from a random sample. They provide the grounds for making claims about the population. A statistically significant result means that the observed difference or relationship within a sample is due to a difference or relationship in the population from which the sample was taken.
However, statistical significance tells you nothing more than that an observed difference or relationship is unlikely to have been caused by random sampling error. It tells you nothing about the size of any difference or about the degree of any relationship.
FOR MORE INFORMATION ON SIGNIFANCE TESTS AND WHICH TO USE IN DIFFERENT CIRCUMSTANCES SEE Section 8.4.3.6
