OrientationObservationIn-depth interviewsDocument analysis and semiologyConversation and discourse analysisSecondary Data
SurveysExperimentsEthicsResearch outcomes
Conclusion
8.3.8 Pilot survey Irrespective of how experienced you are, problems will always arise when carrying out a survey. To minimise the potential problems it is advisable toundertake a pilot survey.The pilot survey will bring to light problems that you can put right before doing the full scale survey.
A pilot survey usually consists of two stages: the pre-pilot and the complete pilot.
8.3.8.1 Pre-pilot The pre-pilot usually just tests the adequacy of the schedule or questionnaire. By asking a small number of people, the pre-pilot indicates whether any of the questions cause problems for respondent or interviewer and whether they flow from one to the next in a logical and meaningful sequence.
It helps if the respondents at this stage are as similar as possible to the people in the final sample.
Activity 8.3.8.1.1
Pre-pilot the schedule you prepared in Activity 8.3.7.5.1 Make a note of any questions that are a problem.
8.3.8.2 Complete pilot
A complete pilot of a survey involves a small-scale trial run-through of the entire survey from data collection to analysis to see if it works and whether the right kind of information is being collected.
There is no fixed size for a pilot sample. It depends on the complexity of the interview schedule and the nature of the population being sampled. If it is to be at all meaningful, then each interviewer should do a minimum of five pilot interviews and the total should not be less than about twenty-five.
Piloting a questionnaire or schedule interview is an extremely important part of the research process. There is a tendency for researchers to skip this part and to get on with the research proper. They subsequently regret having missed the pilot survey out!
The final data collection is a time-consuming and expensive process that you only do once so it is important to get it right. Piloting the research is invaluable in identifying problems such as the wording or ordering of the questions, ambiguities in instructions to interviewers, the adequacy of the introductory statement, the suitability of the coding frame and difficulties in the analysis of the data.
The pilot survey provides an opportunity to assess the reliability (See Section 1.9) and validity (See Section 1.8) of the survey. For example, Rood (2011) who was undertaking a study of employer preferences, constructed an initial survey that was:
refined using a pilot group of professional colleagues who were not part of the study. This was to test for construct validity and to identify any potentially ambiguous items that may threaten the validity or reliability of the instrument.... This resulted in some rephrasing and elimination of some questions and the addition of others, to what became the final survey instrument.
8.3.8.2.1 Survey reliability
Social surveys are presumed to be reliable because they use a schedule of questions to measure the same thing in the same way each time the schedule is applied. If you use a tape-measure to measure someone's height you would expect it to be consistent. If you had a tape-measure that stretched you would regard it as unreliable and probably unusable.
Checking a questionnaire or interview schedule for reliability, however, is not as easy as checking to see whether your tape-measure has stretched.
There is no sure way of testing the reliability of a survey. The problem is that the subject matter of a social survey is both complex and changing.
8.3.8.2.2 External reliability
One method of testing the reliability of a questionnaire over time (so-called external reliability) is by administering it to the same sample again and checking to see if you get the same results. The problem with this test–retest method (See Section 1.9.1.1.2) is that the sample itself will have changed from the first to the second testing. So you will not know whether any difference in answers is due to an unreliable survey or a change in attitude and knowledge of the respondents.
8.3.8.2.3 Internal reliability However, there are various ways of assessing the internal consistency of the survey. By far the best way to do this is to divide the responses on the questionnaire into two groups at random and compare the answers on each half for the sample. A high degree of agreement between the two sets of results suggests, although by no means confirms, a reliable set of questions, that is, a set that is measuring the same thing at any given point in time.
This reliability test is known as the split-half technique (See Section 1.9.1.1.3). Split-half thus gives some idea of the reliability of the questions at one point in time but does not tell us whether the measurement is reliable over time.
Reliability is not enough in itself though. A questionnaire, for example, may be consistent, but it may be consistently measuring the wrong thing.
8.3.8.2.4 Survey validity
The major concern of operationalisation is the problem of validity (See Section 1.8). How can you be sure that the operational measurement still measures the theoretical concept?
For example, do Townsend's (1974) questions in the CASE STUDY about deprivation actually measure poverty?
There is no way in which the validity of a set of questions can be 'tested'. The choice of questions is ultimately a subjective decision of the researcher based on theoretical consideration, personal preference and prior experience.
The pilot survey can show whether a question 'works' but even so there is no guarantee that it means the same thing to all the respondents. Different age groups or different ethnic groups may understand different things by similar questions. The connotation of a word can be quite different for different respondents.
Assessing the answers to questions in the pilot will give you some clue as to when very different interpretations are being put on questions (this usually shows up in 'open' questions). Subtle differences of meaning in closed questions are often very difficult to detect because fitting answers into pre-codings may conceal different meanings. So, in short, there is no simple check on question validity.
There is always a problem of whether or not the questions represent the concept the researcher is trying to find out about from the point of view of both the researcher and the respondent.
Pilot sample
There is no fixed size for a pilot sample. It depends on the complexity of the interview schedule and the nature of the population being sampled. If it is to be at all meaningful, then each interviewer should do a minimum of five pilot interviews and the total should not be less than about twenty-five.
Piloting a questionnaire or schedule interview is an extremely important part of the research process. There is a tendency for researchers to skip this part and to get on with the research proper. They subsequently regret having missed the pilot survey out! The final data collection is a time-consuming and expensive process that you only do once so it is important to get it right. Piloting the research is invaluable in identifying problems such as the wording or ordering of the questions, ambiguities in instructions to interviewers, the adequacy of the introductory statement, the suitability of the coding frame and difficulties in the analysis of the data.
Activity 8.3.8.2.4.1
Pilot the schedule you pre-piloted in Activity 8.3.8.1.1
1. Are there any remaining problems with the schedule?
2. What problems arise in locating respondents?
3. See whether the schedule gives you the sort of data you want and in a way that you maybe able to analyse.
4. Assess the interviews to see whether any form of interview bias arises when carrying out your pilot. If so, consider how you would overcome it.
NOTE: See Section 8.4 for how to analyse the data collected using statistical techniques.
