OrientationObservationIn-depth interviewsDocument analysis and semiologyConversation and discourse analysisSecondary Data
SurveysExperimentsEthicsResearch outcomes
Conclusion
1.8.1 Introduction Validity, in social research, assesses the extent to which a research study addresses the issue that the research was intended to explore.
It asks whether the concepts used in the research represent the theoretical notions the research is grappling with. If someone is researching the extent of alienation among workers, is it adequate to construe alienation as dislike of management?
It asks whether the theoretical concepts are operationalised appropriately. For example, if the theoretical concept being studied is alienation, is a question that asks 'are you happy?' an appropriate operationalisation? (see Section 8 for more on operationalisation)
It asks whether the data collected is appropriate as a measure of the concept being investigated? For example, does Durkheim's use of national suicide rates measure the extent of suicide in different religious communities?
It asks whether the research instrument being used is able to collect the appropriate data? For example, will a focus group of patients provide information on how patients perceive their treatment?
It asks whether a scale (that is, a device for combining answers to questions to measure an attitude, disposition, property or attribute, see section xxx on scales) or a test actually measures the thing it is supposed to be measuring. Does an IQ test measure intelligence or does it measure the ability to do IQ tests?
Validity is also a term used in logic to evaluate an argument. In logic, the form of an argument is valid precisely if it cannot lead from true premises to a false conclusion. An argument is said to be valid if, in every model in which all premises are true, the conclusion is true. For example: "All A are B; some A are C; therefore some B are C" is a valid form. We will not pursue this aspect further. Click here for more on valid arguments in logic.
Detailed discussion of the notion of validity crosses over to other areas of methodological debate, both at the methodic and the epistemological levels. The following is, in places, complex and may require some diversions on the part of the reader to explore specific issues.
Much of the published discussion of validity assumes a positivist (usually quantitative) approach to social research. Validity is usually concerned with measures appearing to measure the concept they were designed to measure, or operationalisations appropriately representing the concepts they are supposed to represent. A lot of validity testing, as we shall see, is about creating scales, that is, a set of questions that represent different ways of expressing a variable and then adding them up in some way, or comparing results of questions designed as check questions.
However, validity is contingent on the epistemological approach being adopted and so there are other analyses (apart from a positivist perspective) of validity from a phenomenological or critical approach. What is valid for a positivist is not necessarily the same for phenomenologist or a critical social researcher. We will begin with the usual positivist accounts.
1.8.2 Positivist approach to validity
There is no simple solution to the issue of whether a piece of research is valid. Positivists use a variety of ways to assess validity. Usually, the issue is not whether research is valid or invalid but an attempt to explore the degree to which it is valid.
One might argue that validity is a binary state and that if research is not valid in every respect it must therefore be invalid. Such a line would render invalid virtually all social science and most cutting-edge natural science.
Positivists refer to a range of aspects of validity and suggest tests to examine each aspect. Although these different aspects focus on different elements of validity from a positivist perspective, there is far from universal agreement on the different categories and they do overlap. So, as a student, it is not essential that you learn all the different positivist ways of constructing validity, rather that you are aware that they exist and, more important, that they are not simple discrete categories into which the concept of validity can be neatly partitioned.
The key thing is to note, for a positivist, is that validity is about whether the research measures the thing it sets out to measure and there are no logical errors in drawing conclusions from the data.
The different notions are as follows (and can be examined in detail by clicking on the link):
1.8.3 Phenomenological approach to validity Phenomenological approaches to validity are not concerned with establishing whether measures of objects, attributes, events or actions correspond with the objective world. They accept that knowledge of the external world is constructed and they are, generally, more concerned that the interpretations made by researchers coincide with the meanings constructed by the social actors.
Unlike positivists who try and construct a distance between the researcher and the subject, phenomenological research takes engagement with the subject as an important element of the process of interpreting the world. Positivists think involvement with research subjects reduces validity. Phenomenologists have the opposite view and that denying one's role within research compromises validity.
Phenomenological social researchers take a variety of positions about the desirability and means of engaging with the concept of validity.
Some phenomenologists have argued that that some form of validation is necessary, at least to convince others of the value of the research.
Morse et al. (2002) argue that validity (and reliability) remain appropriate concepts in phenomenological research and to ignore them marginalises qualitative studies.
Hancock (2002, p. 22) maintains that it is important that qualitative data analysis is reliable and valid and that it is conducted in a rigorous manner, particularly given a common criticism that qualitative results are anecdotal.
Kvale (1983, p. 191) describes his view on phenomenological research as 'whether one has in fact investigated what one wished to investigate', which is not substantively different to the positivist view described above.
Lukiv (2004) proposes that validity can be achieved through a systematic approach:
Phenomenological research, on the other hand, although it also explores phenomena through whatever information seems opportune, should follow a systematic methodology that, described fully, addresses bias, gathering of data, analysis, interpretation, and, if necessary, ethical concerns and sampling procedures. A systematic methodology can establish a study's validity.
Others question whether the notion of validity is appropriate arguing that it is a positivist construct foisted upon phenomenological research. Beck et al. (1994) and Le Roux (2006), for example, argue that phenomenological research is heavily influenced by positivism in its adherence to notions such as validity. For Le Roux, this compromises the phenomenological approach and takes attention away from the anti-positivism of phenomenological research:
Even though qualitative researchers maintain that these constructs [validity and reliability] should be reviewed for their use in an interpretivist (qualitative) approach (Golafshani, 2003:597) and, that the 'usual canons of "good science" require redefinition in order to fit the realities of qualitative research' (Strauss and Corbin, 1990: 250) the mere inclination to have different 'rules' to describe these issues in qualitative research, points to a covert positivist influence. Also, the fact that qualitative research feels it needs to address the criticism of it being anecdotal (and thus the need for rigour in conducting it) is a not-so-covert justification of the contribution of qualitative research through using 'accurate' paradigm-sanctioned rules which are accepted as proof of 'good research'.
Amedeo (2002) argues that validity in phenomenological research is a function of the researcher's subject discipline and subfield and the philosophy of science adopted. He argues that validity does not have the same role within a phenomenological philosophy of science as within positivism and, indeed, it is not particularly important in Husserlian phenomenological approaches.
There is also a view that says validity of phenomenological research is contingent on it capturing the uniqueness of individual perceptions. The very idea of validity, from an interpretivist point of view, for example, is inconsistent with the uniqueness of individual perceptions and experiences. Validity is based on positivist assumptions about truth and knowledge, which value ideas and beliefs that have consistency and persistence over time. This is incompatible with the uniqueness of human experience that phenomenological approaches aim to capture.
Much of the published discussion refers to "qualitative research" or worse "qualitative research paradigm", which, as we suggest (Section 1.5), is misleading.
The ways in which validity in qualitative research is demonstrated are examined below, rather than being framed as forms of potentially measurable validity, as taken in the positivist approach, the phenomenological approach frames validity as based on plausibility, credibility and trustworthiness.
1.8.4 Critical social research approach to validity Critical social research, in its attempt to dig beneath the surface of appearances regards positivist notion of validity as suspect as it fundamentally designed to match measurements with a presumed objective reality. Kincheloe and McLaren, (1994, p. 151), for example, argued that the notions of internal and external validity used by positivists were not appropriate to critical research. They, in similar fashion to some phenomenologists, adopt the notion of trustworthiness, because, for them, critical research is about credibility. However, they admit that assessing trustworthiness is not easy to do.
Critical researchers award credibility only when the constructions are plausible to those who constructed them, and even then there may be disagreement, for the researcher may see the effects of oppression in the constructs of those researched. Thus it becomes extremely difficult to measure trustworthiness of critical research. No TQ (trustworthiness quotient) can be developed. (Kincheloe and McLaren, 1994, p. 151).
Another similarity to phenomenological approaches (and indeed seems to reflect some positivist notions of validity) is the idea that critical research should, in being emancipatory, be recognised by the subjects as reflecting their experiences (Reason, 1988).
1.8.4.1 Catalytic validity
Patti Lather (1986b) argued that critical social research needs to be rigorous and the data must be credible. However, despite the positivist implications, her call was for relevant praxiological research (see Section 2.4.2.4) .
For Lather, critical enquiry is about empowering 'the researched, to build emancipatory theory' but in so doing it is important to 'move toward the establishment of data credibility within praxis-oriented, advocacy research' (Lather, 1986b, p. 272).
She proposed 'catalytic validity' as a measure of validity in critical research. For Lather, catalytic validity
represents the degree to which the research process reorients, focuses, and energizes participants toward knowing reality in order to transform it. . . Efforts to produce social knowledge that will advance the struggle for a more equitable world must pursue rigor as well as relevance. (Lather, 1986b, p. 272)
For Lather, catalytic validity refers to research that not only displays 'the reality-altering impact of the inquiry process' it also empowers the research subjects to 'gain self-understanding and self-direction' (Lather cited in Kincheloe and McLaren, 1994, p. 151).
To her notion of catalytic validity, Lather (1986a, p. 78) adds three other elements that she considers ensures rigour and data credibility:
triangulation of methods, data sources, and theories;
reflexive subjectivity; the documentation of how the researcher's assumptions are related to the data;
face validity; established by recycling categories, emerging analysis, and conclusions back through at least a sub-sample of respondents.
Not a lot has been published on issues of validity in critical social research partly because the concept of validity is so heavily imbued with positivist notions of objective reality and researcher detachment. As such it is a non-concept for critical researchers who are more concerned about engaging with and taking apart so-called objective reality. Furthermore, critical researchers do not subscribe to the notion of the detached researcher, for them, research is about changing the world (see Section 2.4).
Oakley acknowledged feminist methodological problems 'in relation both to the validity of truth-claims and conflict with feminist values'. However, she insists that 'where non-hierarchical relationships between researcher and researched exist, the resulting data are intrinsically more valid' (Oakley, 1998, p. 711, emphasis in original).
1.8.5 Conclusion Validity is about demonstrating that you are researching what you set out to research and that you are explicit about what you have done.
The main issue in this discussion of validity is that in undertaking research one should always be scrupulous in the approach adopted, avoid sloppy mistakes, explain how you conceptualise the issue and present you evidence in a way that will convince your audience. Different audiences need different types of presentation to be convinced.
In the last resort, do not get preoccupied with defining and differentiating types of validity. It is more important to be critical (in a positive sense) of any research and to be self-critical of ones own research.
