Analytic Quality Glossary

Analytic Quality Glossary

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Home

Citation reference: Harvey, L., 2004-24, Analytic Quality Glossary, Quality Research International, http://www.qualityresearchinternational.com/glossary/

This is a dynamic glossary and the author would welcome any e-mail suggestions for additions or amendments. Page updated 8 January, 2024 , © Lee Harvey 2004–2024.

Recipes

_________________________________________________________________

Audit

core definition

Audit, in the context of quality in higher education, is a process for checking that procedures are in place to assure quality, integrity or standards of provision and outcomes.

explanatory context

Audit and institutional audit tend to be used interchangeably. However, there is a distinction between an external and internal audit (the former by an external agency) and between an institutional audit and a sub-institutional audit. There are then, four possibilities, external institutional audit, internal institutional audit, external sub-institutional audit and internal sub-institutional audit. The latter may include an audit of academic programmes, although this is usually referred to as programme assessment, approval or accreditation.

Audit tends also to be shorthand for quality audit, although it might also include, or specifically be about, auditing standards. In a sense, external examining is a standards auditing procedure.

Audits might establish the existence of procedures or may attempt to audit their effectiveness.

Audit may be to ensure conformance with regulations or procedures specified by an external body, such as regulations relating to examination of students.

Audit may aim to ensure compliance to policy objectives or system-wide practices. For the majority of policy objectives, this is probably most effectively done at the institutional level. However, if policy is focused on specific issues such as encouraging problem-based learning, the development of employability skills, developing inclusive curricula, encouraging equal opportunities, then programme-level assessments are useful tools to see to what extent such initiatives have been taken on board.

Normally, audits check whether procedures are in place, which usually requires an institution to specify its internal quality-monitoring procedures, including identification of responsibilities and intra-institutional communication and co-ordination of practices. Such processes will usually be circumscribed by established national or international protocols, codes of practice or tradition (Harvey, 2002, pp. 13–14). The academic audits of institutions formerly undertaken by the Quality Audit Division of the Higher Education Quality Council in Britain (c. 1993), for example, was an audit of internal quality assurance procedures. It made no attempt to evaluate the institution as such, just to ensure that the institution has clearly defined internal quality monitoring procedures that ensure effective action.

In some cases, audit is specifically targeted at improvement plans and processes, such as the quality audits of Finnish polytechnics undertaken by the Finnish Higher Education Evaluation Council (HEEC, 1997). In Sweden, the approach to audit undertaken by the National Agency was to focus on the stated improvement agendas of institutions and explore the efficacy of improvement projects and the approach appears to have aided the development of quality awareness and quality work in the institutions (Askling, 1997; 1998).

Often an audit team (usually of peers) will receive documentation and visit the institution to establish the effectiveness of the stated processes via panel discussion augmented by informal conversations and observation. Audits do not usually attempt to evaluate the institution as such, just to ensure that the institution has clearly-defined internal quality monitoring procedures linked to effective action.

Although higher education quality audit is an approach adopted, in various forms, by several countries, including New Zealand, Australia, Sweden, Finland Norway and the USA (Spanghel, 2001) it was initiated in the UK and is a term designed to focus on procedures rather than quality per se. Webb sets out the historical genesis and intent:

The term ‘audit’ was suggested by Professor Stewart Sutherland, Vice-Chancellor of the University of London and chairman of the CVCP’s Academic Standards Group and of the Management Board of the Academic Audit Unit between 1990 and 1992, partly in order to distinguish academic audit from other kinds of peer reviews exercises, and partly to hint at the intended methodology that would characterize this form of external academic review. Academic audit would seek to borrow and adapt some techniques from financial audit, such as the scrutiny of an institution’s documented procedures and practice, the extermination of these in operation through detailed inquiries and discussions with staff (and students), and the sampling of a wide range of activity and evidence, to establish and test the robustness and effectiveness of academic quality assurance. The analogy with financial audit was known to be far from perfect; it was not meant to imply that a university’s teaching and learning activities are comparable with a company’s balance sheet, or that they are susceptible to similar forms of analysis. Indeed, the starting point for an academic audit was, and remains, an institution’s stated aims and objectives and its stated means of assuring the quality, in particular, of its activities associated whit teaching and learning. This approach has enjoined audit teams to bring an open, non-prescriptive perspective to each institutional audit and to review arrangements for quality assurance agnostically, rather than from a conviction that a given set of procedures or an approach must be right, irrespective of the institutional context or an institution’s declared aims and objectives. (Webb, 1994, p. 47)

analytical review

For INQAAHE (2001, p. 2) audit is about evaluating processes rather than evaluating quality. The definition also identifies external sub-institutional audit:

Audit is the evaluation or review of procedures, processes and mechanism in an HEI or a part of an HEI (e.g., school of Medicine, school of Business & Management). Not quality itself is subject of review, but procedures and mechanism to assure the quality and the processes to achieve the mission and goals… A quality audit evaluates especially the procedures and processes for the assurance of the quality. The main assumption is that if QA-procedures are in place, one may expect that HEI will deliver quality. … As soon as the quality of the core activities (educational activities and research outcomes) are judged too, it is called institutional assessment.

Green, (1994, p.12) also implicitly differentiates assurance and management audits but argues that quality audit:

focuses on the robustness of quality assurance and quality management systems, and is therefore legitimately the responsibility of the institutions themselves.

Fraser (1994, p.106) disassociates audit from any form of evaluation of mission or outcomes:

A scrutiny by a group external to the university checking that the quality assurance and quality control processes are appropriate and working properly has been described as a ‘quality audit’…. Quality Audit is neither concerned with a university’s mission (objectives, inputs) nor with how successfully these objectives have been attained (outputs), but solely with the processes by which the university checks on the relations between its inputs and outputs.

The European Training Foundation (1998, p. 12) defines audit as:

Quality audit: Evaluation of an institution’s processes for quality management.

The Australian Universities Quality Agency (2004) defines audit as:

‘Quality audit’ is defined as “a systematic and independent examination to determine whether activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives” (Australian/New Zealand Standard, 1994). This definition forms the basis upon which AUQA has developed its approach to audit.

University of Canberra (2002), has a slightly different take, however:

The major aim of the audit is to consider and review the processes an organisation has in place to monitor and achieve its objectives and the outcomes being achieved.

Southern Cross University (2003) states:

Academic audits of self-accrediting institutions will be whole-of-institution audits based on a self-assessment and a site visit. The audit will assess the extent to which the university is “fit for its purpose” and achieving its missions and objectives

Westerheijden (2007, p. 8) wrote:

‘‘Quality audit’ is used in the jargon of higher education for an evaluation mechanism that investigates quality management arrangements within higher education institutions.

Woodhouse (1999) reminds us that ISO (Standards New Zealand, 1994) defines quality audit as a three-part process, checking: 1) the suitability of the planned quality procedures in relation to the stated objectives; 2) the conformity of the actual quality activities with the plans; and 3) the effectiveness of the activities in achieving the stated objectives.

The British Standards Institution has a generic definition that focuses on the compliance of processes with planned intent:

Quality audit is a systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives. (BSI, 1989)

Tempus (2001) defines quality system audit as both an internal and external process:

Quality System Audits – systematic and independent investigation with the purpose of determining whether the actions and results referring to quality are in accordance with the set regulations, whether these regulations are suitable for achieving the set goals and whether they are really being implemented. It can be internal and external.

The University of Aberdeen (2002) equates audit with external processes:

Quality Audit: A systematic and independent examination to determine whether quality control and quality assurance are satisfactory, implemented effectively, and are suitable to achieve objectives.

The Quality Assurance Agency for Higher Education (QAA, undated) in the UK states:

Institutional Audit: The process used by QAA until 2011 to report on how, and how well, higher education institutions in England and Northern Ireland safeguarded quality and standards.

Audit of collaborative provision: A process used until 2011 by QAA to assess the academic standards and quality of higher education delivered through a collaborative arrangement between a degree-awarding body in England or Northern Ireland and a partner organisation in the UK or elsewhere (see collaborative arrangements). This process was only used in situations where the collaborative arrangements were too complex to be reviewed by a normal or slightly modified Institutional audit.

Citing the British Higher Education Quiality Councils Learning from Audit 2, Duff et al. (2000, p. 130) stated:

An institutional quality audit is a review of the overall institution in respect of its quality assurance systems. …. The aim of a quality review or audit is to evaluate the academic structures, procedures and standards of the institution, faculty or department in order to satisfy customers of the institution, other stakeholders and the funding authorities that the educational awards of the institution are of the quality claimed

Woodhouse (1999, pp. 30–31) defines audit as:

a check on an organisation’s explicit or implicit claims about itself. When an institution states objectives, it is implicitly claiming that this is what it will do, and a quality audit checks the extent to which the institution is achieving its own objectives. When the claims are explicit (as in financial reporting or if the institution has done a self-quality audit), audit becomes a validation (or otherwise) of those claims. Audit asks “are your processes effective?” (i.e. in achieving your objectives). The output of an audit is a description of the extent to which the claims are correct. An audit is sometimes called a review.

Haakstad (2001, p. 77) sees audits as having an improvement function:

The audits were to evaluate how the institutions accomplish their responsibility for quality assurance and quality work, however in a way that addresses the internal capacity of the institution and motivates efforts at self-improvement and institutional learning.

The QEPSE-Leonardo (2011) Glossary suggests that audits can be undertaken of programmes as well as institutions:

Audit is the process of reviewing an institution or a programme that is primarily focused on its accountability, and determining if the stated aims and objectives (in terms of curriculum, staff, infrastructure, etc) are met.

Audit; Der Prozess zur Überprüfung einer Einrichtung oder eines Programms. Er richtet sich in erster Linie auf die Verantwortlichkeit und Feststellung, ob die angegebenen Ziele (bezüglich Lehrplan, Personal, Infrastruktur, etc.) erreicht werden.

The UNESCO glossary defines quality audit:

Quality Audit: The process of quality assessment by which an external body ensures that (i) the institution of programme quality assurance procedures or (ii) that the overall (internal and external) quality assurance procedures of the system are adequate and are actually being carried out. Quality audit looks to the system for achieving good quality and not at the quality itself. A quality audit can be realized only by persons (i.e., quality auditors) who are not directly involved in the areas being audited. Quality audits can be undertaken to meet internal goals (internal audit) or external goals (external audit). The results of the audit must be documented (audit report). See, also, audit). (Vlãsceanu et al., 2004, p. 50)

The UNESCO definition also suggests, under audit that:

In the United Kingdom, when an audit is an institutional process carried out internally, the process is described (since 2002) as an “institutional review” process. (Vlãsceanu et al., 2004, pp. 22–23)

There is a conflict in the last sentence with the UNESCO definition of institutional audit elsewhere: the quote above implies review is an internal process but Vlãsceanu et al., (2004, p. 23) state that it is a peer review process, implying an external process.

CHEA (2002) defines quality audit as:

Quality Audit: A test of an institution's quality assurance and control system through a self-evaluation and external review of its programs, staff, and infrastructure. Designed to provide an assessment of an institution's system of accountability, internal review mechanisms, and effectiveness with an external body confirming that the institution's quality assurance process complies with accepted standards.

CHEA (2002) has a separate definition of audit which is closer to external sub-institutional audit:

Audit: A process of review of an institution or program to determine if its curriculum, staff, and infrastructure meet its stated aims and objectives. An audit focuses on accountability of institutions and programs. (In the U.K., an audit is an institutional process. The term "audit" is scheduled to be replaced in 2002 by "institutional review" as part of a new academic review process.)

Luckett (2003, p. 15) discussing the (then) new South African process suggests that audit is rather prescriptive [and later the HEQC emphasised 'fitness of purpose' alongside its 'fitness for purpose' approach]:

...the HEQC defines the nature of audit according to the accepted meaning in quality assurance discourse, namely that its purpose will not be to judge quality per se, but rather to evaluate the nature and extent of the quality management system in place at an institution and to judge its effectiveness on the basis of evidence provided by the institution. Quoting from its draft Institutional Audit Framework (2002), the HEQC reminds its readers of its objectives for audit: 1. To enable a higher education institution to assure itself, its stakeholders and the HEQC that its policies, systems and processes for the development, maintenance and enhancement of quality in all its educational offerings are functioning effectively. 2. To enable providers and the HEQC to identify areas of strength and excellence as well as areas in need of focused attention for improvement in the short, medium and long term. 3. To provide for consistency in quality management across the higher education sector and generate a national picture of the role of quality management in the transformation of higher education.... The HEQC seems to be suggesting that its audit system will be able to serve both the evaluative and supportive functions. The HEQC document then goes on to specify specific objectives for the first cycle of audits. The HEQC is clearly setting the audit agenda here and through its specification of criteria for audit intends to ‘signal clearly to the HE community the institutional areas in need of systematic attention...'...In an apparent contradiction of its earlier definition of the nature of audit, in describing the nature of its criteria, the HEQC suggests ...that these should serve a dual purpose, namely to make judgments about quality per se and about quality management, for both audit and accreditation and also for internal and external evaluation. This dual function is described as follows: The criteria should be useful for institutions in developing and enhancing the quality of provision in a way that advances the achievement of national goals and priorities in higher education in South Africa. They should serve diagnostic as well as improvement purposes in respect of the core functions of the institutions.

Kristoffersen (2003) does not talk of audit but of evaluations of quality assurance mechanisms. However, this appears to be similar to audit but involves an assessment of the impact of quality processes.

Evaluations of quality assurance mechanisms aim to assess the quality assurance mechanisms within, typically, a specific system, institution or programme, and how these affect the quality of the educational activities.

Universiti Malaysia Sarawak (2011) unusually refers to programme audits:

Academic programme audit is a systematic assessment in order to ensure that the activities and other related processes are implemented effectively in accordance to the quality assurance planning and would achieve its objectives.

A totally different meaning of audit comes from the United States (which has not been accommodated in the core definition):

Audit: Enrolling in a class on an audit basis means the class would not count for credit or grade point average. In some cases the audit fee is less than the tuition rate. Registration for audit may require the permission of the instructor. (NTNC, 2002)

associated issues

In the review of the contributions in the first 15 years of the international journal Quality in Higher Education, Harvey and Williams (2010) write the following about the contributions on audit:

Quality audit as an approach was examined in several articles. Dill (2000) provided a US perspective on academic audit. He argued that academic audit (first developed in the UK and adapted in Sweden, New Zealand and Hong Kong) was an accountability mechanism that improved the capacity of colleges and universities to independently assure the quality of their academic degrees and student learning. He examined problems in implementing academic audits; including focus; selection and training of audit teams; audit self-studies; conduct of visits; reports; and follow-up. He concluded that audits have: helped initiate development of quality assurance systems within institutions; put the improvement of teaching and learning on institutional agendas; reinforced institutional leaders in their efforts to develop institution-wide quality cultures; provided system-wide information on best practices; offered visible confirmation to the public that attention is being paid to academic quality assurance.

Woodhouse (2003) claimed that a quality audit checks an organisation’s effectiveness in achieving its goals. He described improvements recorded as a result of the activities of the New Zealand Universities Audit Unit and the Australian Universities Quality Agency and argued that external quality audit can augment an institution’s ability to improve.

Cheng (2009) explored how academics in a pre-1992 university in England understood their work as a profession. She revealed that academics’ professionalism has affected their attitudes towards audit-related quality mechanisms and resulted in a tension between professional values and the audit. This tension was caused by the perceived bureaucracy of the audit, its time cost, and the perception that the audit is a symbol of distrust in the professionalism of academics.

related areas