Analytic Quality Glossary

Analytic Quality Glossary

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Home

Citation reference: Harvey, L., 2004-24, Analytic Quality Glossary, Quality Research International, http://www.qualityresearchinternational.com/glossary/

This is a dynamic glossary and the author would welcome any e-mail suggestions for additions or amendments. Page updated 8 January, 2024 , © Lee Harvey 2004–2024.

Recipes

_________________________________________________________________

Assessment

core definition

Assessment is a general term that embraces all methods used to judge the performance of an individual, group or organisation.

explanatory context

Assessment includes assessment of quality, through an external process of some kind as well as assessment of student learning and assessment of teaching and learning including teacher performance. These are dealt with elsewhere.

Also, note the slightly confusing use of assessment in the US context where it relates to the Assessment Movement, discussed below.

Note, also, that assessment of teacher performance is not the same as Teaching Quality Assessment, as used in the UK, which is a quality assessment process explored below.

Quality assessments set out to ‘measure’ the level of quality of inputs, processes and, sometimes, outputs. This may be a judgment of the overall quality of an institution or programme or of specified component elements. In France, for example, the Comité National d’Évaluation (CNE) evaluates each institution holistically (Staropoli, 1991; Ribier, 1995).

Assessment has also focused on research, most notably on research outputs. Finland undertook such evaluations in the early 1980s (Luukkonen and Ståhle, 1990) and Lithuania has evaluated research performance (Mockiene and Vengris, 1995). Probably the most extensive approach is the periodic Research Assessment Exercise undertaken in the United Kingdom, which not only provides a public rating of research for all higher education institutions across more than 60 subject areas but also allocates significant government funding to institutions on the basis of the rating — the higher the rating the more money is allocated (HEFCE/SHEFC/HEFCW, 1993). The Research Assessment Exercise will be replaced by the Research Excellence Framework (REF), which is the new system for assessing the quality of research in UK higher education institutions in the UK. It will be completed in 2014.

analytical review

The International Network of Quality Assurance Agencies in Higher Education (INQAAHE, 2001, p. 2) refer to assessment as follows:

Assessment is the evaluation of the quality itself. Assessment tries to collect data, information and evidence of the quality of the HEI as a whole (institutional assessment) or its core activities (education, research and community service) separately (program assessment). It goes beyond quality procedures (although it will be included) and tries to judge the quality of input, process and output.

The Council for Higher Education Accreditation does not refer to judgements in its definition of [quality] assessment:

A diagnostic form of quality review and evaluation of teaching, learning, and programs based on a detailed examination of curricula, structure, and effectiveness of the institution, its internal review, and quality control mechanisms. (CHEA, 2002)

Westerheijden (2007, p. 8) although not wanting to closely define terms, wrote:

‘Quality assessment’ or ‘evaluation’ is used for external quality assurance schemes when focusing on the ‘measurement’ (quantitatively and qualitatively) of quality.

The QEPSE-Leonardo (2011) Glossary states:

A general term that embraces all methods used to judge the performance of an individual, group or organization.

Bewertung: Ein allgemeiner Begriff, der alle Methoden umfasst, um die Leistung einer Person, einer Gruppe oder Organisation zu beurteilen.

Woodhouse (1999, p. 32) focuses on the grading aspect of assessment:

Assessment is an evaluation that results in a grade, whether numeric (e.g. a percentage or a shorter scale of say 1 to 4), literal (e.g. A to F) or descriptive (excellent, good, satisfactory, unsatisfactory). There may or may not be a pass/fail boundary somewhere along the grade spectrum (or it may simply be a two-point scale). Assessment asks “how good are your outputs?” The output of an assessment is the grade. An assessment is sometimes called an evaluation.

The UNESCO definition suggests two meanings pertinent to higher education quality. The first sees assessment as a judgemental evaluation. The second, is a specific evaluation process focusing on learning and teaching and reflects the issues discussed under assessment of learning and teaching.

1. The process of the systematic gathering, quantifying, and using of information in view of judging the instructional effectiveness and the curricular adequacy of a higher education institution as a whole (institutional assessment) or of its educational programmes (programme assessment). It implies the evaluation of the core activities of the higher education institution (quantitative and qualitative evidence of educational activities and research outcomes). Assessment is necessary in order to validate a formal accreditation decision, but it does not necessarily lead to an accreditation outcome.
2. A technically designed process for evaluating student learning outcomes and for improving student learning and development as well as teaching effectiveness. (Vlãsceanu et al., 2007, pp. 29–30)

Vlãsceanu et al., (2007) state that institutional effectiveness can be assessed via:

Outcomes Assessment: The process of evaluation and improvement of specific results of a higher education institution in order to demonstrate its institutional effectiveness. Assessment may concern the performance of teaching staff, the effectiveness of institutional practices, and/or the functioning of departments or programmes (e.g., programme reviews, budget reviews, etc.). It is a formative procedure used for institutional self-study, financial retrenchment, programme evaluation, and better understanding of the current needs of students.

And effectiveness of student learning outcomes via:

Student Outcome Assessment: The act of assembling, analyzing, and using both quantitative and qualitative evidence of teaching and learning outcomes, in order to examine their congruence with stated purposes and educational objectives and to provide meaningful feedback that will stimulate improvement. (Vlãsceanu et al., 2007, p. 64)

The UNESCO glossary also has an entry for quality assessment, which it equates with review:

Quality Assessment/Quality Review: Indicates the actual process of external evaluation (reviewing, measuring, judging) of the quality of higher education institutions and programmes. It consists of those techniques, mechanisms, and activities that are carried out by an external body in order to evaluate the quality of the higher education processes, practices, programmes, and services. Some aspects are important when defining and operating with the concept of quality assessment: (i) the context (national, institutional); (ii) the methods (self-assessment, assessment by peer review, site visits); (iii) the levels (system, institution, department, individual); (iv) the mechanisms (rewards, policies, structures, cultures); (v) certain quality values attached to quality assessment such as academic values, traditional values (focusing upon the subject field), managerial values (focusing on procedures and practices); pedagogical values (focusing on staff and their teaching skills and classroom practice); employment values (emphasizing graduate output characteristics and learning outcomes). (Vlãsceanu et al., 2004, p. 48)

For Kisuniene (2004):

Quality assessment means a process of external assessment: students’ achievements and performance, school documentation, statistical data, observations, interviews.

Another view relates assessment to subject review, emphasising direct exploration of provision:

The area of quality assurance where standards of teaching and learning in institutions are directly evaluated. This may be by self-assessment, visits by external teams or, usually, some combination of the two. (AEC, 2004)

Quality Assessment : External assessment by peers of actual provision in particular subjects, by scrutiny of institutional documentation and student work, direct observation, interview, and by reference to performance indicators. (University of Aberdeen, 2002)

The latter draws on the UK system of quality assessment (now concluded) that:

examines the wide range of influences that shape the learning experiences and achievements of students. It covers the full breadth of teaching and learning activities, including: direct observation of classroom/seminar/workshop/ laboratory situations, the methods of assessing students’ work, students’ work and achievements, the curriculum, staff and staff development, the application of resources (library, information technology, equipment), and student support and guidance. This range of activities is captured within a core set of six aspects of provision, each of which is graded on a four-point assessment scale (1 to 4), in ascending order of merit. The aspects of provision are: curriculum design, content and organisation; teaching, learning and assessment; student progression and achievement; student support and guidance; learning resources; quality management and enhancement. (QAA, 2004)

One view sees quality assessment as the same as review, measurement or evaluation:

Quality assessment, quality measurement, and review of quality are all taken here to be synonymous with evaluation, especially when there is an external element to the procedure. (Campbell & Rozsnyai, 2002, p. 32).

Quality assessment can also be used in conjunction with quality control, essentially as the vehicle for testing the quality control system:

Quality assessment: a system of actions which guarantees that the quality control will be efficient and includes assessment of products and production, that is, of the measuring system. That is a mechanism used to establish whether the quality system works within the set boundaries of tolerance defined on the basis of the end user’s judgement. (Tempus, 2001)

In the US, the development of the Assessment Movement resulted in a slightly different use of the term assessment, viz. assessment of improvement processes:

Assessment is an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance. When it is embedded effectively within larger institutional systems, assessment can help us focus our collective attention, examine our assumptions, and create a shared academic culture dedicated to assuring and improving the quality of higher education (Thomas A. Angelo, AAHE Bulletin, November 1995, p.7).

associated issues

Assessment and accreditation

Both the INQAAHE and UNESCO definitions state that while assessment does not lead to accreditation, accreditation decisions must be based on accreditation evidence.

Assessment does not necessarily lead to a formal accreditation decision. However, formal accreditation needs to be based on assessment. (INQAAHE, 2001, p. 2)

Assessment is necessary in order to validate a formal accreditation decision, but it does not necessarily lead to an accreditation outcome. (Vlãsceanu et al., 2007, p. 30)

Not everyone agrees that accreditation requires judgement on the level of quality as is implied by assessment, other than perhaps a basic threshold judgment. The concern is that some accreditation processes do not actually measure/assess quality (even at a threshold level) but establish that requirements have been complied with, which is taken as a proxy for a quality judgement.

Assessment: accountability and improvement

Assessment is primarily an accountability process. However, it can also be used to encourage improvement. In the UK, the QAA (2004) asserts that the purposes of quality assessment are:

· to ensure that the public funding provided is supporting education of an acceptable quality,

· to provide public information on that education through the publication of reports,

· and to provide information and insights to encourage improvements in education.

In the United States, it has been claimed that:

No state has yet developed an adequate approach to assessing student learning. And yet surely this is the most important category of all (Bennett, 2001).

The key to assessing quality, it is argued, is the value added by an institution or programme.

Virtually everyone who has thought carefully about the question of assessing quality in higher education agrees that "value added" is the only valid approach (Bennett, 2001).

Assessment measures

Harvey (2002, p. 14–15), noted:

Measurement may be against externally-set criteria (both implicit and explicit), against internally-specified objectives or missions, or a mutually agreed set of criteria. Many assessments are supposedly of fitness for purpose and thus institutions or programmes are assessed against mission-based criteria. In practice, there are a set of overarching expectations and the mission-based variability operates within narrow tolerances. Assessment might include a complex grading system or might be based on a simple satisfactory/non-satisfactory dichotomy….

Assessment thus may focus on inputs (such as teaching staff, learning resources) or process (such as teaching, learning, support services) or outcomes, (such as students academic standards of achievement or professional competence, employment rates, student perception of their learning). Assessment evidence includes statistical indicators, direct observation, direct evaluation of research outputs, student and graduate views, employer views, student performance, self-assessment and other documentation, discussion and interviews with teachers, students and managers, and perceptions of other agencies, such as professional bodies.

Assessment and benchmarking

Assessment may also ‘benchmark’ against other institutions, national norms or against oneself over time.

Student views on quality assessment

Cardoso et al. (2012) write:

quality assessment seems to be seen by students as a legitimate state device for controlling and regulating universities, enhancing accountability, performance and continuous improvement. This device is primarily directed at universities’ governance and management, teaching and learning, students’ progression and graduates’ access to the labour market. However, students also identify some weaknesses in quality assessment, such as a lack of transparency and a delay in producing effects; its bureaucratic character; the inconsistent involvement of some institutional actors, namely students; the insufficient diffusion of its results and reports; and an inadequate implementation of its recommendations (Amaral, 2003; Leite et al., 2006, 2007; Polidori, 2000).

related areas