33 Evaluating the curriculum
The importance of evaluation
• With the current emphasis on quality assurance it is important to ascertain whether the curriculum and the teaching are fit for purpose.
• Curriculum development is an ongoing iterative process, which requires to be informed by an evaluation of current practice and with feedback as to problems identified.
• There is a move to recognise excellence in education in medical schools as well as excellence in research.
Levels of evaluation
An evaluation of the education programme can be conceptualised at different levels:
• At the macro level the mission of the medical school and the extent to which the school is achieving its overall aims should be examined.
• At the micro level the elements of the curriculum should be evaluated. These include the learning outcomes, the teaching, learning and assessment methods, the educational strategies, the education environment, the management of the curriculum and the engagement with students and staff.
The mission of the school
An aspect of the mission of a medical school that has attracted attention is the social responsibility and accountability of the school. A school should not be an ivory tower but should be regarded as excellent only if significant attention is paid to its social accountability and how it relates to the community that it serves. Boelen and Woollard (2009) have argued that for a school to be socially responsible it needs to have a commitment to respond to the priority health needs of citizens and society. The impact of the educational institution on society and the public good should be part of the assessment of its success. It has been shown that the most research-active medical schools demonstrate the least social responsibility. Some schools see as their mission the development of future leaders or researchers.
Curriculum evaluation
Definition
Curriculum evaluation can be defined as ‘a deliberate act of enquiry which sets out with the intention of allowing people concerned with an educational event to make rigorous, informed judgements and decisions about it so that appropriate development may be facilitated’ (Coles and Grant 1985).
Aims of curriculum evaluation
The evaluation of a curriculum can have different purposes:
• Demonstration of the achievement of the minimum standards expected by an accrediting body such as the General Medical Council in the UK, the Liaison Committee on Medical Education (LCME) in the USA or the Australian Medical Council (AMC) in Australia. The World Federation for Medical Education (WFME) has set out minimum standards for educational programmes at undergraduate, postgraduate and continuing education levels.
• Establishing that the programme as set out (‘the planned curriculum’) is in fact happening (‘the received curriculum’). There is often a gap between what is set out on paper and what happens in practice.
• As an essential requirement for curriculum development. Without an evaluation of the curriculum there can be no informed curriculum development. Curriculum evaluation should be concerned not simply with a measurement of the success or failure of a curricular initiative but be a more complex assessment that provides a fuller understanding of the education process. Parlett and Hamilton (1975) described this as ‘illuminative evaluation’.
• The assessment of a curriculum change. A better understanding of a change made to the curriculum can be sought and the extent to which it has been effective measured. The difference between curriculum evaluation and research is sometimes questioned. As argued by Kelly (2004), curriculum evaluation becomes part of curriculum research. In the context of research, the findings of the evaluation must be examined from a generalisable perspective. Curriculum planning and evaluation can be seen as hypothesis to be tested.
• A comparison between and understanding of education programmes in institutions in different countries or with different approaches and admission policies. Such information is useful in the context of student mobility as well as for use in research in medical education.
• The provision of information for potential students. A curriculum evaluation can provide information not only on the excellence of the education programme in a medical school but also on its character, and this might inform a student’s choice of medical school. The publication by the Higher Education Funding Councils in the UK of the ratings of the education programmes in medical schools was associated with significant changes in the pattern of schools to which students applied to be admitted.
Approaches to curriculum evaluation
The ten questions
The ten questions to be addressed in the planning of a curriculum, as described in Chapter 11, can be used also as a framework for a curriculum evaluation. Examples are given of the issues to be addressed relating to each question.
1. The needs which the medical school aims to meet. The evaluation of the mission of the medical school is discussed earlier in the chapter.
The ‘SPICES’ model as described in Chapter 11 is a useful tool to analyse the curriculum and the educational strategies adopted:
6. Teaching and learning methods and opportunities.
9. Communication about the curriculum.
Kirkpatrick’s four levels of evaluation
Each successive level in the model provides a more precise measure of the effectiveness of the educational programme but requires a more rigorous and time-consuming analysis. Kirkpatrick suggests that evaluation can begin at the first level and, as time and resources allow, move to the higher levels (Fig. 33.1).
Level 1: Opinion/reaction. Evaluation at this level examines how participants in the education programme react to it. Are they satisfied? Do they like or dislike it? Does it meet their needs?
Level 2: Competence/learning. This level moves beyond the learner’s satisfaction, and the extent to which there is a difference in the learner’s knowledge, skills or attitudes is assessed. Students’ performance in written or clinical assessments is studied and taken as evidence of the effectiveness of the education programme.
Level 3: Performance/behaviour transfer. Changes in the learner’s behaviour are studied at this level. As a result of the education programme do students communicate more effectively in their day-to-day contacts with patients? Do students apply their knowledge of the basic sciences to understand better the pathophysiology of patients they encounter?
Level 4: Outcome/results. This level, in the context of Kirkpatrick’s work, measures the success of a training programme in terms of the business results. Did sales increase after training was completed? In medical education the question is whether the training of the doctor affects medical practice. Does the training in cardiac auscultation and the interpretation of murmurs, for example, reduce the need for laboratory investigations of patients? Does education of doctors about hypertension reduce the incidence of side effects in patients presenting with raised blood pressure?
Undertaking a curriculum evaluation
Undertaking a curriculum evaluation involves:
• Clarifying the purpose of the evaluation. A number of different aims are mentioned earlier in the chapter.
• Deciding the approach to be adopted. The ‘ten questions’ model and the Kirkpatrick four levels approach are described.
• Engaging the stakeholders. These include the curriculum committee, the teachers, the students, other health professionals, future employers and the public.
• Planning the methods to be used. These almost certainly will include qualitative and quantitative methods.
• Allocating responsibilities for the implementation of the evaluation and the necessary resources to carry it out.
• Planning how the results will be communicated and how action will be taken as a result of the evaluation.
Reflect and react
1. Are you aware of the stated mission for your medical school or educational body? How successfully is this achieved?
2. What purposes described for curriculum evaluation are relevant to your situation?
3. What steps are taken to evaluate your curriculum and are all the stakeholders involved? Is effective use made of the results of the evaluation?
Boelen C., Woollard B. Social accountability and accreditation: a new frontier for educational institutions. Med. Educ.. 2009;43:887-894.
Coles C.R., Grant J.G. Curriculum evaluation in medical and healthcare education. Med. Educ.. 1985;19:405-422.
A description of what is meant by curriculum evaluation in medical education.
Harden R.M. Ten questions to ask when planning a course or curriculum. Med. Educ.. 1986;20:356-365.
A description of the ten questions to be asked when planning a curriculum.
Berk R.A. Thirteen Strategies to Measure College Teaching. A Consumer’s Guide to Rating Scale Construction, Assessment, and Decision Making for Faculty, Administrators, and Clinicians. Sterling, Virginia: Stylus Publishing; 2006.
An amusingly written account of how to assess the effectiveness of teaching.
Kelly A.V. The Curriculum: Theory and Practice, fifth ed. London: SAGE Publications; 2004.
A useful account of the purpose, approaches and theory underpinning curriculum evaluation.
Kirkpatrick D., Kirkpatrick J. Evaluating Training Programs: The Four Levels, third ed. San Francisco: Berrett-Koehler; 2006.
A description of the Kirkpatrick four levels of evaluation.
Parlett M., Hamilton D. Evaluation as illumination. In: Tawney D., editor. Curriculum Evaluation Today: Trends and Implications. Macmillan, London: Schools Council Research Studies, 1975.
