Who should assess the student?

One reason why assessment is complex and the teacher’s responsibilities may be unclear is that there is a range of stakeholders involved. These include:

In medical schools in the UK, the assessment process is overseen by the General Medical Council (GMC) and the implementation is the responsibility of each medical school. Teachers from other schools serve as external examiners and participate in the development of the school’s examinations, their implementation and pass/fail decisions. In contrast, in North America and in some other countries there is a national examination which students are required to pass. Each approach has merits. A national examination, while setting national standards, may stifle innovation in individual medical schools (Harden 2009).

In medical practice, doctors have to take responsibility for assessing their own performance and keeping themselves up-to-date. Not all doctors have the necessary skills or recognise the importance of the responsibility. Students must be prepared for this as part of their undergraduate education with reinforcement throughout their postgraduate training. Problems with unreliability in self-assessment are well recognised and there may also be problems with how students react to the assessment of their own competence. On one occasion, we asked students to mark their own examination paper against a model answer. Some found the procedure so traumatic that they were unable to complete the process and to our surprise required counselling as a result. Training students to become doctors who are inquirers into their own competence, and who are comfortable with this, should be a learning outcome of the curriculum.

Increasing attention is being paid to peer assessment, and the evaluation of students by their peers against certain learning outcomes has become part of some institutions’ assessment strategy. This is particularly valuable in the assessment of attitudes where often the student body has a better understanding of individual students’ strengths and weaknesses than the teachers.

Why assess the student?

At an early stage in planning an assessment programme it is important to consider the purpose of the assessment. An assessment designed to certify a student’s competence to practise as a trainee doctor will be different from the assessment method used to review a student’s progress and provide feedback during course work. Traditionally, assessment has been described as either ‘formative’, where the main aim is to provide the learners with feedback about their progress, or ‘summative’ where the aim is to determine whether the learners have achieved the course objectives. This distinction has become blurred with the recognition that summative assessment can also be used to provide feedback to the learner and summative decisions may be based on evidence collected during the training programme.

The purposes that can be served by assessment include:

What should be assessed?

A key feature of outcome-based education, as discussed in Section 2, is that assessment is matched closely with the specified learning outcomes. This is referred to as competency-based assessment. What we choose to assess in the education programme demonstrates what we value and many problems encountered with assessment arise from an inadequate consideration of what is being assessed. Assessment drives learning as we have described above. In the absence of a set of learning outcomes what is assessed becomes, for the students, the course objectives. In the past the emphasis in assessment was on the knowledge domain with less attention paid to skills and attitudes. There were a number of reasons for this. Mastery of knowledge was traditionally regarded as of greater importance than the development of attitudes. Knowledge was also easier to assess than other domains and there was a natural tendency to assess what was easy to assess and to shy away from areas where assessment was contentious or difficult. Written assessments, including MCQ papers which tested the knowledge domain, dominated assessment practice. However, someone who can answer correctly a set of MCQs is not necessarily a good doctor and there has been a move to assess in the student or doctor more complex achievement, higher order thinking, clinical skills, attitudes and professionalism.

The introduction of the OSCE stimulated the assessment of psychomotor and other performance related skills, and more recently the adoption of portfolio assessment and multi-source feedback has recognised the importance of the assessment of independent learning and self-assessment skills, attitudes and professionalism.

It is important that, with the many changes advocated in medical education and the different expectations we have of our students in today’s curricula, assessment does not lag behind. What we assess must closely match what we expect students to learn.

How should the student be assessed?

A wide range of tools or instruments are now available that can be used to assess the student’s competence (Fig. 28.3). Some of these are described in the chapters that follow. Just as important as the assessment tool selected, if not more so, is the way in which the tool is employed. A good tool badly used will yield inappropriate or misleading results. We now have a better understanding of what makes a good assessment:

• The method should be reliable and consistent. This relates to the certainty with which a decision can be made about the student’s performance on the basis of the test results. For a reliable assessment the measurement instrument must be relatively stable. It would not be good practice, for example, to use an elastic tape measure to measure length. The measurement would not be reliable.

Fig 28.3 A range of assessment tools are available.

A factor that led to the widespread use of MCQs was that their high reliability could be demonstrated. Problems with reliability associated with tests of clinical competence were highlighted when we found that examiners watching the same clinical performance awarded different scores to the examinees and, watching a video of the students’ performance some weeks later, they were not consistent with the marks they had awarded. The problem of reliability was a factor which stimulated us to develop the OSCE.

• The method should be valid. In other words the assessment method should measure the learning outcomes intended (Fig. 28.4). The test should be an ‘honest’ one, testing what it purports to measure. There may be a trade-off at times between reliability and validity. This is illustrated in the classic story of the drunk man who was seen at night looking under a street light for his car keys which he had dropped. When asked why he was looking there when he had dropped them a short distance away, he replied that it was easier to see what was on the ground under the light. His search strategy, although having merit, was not valid in his situation.

Fig 28.4 Tests should be valid as well as reliable.

Reliability has been emphasised in the past at the expense of validity. What is needed is a test that is both valid and reliable. A test may be reliable but it is of no value if it does not measure what we want to measure. Unfortunately, the more simple a test, the more likely it is to be reliable while at the same time the less likely it is to be valid. Medicine by its nature is a complex subject and assessment of professionalism or communication skills is by necessity complex if it is to be valid.

• The method should be feasible in terms of the resources available and the number of students to be examined. The assessment scheme should not be overly complex and should be capable of being implemented by the teacher in routine practice.

• The assessment should have a positive impact on the student’s learning. When we first introduced the OSCE in the final examination and included some stations with family physicians as examiners, we found that students spent less time in the library revising their theoretical knowledge and more time learning on the wards and in the community setting. This was in line with the aims of the curriculum.

When should the student be assessed?

Traditionally learners were assessed on their mastery of the subject in a set examination at the end of the course. Increasing emphasis has been placed on collecting evidence about the learner’s achievement of the expected learning outcomes during their training or course of study. There are a number of reasons for this. Without the time constraints of a final examination, a much wider sample of the students’ performance can be assessed, increasing the reliability of the examination. In addition, the assessment may be more valid in that the assessment tools that can be adopted during the course may assess learning outcomes difficult to assess in an end-of-course final examination. An important additional benefit of in-course assessment is that it provides feedback to the student and teacher and allows time for remediation.

Less frequently undertaken is the assessment of the student at the beginning of the training programme. A number of years ago, we asked third-year students at the beginning of their course in endocrinology to complete the end-of-course examination on day one of the course. The results were surprising. Some students scored less than 10% while other students scored almost 50%, the pass mark for the examination. This provided important evidence as to the need for a course in endocrinology which could be tailored to suit the abilities of the different students. Independent learning modules were developed which were used successfully to replace some lectures on the topic. It could be argued that our first student assessment is when we select students to enter medical studies, or select graduates for a postgraduate training programme (see Chapter 32).

Where should the student be assessed?

Students have been assessed traditionally using written papers in an examination hall environment, or with formal clinical examinations in the hospital ward. With moves towards greater authenticity in assessment, and with assessment seen more as a continuous process, increased attention is being focused on assessment in the work place and in the wide range of contexts where teaching takes place, whether in the hospital or community environment. This is discussed further in Chapter 30.

React and reflect