Written and computer-based assessment

Published on 01/06/2015 by admin

Filed under Internal Medicine

Last modified 22/04/2025

Print this page

rate 1 star rate 2 star rate 3 star rate 4 star rate 5 star
Your rating: none, Average: 0 (0 votes)

This article have been viewed 1589 times

29 Written and computer-based assessment

Written questions have a role to play alongside other assessment methods.

Written assessment has a role to play

Written approaches to assessment are well established and have been widely used to assess learners’ competence in all spheres of education. Despite the greater emphasis being placed on performance assessment, written approaches still have an important role to play. The use of written examinations has come under scrutiny with regard to their match with the expected learning outcomes and their impact on the learners’ behaviour. Alternative written assessment methods have been developed as a response to these challenges and increasingly computers have replaced paper and pencil techniques for the delivery of the assessment.

image

Fig 29.1 A written examination.

The elements in written assessment

It is helpful to think of written assessment in terms of the stimulus or task that necessitates a learner’s response, the response expected of the learner, the assessment of the student’s response and the media or technology used.

The stimulus

The stimulus for the learner’s response may be:

• a short statement or question, e.g. ‘In which of the following pathologies does a patient typically present with weight loss and an increased appetite?’ or ‘List the 3 options in the management of a patient with hyperthyroidism.’

• a statement or question with accompanying diagrams or charts, e.g. ‘In the diagram which structure is labelled ‘A’?’

• a short clinical scenario with a patient’s presentation followed by a question, e.g. ‘Mrs Wilkie, a 35-year-old waitress complaining of tiredness and nervousness… ’

• a more extended patient management problem that develops over a period of time.

The student’s response

The response expected of the learner can be categorised into:

• Constructed response questions where the student has to write a long or short narrative in response to the stimulus. These include essays, short essay questions and short answer questions.

• Selected response questions where the student has to make a selection from a range of options provided. These include multiple choice questions (MCQs), for example the one best answer or multiple true/false questions and extended matching items.

The assessment of the student’s response

The learner’s response may be scored:

• Automatically correct or incorrect, as in an MCQ where there is an agreed correct answer. This also applies to short answer questions where the expected answer is limited to a few words. In this case, agreement has to be reached with regard to alternative wording and spelling that is acceptable.

• In a constructed response question, by an examiner based on a holistic impression of the student’s response, or with an assigned structured marking scheme.

• In the context of the answers to the questions provided by members of a panel of experts. This is the strategy used in the script concordance test as described below.

Standard setting

The subject of standard setting was introduced in the previous chapter. A range of methods have been used in a written examination to determine the mark above which students will pass the examination and below which they will fail (Bandaranayake 2008).

It is not uncommon for a pass mark for an MCQ paper to be set at 60–70%. This implies, however, that students need to know only two thirds of the area covered and that it does not matter which third they do not know. Although it is not usual practice, consideration needs to be given to an examination or part of an examination that assesses essential core knowledge, with a pass mark of more than 90% expected for that part of the examination.

Types of written assessment

Essay question

While the essay remains a common assessment tool in many fields, it is less commonly used in medicine. Provided the question or stimulus is appropriate, the essay can test:

• the learner’s general understanding of the topic

• higher level skills including synthesis, organisation of information, analysis, problem solving and evaluation

• written communication skills, a competence often not tested by other written assessment methods although portfolio assessment may be used for this purpose

• aspects relating to attitudes and medical ethics.

Essay questions do have a number of disadvantages as an assessment tool:

• The content area sampled is small compared to an MCQ paper.

• The scoring of the questions is subjective and time consuming.

Short essay question (SEQs)

Short essay questions are designed to sample a wider range of content than the essay question. A student may have to answer twelve 10-minute short essay questions instead of four 30-minute or two 1-hour essay questions. The SEQ has many of the advantages and disadvantages of the essay question.

Short answer questions (SAQs)

In a short answer question, rather than selecting from a list of choices as in a MCQ, the student is expected to respond to the question in one, two or a limited number of words. The format has the advantages of an MCQ in that a wide range of content can be sampled and has the added advantage that it does not simply test recognition of the correct answer from a list of options. It is also easier to set questions that test core or basic knowledge without having to cue the learner with the responses on a list. MCQs offer the advantage that they can be automatically marked but SAQs share this advantage and it is possible to mark an SAQ using the appropriate computer program, although agreement needs to be reached with regard to the answers that are acceptable. SAQs are now routinely used in examinations such as the progress test at the medical school in Dundee and merit wider consideration and adoption as an alternative to the MCQ.

Multiple choice questions (MCQs)

Multiple choice questions can sample objectively a wide range of a student’s knowledge and understanding. The downside for the examiner is not the marking of the responses but the setting of the questions. Several question banks have been developed in medicine and these allow schools to share questions.

Many MCQ examinations test mainly recall of knowledge rather than in-depth understanding or application. This has had a detrimental effect on how students study. To counteract this, there has been a move to introduce greater authenticity to the MCQ through the use of clinical scenarios in the stem or stimulus.

Many formats have been described but two approaches have dominated:

• Single best option where the learner is required to select the best response from four or five alternatives.

• Multiple true/false questions where the student has to categorise as true or false each of five statements relating to the stem. Guessing is a more important consideration with this type of question and as a result the marking scheme may be more complicated. There has been a move away from using this type of question although it does test a wider range of knowledge and the questions are easier to set without the need for distracters as in the single best option type of question.

Extending matching questions (EMQs)

The EMQ is a type of written question in the selected response category that provides an alternative to the standard MCQ. The EMQ consists of a list of around 20 options relating to a theme or topic. This may be a list of drugs, diseases, laboratory investigations, symptoms, explanations or pathologies. Following the lead-in stem or stimulus, usually in the form of a patient scenario, the student has to select the most appropriate answer from the list of options. There may be one, two or more questions using the same list of options. An advantage is that, with the extended list of options from which the student has to select the answer, the effect of cuing as found in the MCQ is minimised. The questions are less time consuming to produce as several scenarios can use the same list of options. As with MCQs a computer can be used to score the answers.

Modified essay questions (MEQs)

The MEQ is a sequence of questions based on a patient scenario that develops over a period of time. After the first question is answered, further information is provided about the patient and this is followed by another question. The typical MEQ may have six or more questions and both SAQs and MCQs may be used. There are difficulties with scoring this format and it is now rarely used.

Script concordance test (SCT)

This is a relatively new form of written assessment. Students are given a brief patient scenario and asked to make judgements regarding diagnostic possibilities or management options. To allow decisions to be made that reflect medical practice, the student is given sufficient clinical detail but a certain amount of uncertainty, imprecision or incompleteness is deliberately built into each case in order to simulate real-life clinical situations. The scoring is based on the amount of agreement between the student and a panel of experts. The approach may be useful, particularly in the assessment of a student’s clinical reasoning.

The technology

Questions can be posed, responses recorded and examinations scored either on paper or on a computer. Computers are now widely used for MCQs: the students answer on a computer and their responses are automatically scored. There is some interest in adaptive testing where the questions presented to the students on the computer are determined by their responses to earlier questions. A wrong answer to a question may generate additional related questions that allow the student’s understanding or lack of understanding of the area to be explored further. This may increase the reliability of the examination by increasing the number of questions used in the areas where the student’s understanding is in doubt. This approach is not in routine use at the moment.

Reflect and react

1. Given the learning outcomes to be tested and the resources available, which format of written test is appropriate to your setting?

2. To what extent does a written examination in your course test more than factual recall?

3. Consider how you can best provide feedback to students about their performance in a written examination. Simply providing a mark, for example 62%, is not sufficient.

Exploring further

If you have a few hours

Amin Z., Boulet J.R., Cook D.A., et al. Technology-enabled assessment of health professions education: consensus statement and recommendations from the Ottawa 2010 Conference. Med. Teach.. 2011;33:364-369.

This paper highlights the changing nature of ICT in assessment and the challenges that need to be addressed when technology is incorporated into assessment.

Bandaranayake R.C. Setting and maintaining standards in multiple choice examinations. AMEE Guide No. 37. Med. Teach.. 2008;30:836-845.

An account of the advantages and disadvantages of the more commonly used methods of setting standards in MCQ examinations.

Case S.M., Swanson D.B. Constructing Written Test Questions for the Basic and Clinical Sciences, third ed. (revised). Philadelphia: National Board of Medical Examiners; 2002.

A classic description of how to prepare MCQs..

If you have more time

Schuwirth L.W.T., Van der Vleuten C.P.M. Written assessment. In Cantillon P., Wood D., editors: ABC of Learning and Teaching in Medicine, second ed., Chichester: John Wiley and Sons, 2010. (Chapter 9)

A useful overview of the use of written examinations in medicine.

Related posts:

What is a good teacher? Ten questions to ask when planning a curriculum Building learning around problems and clinical presentations Default ThumbnailAssessment for admission to medicine and postgraduate training Simulation of the clinical experience Making the apprenticeship model and work-based learning more effective