Over the past decade, medical schools have made new efforts to provide accurate, reliable, and timely assessments of the competence of trainees. All methods of assessment have strengths and intrinsic flaws. Written examination questions forms an important item in knowledge assessment and are typically classified according to whether they are open-ended or multiple choices. At present questions are prepared casually just before the examination and are not put through any quality check to correct any possible mistakes or ambiguity in the questions. This may lead to confusion or wrong understanding of the questions by the students which will be reflected in their answers as well. Question Vetting is the process when an expert person examines or evaluates and edits questions to make it free of any mistakes. The importance of vetting and re-vetting of examination questions cannot be overstated in the present scenario of medical education. There is a prescribed structure and protocol of question setting and vetting for examinations in medical education programmes. The areas that need to be examined during question vetting are for technical accuracy, content level and language aspects. The use of the prescribed structure and protocol of question vetting ensures a consistently high standard of question presentation during examinations which will help the students to understand the questions better and to answer them correctly. This article is intended to highlight the importance of introducing question vetting in our medical education scenario in order to improve the quality of assessment of medical students.
Introduction
The teaching faculty in medical colleges is entrusted with the responsibility of carrying out the academic activities of undergraduate, postgraduate and paramedical students, apart from their regular clinical and patient care services. This is a balancing act by the medical college teachers to meet the demands from both the responsibilities. The academic activity mainly consists of teaching and training the students based on the subjects handled by each department. As per the curriculum requirements, evaluation is to be done to assess the student’s level of knowledge and skills acquired during the study period. The student evaluation is done based on the three domains of the learning process namely cognitive, affective and psychomotor domains.
Assessment is usually done at two levels: internal, periodic assessment (formative assessment) by the department and end of posting / final examination (summative assessment) conducted by the University. For these examinations, question paper setting or preparation is an important task which is usually entrusted to the concerned middle level or junior teaching faculty who is responsible for the batch of students as course coordinator. Usually the questions are prepared casually just before the examination and are not put through any quality check to correct any possible mistakes or ambiguity in the questions format. This may lead to confusion or wrong understanding of the questions by the students which will be reflected in their answers as well.
In order to overcome these difficulties, rapid changes have been happening in assessment of medical students which have resulted in the development and implementation of several new methods of assessment which focused on clinical skills, communication skills, procedural skills, and professionalism [1]. The medical knowledge and clinical skills of medical students are often assessed using different forms of assessments.
Different forms of assessment now followed in most medical institutions includes.
The written examination
- free response (essay, short answer questions)
- fixed response (objective type, e.g. MCQs)
The oral examination (viva-voce)
The practical examination
The clinical examination
Written examination questions forms an important item in knowledge assessment and are typically classified according to whether they are open-ended or multiple choices, which are graded by hand [2]. The practical examinations typically required the student to go to a patient’s bedside, gather information, and then present a diagnosis and treatment plan to assessors who asked questions and made judgments about the performance [1]. In this perspective the written examination plays a major role in assessing the knowledge level of the students.
To ensure and maintain high quality standards in medical schools, the assessment methods have become challenging tasks. From this viewpoint, vetting of assessment tools is a crucial process. Question vetting is the process of reviewing and evaluating question items according to specified criteria with the intention to detect flaws and to edit them accordingly to improve their quality [3]. This step is important to sustain the validity of test items [questions] and avoid or minimize the threats. Flawed or badly written items are the major threats. They are frequently encountered in any form of tests. Most questions, even those produced by experienced item writers are still flawed in some ways. So, once an item is constructed, it should undergo a critical review by a review (or vetting) committee. The aim is to treat test items appropriately by removing flaws and making them as clear and as understandable as possible [3].
The role of such a review (or vetting) committee was described many years ago and is being practiced by many medical institutions globally. An item/question review process includes several activities. They include.
Item/question-writing principles review; in which items must be ensured to adhere to identified item-writing guidelines,
cognitive demand review; in which the cognitive level of items are assessed,
content review; in which the content of each item is matched with what is intended to be measured (testing blueprint),
editorial review; where items are checked for any errors in spelling, grammar, and punctuation,
sensitivity and fairness review; through which personally, culturally and ethnically offensive terms are removed and substituted by suitable ones,
answer key check; where each item is checked for accuracy of the correct answer,
answer justification; this is done by listening to examinees’ view for their choices during test and accepting their choices if clearly justified,
think-aloud; in which a comprehensive discussion is done with test takers to identify relevant information about quality of test items, their contents and cognitive levels being measured.
There is a prescribed structure and protocol of question setting and vetting for examinations in both undergraduate and postgraduate medical programmes. [4] This prescribed protocol differs for internal (formative) assessments and final professional (summative) examinations.
Internal [Formative] Assessment
Often before the End-of-Posting examinations, the coordinators will call for questions from relevant lecturers, giving information regarding subject area/discipline, type of questions (whether MCQ Multiple True/False, MCQ-Best Answer Type, Modified Essay Question [MEQ], (Objective Structured Clinical/Practical Examination) OSPE or OSCE) and number of questions for each subject area and type of questions. This information is sometimes distributed in the form of a matrix/table. A date is fixed for submission of questions to the coordinator and another date for vetting of the questions. All lecturers involved in the construction of questions are invited for the vetting session. The number of vetted questions in each subject area/discipline selected for the examination will be proportional to its lecture/contact hours.
The coordinator usually chairs the vetting session and various aspects of the questions will be paid attention to. The vetted questions are then sent to the Examination Unit for formatting into an examination paper draft. This draft will then be checked for technical and language compliancy preferably by a representative from the Medical Education Unit. The coordinator will check the final draft before the Examination Unit processes it into examination papers.
Final Professional (Summative) Assessment
Separate question vetting committees for different forms of question papers will be appointed by the Controller of the University for the Professional Examinations. The members of the committees include representatives from various disciplines. The Controller calls for questions from all relevant senior faculty involved in the particular phase, giving information regarding subject area/discipline, type of questions, etc. A date can be fixed for submission of questions to the controller. The Chairpersons of the vetting committees will schedule date(s) for vetting these questions.
All members of the committees and faculty involved in the submission of questions are invited for the vetting session (s). The Chairperson of the vetting committee chairs the session and various aspects of the questions will be paid attention to. Questions that are to be revised are referred back to the original authors for endorsement of the revisions. The vetted questions are submitted back to the coordinators, who will select and arrange the questions in a specific order. These questions are sent to the Examination Unit, for formatting into an examination paper draft. This draft will be checked for its technical and language compliancy by the Controller before the Examination Unit processes it into examination papers.
Question Vetting Process
Many variables are given careful attention during the vetting process to ensure that the examination questions are correct, fair, valid and reliable. These variables can be divided into the following three aspects related to the technical, content and language accuracy. Compliance of the questions with these three aspects ensure both content and construct validity, besides ensuring that the examination measures understanding of prescribed content/process and not understanding of the question itself.
Aspects Of Questions Taken Note Of During Vetting [
4]
Technical Aspects
There should be technical accuracy, that is, the question complies with the recommended format and requirements of its type. The task or demands of question is appropriate with the provision of enough time and the marks awarded for each question or the total marks allotted for each question type is as required.
MCQs, there are five options for the correct answer to choose from, A, B, C, D and E
For MEQs the space and time allotted for the answers to particular questions is appropriate to task with appropriate length of MEQ text.
Content Aspects
The content aspect looks into the relevance of the question to subject area/discipline covered. The distribution of questions should be in relation to the subject area/discipline and according to specified rubrics, such as
Information in the question is valid, appropriate and current
Content being tested is within the syllabus, is deemed important, does not contravene any regulations or laws and is not culturally or racially sensitive
The difficulty level of each question commensurate with the ability level of the group to be tested and there should be a range of difficulty, from easy to difficult questions
There should be a proper mix of skills tested, from factual to understanding, prediction, inference or implication and there should be no cues in the text or stem towards the right answer
There should be no overlap of exact content being tested in the questions within and between question types.
For Multiple Choice Questions (MCQs)
Each MCQ should be integrated rather than single subject/discipline oriented
The stem is clear, short and precise and there is a definite, explicit and singular question/problem in the stem
Questions with negative stems, for example, NOT, EXCEPT etc. are to be avoided or its number kept to a minimum and when used, the negative word is underlined/capitalized/bold
All options are of similar length and have a reasonable chance of being selected as answers
All options are consistent in terms of aspect or perspective, that is, distracters are of the same class as the key
All options are approximates rather than opposites and options which are mutually exclusive are avoided
Options which are synonymous with, include or overlap others are avoided and are presented in some logical order, for example, chronologically, most to least, least to most, etc.
Distracters relate to some superficial logic or a popular misconception /belief and have a ‘ring of truth’
There is only one unequivocal key or correct/best answer and the key should not be too obvious (test taker does not need to read the text/stem further or read the other options)
Vary the location of the answer/key as random a basis as possible
Vague qualitative modifiers such as “many”, “large”, “much”, “small”, “old” and “important”, etc. are avoided
Cues to the key such as “always”, “never”, “all”, etc. are avoided and there should be no double negatives in both stems and/or options.
For Modified Essay Questions (MEQs)
MEQ are broken down into scenarios containing facts for thinking from multidisciplinary approach
The sequence of scenarios should depict development of different stages of problem or issue
Each scenario should provides relevant new information and requires clinical appraisal, problem solving or decision making.
Language Aspects
Standardization of spelling norms should be either British/American orthography
Terminology used is the current version and punctuation marks are correct and proper
Language used is simple, clear and direct and is not colloquial or oral oriented
Consistent use of tenses
Appropriate ‘action’ verbs are used, for example, indicate, show, illustrate, give, etc.
No spelling errors, no typography errors and no grammar mistakes.
Amin Z et al., describes that, traditionally, assessment is viewed as a “necessary evil” in the curriculum - an act that most medical teachers carry out by compulsion. If it is properly planned and implemented, has a powerful positive steering effect on learning and it conveys what the teachers value as important and acts as the most cogent motivator of student learning [5]. Regular question vetting helps a lot in improving the assessment process.
Conclusion
The assessment methods have become a challenging task for teaching faculty in order to maintain high quality standards in medical schools. Written and practical examinations are an important part of knowledge and skill assessment of any student. So properly prepared question papers / items for evaluation is a must to guide the students to understand the questions properly and help them to answer appropriately. Question vetting plays a major role in this process. Based on the highlighted points described in this article, it can be seen that question vetting is a major part of quality assurance in medical education. However, it consumes a substantial amount of time and effort. The use of the prescribed structure and protocol of question setting and vetting by the teaching faculty ensures a consistently high standard of question presentation during examinations. This will be beneficial for different strata of students to understand the questions correctly and properly and helps them to give their best answers.
[1]. Norcini JJ, McKinley DW, Assessment methods in medical education Teaching and Teacher Education 2007 23:239-50. [Google Scholar]
[2]. Epstein RM, Assessment in Medical Education N Engl J Med 2007 356:387-96. [Google Scholar]
[3]. Wadi M, Question vetting: theory and practice Education In Medicine Journal 2012 4(1)DOI:10.5959/eimj.v4i1.29 [Google Scholar]
[4]. Hoon O P, The process and guidelines for question vetting at the Faculty of Medicine and Health Sciences, UNIMAS South East Asian Journal of Medical Education 2009 3(2):74-76. [Google Scholar]
[5]. Amin Z, Seng C Y, Eng K H, Practical Guide to Medical Student Assessment, Section 1, Assessment in Medical Education: An Overview, Chapter 1 2006 SingaporeWorld Scientific Publishing Co. Pte Ltd:3-7. [Google Scholar]