Introduction of Structured Oral Examination as A Novel Assessment tool to First Year Medical Students in Physiology
Mrunal R. Shenwai1, Krishnakant B. Patil2
1 Assistant Professor, Department of Physiology, Smt. Kashibai Navale Medical College & General Hospital, Pune, Maharshtra, India.
2 Professor & Head, Department of Physiology and Deputy Dean, Smt. Kashibai Navale Medical College & General Hospital, Pune, Maharashtra, India.
NAME, ADDRESS, E-MAIL ID OF THE CORRESPONDING AUTHOR: Dr. Mrunal R. Shenwai, G/701, Kumar Padmaja, Near Ashish Garden, Opposite Mahesh Vidyalaya, Kothrud, Pune-411029, Maharashtra, India.
Phone: 91-9923444740,
E-mail: drpmrunal@yahoo.com
Background: The traditional viva examination which is still predominantly used in most of the medical institutions as one of the assessment methods has some demerits like lack of standardization, objectivity & reliability. The process involves many faculty members from respective departments. Hence there can be variations in the time allotted to each student, number of questions asked, and difficulty level of the questions. These can be resolved by structuring the oral examination to make it a better assessment tool. The present study was conducted to introduce structured oral examination (SOE) as a novel assessment tool to first year M.B.B.S. students in Physiology and evaluating the process by taking feedback from the students and faculty.
Material and Methods: All the subjects [first year M.B.B.S. students (n=100)] were initially assessed by traditional viva. Feedback in the form of a questionnaire was collected from the students. Questionnaire included various questions based on Likert scale and numerical value for each response was decided. Topic for the structured oral examination was decided by the faculty members. Students were intimated about the viva process well in advance. A checklist of questions to be asked in the structured viva was prepared and their probable/most correct answers were discussed with the faculty in advance. Feedback was again collected from the students after the viva session. Statistical analysis of the questionnaire was done using ‘paired t–test’.
Results: Questionnaire analysis depicted that students were overall satisfied with the structured viva and felt it better than the traditional viva. Statistically significant differences (p=0.0001) were observed in terms of uniformity of questions asked, syllabus coverage, reduction in the anxiety levels etc. between these two assessment methods. Faculty members also expressed that structured oral examinations are better in terms of reducing bias, minimising luck factor and uniformity of questions makes SOE a fair assessment tool.
Conclusion: Structured oral examination can be a better assessment tool and with some modifications in blueprinting it will be acceptable to the students as well as faculty.
Introduction
Oral examinations are being used as a mode of assessment of medical students for years. Traditional oral examinations consist of a dialogue or discussion with the examiner who asks questions to which candidate must reply. Orals give the examiner the unique opportunity to explore students’ depth of knowledge as well as their ability to express it in a precise manner.They are used for their flexibility and potential for testing higher cognitive skills [1]. Oral examinations are appealing because of their high face validity, their flexibility and the possibility that they measure aspects of clinical competence that are perhaps not tapped in written examinations [2]. In some British schools, short orals or vivas have been used in the past to test candidates who were borderline on written examinations [3]. Orals provide the students an incentive to explore topics, give them a chance to interact one on one with examiners and get excited about learning [4].
Despite this, there are some challenges often faced in the traditional viva examinations. The atmosphere during traditional oral examination is often threatening and at times the dialogue takes the shape more of a confrontation than discussion.The subjectivity in the traditional viva can at times be intimidating to the students. Questions asked vary from examiner to examiner and may not cover the syllabus. Most of the times questions are of recall type rather than those which test the analytical & problem solving ability of the students. As there is no uniformity of questions and their difficulty level, the assessment of the students based on these questions may not be fair. Besides this, there can be some personal biases and carry over effect (performance of the previous student affecting score of the next) which are more likely to occur in a traditional oral examination. As studied by Holloway et al., there is an inverse relationship between anxiety and performance in the oral examinations [5]. A significant part of the error in oral performance ratings is due to the tendency for some evaluators to be lenient and others to be stringent in their assignment of ratings. Correcting for such errors would change the pass/fail decisions for about 6% of the examinees [6]. Marks awarded to candidates by different examiners indicates low reliability between ratings and agreement between examiners is often poor [7].
All these problems may be overcome by replacing the traditional viva by structured oral examination (SOE). This can be done by pre deciding the syllabus to be covered, competencies to be measured and preparing a blueprint/checklist of questions to be asked in the viva. Although the implementation process is onerous, but once in place, it can become an efficient assessment tool [8]. In structured oral examination (SOE), as the Question, answers and scores are noted by the examiners for each candidate, a feedback can be given to them later, where they scored and where they did not do well.
Structured oral examination being a novel concept with very few studies done especially in the Physiology subject, the present study was undertaken with an aim of making oral examination/ viva structured one for some selected topics in Physiology and its introduction as an assessment tool to Ist year M.B.B.S. students.
Material and Methods
Participants for the research project were first Year Medical students. (n=100). Study was carried out in Department. of Physiology, Smt. Kashibai Navale Medical College & GH, Pune, Maharashtra. Initially all the students were assessed by traditional viva as a part of their internal assessment examination. Four faculty members separately conducted the viva without any checklist. Feedback in the form of a Pre-test questionnaire was collected from the students. Questionnaire included various questions (based on likert scale) to get their views on the overall viva process, anxiety levels, student friendly environment, uniformity of questions, coverage of syllabus, language barrier, carry-over effect, any sort of bias they experienced etc. Feedback from the faculty members involved in the viva process was also taken.
Topic for the structured oral examination was then decided by the faculty members [Sensory division of CNS] Students were intimated about the viva process well in advance. The questions were constructed by a group of faculty with inputs from all those who have participated in the teaching process. A number of questions from each topic covering the content area of varying difficultly among the learning objectives were jotted down. About 10-15 questions from each topic were shortlisted and final template /checklist of questions to be asked in the viva was prepared. The probable/ most correct answers for these questions were also discussed. The process involved all the faculty members from the department (Assistant professors & above). Checklist was prepared by taking into consideration the ‘must know’, ‘nice to know’ & ‘desirable to know’ aspects of their curriculum as per Maharashtra University of Health Sciences, Nashik (MUHS). Some of these questions were of recall type and some questions tested analytical & reasoning power of the students. The questions were arranged in an ascending order according to their difficulty level. All the faculty members involved in the viva exam were provided with the checklist in advance, with written instructions regarding the content area to be covered and the nature of competence to be measured.
Next part of the project was introduction of this structured pattern of viva to Ist year students who have previously undergone traditional viva. It was conducted as a separate practice viva session. In the Structured Viva, all the faculty members conducted viva with the checklist. Students who finished their viva were strictly kept separate from rest of the students. Total no of students who attended the viva was ninety three. Post test Feedback was collected from the students & the faculty regarding their experience about structured oral examination.
The study was given ethical exemption by the institutional ethical committee as it was an educational project and vivas are a routine part of students’ assessment.
Results
Statistical analysis of the feedback questionnaire was made applying ‘paired t–test’ [Table/Fig-1]. Each response of the questionnaire was assigned a numerical value on Likert scale and mean/average was calculated.
Statistical analysis of the questionnaire
| Question No. | Traditional viva | SOE | t value | p–value |
|---|
| Mean | S.D. | Mean | S.D. |
|---|
| 1. (Overall experience) | 3.484 | 0.601 | 4.043 | 0.7359 | 5.612 | 0.0001, HS |
| 2. (Student friendly environ.) | 2.796 | 0.4054 | 3.344 | 0.5613 | 7.733 | 0.0001, HS |
| 3. (Satisfaction level) | 2.581 | 0.558 | 3.215 | 0.4847 | 8.693 | 0.0001, HS |
| 4. (Uniformity of ques.) | 2.376 | 0.5882 | 3.581 | 0.7873 | 11.442 | 0.0001, HS |
| 5. (Anxiety levels) | 3.269 | 0.6104 | 2.667 | 0.7273 | 5.676 | 0.0001, HS |
| 6. (Syllabus coverage) | 2.613 | 0.5323 | 3.398 | 0.5737 | 9.559 | 0.0001, HS |
| 7. (Carry over effect) | 2.892 | 0.7292 | 2.452 | 0.6171 | 4.478 | 0.0001, HS |
| 8. (Gender bias) | 1.71 | 0.7006 | 1.624 | 0.5089 | 1.051 | 0.296, NS |
| 9. (Language barrier) | 2.032 | 0.8902 | 1.871 | 0.7693 | 1.296 | 0.198, NS |
| 10. (Time allotted) | 2.892 | 0.5798 | 2.957 | 0.658 | 0.705 | 0.482, NS |
Analysis of the questionnaire showed significant differences in students’ perceptions about traditional and structured oral examinations. Students felt that the overall viva session in structured oral examination was better than the traditional viva. The atmosphere was less threatening and more students friendly during structured oral examination. Due to uniformity of questions to all the students ‘luck factor’ or ‘carry over effect’ was minimised. They expressed that the questions designed were good and coverage of the syllabus was better in structured oral examination as compared to traditional viva. There was less anxiety amongst the students during structured oral examination. Regarding gender bias majority of students disagreed with any such bias during both the viva sessions. Time allocated to each student was also equal in both the sessions. For % distribution of students’ responses to various questions (based on likert scale) refer [Table/Fig-2 and 3].
% distribution of students’ responses to various questions (based on Likert scale) in the questionnaire
| N=100 | ITEM | Excellent 5 | Good 4 | Average 3 | Poor 2 | Very poor 1 |
|---|
| | Trad. viva | SOE | Trad. viva | SOE | Trad. viva | SOE | Trad. viva | SOE | Trad. Viva | SOE |
|---|
| Q1 | Overall viva session (rate on the scale of 1-5) | 4% | 27.95% | 44% | 49.46% | 50% | 21.5% | 2% | 1.07% | 0 | 0% |
| ITEM | Highly satisfied 4 | Satisfied 3 | Unsatisfied 2 | Highly unsatisfied 1 | |
| | Trad. viva | SOE | Trad. viva | SOE | Trad. viva | SOE | Trad. viva | SOE | | |
| Q3 | Satisfaction with the viva process | 0 | 24.73 | 63 | 72.04 | 33 | 3.22 | 4 | 0 | | |
[Response scale: Strongly agree=4, Agree=3, Disagree=2, Strongly disagree=1
Highly satisfied=4, satisfied=3, Unsatisfied=2, Highly unsatisfied=1
Excellent=5, Good=4, Average=3, Poor=2, Very poor=1]
% distribution of students’ responses to various questions (based on Likert scale) in the questionnaire
| Q. No. | ITEM | Strongly agree 4 | Agree 3 | Disagree 2 | Strongly disagree 1 |
|---|
| (%) | Trad. Viva | SOE | Trad. viva | SOE | Trad. viva | SOE | Trad. viva | SOE |
|---|
| Q2 | Overall environment was student friendly. | 0 | 38.70 | 79 | 56.98 | 21 | 4.30 | 0 | 0 |
| Q4 | There was uniformity of questions to all the students | 0 | 52.68 | 35 | 40.86 | 59 | 6.45 | 6 | 0 |
| Q5 | Felt anxious /depressed about the questions. | 37 | 8.6 | 55 | 59.13 | 8 | 24.73 | 0 | 7.52 |
| Q6 | Viva Q. covered all the must know aspects of the curriculum. | 2 | 44.08 | 61 | 51.61 | 35 | 4.30 | 2 | 0 |
| Q7 | ‘Carry over effect’affected the viva performance. | 17 | 3.22 | 57 | 45.16 | 23 | 47.31 | 3 | 4.3 |
| Q8 | You felt that there was a gender bias. | 3 | 0 | 4 | 1.07 | 52 | 60.21 | 41 | 38.7 |
| Q9 | There was a language barrier in expressing their answers. | 8 | 1.07 | 15 | 20.43 | 47 | 43.01 | 30 | 35.48 |
| Q10 | Equal time was given to each student. | 11 | 16.12 | 67 | 66.66 | 22 | 13.97 | 0 | 3.22 |
Feedback from the faculty members is summarised below.
Traditional viva
Are easy to conduct (no ground work is needed).
The assessment method offers lot of flexibility for the examiner in order to judge the comprehensive knowledge of the students about the subject.
But there are some shortcomings in terms of uniformity of questions asked, their difficulty level, due to which this assessment may not be fair.
Structured oral examination (SOE)
Are better in terms of reducing bias and minimising luck factor. Anxiety/fear amongst the students is also less.
Uniformity of questions makes SOE a fair assessment tool. Concurrent marking ensures that the examiner does not make an overall assessment.
Monotony of asking same questions to all the students especially when the students’ population is larger can be avoided by creating more such sets of questions/checklists with properly adjusted difficulty level of the questions. This will maintain uniformity and objectivity besides imparting some flexibility to the examiner.
Discussion
The oral exams format enables the instructors to test the students on all five cognitive domains of Bloom’s taxonomy i.e. knowledge, comprehension, application, analysis, synthesis and evaluation [9]. In practice, they are used not as a substitute but as a complement to written exams [10].
Reliability of the viva examination is often questioned but according to Sharmila Torke et al, reasonable reliability has been demonstrated with structured, standardized orals using hand-picked examiners [2]. Their study also revealed that there was not much correlation between performance of the students in theory and in viva. Accordingly they modified their viva process so that it is being conducted only for borderline failed and distinction students. Another study done in an Indian setup in Anatomy showed that students liked the structured viva over the traditional viva exam as it minimised the luck factor and reduced bias [11].
Although there are examples of organisations restructuring their oral examinations to a structured oral examination format, recent research describing the students’ response to SOE has been limited especially compared to Objective Structured Clinical Examination (OSCE) formats [12]. These are multi-stationed clinical examinations which are effective in testing students’ ability to integrate the knowledge, skills and attitudes acquired during their pre-clinical and clinical training and experiences [10]. Most of the medical institutes in Maharashtra still conduct the viva by traditional method. As per M.U.H.S. curriculum in Physiology it carries weight age of 20 marks and is conducted directly in the university exams. Students are scared of facing the examiners as the syllabus is vast. The present study was conducted to introduce structured oral examination (SOE) as an assessment tool to Ist year M.B.B.S. students in Physiology and to understand their perspective on both traditional viva and structured oral examination. Our goal was to standardize the viva process for one system in physiology by use of checklist and pre deciding the marking system. Students’ feedback was taken on both methods of assessment. Physiology being a vast subject one of the challenges was whether it would be possible to structure the viva. But it could be possible due to positive contribution from the faculty members. From the students’ comments [Table/Fig-4 and 5] in the open ended question of the questionnaire it was evident that they were overall happy with the SOE experience. For many of them it was a morale booster, anxiety reducing experience and they suggested that such sessions should be implemented routinely and frequently. They were more comfortable with SOE as they knew that all the students will be assessed by same set of questions. They felt that it reduced the luck factor and made the assessment fairer. Personal biases were also minimised. Faculty also expressed that SOE can be implemented for rest of the topics in Physiology with some modifications in the blueprinting of questions. Overall it was a positive response from both students & faculty. We feel that, though the ground work in preparing the checklists is extensive, once done, the structured viva can be implemented successfully.
Students’ comments about the open ended question in Traditional viva Questionnaire:
| Remark / Feedback of the students (PRE TEST SESSION) [No of students = 100] | No. of students of such opinion |
|---|
| Overall viva session was good/ nothing to be improved/ | (%) |
| got insight from the questions asked | 29 |
| Teachers / atmosphere should be more student friendly to reduce anxiety, fear/ teachers should be encouraging not demotivating/ should help the students to get at the answer | 22 |
| Questions should test the integrative power/more conceptual / application based questions/ should be more of a discussion session | 12 |
| difficulty level of the questions should be properly adjusted | 12 |
| coverage of the syllabus inadequate | 10 |
| should be 1:1 viva | 8 |
| Panel of examiners should be same for all/ at least one examiner should be unknown to the students / students should have choice to whom they should go for viva | 8 |
| Feedback should be given to students right there | 3 |
| Time allotted to each student should be equal | 2 |
| one topic should be continued instead of jumping on different topics | 1 |
| Other (more such sessions should be kept / vivas should not be kept on practical days) | 7 |
| Viva not needed/ not good for assessment | 5 |
| Left unanswered | 12 |
(Q. - What could be improved in the viva process? Give your opinion)
Students’ comments about the open ended question in Structured Oral Examination (SOE) Questionnaire: (Q. - What differences you observed in SOE as compared to the traditional viva? Give your opinion.)
| Remark / Feedback of the students (POST TEST SESSION) [Total no of students = 93] | No. of students of such opinion |
|---|
| Overall viva session was good/ better than the conventional viva / confidence booster / should be kept after every system | 41 |
| Teachers / atmosphere was more student friendly which reduced anxiety,fear / less scary / teachers were supportive / felt comfortable, relaxed & confident. | 32 |
| Questions were precise, well designed / Uniformity was there which minimised luck factor / conceptual & co relational /difficulty level of the questions was properly adjusted. | 24 |
| Much better as it was 1:1 viva | 12 |
| coverage of the syllabus was good | 10 |
| enough time to prepare as there was separate timing for the viva (not on practical days) | 9 |
| conventional viva was better | 8 |
| Both conventional & structured viva are good | 3 |
| 1:1 session is tougher | 2 |
| Left unanswered | 9 |
Recommendations for better implementation of structured oral examinations:
Proper orientation of the faculty involved in the viva regarding the competencies to be measured and the marking system.
Pairing of the examiners, new one with the more experienced one.
Multiple sets of checklists should be prepared to reduce monotony in the process.
Checklists should have clear and straightforward questions so that evaluation of student’s performance becomes easy.
Each student can be marked on the same checklist by 2 or more examiners and scores can be averaged to impart higher degree of validity & reliability.
We feel that this was just a beginning of a continuous ongoing process. Extensive ground work is needed to bring about a shift in students’ assessment from traditional viva to structured orals. The change should not only be restricted to one subject but needs to get extended to all other medical subjects.
[Response scale: Strongly agree=4, Agree=3, Disagree=2, Strongly disagree=1Highly satisfied=4, satisfied=3, Unsatisfied=2, Highly unsatisfied=1Excellent=5, Good=4, Average=3, Poor=2, Very poor=1](Q. - What could be improved in the viva process? Give your opinion)
[1]. Wakeford R, Southgate L, Wass V, Improving oral examinations: Selecting, training and monitoring examiners for the MRCGP BMJ 1995 311:931-935. [Google Scholar]
[2]. Torke Sharmila, Abraham Reem Rachel, Ramnarayan K, Asha K, The impact of viva-voce examination on students’ performance in theory component of the final summative examination in physiology J. Physiol. Pathophysiol 2010 1(1):10-12. [Google Scholar]
[3]. Wass V, Van Der Vleuten C, Shatzer J, Jones R, Assessment of clinical competence Lancet 2001 357:945-49. [Google Scholar]
[4]. Rangachari P.K., The targeted oral Adv Physiol Educ 2004 28:213-14. [Google Scholar]
[5]. Holloway PJ, Hardwick JL, Morris J, Start KB, The validity of essay and viva voce examining technique Br Dent J 1967 123(5):227-32. [Google Scholar]
[6]. Raymond Mark, Webb Lynn C, Houston Walter M, Correcting Performance-Rating Errors in oral examinations Eval Health Prof 1991 14(1):100-22. [Google Scholar]
[7]. Thomas CS, Mellsop G, Callender J, Crawshaw J, Ellis PM, Hall A, The oral examination: a study of academic and non-academic factors Med. Educ 1993 27:433-39. [Google Scholar]
[8]. Wiggins MN, Harper RA, Implementing a structured oral examination into a residency program: getting started Ophthalmic Surg Lasers Imaging 2008 39(1):40-48. [Google Scholar]
[9]. Bloom BS, Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain 1956 New YorkDavid McKay Co Inc [Google Scholar]
[10]. Rahman Ghousia, Appropriateness of using oral examination as an assessment method in medical or dental education J Educ Ethics Dent 2001 1(2):46-51. [Google Scholar]
[11]. Kshirsagar S.V., Fulari S.P., Structured Oral Examination – Student’s Perspective Anatomica Karnataka 2011 5(2):28-31. [Google Scholar]
[12]. Anastakis DJ, Cohen R, Reznick RK, The structured oral examination as a method for assessing surgical residents Am J Surg 1991 162:67-70. [Google Scholar]