We adopted a qualitative study design as this method could explore the experiences of the stakeholders that would not be sampled by close-ended questionnaire surveys. We used two methods of data collection; focus group discussions for interns and in-depth interviews for internal medicine examiners. The subjects for focus group discussions were 3 groups of interns with 8 members in each group. They were purposively sampled to include varied range of academic performance. They included those who completed MBBS in Government Medical Colleges and Private Medical Colleges from two sites.

The subjects for in-depth interviews were 6 teachers of Internal Medicine who were examiners at least thrice for final MBBS practical exams. They were purposively sampled to the point of saturation. These belonged to 3 different medical colleges both government and private.

Approval of institutional ethical committee was obtained.

The first and seventh author initially made 2 sets of questions pertaining to the various attributes of assessment for use in focus group discussions of interns [Table/Fig-1] and in-depth interviews of internal medicine examiners [Table/Fig-2]. The questions were set to explore the aspects of pre-exam preparation, validity, reliability, feasibility and educational impact of the cardiovascular long case presentation in final MBBS exams. These sets were then pre-validated by conducting mock group discussion with third, fourth and sixth authors and interview of second author. Ways to probe the responses further were also determined. The questions were designed after reading “Qualitative Methods in Public Health” [3].

“The sets of questions are appended” must be deleted.

Focus group discussions were held with interns in groups of 8 members which were recorded on voice recorder application of mobile phones. Informed consent and verbal consent were obtained. Pledge for ensuring confidentiality was undertaken by the investigators and participants alike. Discussions were conducted on Sundays at places familiar to them after completion of duty hours excluding those who were on emergency duty. Notes were taken verbatim. Summarisation was done at the end of group discussions. No remuneration was paid.

In-depth interviews of General Medicine teachers were held telephonically or personally based on the questions [Table/Fig-2], most of which were open ended, by 3 authors (trained for interviewing by prior discussions on phone and e-mails) after obtaining verbal consent and pledging confidentiality. The interviews were recorded using voice- recorder application of android mobile phones. Notes were taken verbatim.

Data analysis was done by a process of immersion-crystallisation [3]. Interviews and group discussions were transcribed. There was no need for translation as the medium of communication was English. Some Hindi or Kannada words were translated. Data from multiple sites was pooled. Data was read repeatedly till characteristics emerged. The characteristics were coded using different coloured high-lighters. Coding sorts was done manually. Themes that emerged were grouped into categories. Interpretations were drawn subsequently. No statistical package was used as the sample size was small.

Themes that emerged were categorised under the heads pre-exam preparation, validity, reliability, feasibility and educational impact. All these aspects were differently perceived by the interns and examiners.

Pre-exam preparations: Most of the interns admitted to a lack of confidence before their final MBBS exams. This was attributed to difficulty in diagnosing cases with murmurs. They expressed helplessness due to a lack of choice and the vastness of syllabus. Many interns felt that the teaching in preparation for exams employed inappropriate tools. Very few had learnt in skill laboratories. Some had purchased compact discs of heart sounds and murmurs. One of the interns expressed it as “They teach murmurs by verbal description and we have to imagine them in sounds. Remember the story of six blind men and the elephant?”Another intern replied that there should be devices which allow simultaneous auscultation by teachers and students of the heart sounds.

On enquiring about resource material, all felt that textbooks written by Indian authors were preferred over those written by foreign authors because of simpler language. However some doubted about the authenticity of content. In contrast the examiners admitted that they were more worried about arrival of examiners, arranging transport to the venue and playing host. Some said there was a dearth of typical rheumatic heart disease (RHD) cases as most of the patients directly go to the cardiologist through government schemes. They said that sometimes patients had to be called from other hospitals and paid remuneration for participating in exams. None of them expressed the need for training the patients. Many said that the patients experienced burnout over 3 days of practical exams and replacing them with similar cases was difficult. Some also expressed concern about difficulty in arranging examiners as most of them also had lucrative private practice which they would not forego for 3 to 4 days.

Validity: The interns said that there was often a deviation from actual case allotted in asking questions which they found unnerving. Many also expressed that the questions were limited to history or physical examination of the cases and ignored the aspect of management. One of the interns expressed that he was asked only about the design of the stethoscope which caused him anxiety and moral decline during exams.

Almost all said that inability in diagnosing murmurs was the most important factor which resulted in reduced confidence. However, only one academic achiever expressed the contrary. One of the students said that it was easy to impress the examiners by using common phrases describing murmurs such as “rough, rumbling, heard in lateral position, etc” without actually being sure of its presence or nature.

Some wondered why cases of ischemic heart disease, congestive heart failure, hypertensive heart disease or atrial fibrillation that were more often admitted in the wards than rheumatic heart disease were not kept in exams more frequently. Some pointed out that the only assessment in practical exams about ischemic heart disease was in viva regarding electrocardiogram or drugs for one or two minutes. Another intern added that they could not remember drug dosages even if they “rattofied” (memorized). She observed that dosages were remembered only when they administered the drugs. An intern in his response to this statement expressed a need for student-doctor model just like the student –nurse model to “learn by doing”. Most of them were not aware of assessing for competencies. When probed, they said communication skills and working in a team were not assessed at all as there was no scope in the present practical exams for the same. They appreciated the importance of systems approach in handling issues of referral and follow-up before and after surgery through government health insurance schemes. However they could not figure out how that skill would be assessed in exams.

The examiners in their interview said that cardiovascular cases with murmurs were kept because those cases gave an opportunity to assess the clinical skills and the understanding of cardiac cycle. They admitted to not having consensus about the level of difficulty of a case. Some of them said that asking questions unrelated to the case was to discourage the students from ’match-fixing’ and non-random allocation of cases. Three examiners said that it would be more valid to assess the students’ knowledge of implications of murmurs such as infective endocarditis and thromboembolism rather than seeking a precise description of a cardiac murmur. Some could not figure out how to allot a case of ischemic heart disease as the patient would be generally requiring intensive care services. Two of the examiners expressed the need for a detailed blueprinting for practical exams with appropriate weightage assigned to various diseases based on epidemiologic burden.

Reliability: The interns felt that there was a variation in assessment of their case presentation by different examiners. According to them the examiners used different yardsticks of performance depending on whether they were from private or government institutions. The examiners also assessed the candidates differently when the same case was presented by two candidates by asking trickier questions to the second candidate. Some told that the day of exam influences the results as the external examiners come late on the first day and wish to leave early on the last day.

The examiners expressed that the assessment was reliable despite of the above mentioned shortcomings. One said it was enough to test a grain of rice to check if it was cooked. Four of them admitted that they did get tired over the day and the assessment could be influenced by it. Two examiners suggested use of checklists to bring in uniformity. One of them suggested adopting new methods like objective structured clinical exams (OSCE) and objective structured long examination record (OSLER) for improving the objectivity. When probed about the need of prior training of examiners to record consensus in diagnosis of the cases most of them were surprised and said it would only waste time. When asked if a question bank could be created exclusively out of ’must know’ areas and used, many dissented for the reason of trivialising education. However they all agreed to the setting of benchmarks.

Feasibility: The interns admitted that they were concerned about the vastness of the syllabus when they appeared for the exams. They wished that there were two exams at least 1 year apart. Assessing three years’ acquisition of knowledge and skills on one single day was not feasible according to them.

The examiners expressed that the present system of long case presentation survived only because of its feasibility. Feasibility came in the form of assessing 25 students per day by 4 examiners. By this each examiner could assess 12 or 13 long cases and 25 short cases. By restricting themselves to 8 minutes for long case and 4 minutes for short case this enormous task could be accomplished. Because of the shortage of time examiners said they did not have an opportunity to observe the history taking and clinical skills.

Educational impact: The interns were probed about the outcome of cardiovascular case assessment in preparing them to be health care providers to the rural population as rural service has been made mandatory in Karnataka. The interns felt that they learnt to handle cardiovascular emergencies and to provide ambulatory care for patients with ischemic heart disease predominantly during internship. They expressed that most of what they learnt preparing for exams was not relevant to their future practice.

The examiners also were of the same view and two of them said that there was a need to overhaul the assessment method by aligning it to the expected competencies.

To quote van der Vleuten C, “Assessment drives learning in at least four ways: its content, its format, its timing and any subsequent feedback given to the examinee” [4].

Assessment of cardiovascular long cases in general medicine practical exit exam is often limited to simple history taking and physical examination. Other competencies like communication skills, systems based practice; self-improvement and patient care are not assessed. The cases kept in the exams are not representative of the epidemiologic burden. In fact there is no blueprinting for assessing practical skills and assessment is done as done traditionally by the veteran teachers. Standards of assessment are not uniform. This makes assessment less valid and not reliable. However the long case presentation has stood the test of time because it is feasible. Feasibility comes in the form of 4 examiners being able to assess 25 students each day. Each examiner has to assess 12 or 13 long cases and 25 short cases. This means he has to assess a long case for about 8 minutes and short case for not more than 5 minutes so as to allow sufficient time for afternoon viva-voce examination. So the long case assessment which could have been an excellent tool owing to its semblance with real-life case encounters becomes mired as a tool of poor validity due to limitation of time.

Most of the teachers in general medicine also have private practice because of which it is difficult to procure examiners for consecutive 3 or 4 day practical examinations. So though feasible, the long case presentation becomes “luck of draw”. In ’Principles of assessment in medical education’ Tejinder Singh in his chapter ’Improving the long case’ published in 2012; noted that there are no studies regarding validity and reliability of long case assessment in India [5]. Since then very few articles are published in India about long case presentation though it is the method of choice in assessing practical skills [6]. New methods of assessing clinical skills like OSCE, OSLER and MINIcex are used non-uniformly in some medical colleges only in formative assessment [7–9].

There are no studies of perception of internees and general medicine examiners regarding long case presentation in final MBBS general medicine exit exams in India. In particular, there is limited validity of cardiovascular long case presentation considering the changing cardiovascular epidemiology in India. The experience of two of the stakeholders that is interns and examiners suggests a need to change the present pattern of examination if important competencies are to be addressed in education. Our study has evaluated level 3 of Kirckpatrick’s model by using a qualitative design.

A lack of orientation of two categories of stakeholders to the outcome of education and its assessment was palpable. Both the parties took examination for examination’s sake. The perceptions of interns differed from that of examiners as the former were concerned about content underrepresentation while the latter were concerned about feasibility.

Our study is limited by a small sample size. A large sample could not be included as we did not have software for analysis of qualitative research.

It is also a study designed to explore the past experiences of stakeholders regarding a system of assessment. Further research to actually measure the validity and reliability would guide any intended change in the system of assessment.

Since this is a qualitative study, the opinions of the interviewers could have influenced the results.

There is a perceived need to make the traditional long cases structured, observed, more frequent and employing multiple assessors. There is also a need to blueprint practical examinations in medicine topic-wise with due changes made at predefined intervals for keeping up with changing epidemiology of cardiovascular illnesses.

Financial or other competing interests: This study was undertaken as a project for MCI’s advanced medical education technology course of 2014 by the first author. The authors thank the faculty of JNMC Belagavi’s MCI Nodal centre for the training received about qualitative methods.