Evaluation of Community Medicine Postgraduate Theory Examination to Measure the Content and Cognitive Domain Coverage- A Retrospective Analysis
Correspondence Address :
Vinayagamoorthy Venugopal,
Assistant Professor, Department of Community and Family Medicine, All India
Institute of Medical Sciences, Deoghar, Jharkhand, India.
E-mail: drvinayagamoorthy@gmail.com
Introduction: Assessment is a critical step in learning process. Postgraduate theory examination ideally should assess higher levels of cognition. In order to maintain high standard of public health the postgraduates of the specialty should be certified properly using the appropriate assessment methods. Since the assessment drives learning, ideally it should be aimed at judging a learner’s attainment of curriculum outcomes.
Aim: To evaluate the postgraduate theory exam question papers from 2014 to 2018 in Community Medicine (CM) of various South Indian Universities and to measure the actual content coverage and cognitive domain coverage as per Bloom’s taxonomy.
Materials and Methods: The present retrospective analytical study was conducted from April 2019 to July 2019 to assess the question papers from 2014-2018 in community medicine. Sixty theory exam question papers (20 per University) of last 5 years of three South Indian Universities namely Pondicherry (PU), Tamil Nadu (TN), and Kerala (KR) of CM subject were analysed for content coverage and to categorise the level of knowledge assessed as per Bloom’s taxonomy. The significance of difference between the data of various years was tested using Chi-square test.
Results: Questions testing higher cognitive domain was found only in 15%, 3.75% and 2.5% of PU, TN and KR Universities, respectively. Epidemiology of communicable and non communicable diseases was the content that received maximum coverage of total marks, 25%, 21.25% and 18.5% in PU, TN and KR Universities respectively. Certain topics were not covered in many papers. There was no statistical difference in distribution of marks among various Universities in South India.
Conclusion: Findings of this study may be used to redefine the distribution of contents and cognitive domain tested across universities ensuring the validity and reliability of the assessment.
Assessment, Bloom’s taxonomy, Blueprinting
The Department of CM is a branch of healthcare that aims to develop a well-rounded and holistic medical professional, who will demonstrate knowledge and competence with compassion in dealing with primary healthcare, desire for lifelong learning, evidence-based practice, interdisciplinary teamwork, professional and ethical behaviour in practice in order to improve and sustain the health of the population (1),(2). In order to maintain high standard of public health the postgraduates of the specialty should be certified properly using the appropriate assessment methods. Since the assessment drives learning, ideally it should be aimed at judging a learner’s attainment of curriculum outcomes (3).
The summative assessment for postgraduate in CM across various Indian Universities has written theory examination and clinical examination. Long Essay Question (LEQ) and Short Essay Question (SEQ) are the type of assessment tools of the written exam (4),(5). Ideally postgraduate theory exam needs to test higher level of cognitive domain namely critical thinking, evaluation and synthesis skills. Due weightage has to be given to all contents of the curriculum that ensures validity of the test paper. However, there are drawbacks in setting theory papers namely subjectivity in setting papers, loss of uniformity, lower cognitive domains predominantly assessed, no prevalidation by peer reviewers, and specific learning objectives not clearly defined (6). The evaluation of examination question papers is hence crucial in educational institutes since examination helps to evaluate student’s achievement and proficiency in specific course.
In this study, an attempt was made to evaluate the level of assessment of postgraduate CM question papers of selected South Indian Universities namely Puducherry University (PU), Tamil Nadu-Dr. MGR University (TNMU) and Kerala University of Health Science (KUHS) for the levels of knowledge as well as weightage given to various topics which helps to give evidence-based suggestions to improve the validity of the theory exam.
The present retrospective analytical study carried out in the department of CM, Sri Manakula Vinayagar Medical College and Hospital (SMVMCH), that trains and certifies undergraduate and postgraduate students, located at rural Puducherry. Postgraduate examination was conducted at the end of three years of Doctor of Medicine (MD) course. PU conducts two examinations annually that is one examination in the month of March-April and other examination in the month of October-November. The theory question papers of PU, TNMU, KUHS were publicly available in their respective websites after examination. Total time period of study from data collection to analysis and interpretation was for four months during the months of April and July 2019. Question papers from 2014 to 2018 in CM were reviewed by the first and second authors of the study who were guided by the third author. Total of sixty MD degree theory exam question papers (20 per University) of last five years from PU, TNMU, KUHS were included for content analysis.
Theory question papers of CM postgraduation theory examination from three selected South Indian Universities (PU, TNMU, KUHS) were the samples. The study variables included name of university, year of examination, type of assessment tool, number and marks of the tools, contents of the curriculum assessed, domain of the cognition assessed.
Study Procedure
The question papers were reviewed, and data extraction sheet was prepared. Question papers were retrieved from respective university websites and from college libraries. The time spent for reviewing each question paper was 10-15 minutes. The parameters which were extracted from the question paper were categorisation of essay questions as per Bloom’s taxonomical domains and weightage given to topics in various universities (7). Classification of levels of Bloom’s taxonomical domain was based on commonly used action verbs. The verbs used in Level-1 Bloom’s taxonomical domain include “define, identify, explain, describe, summarise, interpret, classify, compare and contrast”. Level 2 includes “solve, relate, apply, construct, develop, plan, utilise”. Level 3 includes “analyse, categorise, classify, compare, contrast, distinguish, divide, determine, interpret, criticise, compile, imagine, predict, and propose”. Any controversy in classifying the verb or topics was intervened and facilitated by the second author.
Details of each question paper: Four theory papers (I, II, III, IV), each with maximum mark of 100 were conducted in all the selected Universities. Duration of theory examination of all Universities was three hours. However, type of question varied within universities. In Pondicherry University, there were two LEQs, each carried 25 marks and five SEQs, each carried 10 marks. In Tamil Nadu Dr MGR University, there were two LEQs, each carried 15 marks and seven SEQs, each carried 10 marks. In Kerala University there was one LEQ for 20 marks and eight SEQs, each carried 10 marks.
Contents of theory examination question paper remain same in all Universities. Topics covered in theory paper-I include epidemiology, behavioural sciences, population sciences, demography, environmental health and research methodology and biostatistics. Paper-II includes epidemiology, prevention and control of communicable and non communicable diseases, health education, and behaviour change communication. Healthcare of special groups and nutrition were covered in paper-III. Paper-IV covers health services, health administration, primary healthcare, national health programmes, international health and health legislation (8).
Content representation in each university: Total number of theory examination papers reviewed from 2014-2018 in PU, TNMU, KUHS were four (Paper I, Paper II, Paper III, Paper IV) in each year. All four papers were selected from each year and each University. Marks allotted for each LEQ (25, 15, 20) and each SEQ (10, 7, 10) in Pondicherry, Tamil Nadu and Kerala respectively. Total marks in each paper were 100 in all universities.
Statistical Analysis
The software used for data analysis was Statistical Package for Social Sciences (SPSS) version 24.0. Description of categorical variables was expressed as frequencies and percentages. The significance of difference between the data of various years was tested using Chi-square test.
Total of 60 MD degree theory exam question papers (20 per University) of last five years were analysed for understanding the comprehensiveness of topics and to categorise the level of knowledge as per Bloom’s taxonomy. Among all the three Universities, a total of 100 LEQ and 460 SEQ were reviewed.
Categorisation as per Bloom’s taxonomical domains: More than half [290 marks (72.5%), 370 marks (92.5%), 360 marks (90%)] of the essay questions tested the remembering and understanding skills (Level 1 of Bloom’s taxonomical domain) of the students in PU, TNMU, KUHS respectively. Level 2 (applied type of question) accounts for [50 marks (12.5%), 15 marks (3.75%) and 30 marks (7.5%)] in various universities. Level 3 (creating, evaluating and analysing) type of questions accounts for [60 marks (15%), 15 marks (3.75%) and 10 marks (2.5%)] in PU, TNMU, KUHS, respectively. There was a significant difference in distribution of level 1 and level 2 Bloom’s taxonomical domain among the three universities using Chi-square test, which shows blueprinting was not followed properly (p-value=0.032) (Table/Fig 1).
Weightage to topics in PG Community Medicine (CM) examination: In total, there were four papers in PG CM examination in all universities. Total marks allotted for each paper was 100 in all the Universities. Average marks allotted in each University (PU, TNMU, KUHS) from 2014-2018 were given in (Table/Fig 2). Epidemiology of communicable and non communicable diseases had maximum average marks [100 (25%), 85 (21.25%) and 74 (18.5%)] allotted in PU, TNMU, KUHS respectively. Certain topics which were less covered or not covered in some question papers were international health and health legislation, mental health, disaster management, man and medicine, genetic health, communication for health education and behaviour change communication. There was no statistical difference in distribution of marks among various universities of South India.
Comparison of weightage given to topics in various universities each year: It was observed in (Table/Fig 3) that there was significant difference (p-value 0.001, 0.005, 0.001, 0.001 and 0.008) in weightage given to topics such as demography and family planning, epidemiology of communicable and non communicable diseases, communication for health education and behaviour change communication, healthcare of special groups, healthcare services and primary healthcare in each year in Pondicherry University. In TN Dr. MGR University topics such as healthcare of special groups, nutrition, healthcare service, primary healthcare showed significant difference (p-value 0.001, 0.010 and 0.001) in weightage over the years. Environment and health, healthcare of special groups, nutrition, health planning and management and recent advances and updates were found to be significant (p-value 0.001, 0.001, 0.041, 0.001 and 0.026) in KUHS. This shows that blueprinting was not followed in last five years of examination in all the three universities.
Content analysis of theory exam question papers revealed that questions testing higher cognitive domain were found only in 15%, 3.75% and 2.5% of PU, TN and KR Universities respectively. Epidemiology of communicable and non communicable diseases was the content that received maximum coverage of total marks, 25%, 21.25% and 18.5% in PU, TN and KR Universities respectively.
It was observed that majority (72.5%, 92.5%, 90%) of essay questions tested the Level-1 of Bloom’s taxonomical domain in postgraduate CM examination in all the Universities (PU, TNMU, KUHS) respectively. Similar to the current study’s finding, previous studies done also expressed that the lower order questions were asked more than higher order questions (Table/Fig 4) (9),(10),(11),(12),(13). A good and reasonable postgraduate examination paper must consist of various difficulty levels to accommodate the different capabilities of students. Therefore, there is a crucial need to construct a balanced and high-quality exam, which satisfies higher cognitive levels. There has to be a system that automatically handles the classification of questions in papers before finalising it in accordance with Bloom’s taxonomy. It is high time to make use of preplanned software program to auto check the papers based on the examination type.
In the present study, it was found that non communicable and communicable diseases chapter were given more weightage and many topics namely international health and health legislation, mental health, disaster management, man and medicine, genetic health, communication for health education and behaviour change communication were ignored in most of the universities. Validity is an important characteristic of good assessment and construct under-representation is one of the major validity threats in medical education. Under-representation is under-sampling or biased sampling of the curriculum or course content (11),(12),(14). Blueprinting is a guiding map which specifies the assessment program and curriculum over a specified period of time (15). Blue printing helps to reduce the two major threats to validity such as construct under-representation (CU) and construct irrelevance variance (CV) (16),(17),(18),(19),(20),(21). Few of the previous studies that evaluated question papers revealed that blue printing though existed was not followed in their specialties (11),(16). It is recommended that a system should be developed for centralised moderation of question papers at national level to avoid variation in percentage of questions asked on each topic among different colleges (22),(23),(24).
It was clear that the weightage given to higher cognitive domains was negligible in the summative written exam. However, the formative assessment might have tested higher cognitive domains but they could not be evaluated and that was one of the limitations. The content coverage was not uniform across universities and also many chapters were ignored and under-represented. One might argue that as it was postgraduate examination students should be prepared to answer any variations in contents asked in the test paper but there have to be an acceptable range of variation. As there was no standard recommendation about these variations our evaluation could not account for it.
Limitation(s)
The results cannot be generalised across nation as only three South Indian University question papers were evaluated.
It was evident that the theory test papers testing the higher cognitive domains were negligible across all three universities evaluated. The content under representation and lack of uniformity of content coverage across universities was another issue identified. Findings of this study may be used to redefine the distribution of contents and cognitive domain tested across universities ensuring the validity and reliability of the assessment. Measures must be taken during system increasing moderation of university professional question papers to look into the identified issues in setting theory examination question paper for postgraduates.
DOI: 10.7860/JCDR/2022/51597.16108
Date of Submission: Jul 27, 2021
Date of Peer Review: Nov 26, 2021
Date of Acceptance: Jan 10, 2022
Date of Publishing: Mar 01, 2022
AUTHOR DECLARATION:
• Financial or Other Competing Interests: None
• Was Ethics Committee Approval obtained for this study? No
• Was informed consent obtained from the subjects involved in the study? No
• For any images presented appropriate consent has been obtained from the subjects. NA
PLAGIARISM CHECKING METHODS:
• Plagiarism X-checker: Jul 28, 2021
• Manual Googling: Jan 08, 2022
• iThenticate Software: Jan 20, 2022 (6%)
Etymology: Author Origin
- Emerging Sources Citation Index (Web of Science, thomsonreuters)
- Index Copernicus ICV 2017: 134.54
- Academic Search Complete Database
- Directory of Open Access Journals (DOAJ)
- Embase
- EBSCOhost
- Google Scholar
- HINARI Access to Research in Health Programme
- Indian Science Abstracts (ISA)
- Journal seek Database
- Popline (reproductive health literature)
- www.omnimedicalsearch.com