Art of Paper Setting: The Blueprinting of MBBS Physiology Competency Based Curriculum

1. Associate Professor, Department of Physiology, Armed Forces Medical College, Pune, Maharashtra, India. 2. Clinical Tutor, Department of Physiology, Armed Forces Medical College, Pune, Maharashtra, India. 3. Assistant Professor, Department of Physiology, Armed Forces Medical College, Pune, Maharashtra, India. 4. Professor, Department of Physiology, Armed Forces Medical College, Pune, Maharashtra, India. 5. Professor, Department of Pharmacology, AIIMS Mangalgiri, Andhra Pradesh, India.

Abstract

Introduction: Written examinations are integral component of students’ assessments at formative and summative level. There is no blueprint of question paper available for the new curriculum in Bachelor of Medicine and Bachelor of Surgery (MBBS) Physiology curriculum which may lead to variations in question paper setting. This lack of consistency in question papers affects the preparedness and performance of students.

Aim: To prepare the blueprint of MBBS Physiology Competency Based Medical Education (CBME) curriculum and to do content validity by comparing with Preliminary Examination (PE) and University Examination (UE).

Materials and Methods: This was a descriptive study conducted over a period of six months from July 2020 till December 2020. The study was conducted in Armed Forces Medical College, Pune. There are 15 topics and 140 outcomes for Physiology MBBS course as per new CBME curriculum is divided into paper I and II according to Maharashtra University of Health Sciences (MUHS) syllabus. The main topics in each paper were further subdivided into subtopics for preparing learning objectives. Marks were allotted to each system taking into consideration optimum marks 97 for subjective with included options and 20 marks Multiple Choice Questions (MCQs). The papers of UE and PE conducted with new format were analysed for content validity.

Results: The blueprint of paper I (subjective) shows marks allotted for General Physiology 12, Haematology 13, Respiratory System (RS) 15, Cardiovascular (CVS) 17, Exercise 05, Renal system 15, Gastrointestinal System (GIT) 12, Lifestyle Aging and Meditation 03 and Attitude, Ethics and Communication. (AETCOM) 05 marks. The blueprint of paper II (subjective) shows marks for the Nerve-Muscle 16, Central Nervous System (CNS) 25, Special senses 14, endocrine system are 18, reproductive system 15, and temperature 09. MCQs were also allotted proportionate marks for each topic. Content analysis of papers showed exercise Physiology and body temperature regulation were not assessed in UE.

Conclusion: There was disproportionate representation of topics in formative and summative examinations in absence of blueprint. The blueprint should be an integral part of assessments.

Assessment, Competency based medical education curriculum, Syllabus

A new CBME curriculum was implemented by the Board of Governors for MBBS course from 2019 onwards. The Medical Council of India rolled out subject wise competencies with thrust on horizontal and vertical integration (1). A very important component of curriculum is imparting correct education to students. Assessment is a systematic way of obtaining information about learning by students. ‘Assessment drives learning’, a statement defines the role of assessment in any form of education (2). The assessments in CBME are competency based (3). Every evaluation should be valid which implies that candidates achieving the minimum performance level have achieved the level of competence set out in the learning objectives. As students learn from what is being asked in the examinations, it is important to ask the valid questions (4),(5). For high quality assessment it is better to align with learning outcomes and compare with common testing modalities (6).

Students often point out in their feedback after theory examinations on inappropriate framing of the question papers with comments such as ‘it was very lengthy and time was not adequate,’ ‘it has not covered the whole syllabus’, and ‘it has missed the important topics’, etc., (7). Constructing a quality assessment is a challenging task (8). In the traditional assessment, question papers are set by one teacher/examiner and practical examinations are conducted by some other teacher without any alignment to objectives. For a written examination to be valid, it should match the contents of the syllabus and should give proportional weightage to contents. The blueprint of the syllabus can improve validity of examinations. Blueprint is a matrix that relates the assessment item to its objective and defines the number of items in each of the assessed domains according to its weight in the curriculum and is the appropriate tool for measurement (9). It is essential that a periodic evaluation of the question papers is done for content validity for the criteria being fulfilled. The content validity of evaluation with its congruence with learning objectives and learning experience can be facilitated by using blueprint (10).

Several methods to construct a blueprint are described across various reviews of literature and each discipline choose their appropriate method based on design of the curriculum implemented by their governing Medical council and health University. The first step in blueprinting is to define and tabulate the curricular content (11). After this tabulation, each curricular content could be addressed to learning objectives or cognitive skills. The two-dimensional matrix will be developed in which the contents represent one dimension and the learning objective or weightages allotted represent the other dimension (12).

There is no blueprint of question paper available for the new curriculum in MBBS Physiology curriculum which may lead to variations in question paper setting every year (13). This lack of consistency in question papers affects the preparedness and performance of both average and diligent students. Every mark obtained or lost is crucial for both categories of students. With this background, this study was formulated to prepare the blueprint of the MBBS Physiology as per new CBME curriculum and content validity with the formative and summative assessments.

Material and Methods

This was a descriptive study conducted over a period of six months from July 2020 till December 2020 at a Medical Institute in Western Maharashtra with 150 MBBS seats every year. The study was conducted in Armed Forces Medical College, Pune, Maharashtra, India. There are 15 topics and 140 outcomes for Physiology MBBS course as per new CBME curriculum. The main topics in each paper were further subdivided into subtopics. The entire Physiology curriculum is divided into paper I and II according to the Maharashtra University of Health Sciences (MUHS) syllabus. A total of five faculty members prepared learning objectives and categorised them in domains of learning levels by consensus. Specific learning objectives were prepared according to clinical importance of competency to be achieved and marks were distributed according to proportional weightage. These subtopics were then allotted weightage according to impact and frequency. The impact and frequency are weighted in the scale of 1-3 as (Table/Fig 1). The weightage of topic is calculated as:

Weightage score=Impact×Frequency

Subsequently, a two-dimension table of syllabus content assessment tools and the weightages allotted to them was prepared.

Decide the Impact of Each Content Area and Allot an ‘Impact Score’ (I) to each subtopic

The Impact score (I) range was from 1 to 3. Impact score 1 was given for topic with less public health importance and having “nice to know” content areas for students. Impact score 2 was given for topic with moderate public health importance and “desirable to know” content area and impact score 3 for topic with high public health importance and “must know” content areas for students (14).

The frequency of asking questions on that content areas and give ‘Frequency Score’ (F) to each subtopic

Frequency score ranges from 1 to 3. Frequency score 1 means less frequently asked question, frequency score 2 means moderate frequency of asking questions and frequency score 3 means high frequency of asking questions.

Decide Weightage coefficient of the Each Content Area (W): Following steps were conducted for deciding weightage to each content area.

a. Calculate I × F: Impact of topic × Frequency of asking questions from each topic
b. Calculate total summation of all I × F and this will be labelled as “T”.
c. Weightage coefficient (W) will be calculated as I × F/T
d. Multiply the Weightage coefficient (W) by total no. of items for subjective paper, if options are included i.e., 23. And “Calculate adjusted weightage of each content areas as per total marks of subjective questions, if options are included i.e., 97.

The cumulative impact and frequency were calculated as the average of subtopics for each system. Then Weightage coefficient was calculated as:

Weightage of a Topic (in %)=Weightage of topic/Total weightage of topic×100.

Then marks were allotted to each system taking into consideration optimum marks 97 for Subjective and 20 Blueprint Marks MCQs. Total number of items in each paper were 43 (23 Subjectives+20 MCQs).

Skeleton of the Assessment Tool

As per the norms of MUHS guidelines, total allotted marks for Physiology theory is 200 marks. Thus, each paper is of 100 marks (as shown in (Table/Fig 2)). Each paper will have following marks as per section: Total marks for each paper: 100 (97+20=117 including marks of option questions).

Duration of each paper: three Hours

Section A: MCQ: 30 minutes

Section B: 2 hours and 30 minutes

Section “B”

1) Brief Answer Questions (BAQ) (any ten out of eleven) 10×2=20 marks
2) Short Answer Questions (SAQ) (any eight out of nine) 8×5=40 marks
3) Long Answer Questions (LAQ) (any two out of three) 2×10=20 marks

Theory systems to be included were

Paper I

General Physiology, Haematology, Respiratory Physiology, CVS Physiology, Exercise, Renal Physiology, Alimentary system, Lifestyle Aging and Meditation, AETCOM 1.2,1.3.

Paper II

Endocrine Physiology, Reproductive System, CNS, Special senses, Temperature Regulation, Nerve muscle physiology.

Each paper had 23 items of 97 marks for subjective questions out of which student will have to attempt for 80 marks and 20 MCQ of 20 marks.

Content validity: The UE question papers of winter examination 2020 as summative assessment and PE conducted as formative assessment in department with new format were analysed for content validity for number of marks allotted to each system and compared with a prepared blueprint with accepted variation of 10%. Comparison of MCQs of PE was done but MCQ paper of 20 marks of UE MCQ question was not available hence content validity couldn’t be done.

Statistical Analysis

Data was analysed on Microsoft excel.

Results

There were a total 75 Competencies or outcomes in paper I and 65 Competencies or outcomes in paper II. The outcomes were divided by Learning objectives by faculty members and allotted impact and frequencies to calculate cumulative weightage and marks. The blueprint of paper I shows marks allotted for General Physiology 12, Haematology 13, RS 15, CVS 17, Exercise 05, Renal system 15. GIT 12, Lifestyle Aging and Meditation 03 and AETCOM 05 marks as shown in (Table/Fig 3). The blueprint of paper II shows marks allotted for Nerve Muscle 16, CNS 25, Special senses 14, endocrine system 18, reproductive system 15 and temperature 09 marks as shown in (Table/Fig 4). The MCQs paper carried 20 in paper I and II. The MCQs were also allotted proportionate marks for each topic. When compared between paper I and II, paper I contains 09 systems and paper II contains 06 systems because of which systems of paper I have less marks allotment on systems as compared to paper II. The blueprint prepared was complete for content validity.

On the content validity of paper I, it was observed that blueprint marks of General Physiology are 12, while PE included 10 marks question while UE included 14 marks question. There was no question asked on Exercise Physiology in the UE. In assessment of GIT blue printed marks are 12, PE included questions of 09 marks while UE asked questions on 14 marks. Lifestyle and meditation topics were allotted blueprint marks of 03 while UE asked questions on 05 marks. The questions on AETCOM were 05 marks in blueprint, PE as well as UE as mandated by University (Table/Fig 5), (Table/Fig 6). The MCQs marks for paper I were almost similar for all systems in PE exam as blueprint marks except RS and CVS.

In paper II, questions from Nerve Muscle were given 16 marks, PE included 09 marks while UE included 12 marks. The blueprint marks of CNS are 25, while PE included 31 marks and UE included 29 marks, Endocrine system blueprint marks are 18, while UE and PE included 19 marks which was near close representation, the reproductive system blueprint marks were 15, while PE included 24 marks and UE included 16 marks. The body temperature regulation included nine blueprint marks while PE included 05 marks and UE did not include any questions (Table/Fig 7), (Table/Fig 8). The MCQs marks for paper II were almost similar in PE exam as blueprint marks.

Discussion

Blueprint is a map for an assessment which contains important aspects of the curriculum and educational domains are covered by assessment (15). The term “blueprint” is derived from the field of architecture which means “detailed plan of action” (16). The blueprint is a guide for the faculty who sets the question paper, knows which question test which objective, and marks allotted to it. The present study was conducted to provide the quality of the questions prepared based on the topics they covered and the proportionality of the marks allotted to each of them.

The weightage of marks given to each content area with impact on public health importance and frequency of asking the question in accordance with the subjective consensus of the faculty of Physiology department. A study by Patel T et al., prepared a blueprint in Pharmacology with learning objectives identified and using a proportional weighting system based on clinical importance (17). After thorough brainstorming, the blueprint of the new MBBS Physiology curriculum was prepared for theory paper I and II. There were 75 outcomes for 09 topics in paper I and 65 outcomes for 06 topics in paper II. As there were a greater number of topics and outcomes in paper I, each topic was allotted less marks as compared to paper II. Paper I included blueprint marks for Lifestyle, meditation, Aging and Exercise Physiology as three and five respectively. As these topics are part of the syllabus and included in the blueprint as well, these topics won’t be missed while preparing question papers. Maximum marks were allotted to the CNS i.e., 25 in paper II.

The content analysis blueprint of the paper was conducted with PE exam conducted in department as well as UE for 150 MBBS first year students as formative and summative examination respectively. Content analysis of the paper I showed there was wide variation in Blueprint marks of General Physiology and GIT and marks in PE and UE. Less marks were allotted in PE and more questions were asked in UE as compared to blueprint marks for General Physiology and GIT. There was no question asked on Exercise Physiology in UE and no question asked on lifestyle and meditation in PE. This shows some topics were completely missed in assessment and there is mismatch between PE and UE marks weightages. A study by Gill JS and Sen S also showed that there was overrepresentation and underrepresentation of many topics across all the last 5-year University papers in the subject of microbiology in absence of blueprint (18). A study by Bhandare NN and Bhandare PN also showed all subdivisions of Pharmacology were not covered in theory assessments (5).

Content analysis of paper II shows blueprint marks of nerve muscle were 16 while PE were nine marks and UE were 12. This was a wide variation in PE and UE marks of Nerve muscle topic. The CNS marks were 25 while PE exam contained questions on 31 marks and UE exam contained questions on 29 marks that means both exams over assessed CNS. The Reproductive system blueprint marks were 15 while in PE 24 and UE 16. The body temperature blueprint marks were 09, PE were 05 while UE did not include questions on body temperature regulation. The MCQs marks were similar in PE exam as blueprint marks. The content analysis showed that there is wide variation in blueprint marks and marks in PE as well as UE for some topics for paper II. Some topics were completely missed in UE. These formative and summative examinations were conducted without use of blueprint and we found wide variation in marks. After creating a reliable blueprint, content validity of the blueprint should also be done when the blueprint is used to guide course design and evaluation (11). Hence, blueprint should be an integral part of assessment and it should be available to teachers as well as learners to increase fairness of the evaluation process (19).

Assessment is an integral part of the curriculum. It is very difficult to develop valid and reliable assessment of competence set out in learning objectives by the University. The initial investment of time and effort involved in making a blueprint can produce rich dividends over the long run. A study by Patil SY et al., concluded that a well-constructed and reliable blueprint is a valuable educational tool to align objectives with assessment that can include all aspects of assessment and helps in distribution of appropriate weightage and questions across the topics- benefiting both teacher and learners (14). Our blueprint of syllabus can help in setting question papers at Institute as well as University level.

Limitation(s)

The limitation of the study was that we couldn’t do content validity for MCQ question paper of UE.

Conclusion

Blueprints of the course curriculum need not be obligatory, and the paper setter should have freedom while setting the paper. Availability of blueprints make it easier for students to prepare for assessments as well. We have prepared a theory examination blueprint which is a realistic and objective measure of the relative importance of topics. The results of our content analysis showed that assessments were not matching at formative and summative assessments with blueprint marks. This blueprint will provide validity and reliability to the assessment process and should be an integral part of assessment. The blueprint prepared by us is as per the syllabus divided into two papers as per MUHS University guidelines. This blueprint may not be useful at other Universities but may guide them in setting up paper.

DOI and Others

10.7860/JCDR/2021/48372.15118

Date of Submission: Jan 03, 2021
Date of Peer Review: Mar 06, 2021
Date of Acceptance: May 06, 2021
Date of Publishing: Jul 01, 2021

AUTHOR DECLARATION:
• Financial or Other Competing Interests: None
• Was Ethics Committee Approval obtained for this study? Yes
• Was informed consent obtained from the subjects involved in the study? NA
• For any images presented appropriate consent has been obtained from the subjects. NA

PLAGIARISM CHECKING METHODS:
• Plagiarism X-checker: Jan 04, 2021
• Manual Googling: Apr 19, 2021
• iThenticate Software: May 28, 2021 (16%)

ETYMOLOGY: Author Origin