Teaching and Assessing Reflecting Skills among Undergraduate Medical Students Experiencing Research

Today’s medical students who will become tomorrow’s doctors should be trained to confront the various dilemmas and intricacies awaiting their professional lives. In this regard, reflection serves as a model for life-long learning [1–6] by engaging students in thinking what they learnt in a given context and how they will apply that learning in future. Critical reflection is the “process of analyzing, questioning, and reframing an experience in order to make an assessment of it for the purposes of learning (reflective learning) and/or to improve practice (reflective practice)” [7]. The deeper critical reflection moves beyond merely seeking an alternate plan for future similar experiences or identifying reasons for the outcome. It involves questioning underlying conceptual frameworks [8]. The ability to reflect is considered as one of the outcomes of undergraduate and postgraduate medical education [7–10]. Educators who have attempted to teach reflection to health care professional students have echoed the need for incorporating opportunities for students in the curriculum, to develop these skills [11]. Recently, researchers have demonstrated reflection as an attribute for students’ academic performance, where students’ reflection had significant effect on the quality of clinical case solving [12]. Reports suggest that reflective learning may improve professionalism, clinical reasoning, critical thinking and health care [7,13–16].

For the reflection to be effective it has to be guided and appropriate educational approaches are to be designed [2]. Literature suggests a diversity of practical approaches for effective implementation of reflection in undergraduate curriculum [7,8,10,17,18]. However, there is no definite evidence on the best way to teach reflection.

The context used for the reflection exercises should be complex which cannot be resolved by previous problem solving strategies [19]. As engaging in research, in an already ‘packed curriculum’ is a challenging context for undergraduate medical students, reflecting on those experiences may help them to understand their learning process better. For teachers, students’ reflections may serve as a useful route to give feedback to students on their attitude, learning goals and reflective skills. To the best of our knowledge, a systematic attempt for inculcating reflective skills among undergraduate medical students using research as a context has not been reported.

The positive evidence that reflection, like any other skill can be taught [20] and the growing literature proclaiming the irrefutable impact of incorporating reflective skills in medical education [1,21,22] has motivated us to venture on determining the feasibility of imparting this skill to undergraduate medical students in our medical school. In this connection, by setting the Mentored Student Project (MSP) program at our institution as the context for enhancing reflective skills, the present study was conducted with the aim to evaluate the feasibility of incorporating a teaching-learning activity on reflective skills of undergraduate medical students and to determine whether the reflective skills of students improve upon training.

At Melaka Manipal Medical College (MMMC), Manipal Campus, Manipal University, India, there are twice a year admission in March and September. The MSP program was blended in the undergraduate medical curriculum (Bachelor of Medicine and Bachelor of Surgery; MBBS) from September 2007, with an intention of developing research skills in students. In MSP, second year students in groups (n=3 to 5) undertook a research project under the guidance of their faculty mentors. In addition, students wrote a scholarly report and presented their work as a poster. Institutional MSP coordination committee guided students with timely orientations pertaining to research methodology, protocol writing, ethical guidelines, project report writing and poster presentation [23,24]. Apart from these, reflective writing on students’ research experience was introduced as a new initiative from the year 2015 onwards. Two MSP coordinators with Foundation for Advancement of Medical Education and Research (FAIMER) fellowship degree, who had reflection as a part of their training program were involved in the development and teaching of reflection to medical students.

The study design was experimental and was conducted after obtaining Institutional Ethics Committee permission. Senior batch of medical students in the second year of the course who were at the end of their MSP process were considered as the test group and subsequent junior batch which was in middle of the MSP process was the control. Convenience sampling was done for the control group and students who volunteered to take part in the study were included. Out of 155 students in junior batch, 38 participated voluntarily whereas, all students in the senior batch participated as a part of their research training. The test group (n = 173) was oriented regarding the process of reflection in a lecture class while they undertook their research projects in groups. Students were posed relevant questions during orientation which enabled them to reflect. Following this, students were provided an opportunity to write a reflective summary by themselves, as part of training using project title selection phase of MSP as a context. A few students (n=10) volunteered to share their reflection with the class. Subsequently, constructive feedback on the quality of reflection was given to all students based on this. Students were asked to write reflective summary on experience of doing research in 500 words to measure the impact of training on their reflective skills. The control group (n= 38) who did not receive any training on reflection were also requested to write reflective summaries. Reflective summaries were later analyzed by two authors independently using a newly developed rubric to identify level of reflection. Perspective regarding this teaching-learning activity was collected from the test group using a questionnaire. The items in the questionnaire denoted students’ perceptions regarding the quality of content and instructional method, authenticity in grading and the impact of the activity on their attitude regarding the relevance of reflection. The content validity of the questionnaire was checked prior to its administration, by two medical education experts of the institution who had an experience in validation methods.

The goal of the teaching session was to make students aware of the phases and indicators of critical reflection which will help them to integrate past with present and present with their future research experiences.

The module was designed and implemented by the MSP coordinators of the institution as per the tips given by Aronson L [7]. Students were provided information regarding critical reflection [7] and the benefits of reflection [7,8,17,19]. The different phases of reflection were described using eclectic model of Koole S [10]. This model describes reflection in three phases: A. Reviewing the experience, B. Critical analysis, and C. Reflective outcome. In addition, each phase has been described using indicators, which in turn directs the reflectors to put the model into practice [10]. The module was reviewed by two other faculty members with Foundation for Advancement of Medical Education and Research (FAIMER) fellowship degree for content validity.

To grade students’ reflective summaries, a rubric was developed based on the grading criteria proposed by Association for Medical Education in Europe guide adopted from Moon 2004 for reflection [7,8] [Table/Fig-1]. Though we used Koole’s indicators to describe the process of reflection, for grading students’ reflection, grading system developed by Moon was used as we could easily incorporate the indicators in it. Two experts with FAIMER fellowship degree reviewed the rubric independently and the content validity was established. Based on their feedback, the rubric was suitably revised and approved by the researchers of this study.

Each reflective summary was graded by first and the second authors of this study as A to F based on the level of reflection [Table/Fig-1]. The grades were designated with scores as follows: A= 60; B = 50; C = 40; D = 30; E = 20; F=10. Construct validity was tested by comparing the scores of students who are anticipated to have less skills and those with more skills. If the rubric measures students’ reflective skills accurately then there should be differences in the level of reflection between students of test and control groups. Inter-rater reliability of the rubric was established by comparing the scores given by two examiners for each reflective summary.

Mean reflective summary scores of control and test groups were expressed as mean±standard deviation. Inter-rater reliability was analyzed using Kappa statistics using SPSS, version 16, statistical analysis programme (SPSS, Inc., Chicago, IL). Students’ perception was first expressed as median and interquartile range. Frequency analysis of responses was done and expressed as a cumulative percentage of ‘agree’ and ‘strongly agree’ responses.

Quality of reflective summaries, construct validity and reliability of rubric: Out of 173 students in the test group, 55 (32%) had A grade which depicted vision and plans for future research. Ninety two (53%) had B grade which showed judgement on present research experience. Twenty six students (15%) had C grade which described lessons learned during the research experience. Whereas, in the control group (n=38), five students had F grade, while 11 students each scored C, D and E grades. There was a statistically significant (p<0.001) difference in the mean reflective summary scores between control (26.45±9.43) and test (51.66±6.56) groups which reflects the construct validity of the rubric. Inter-rater reliability for the raters was found to be Kappa = 0.78 (p <0.001), 95% confidence interval (0.84, 0.72) denoting substantial agreement among two authors. Retrospective analysis with the study sample size revealed the power for 5% significance as 100%.

Feasibility data and students perceptions: The initial orientation on reflection for the test group lasted for an hour and the practice session followed by feedback for one more hour. Orientation and practice session was conducted as a large group teaching-learning session encouraging student interaction by posing relevant questions. Reflective summaries written by students during practice sessions were graded by their peers, in 10 minutes using the newly developed rubric. All students in the test group wrote a reflective summary and submitted it before the deadline. It took around five minute for the authors to grade each reflective summary. While grading, only the level of reflection was considered and not the quality of English and creative writing skills. Perceptions revealed acceptability of module (median 4, scale 1-5) [Table/Fig-2].

Inculcating skills and attitude for life-long learning is an imperative requirement of Malaysian Medical Council to which MMMC is accredited. Reflective practices can boost learners’ self-directed learning skills, which in turn is an essential attribute of life-long learning [11]. This study showed that it is feasible to use research as a context to help students to experience the process of reflection and thereby gain reflective skills in order to enhance their self-directed learning skills. The use of reflective summaries to facilitate reflective practice is well documented [25–28]. Literature reports the benefits of written reflection as an instructional method, as it guides students’ critical thinking of their own learning process [7]. In the present study, the control group could not engage in a systematic analysis of how and what they learnt from the research experience. Whereas, the test group’s improved reflective capacity was reflected by the streamlined analysis of their research experience. More of such exercises may inculcate high quality reflecting skills in students.

The module was highly commended by experts in medical education with respect to its relevant content and the superior quality of the rubric which reflected the intended learning outcomes. This established its face and content validity. While establishing construct validity, students with minimal reflective skills would be anticipated to score less than those with some training. In our study the control group was not exposed to any type of training in reflection and probably led to their low scores compared to the test group who had undergone training and had high scores. This implies that students acquire better reflecting skills after undergoing training and the criteria for construct validity therefore seems to be fulfilled.

Even though studies report the use of rubrics in patient care [29], the best of our knowledge, there are no rubrics reported till date, which can measure reflective writing regarding research experience. Hence, there was need for a rubric which could be used for assessing reflective writing on research experience. Though the content of the reflective summary was context specific, the extent to which a student goes into deeper reflection depends on their reflecting ability [8,12]. Hence, the grades obtained, manifested students’ generic reflective skills. The grading system differentiated summaries with mere habitual actions, from critical reflection, and helped us to grade the summaries objectively based on quality. Our study subjects did not have prior experience with reflective skills pertaining to their exposure to neither patient nor research. As a curricular component, students were not exposed to the process of reflection. Hence we assume that students’ grade, reflected the reflective skills they gained through the training. The high ICC value of scores given by two assessors independently, demonstrated that the rubric used is a reliable instrument. The grade also reflected the core processes used by students for the reflection.

It was encouraging to note that all students had submitted their assignment on reflective summary. The finding that majority (53%) of students had grade B, and 15% having grade C [Table/Fig-1] affirms that, reflective writing as a first time experience was a creditable skill which they accomplished. A similar study done on physiotherapy students during clinical internship with patient care as context had shown similar results with majority of students achieving descriptive- analytical and analytical levels of reflection which is comparable to A and B grades of reflection in our study [11]. In a study done on first year medical students who had not received any form of systematic training on reflection, demonstrated low level of reflection which is comparable to the reflection level of control group in our study [30]. Despite the notification about prizes for 10 best reflective summaries, intended to motivate them, only 32 % of students had A grade. This shows that factors like interest, fluency in English language, writing skills etc., might have influenced students’ performance, even though they were well informed that these factors will not be taken into consideration while grading. In addition, only 55% of students opined that they are confident in writing reflective summaries in future [Table/Fig-2]. This module was the first of its kind and more practice sessions and repeated exercises along with constructive feedback from mentors may inculcate deep reflecting skills leading to more fruitful transformative learning in students.

We found it challenging to fit the module on reflection into already packed teaching schedule. In addition, students were not serious about the relevance of the module as their grade in reflective writing was not considered during summative assessment. To tackle this issue, we are in the process of considering students’ grade in their reflective summaries as a mandatory requirement of completion of MSP. There was a gross disparity in the sample size between control (n=38) and test group (n=173). With the S.D of 9.5, power of 80% and the minimum difference in the reflective score between test and control as 15, the estimated sample size is found to be 7 in control group and 21 in test group. Hence, we are confident that sample size was appropriate to make the study conclusion. A follow up study could not be conducted to investigate whether repeated exercises inculcate deep reflection in more number of students because our students undertake the research project in second year (one year duration) following which they enter the clinical phase of six months. They complete the succeeding two and a half years in Malaysia. So it becomes a practical difficulty for us to conduct a follow up study.

This study describes a module on reflection that was acceptable to students. In future, the rubric developed in this study for a summative purpose could be used formatively to help students in developing deeper level of reflection. Our model of teaching reflection is feasible in a curriculum where undergraduate students are exposed to research. This model can also be generalized to other context with suitable modifications.

[1]. Driessen EW, Tartwijk J, Overeem K, Vermunt JD, Van Der Vleuten CP, Conditions for successful reflective use of portfolios in undergraduate medical education Med Educ 2005 39:1230-35.  [Google Scholar]

[2]. Grant AJ, Vermunt JD, Kinnersley P, Houston H, Exploring students’ perceptions on the use of significant event analysis, as part of a portfolio assessment process in general practice, as a tool for learning how to use reflection in learning BMC Med Educ 2007 7:5  [Google Scholar]

[3]. Epstein RM, Reflection, perception and the acquisition of wisdom Med Educ 2008 42:1048-50.  [Google Scholar]

[4]. Mann KV, Reflection: Understanding its influence on practice Med Educ 2008 42:449-51.  [Google Scholar]

[5]. Boerboom TB, Jaarsma D, Dolmans DH, Scherpbier AJ, Mastenbroek NJ, Van Beukelen P, Peer group reflection helps clinical teachers to critically reflect on their teaching Med Teach 2011 33:e615-23.  [Google Scholar]

[6]. Smith S, Martin J, Practitioner capability: Supporting critical reflection during workbased placement – A pilot study Higher Education Skills and Work-Based Learning 2014 4(3):284-300.  [Google Scholar]

[7]. Aronson L, Twelve tips for teaching reflection at all levels of medical education Med Teach 2010 :1-6.  [Google Scholar]

[8]. Sandars J, The use of reflection in medical education: AMEE Guide No. 44 Med Teach 2009 31:685-95.  [Google Scholar]

[9]. Dekker H, Schönrock-Adema J, Snoek JW, Molen van der T, Schotanus J, Which characteristics of written feedback are perceived as stimulating students’ reflective competence: an exploratory study BMC Med Edu 2013 13:94  [Google Scholar]

[10]. Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, Factors confounding the assessment of reflection: A critical review BMC Med Edu 2011 11:104  [Google Scholar]

[11]. Mori B, Batty HP, Brooks D, The feasibility of an electronic reflective practice exercise among physiotherapy students Med Teach 2008 30(8):232-38.  [Google Scholar]

[12]. Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, Does reflection have an effect upon case-solving abilities of undergraduate medical students? BMC Med Edu 2012 12:75  [Google Scholar]

[13]. Mamede S, Schmidt HG, Penaforte JC, Effects of reflective practice on the accuracy of medical diagnoses Med Educ 2008 42:468-75.  [Google Scholar]

[14]. Moon JA, A handbook of reflective and experiential learning: Theory and practice 2004 AbingdonRoutledge Falmer  [Google Scholar]

[15]. Plack MM, Greenberg L, The reflective practitioner: Reaching for excellence in practice Pediatrics 2005 116:1546-52.  [Google Scholar]

[16]. Stern DT, Papadakis M, The developing physician—Becoming a professional N Engl J Med 2006 355:1794-99.  [Google Scholar]

[17]. Kolb DA, Experiential learning: Experience as the i of learning and development 1984 New JerseyPrentice Hall  [Google Scholar]

[18]. Macaulay CP, Winyard PJW, Reflection: Tick box exercise or learning for all? BMC Careers [online] http://careers.bmj.com/careers/advice/view-article.html?id=20009702 [16 Nov 2012]  [Google Scholar]

[19]. Mann K, Gordon J, Macleod A, Reflection and reflective practice in health professions education: A systematic review Adv Health Sci Educ Theory Pract 2007 14:595-621.  [Google Scholar]

[20]. Ker JS, Developing professional clinical skills for practice-the results of a feasibility study using a reflective approach to intimate examination Med Educ 2003 37:34-41.  [Google Scholar]

[21]. Fischer MA, Harrell HE, Haley HI, Cifu AS, Alper E, Johnson KM, Between two worlds: A multi-institutional qualitative analysis of students’ reflections on joining the medical profession J Gen Intern Med 2008 23:958-63.  [Google Scholar]

[22]. Kind T, Everett VR, Ottolini M, Learning to connect: Students’ reflections on doctor-patient interactions Patient Educ Couns 2009 75:149-54.  [Google Scholar]

[23]. Devi V, Abraham RR, Adiga S, Komattil R, Mentored student project for inculcating research skills Med Edu 2010 44:496  [Google Scholar]

[24]. Devi V, Abraham RR, Adiga S, Komattil R, Kamath A, Fostering research skills in undergraduate medical students through mentored student project: Example for an Indian medical school Kathmandu Univ Med J 2010 8:294-98.  [Google Scholar]

[25]. Wald HS, Davis SW, Reis SP, Monroe AD, Borkan JM, Reflecting on reflections: Enhancement of medical education curriculum with structured field notes and guided feedback Acad Med 2009 84:830-37.  [Google Scholar]

[26]. Brady DW, Corbie-Smith G, Branch WT, “What’s important to you?”: The use of narratives to promote self-reflection and to understand the experiences of medical residents Ann Intern Med 2002 137:220-23.  [Google Scholar]

[27]. Kumagai AK, A conceptual framework for the use of illness narratives in medical education Acad Med 2008 83:653-58.  [Google Scholar]

[28]. Levine RB, Kern DE, Wright SM, The impact of prompted narrative writing during internship on reflective practice: A qualitative study Adv Health Sci Educ Theory Pract 2008 13:723-33.  [Google Scholar]

[29]. Wald HS, Borkan JM, Taylor JS, Anthony D, Reis SP, Fostering and evaluating reflective capacity in medical education: Developing the REFLECT rubric for assessing reflective writing Acad Med 2012 87:41-50.  [Google Scholar]

[30]. Abigale LO, Dario S, Gloria TB, Wojciech P, An analysis of reflective writing early in the medical curriculum: The relationship between reflective capacity and academic achievement Med Teach 2016 38:724-29.  [Google Scholar]