The present prospective, single-group, pre-post intervention study was carried out in the Department of Pharmacology, Gujarat Medical Education and Research Society Medical College, Gotri, Vadodara, Gujarat, India, over a period of five months from April 2016 to August 2016. The study was approved by the Institutional Ethics Committee, GMERS Medical College, Gotri. The informed consent was obtained from the second year undergraduate medical students and faculty before their participation and collection of feedback.

Initially, the informal discussion with pharmacology faculty was conducted to decide the topic for the e-learning activity. Based on a suggestion, the pharmacovigilance (a must know topic of pharmacology) was selected to prepare e-learning module. This had been traditionally taught in pharmacology practical.

The standard textbooks on pharmacology, websites of WHO and Indian Pharmacopoeia Commission, Uppsala monitoring center were used to prepare resource materials for e-learning module. The contents were divided into the five units as per the learning objectives [Table/Fig-1].

Multimedia presentations (Microsoft Office PowerPoint, 2007; Microsoft Corporation, Redmond, Washington, United States) were prepared as per the learning objectives of each unit. The learning objectives were included at the beginning of each presentation. The animations, pictures, case-based exercises were added in the presentation. The content was validated with the help of pharmacology faculty. With the help of iSpring Suite 8.3 software free trial version (iSpring Solutions Inc., Alexandria, Virginia, United States), audio was added in the presentation and converted into a video (Unit 1,2a,3,4,5). One role play video of doctor-patient conversation was also prepared (Unit 2b). Learning objectives and audio-visual contents of the presentations were pre-validated.

Designing of Website: The website was designed to upload the learning resource materials (Address: http://pharmacologygmersgotri.weebly.com/adr-module.html).

To assess the learning gain, SAQ and CBE to fill the ADR form were prepared. The suspected ADR reporting form of Central Drug Standard Control Organisation, India was used in ADR form filling exercise. A structured checklist was prepared to assess the ADR reporting form. All assessment questions were prepared as per present learning objectives of the module to ensure content validity.

The closed and open-ended feedback questionnaires were prepared to collect students and faculty perceptions through the online version of the SurveyMonkey tool (Survey Monkey, 2012). Questionnaires focused on e-learning activity, utilisation of learning resource materials, future performance, learning outcome and their recommendation for future batches. The close-ended questionnaires were prepared using the five-point Likert scale (1=strongly disagree, 2=disagree, 3=undecided, 4=agree, 5=strongly agree). Suggestions and comments were also asked from the participants. All the feedback questionnaires were pre-validated.

In the first contact session, students were informed about learning objectives of the course module, its duration, assessment and feedback procedures. All willing students were asked to register their name, roll number, mobile number and email ID for communication. The ‘WhatsApp’ group was created to inform the participants. In the next contact session, pre-test (SAQ and filing of ADR form based on CBE) was conducted to collect baseline data. In the department, only four pharmacology faculties were available to participate and provide the feedback for the module after excluding investigators and faculty involved in validation. So, pharmacology faculty of other institute (Government Medical College, Bhavnagar) of all academic positions (tutor, assistant professor, associate professor and professor) were approached telephonically to participate and informed about study objectives and feedback procedures. All willing faculty were asked to register their details (their name, designation, mobile number and email ID) and communicated through e-mail/WhatsApp as per their preferred mode of communication.

Six videos of learning resource material were uploaded on YouTube and their links on the specially designed website [Table/Fig-1]. Participating students were duly informed about the videos through email and WhatsApp group. They are supposed to download the videos from any of these resources. Students were provided 14 days to use e-learning resources before the post-test assessment. Similarly, the participating faculty were informed about the resources.

On the third contact session, a post-test assessment of students was conducted to assess the effectiveness of the module. The same pre-assessment questions were used in the post-test. The feedback questionnaires were delivered by using the SurveyMonkey tool to students and faculty. They were requested to complete it within a week.

The level 1 and 2 of Kirkpatrick’s Model were used to check the effectiveness of e-learning [15]. In a level 1 evaluation, student and faculties’ perception were analysed based on their feedback. Data of close-ended questionnaires were presented in percentage. Data were presented in three categories: ‘disagree’, ‘undecided’ and ‘agree’. The ‘strongly disagree’ and ‘disagree’ categories were merged into ‘disagree’; ‘agree’ and ‘strongly agree’ were merged into ‘agree’ category. The open-ended responses were organised based on questions and coded according to their answer. The descriptive analysis was used for the open-ended questionnaires.

In the level 2 evaluation, cognitive learning gain of the students was analysed by comparing the pre and post-test assessment score. The pre and post-test assessment were evaluated simultaneously. The answer sheets of both tests were coded to hide the identity of the assessment (pre and post) as well as students using a single block technique through random allocation software 2.0. Data of those students who appeared for both pre and post-test assessment were analysed using Student’s paired t-test. The absolute learning gain (% post-test score-% pre-test score) and relative learning gain (% post-test score-% pre-test/% pre-test score) were calculated. The effectiveness of the intervention was evaluated by using class average normalised gain (g=(% post-test score-% pre-test score) /(100-% pre-test score). The class average normalized gain of 0.3 (30%) was considered significant, as per Hake’s criteria for the effectiveness of educational interventions [9]. Statistical analysis was done by using GraphPad Prism 6.0 demo version (GraphPad Software, Inc., La Jolla, California, United States). p<0.05 was considered as the statistically significant difference.

A total of 127 undergraduate students (second year) consented to participate in the study. Of the participated students, 53.54% were female and 46.46% male. The students used their personal internet connection to use e-learning module and feedback. A total of 17 faculty members gave their consent to participate and provided the feedback.

A total of 125 students (98.43%) gave the online feedback.

Student’s perception of e-learning activity: As shown in [Table/Fig-2a], most students perceived that the amount of time took to complete this module was appropriate for the content (73.6%), the pace of learning was proper (84.0%), the module was interesting (85.6%), and important resources for the self-directed learning (98.35%). Most of the students perceived that audibility (73.6%), visibility (89.6%) of multimedia and overall rating of entire e-learning module (90.0%) were very good.

[Table/Fig-2]:

a) Students’ perception of e-learning activity; b) Students’ perception of utilisation of learning resource material.

Student’s perception of utilisation of learning resource material: As shown in [Table/Fig-2b], most students agreed that the learning resource material was user-friendly (93.6%), informative and logically structured (96.8%) and makes studying easier (88.3%). They also felt that it had increased their understanding of the subject-matter (95.2%). They found the use of problem-based exercises helped to gain a clear understanding of the content (98.4%).

Student’s perception of future performance, learning outcome and recommendation: Majority opined that e-learning will enhance their performance in future assessment (91.7%) and they have achieved the learning outcomes (98.3%). When students were asked for the suggestions for future batches, most opined that module should be taught online only (43.7%) or adjuvant to the classroom (40.3%).

Analysis of general comments: Open-ended questions were analysed qualitatively and categorised as barriers, facilitating factors and suggestions [Table/Fig-3].

A total of 16 faculty (94.1%) gave the online feedback.

Faculty perception of e-learning activity: As shown in [Table/Fig-4a], majority of faculty agreed that the amount of time allotted to complete this module was appropriate for the content (81.3%), the pace of learning was proper (93.8%), the module was interesting (81.3%) and important resources for the self-directed learning (81.3%). Most of the faculty said that audibility (81.3%), visibility (93.8%) of multimedia and overall rating of entire e-learning module (93.8%) was very good. Most of the faculty opined that the module should be taught as an adjuvant to the classroom (75.0%) for future batches.

[Table/Fig-4]:

a) Faculty’ perception of E-Learning Activity; b) Faculty’ perception of utilisation of learning resource material.

Faculty’ perception of utilization of learning resource material: As shown in [Table/Fig-4b], majority of faculty said that course content of e-learning module was informative and logically structured (100.0%), clearly explained and consistent with the learning objectives (93.8%) and can achieve the stated learning objectives (93.8%).

Analysis of general comments: Open-ended questions were analysed qualitatively and categorised as barriers, facilitating factors and suggestions [Table/Fig-3].

A total of 124 students (97.6%) completed both pre and post test for the level 2 evaluation-learning gain. Mean test scores of SAQ and ADR form filling exercise improved significantly from 31.3% to 62.5% and 38.8% to 85.3%, respectively. Mean test scores of absolute learning gain and relative learning gain are presented in [Table/Fig-5]. Medium and high class-average normalised gains were observed in SAQ and ADR form filling exercise, respectively.

In the present study, the e-learning module of pharmacovigilance was prepared and its effectiveness was analysed among the undergraduate medical students through one-group pre-test/post-test study design. Its acceptability and feasibility were assessed through the perception of the students and faculty.

In line with the earlier studies, a significantly higher post-test score of e-learning session (p<0.001) was observed [10,11]. The effectiveness of present intervention was observed in terms of absolute learning, relative learning and class average normalised gain. An earlier literature review of e-learning in pharmacy education reported learning gain of 7% to 46% [16]. The absolute learning gain depends on a student’s pre-test score. Students with lower pre-test scores tend to show more gain than students with higher pre-test scores. The relative learning and class average normalised gain overcome this inherent bias of absolute learning gain. Although, relative learning gain corrects absolute learning gain by the pre-test score, it can show the skewing effect of outlier students with very high or very low pre-test scores [17]. The class average normalised gain is independent of the study group’s pre-test level of knowledge. It corrects absolute gain with maximum possible gain achievable by each student. Earlier studies used the class average normalised gain to measure the effectiveness of the educational intervention [3,9,17]. In the present study, a class average normalised gain for short answer questions and ADR form filling exercise were off 46% and 76%, respectively. Hakes PR, criteria of educational intervention suggest present e-learning module was moderately to highly effective in terms of cognitive gain [9].

The findings of present study need to be interpreted in terms of extraneous variables which can affect the pre-test/post-test assessment model. It includes the halo, Hawthorne, and practice effect [17]. The “halo effect” represents observer bias in which positive ratings in one domain affect other domains and inconsistency across faculty raters [18]. The blinding and structured checklist in assessment was used to minimise this effect. The “Hawthorne effect” is a type of reactivity in which results or behaviour may be affected by knowing of participant that he is being tested or observed. The literature suggests its existence, but little understanding prevails about their operating conditions, mechanisms and magnitudes [19]. Third extraneous variable includes the practice effect-improvement in performance due to earlier exposure to the test [20]. The meta-analysis suggests retesting produces practice effects on cognitive ability. It is heterogeneous in nature, increases with the number of retesting and not affected by study context. However, it is not clear how retesting changes the valid inferences that can be drawn from test scores [21].

The positive perception of students was observed for the e-learning activity and use of learning resource material. As per “technology acceptance model”, perceived usefulness and ease of use are the key variables that influence users attitude, intention, and actual behaviour to use a new technology [22,23]. Atkins S et al., in a survey of African and Asian students observed the perceived usefulness and experience of no technical problems as key factors associated with a more positive perception of the usefulness of the online component of the blended courses [24]. In the present study, the students showed a positive perception toward the learning activity, use of learning resource material, future performance and learning outcome. The “unified theory of acceptance and use of technology” suggests performance expectancy, effort expectancy, social influence, and facilitating conditions influence user intentions to use an information system [25]. Students felt e-learning module simple, systematically explained and interesting. In the study, e-learning was facilitated with the use of social media like WhatsApp. Students found a self-learning opportunity in e-learning. Through self-directed learning, students identify their learning needs, formulate goals, find learning resources, make appropriate learning strategies and evaluate learning outcomes [26]. It motivates the students to learn independently [27].

The present module was well perceived by the teaching faculty. They felt e-learning informative, easy to understand, interesting and attractive. These video-based e-learning modules are easy to prepare with the help of available resources across medical colleges in India. This e-learning module was prepared using Microsoft PowerPoint and the free software. It was uploaded into free website and YouTube. Higher participation (127 vs. 42) and completion rate (97.7% vs. 38.1%) were observed in the study as compared to earlier e-learning Indian study involving undergraduate medical students [3]. In an earlier study, the majority of students depended on the institutional internet facility and found hindrance in accessing resource material [3]. The successful implementation of e-learning activity requires easy access to a computer, internet facility and positive attitudes toward e-learning. In the present study, almost all participated students could readily access to personal mobile (tablet, smartphone) or computer (desktop, laptop) internet. Another reason could be due to the type of e-learning tools used among the studies. Gaikwad N and Tankhiwale S, used the kiosk mode PowerPoint presentations and only used the website to deliver the contents [3]. A recent United States study using PowerPoint with visual aids and recorded narration did not show the benefit of e-learning [28]. This module was prepared as a smartphone friendly video-based e-learning tool and was available to students via website, email, YouTube and WhatsApp during e-learning session.

The present study has several limitations. The impact of e-learning was not compared with the traditional teaching methods in the study. Only single topic of pharmacology was covered. Only short-term cognitive gain was assessed. The long-term gains and behavioural change among the students were not assessed. In the study, e-based interactivity in the form of self-evaluation exercises and formative feedback interactivity between students and faculty were not explored. E-based interaction encourages the students in an active learning process [29]. The present study has barriers like lack of interaction with faculty. Preparation of the e-learning module is a time consuming task. This limits its feasibility to cover the vast subjects in the medical course.

The e-learning module was moderately to highly effective in terms of acquiring cognitive gain. It has attracted interest from students and faculty. Both perceived it easy, informative and facilitating method for self-directed learning. This supports acceptability and feasibility of e-learning as an educational tool.