After taking approval from the Institutional Scientific Advisory Committee and Ethical Committee, a cross-sectional study was carried out involving final year BDS students during their clinical posting in the Department of Periodontology. As per the University norms, BDS students have three clinical postings in Department of Periodontology, two in their third year BDS and one in Final year. Final year students were considered for inclusion in this study as they are sufficiently trained for routine periodontal procedures during their third year and are expected to appear for the summative evaluation in fourth year BDS. Project was explained to students and informed written consent was obtained. Three batches i.e., 40 students participated in study. At the same time, sensitisation of faculty of the Department of Periodontology regarding the project was also done. Four faculty members volunteered to participate in the study amongst whom two were assigned as Conventional Assessors (CA1 and CA2) and two as DOPS Assessors (DA1 and DA2) by lottery method.

Checklist for DOPS assessment of SRP procedure was generated [Annexure 1] and process of internal validation was completed. To generate a valid checklist for DOPS, first of all the competencies required to perform SRP were identified and accordingly Specific Learning Objectives (SLOs) were prepared. Based on these SLOs, checklist for DOPS was generated. While doing SRP, along with dexterity to perform procedure on patient without injury, knowledge about instruments and principles of instrumentation is also needed. Above all, patient doctor interaction in terms of greeting patient, assurance and directions given to patient during procedure, answering patient’s doubts and post-procedural instructions are most important and an integral part of the procedure. So, checklist was designed and pre-validated by peers and senior faculty members to assess important aspects of all domains i.e., cognitive, psychomotor and affective. In addition, total score (65) used for assessment was same as score allotted by Maharashtra University of Health Sciences (MUHS) for university examination (summative assessment). A total of 13 sections were made in checklist, each section having weightage of score 5. Further, in each section, performance was rated on scores from 1 to 5 with 1 being the minimum and 5 being the maximum. In addition, feedback forms for students and faculty were prepared. The feedback forms for both the students and faculty had five questions each, pertaining to their perception regarding feasibility and acceptability of DOPS as assessment tool in comparison with conventional or the existing method [Annexure 2]. These feedback forms were personally handed over to the students and faculty and collected back.

The study was designed in such a way that reliability estimation was feasible. Each student was assessed using conventional method by two assessors (CA1 and CA2) and using DOPS by other two assessors (DA1 and DA2). Two such assessments were carried out before giving feedback to students, thus it was possible to calculate inter and intra examiner reliability and correlation for both the methods of assessment.

Students were subjected to both kinds of assessment i.e., conventional and DOPS by different examiners on three days of their clinical posting; Day 1, 2 and on the last two days (13^th and 14^th day) of their posting. Feedback was given to the students after day 2 that is the second assessment. Descriptive analysis and ANOVA was used to determine and compare the scores given by Conventional and DOPS assessors. To check pair-wise difference between both the groups, Tukey’s Post-hoc was applied.

Forty (35 females and 5 males) final Year BDS students and four staff members of Department of Periodontology participated in the study. [Table/Fig-1] shows descriptive statistics and [Table/Fig-2] shows pair-wise comparison between scores assigned by conventional assessors and DOPS assessors on day 1, day 2 and after feedback (three assessments for each student). Comparison of the mean scores given by examiners (conventional and DOPS) showed significant difference in the scores given by conventional and DOPS assessors except for pair 5. [Table/Fig-3] represents inter-examiner correlation and reliability of Conventional and DOPS method. As evident, DOPS showed better correlation between different examiners and better reliability as compared to conventional method. As far as intra-examiner reliability was concerned, all the DOPS assessors and conventional assessor 2 had intra-examiner reliability above 0.7(70%) but conventional assessor 1 has achieved only 0.379 i.e., 37.9% reliability between his own two set of scores. ANOVA was used to compare the assessment scores on all days. ANOVA showed that groups (day1, day 2 and after feedback) were significantly different for both DOPS assessors but not for conventional assessors. Further to check pair-wise difference between groups, Tukey’s Post-Hoc was applied [Table/Fig-4]. Statistically significant improvement after feedback was seen as compared to Day 1 and Day 2 for both DOPS assessors (DA1, DA2). Although improvement was there after feedback as compared to Day 1 and Day 2 with conventional assessment also (CA1, CA2) [Table/Fig-3] but it failed to reach the level of significance.

Validity is a unitary concept requiring evidence from a variety of sources (content, response process, internal structure, relationship to other variables, and consequences) [9].

Valid assessment method means it is assessing what it meant to assess. Although improvement in scores were seen by both methods of assessment after giving feedback to students based on their two performances, this improvement was significantly more in DOPS score.

According to Downing SM, reproducibility of assessment data or scores over time or occasions is a major source of validity evidence [10]. When inter-examiner statistics were applied, the Pearson’s correlation between two conventional examiners were found to be 0.238 and 0.55 on day 1 and day 2 of assessment, respectively. On the other hand, correlation was found to be 0.744 and 0.711 on day 1 and 2 respectively between two DOPS examiners. It concludes that correlation between DOPS examiners (74%, 71%) were significantly higher as compared to correlation between conventional examiners (23%, 55%). The reliability of the whole test can then be predicted using the Spearman-Brown formula. Downing expects reliability in the “range of 0.70-0.79 or so.” [10]. With DOPS it is found to be 80-90%, well within required range and with conventional method 50-70%, which is below the range suggested by downing as shown in [Table/Fig-3].

When intra-examiner reliability was analysed, it is a found to be above 70% for all examiners (CA2, DA1, DA2) except CA1 for which it was just 37.9%.

DOPS assessment was successfully implemented and was well appreciated by students and faculty. When students as well as faculty were asked about the feasibility of this method in feedback, all had given positive response.

Based on feedback, 97% students and all faculty members had chosen DOPS over conventional method, showing its acceptability. At last, all the students and faculty members agreed that experiencing DOPS during formative assessment will help students to better prepare for summative assessment i.e., their final university examination.

Improvement in scores of all students after feedback was a clear indication of its educational impact. Performance of students significantly improved after feedback, when their DOPS scores were considered. Students were able to identify their weaker areas: 90% pre-procedural preparation, 50% patient position, 70% implementation of principles of instrumentation, 85% doctor patient interaction, 40% post procedural instructions. Assessment drives learning, thus 90% students had tried to improve.

Teachers were also able to identify problematic areas for students and were able to give timely specific feedback to students. In their feedback, teachers suggested few additions in teaching learning methods to improve problematic areas of students: 1) To improve doctor patient interaction, a live demonstration on patient will be helpful; 2) For post-procedural instructions (plaque control instructions) in addition to discussion and demonstration on jaw models, a simulated patient can also be used; 3) Demonstration on pre-procedural preparation should be included in teaching schedule; 4) Students should be encouraged to learn regional language to improve doctor patient interaction. Implementation of these proposals by teachers was suggested and further assessment of its impact on student’s performance is recommended.

The assessment of actual performance that is what the doctor does in practice is the ultimate goal for a valid assessment of clinical competence. Unfortunately, traditional assessment method of post-procedural case presentation is limited in assessing this aspect of clinical skill [11]. DOPS is a well-established assessment tool in the context of work place, especially in Post Graduate (PG) setting in Medical field [12]. This study focuses on evaluating DOPS as an assessment tool for undergraduate students in dental setting in terms of the cardinal attributes of assessment methods that are: validity, reliability, feasibility, acceptability and educational impact.

Results showed statistically significant difference between scores given by conventional method of assessment and DOPS on all the days of assessment (except pair 5). It means same level of performance is being given different weightage by conventional and DOPS assessors. Overall, by conventional method score given on day 1 and 2 was more as compared to DOPS, reason being conventional method has not assessed the way procedure was performed and only the outcome of procedure was evaluated. Thus, it may be inferred by this study that in conventional method of assessment, score depended on trends of scoring in department and generosity of the examiner. On the other hand, DOPS assessed each and every step of procedure, thus score given is less, depending upon the performance of students in different areas and this score has increased only after feedback. It also resulted in improvement in performance of students and was actually reflected in their improved scores. On the other hand, this improvement was not significant when their conventional score were considered because scores given before and after feedback were not significantly different although performance of students has improved after feedback. So, it can be inferred that conventional method of assessment failed to appreciate the improvement in the performance of students.

Use of a checklist enabled the faculty members to provide student-specific and procedure-specific timely feedback to students. This enabled students to identify their weak areas and improve their performance as well as scores which is the main objective of any formative assessment, thus further confirming the validity of DOPS checklist.

In DOPS, use of checklist to score all important areas of procedure decreased the subjectivity; further categorising level of performance i.e., scoring 1-5 might have increased correlation between two examiners. It means that DOPS assessors were better in reproducing their assessment, reason being use of check list that has led to uniformity in assessment. However, one conventional examiner has also achieved reproducibility above 70% reliability which means that even conventional assessment can be made reliable if specific learning objective are clearly defined and task of assessment is performed with responsibility.

Despite its educational effectiveness and simplicity, the formal implementation of DOPS in dental education is still lacking and very few studies have been reported so far. Concerns may be raised regarding DOPS being time taking and less feasible, as mentioned by some previous authors [13]. However, it does not need any extra infrastructure but preparation and planning is required in terms of generation of valid checklist, sensitisation and involvement of faculty and supporting staff. Sensitising students and pre-hand sharing of checklist with them will further improve feasibility. Time management may prove to be a critical factor. According to our experience, DOPS requires approximately 30 minutes per student which is much higher as compared to conventional method which requires approximately 5 minutes per student. So, number of days required to assess whole batch and human resources (examiners and supporting staff) will be more. Still, it is recommended that DOPS should be used at least twice for each batch of students, first one to identify their weaker areas and to give feedback during clinical posting and second one to assess improvement at end-posting.

As far as feasibility and acceptability were concerned, almost all the participants found the DOPS to be a feasible and acceptable assessment tool. This has been earlier also concluded in a similar study done by Singh et al. that the faculty was comfortable using DOPS and found it to be feasible [13].

In fact, DOPS has been found to be useful as assessment learning tool in previous studies as compared to traditional methods and has been recommended to be used for formative assessment [14,15]. However, as with any other assessment tool, the quality of utility depends upon what is assessed rather than how it is assessed.

This study had certain limitations, such as lack of systematic internal validation and small sample size. A further study with pre-validated tools on larger sample size of students is recommended.

Within the limitations of this study, DOPS was found to be valid and reliable method for skills assessment. A positive feedback from students and faculty makes it feasible and acceptable too. As it is time consuming, a great deal of support from faculty is needed for which sensitisation sessions for examiners and students are mandatory. Improvement in performance of students after feedback further reinforces the statement “assessment drives learning”.