The validity reliability study was conducted in one of the private university in Malaysia, UTAR in the year 2016. The study was duly approved by Scientific and Ethical Review Committee of the University, with the approval number of U/SERC/71/2016.

To construct the PSCAT, the following tools were referred, American Physical Therapist Clinical Performance Instrument (PT CPI), Canadian Physiotherapy Assessment of Clinical Performance (ACP), Assessment of Physiotherapy Practice (APP) from Australia, Clinical Competency Evaluation Instrument (CCEVI) from UKM, Malaysia and Common Assessment Form (CAF) from Irish [2,3,15,21]. Two Lecturers and a Physiotherapy student of final year was involved in the construction of the initial version of the tool which contained 35 item under 7 components: (1) Professional practice and behaviour; (2) Assessment; (3) Clinical reasoning; (4) Intervention; (5) Risk Management; (6) Communication skills; (7) Teamwork. It was structured on a five-point rating scale of Poor; Fair; Average; Good and Excellent and weighted, 1; 2; 3; 4 and 5 respectively. Each component after marking was converted into percentage. Component mentioned in the tool were referred from World Confederation for Physical Therapy (WCPT) guidelines. The tool also included component of feedback and self-evaluation. The tool was made so that it is user-friendly, with an easy to understand language, not include too many instructions. It was also taken into consideration that it could be used for the overall clinical performance. The tool is flexible to be used in all year of study in physiotherapy programme, as there is an option of “NA” for the items which are not included in the student’s learning outcome.

Ten clinical instructors from UTAR who had more than four years’ experience as clinical instructors in each of their specialised fields were involved in the validation of the initial version of PSCAT. A perception and satisfaction questionnaire was used to study the validity of the initial version of PSCAT by CI of the university where the study was conducted. Since, the clinical assessment tool also needed to be understood by students, therefore the same perception satisfaction questionnaire was circulated among second year and fourth year students by convenience sampling. Each item’s face validity, content validity, criterion-related validity and construct validity was tested with a list of questions shown in [Table/Fig-1]. The CIs who were involved in this study for validation and reliability testing had an average of more than eight years of experience as clinical instructors. After the initial validation, a revised version of PSCAT was developed. The revised PSCAT included the suggestion and feedback received after evaluating the initial PSCAT. The next step involved, 11 CIs from various universities, who participated in this validation of the revised version of PSCAT. The CIs were from various universities that offered bachelor degree for physiotherapy for at least 4-years. The same perception and satisfaction questionnaire used for the initial version of PSCAT was utilised for assessing the validity of the revised PSCAT. After the validation of the revised version of PSCAT, the final PSCAT was developed. The reliability of PSCAT included 40 students from fourth year of University “UTAR”. Within the duration of three months of the study, fourth year was attending their clinical posting. Therefore, students of Bachelor of Physiotherapy in fourth year cohort were selected by convenient sampling. The exclusion criteria were; student who were not posted in clinical area during the study period. The procedure included assessment of each student in a clinical placement area by two CIs. The ranges of age of these students were 21-25 years.

The Validity was analysed by ACP or validity index [23,24]. The response of participants was in “Yes” and “No” for each item. In the Microsoft Excel, “yes” key in as while “no” key in as “0” to determine the validity index. The validity index for each item on the PSCAT was calculated as the number of validators giving a rating of “yes” divided by the total number of validator. The validity index for the entire PSCAT was calculated again based on the percentage of total items rated by the experts.

The items were considered valid if it achieved more than 90% or 0.9 of validity index in each category (face validity, construct validity, content validity and content validity). Overall validity in each component was recorded as well.

For the inter-rater reliability, the two-way random effect model ICC at a 95% confidence interval was used. Eighty PSCAT were collected at the end of clinical placement and key in the total marks as well as the mark of each component in percentage.

The characteristic of clinical instructors who involved in the validation of PSCAT.

The construct validity index in initial version of PSCAT was the lowest [Table/Fig-2]. It also had the highest validity index in criterion-related categories. Validators mentioned in their comments that they were unable to understand the item number 28 (demonstrates collaborative practice with other health care provider or profession with understanding their job scope) and also item number 29 (able to analyse and interpret findings with the help of other health care professionals) because the items were not clear and precise. Validators also suggested combining the items number 28, 29, 30 into two items to shorten the clinical assessment tool. Suggestions were incorporated in the edited version of PSCAT.

In the perception and satisfaction forms, there was a column for additional feedback. One of the clinical instructors suggested adding the days of absence during clinical placement. Another suggestion was to add one more item in the assessment component “the ability of the student to perform physical examination or relevant tests that focuses on the hands-on skills”. After analysing the feedback and comments given from the validators, further discussion was undertaken with validators.

This lead to a revised version of PSCAT which contained 30 items was developed.

All of the items of PSCAT in the criterion-related category were valid because all of them has achieved equal or >0.9. However, there were some items that were not valid in the face validity category, for example, item number 2 (provides constructive feedback even in a very challenging situation), 3 (respects the ethical, legal, and cultural practices of the client), 4 (respects the confidentiality by providing proper explanation and seeks consent from client before giving any treatment), 6 (demonstrates self-reflection in clinical posting) and 13 (able to select and perform the appropriate test according to client’s condition). There was significant improvement in construct validity index where it had increased by 0.15 from the original 0.82 PSCAT. However, 37% of the items in the content validly category did not have a value of 0.9. Overall index of all type of validity assessed were adequate. The final PSCAT achieved sufficient construct (0.97), face (0.96), criterion-related (0.99) validity but had weak content validity (0.89) [Table/Fig-3].

[Table/Fig-3]:

Comparison of overall validity index in different categories in original version and revised version of PSCAT.

The result showed, approximately 50% of the clinical instructors preferred to have training or detailed guidelines for PSCAT. All of the clinical instructors had agreed that a few features of PSCAT were helpful in providing or receiving feedback on their clinical performance. The feedback denoted that all the items were clear and precise and were suitable for the components to be tested. Among the feedback received from the CI, it was derived that if the calculation of the scores was measured in percent, the tool would be even more user-friendly. The clinical instructors also acknowledged that the time taken to complete PSCAT is acceptable and it is user-friendly [Table/Fig-4].

[Table/Fig-4]:

Overall performance of revised version of PSCAT.

There were 11 clinical instructors involved assessing 40 students in different hospital setting. The [Table/Fig-5] Table shows the ICC for each components and total marks of PSCAT, From the table, it can be seen that most of the component and total score of PSCAT showed moderate reliability which has an ICC of a range of 0.3-0.69. Component of risk management and teamwork had ICC that is lower than 0.30 which means that there is weak inter-rater reliability.

In all of the components, “assessment” has presented the highest inter-rater reliability while component of teamwork showed the weakest of inter-rater reliability.

The items in this PSCAT tool reflected the technical and non-technical skills needed in physiotherapy students to provide better service to their client and health care. A summary of comparison of various tools has been shown in [Table/Fig-6].

As observed in the study, the content validity was not as adequate as the other items in validity; the reason could be attributed to misinterpretation of the questions which stated, “Is it suitable to all levels of assessment?” An in-depth interview was conducted with the CI, where the reason was clarified that the tool has the ability to analyse students’ clinical competence regardless of their year of study.

Overall, result showed that PSCAT is a user-friendly and tool which easy to review. PSCAT has lesser items compare with as Clinical Competency Evaluation Instrument (CCVEI) [15] which contains 40 items.

Summarising the validity (shown in [Table/Fig-6]), the PSCAT had sufficient validity ranging from 0.89 to 0.99. The values of validity were not reported for American Physical Therapist Clinical Performance Instrument (APTCPI) and Canadian Physiotherapy Assessment of Clinical Performance (CPACP). The validity of Assessment of Physiotherapy Practice and Clinical Competency Evaluation Instrument was 0.91 and 0.96 respectively. Though the validity was close to the PSCAT we developed, the components in all the tools did not include students’ feedback. American Physical Therapist Clinical Performance Instrument (APTCPI), Assessment of Physiotherapy Practice, and Clinical Competency Evaluation Instrument were very lengthy and time consuming compared to PSCAT.

For the inter-rater reliability, most of the items showed moderate reliability and few of them shows weak reliability. The factors that lead to moderate and weak reliability could be attributed to the schedules of CI. The CIs clinical schedule was not in a regular basis for the posting area, therefore, some CI had visited a clinical area to supervise the students several time while some CI had limited number of posting schedule with the same students. The specialisation of the CI could have also led to the bias in marking. It was eventually realised that since every clinical instructor has their own specialised field, they may have higher expectation in the students attending such cases of speciality. Their perception of the student’s clinical performance may be different compared with others and therefore may influence their marking method.

Assessment of Physiotherapy Practice (APP) is one of the clinical assessment tools which has achieved 0.92 ICC for overall performance as mentioned by Dalton M [3]. The study provided training to all user of APP to make sure they had the same standard and perception when marking the student’s clinical performance. Other clinical assessment tool had lesser reliability (PT CPI, ACP, CCEVI, CAF). Therefore, it is acknowledged that the reliability of a tool can be improved by giving training to all clinical instructors and to standardise the marking method. Similar process was seen in one of the articles from Canada, where they had provided training before the inter-reliability test and found the tool having good reliability [21]. In the present study too, 50% of participants had mentioned that training or detailed instruction needed for better utilisation of the tool, however, there were other 50% who said that the description provided was adequate to utilise the tool. [Table/Fig-6] compares the various tool explained with the PSCAT developed.

Therefore, the authors believe that the description given in PSCAT is adequate for its utilisation and further reliability study need to be conducted where the contact time of each CI with each student is uniform.

Time constraints and limited number of CI lead to the non-uniform distribution of contact time with students.

Physiotherapy Student Clinical Assessment Tool (PSCAT) is valid tool to analyse student’s clinical performance during their clinical placement. The reliability of the tool is moderate; therefore, further study is recommended to analyse its reliability overcoming the limitation of the study.