This cross-sectional study was conducted at Tehran University of Medical Sciences (TUMS), School of Medicine, one of Iran’s largest and oldest medical schools, in 2013-2014. The undergraduate medical curriculum at TUMS is divided into three phases; including two and half years of studies in basic sciences, one year of pathophysiology, and three and half years of clinical phase. Since 2006, TUMS commenced to develop and implement a newly revised curriculum for delivering undergraduate medical education. One main feature of the revised curriculum is to pay more attention to the integration of critical thinking as cross-cutting theme for training and assessing medical students [22].

In this study, data were collected from a group of medical students involved in the completion of California Critical Thinking Skills test (CCTST) [23], to measure general critical thinking ability and a researcher-developed test which consists of both open and closed-ended questions. The initial convenience sample was 159 third-year students enrolled in medical undergraduate curriculum at TUMS. Third-year medical students were chosen to participate in this study because they had completed the mandatory critical thinking course. Out of these 159 students, 102 completed both the tests. The reason of having incomplete data was voluntary participation and exclusion of participants who did not complete all tests.

Multi-response format test: A researcher-developed 48-question exam (The Persian version), consisting of short answers and essay questions, Multiple-Choice Questions (MCQ), and True-False (TF) questions, was used to measure critical thinking skills of third-year medical students. While the content of the questions was focused on medical-related subject matter, these were developed to measure a range of critical thinking abilities (assumptions, analysis, inference, evaluation, cognitive biases etc.,). Once complete, the research team reviewed the questions and then sent them to the medical education specialists and critical thinking experts for review of content. Additionally, internal consistency was examined using Cronbach’s alpha (0.69) computed for the total score. We administered the test assessments to medical students. Respondents, with an aim of determining evidence of critical thinking ability in relation to a medical context, using multi-response format test, answered the test consisting of open-ended questions (8 short answer questions and 5 essays) and closed-ended questions (19 MCQ which was followed by 4 or 5 options and 16 TF questions). In a bid to motivate students to participate, a battery of questions was designed which were related to the students’ background, promising them to provide feedback on the results in due course.

California critical thinking skills test [24]: The CCTST questionnaire (The Persian version of CCTST-form B) was used to evaluate general critical thinking skills of medical students. The CCTST contains 34 multiple-choice questions (analysis, inference, evaluation, deductive reasoning, and inductive reasoning) with a correct answer (0-1 score) that targets those general critical thinking skills, considered to be essential elements in higher education [Table/Fig-1]. Reliability and validity of the CCTST assessment has been reported in previous publications [23]. Each correct response was assigned one score and total score on CCTST ranged from a minimum of 0 to a maximum of 34.

The multi-response format test was completed in about 75 minutes. Medical and health problems were chosen due to their multifaceted nature, and the fact that they were related to the students’ background. Students took the test in an exam hall, and each put forth the effort to write down his/her responses. There were tight invigilation rules, and communication and collusion between candidates (by copying, whispering or any kind of signal, exchange of paper or objects) was not allowed. To establish the degree of content independent critical thinking ability, CCTST (Form B) was conducted two weeks later. It took students about 45 minutes to complete the instrument.

This paper is a part of a thesis submitted for the degree of PhD in Medical education. Research Ethical Committee of Tehran University of Medical Sciences approved the study. Students received a full explanation prior to participation which was voluntary and anonymity was maintained and guaranteed.

For all statistical analyses SPSS 22.0 was used. Descriptive statistics (means, standard deviations, and frequencies) were used to describe the sample. Reliability of the researcher-developed test was examined using Cronbach’s alpha. Pearson correlation co-efficient was employed to explore the association between the total scores of tests and subtests.

One hundred and fifty nine students were included in the study which comprised 81 females (51%) and 78 males (49%) with an age range of 20±2.8 years (mean 21.2 years). One hundred two medical students completed both the CCTST and multi-response format test. Data from matched pairs were analysed. The response rate was 64.1%. The possible range for instrument scores is shown in [Table/Fig-1].

Pearson correlation was computed to explore the association between the CCTST and multi-response format test subscales. [Table/Fig-2,3] present the correlation between multi-response format test and CCTST scores and subtests scores. While there were positive correlation with all multi-response format test subtests, the relationship of MCQ (r=0.82) and essay questions (r=0.77) was strongest. CCTST total score correlated best with total scores for MCQ questions (r=0.25) [Table/Fig-2].

The significant positive correlations between multi-response format test and CCTST’s subscales were seen in analysis, evaluation, inference and inductive reasoning. Significant correlations were found between CCTST total score and scores for nearly all its subscales. The correlations between CCTST score and its subscales scores were between 0.56 and 0.74 [Table/Fig-3].

In general, descriptive analysis of CCTST and multi-response format test total scores are presented in [Table/Fig-4]. There was a weak positive correlation between CCTST with multi-response format test total scores, which was not significant (r=0.22, p=0.06).

The combination of two-response format (open and closed-ended questions) into one test is viewed as the current trend in the assessment of critical thinking. Ku KY has mentioned that any measurement of critical thinking that utilises a single-response format is neither sufficient in reflecting students’ true critical thinking ability, nor compatible with the conceptualisation of critical thinking [25]. In this study, closed and open-ended questions were utilised to indicate a better understanding of medical students’ critical thinking abilities while facing problems in the field of medicine.

The results showed changes in the assessment of medical students’ critical thinking by multi-response format test consisting of MCQ, essay, TF and short-answer questions. Our results showed that students’ scores of MCQ and essay questions significantly correlated with total scores of medical students’ critical thinking abilities. This does confirm that well-constructed MCQ can also assess higher levels of cognitive skills of medical students [26]. These findings that MCQ and essay can be used to assess critical thinking were similar to those found in previous studies. The Halpern Critical Thinking Assessment Using Everyday Situations (HCTAES) is a general test that incorporates both multiple-choice and essay questions into a single test [19]. Stein and Haynes developed a measurement tool including standardised multiple-choice tests, essay tests, and faculty-developed rubrics for evaluating student work [27].

In current the study, the mean CCTST score for medical students was 18.25. Athari Z et al., and Haghani F et al., have reported marginally lower means CCTST in medical students [28,29]. Although, the mean score gained in the area of critical thinking proved not to be high enough, by the medical students in our study, it was higher than the national scores as compared to the similar international studies. This can reflect the effects of critical thinking training program in renewal of medical curriculum.

In addition, a positive correlation was observed between total scores of CCTST and multi-response format test during the study, though being quite small. These findings indicate that performance on the multi-response format test in medicine is little related to general measures of critical thinking skills. In our study, the CCTST was used to measure critical thinking as a general competency [30], while the multi-response format test was used to measure critical thinking in medicine discipline. This can reflect that medical students’ multi-response test score is not explained by these other critical thinking tests, which rely on multiple-choice questions [27].

Performance on the multi-response format test generally correlates with scores on analysis, evaluation, inference, and inductive reasoning of CCSTS. However, the relative low correlation between the multi-response format test and subtests’ CCTST scores might suggest that the two tests assess different components of critical thinking via different test contents.

Several limitations of this study prevent cautious transfer of the findings to other contexts. The time needed to complete the two tests was long. This may have caused the unwillingness of some medical students to answer the tests properly. Future researches are necessary to validate the findings of this work.

This study highlights the importance of considering the right format of response in critical thinking assessment. An important feature of the multi-response format test is that it relies primarily on different types of methods to assess critical thinking, unlike many standardised tests that rely on multiple-choice questions. It consists of MCQ, essay, short answer and true-false questions for evaluating medical students’ critical thinking. The results showed that medical students response to essay and well-constructed MCQ questions reveal their critical thinking when encountering medical problems.