Receiver Operator Characteristics (ROC) Analyses of Complete Blood Count (CBC) Delta
S Balamurugan1, V Rohith2
1 Professor, Department of Pathology, Chettinad Academy of Research and Education, Kancheepuram, Tamil Nadu, India.
2 Student, Department of Pathology, Chettinad Academy of Research and Education, Kancheepuram, Tamil Nadu, India.
NAME, ADDRESS, E-MAIL ID OF THE CORRESPONDING AUTHOR: Dr. S Balamurugan, Kelambakkam, Kancheepuram, Tamil Nadu, India.
E-mail: ambikayal@yahoo.co.in
Introduction
Delta check is a quality control tool that involves comparison of lab test result of current sample with that of previous sample from the same patient based on specified criteria. Though chemical analytes have been studied extensively for delta checking, literature is limited for haematology tests.
Aim
In this ‘simulation’ study, we aimed to evaluate the performance characteristics of Complete Blood Count (CBC) tests in delta checking of specimen mix-ups/mis-identification using Receiver Operator Characteristics (ROC) analyses.
Materials and Methods
Retrospective data from hospital patients (aged >18 years) for whom CBC tests were done within 14 days of primary testing were collected. Two groups were created: actual delta and simulated pseudo delta, the latter group simulating the real-time misidentified or mislabeled specimens. Performance characteristics of CBC analytes in detecting specimen mix-ups were evaluated by ROC curves and Areas Under Curves (AUC) using MedCalc statistical software.
Results
AUC values observed for CBC analytes were: MCV (0.90), MCH (0.87) and RDW (0.82), Haematocrit (0.76), Haemoglobin (0.75), RBC count (0.75), Platelet count (0.72), MCHC (0.61) and Total leucocyte count (0.64). Indices of individuality (II) for CBC analytes were: MCH (0.27) <MCV (0.29) <Haemoglobin (0.42) <Haematocrit (0.42) <Platelet count (0.42) <RBC count (0.51) <Total WBC count (0.54) <RDW (0.61) <MCHC (0.88).
Conclusion
MCV and MCH are the most ideal CBC analytes for delta checking of specimen mix-ups/mis-identification as they have low indices of individuality/Reference change values (RCV) and high AUC values. Integration of delta check in the Lab Information System (LIS) is an effective quality practice that can monitor release of erroneous lab results.
Introduction
One of the routinely used tests in Haematology is the Complete Blood Count (CBC). The reports of the complete blood count tests are often essential for the clinicians to monitor overall health, to screen for some diseases, to confirm a diagnosis, and to review the changes following medical treatments [1]. Any discrepancies in the report may lead to false diagnosis or wrong treatment to the patients.
However, due to remarkable advancements in the medical field and the introduction of automated counters, errors have been minimised. But still, errors do exist; which may be classified into Pre-analytical, Analytical, Post-analytical errors. Pre-analytical errors can be due to incorrect specimen collection, improper specimen handling; incorrect specimen labeling; or misidentification of the patient. Post-analytical errors include failure to correct test values for dilution, transcription errors, and misinterpretation of test results [2].
Delta checks involve comparison of lab test results of current sample with that of previous sample from the same patient based on specified criteria. Such changes when they exceed the specified limit (delta limits) may indicate changes in patient clinical condition or pre-analytical, analytical or post-analytical lab errors. Delta check is widely used in clinical laboratories as a patient-based quality assessment tool to detect errors and provide a safety net for identifying testing errors that might otherwise go unnoticed. Contamination of blood by intravenous fluids might be detected only by a delta check alert. Delta check not only acts as a quality control tool in the laboratory but also may allow automated release of results of haematology analysers that pass the delta check [3-5].
Although delta checks are in vogue for over 40 years, the practice of lab haematology has changed of late with the advent of newer and advanced automated haematology analysers. There is limited literature on the use of CBC analytes for delta checking [6]. In this simulation study, we evaluated the performance characteristics of routine CBC analytes for delta checking of specimen mix-ups/mislabeling.
Materials and Methods
This is a retrospective study of data from Chettinad Hospital, Kancheepuram Tamil Nadu, India done from January 2017 to December 2017.
Analytes for delta checking: Following routine CBC parameters were studied: Haemoglobin (Hb), Haematocrit (Hct), RBC Count, Mean Corpuscular Volume (MCV), Mean Corpuscular Haemoglobin (MCH), Mean Corpuscular Haemoglobin Concentration (MCHC), Red-cell distribution width (RDW), Total WBC count (TWBC) and Platelet Count. CBC tests were performed by Coulter LH 750 automated haematology analyser.
Intra and Inter-individual variations for each of the CBC analytes were assessed by calculating their: (1) Index of Individuality (II); and (2) RCV and their values are indicated in [Table/Fig-1].
Indices of Individuality (II) and Reference Change Values (RCV) for Complete Blood Count (CBC) tests.
| CBC test | CVI | CVG | CVA | Index of Individuality (II) | Reference Change Value (RCV) at 98% |
|---|
| Haemoglobin | 2.85 | 6.8 | 1.4 | 0.42 | 11.59 |
| Haematocrit | 2.7 | 6.41 | 1.5 | 0.42 | 11.27 |
| RBC Count | 3.2 | 6.3 | 1.5 | 0.51 | 12.89 |
| MCV | 1.4 | 4.85 | 1.2 | 0.29 | 6.73 |
| MCH | 1.4 | 5.2 | 2.1 | 0.27 | 9.21 |
| MCHC | 1.06 | 1.2 | 1.7 | 0.88 | 7.31 |
| RDW | 3.5 | 5.7 | 1.7 | 0.61 | 12.77 |
| Total WBC Count | 11.4 | 21.3 | 3.8 | 0.54 | 43.84 |
| Platelet Count | 9.1 | 21.9 | 3.4 | 0.42 | 35.44 |
CVI and CVG: Intra and Inter-subject biological variation co-efficients from lliterature [9]
CVA: Analytical variation co-efficient from internal quality data of lab
Data Collection and Grouping
Retrospective data from adult patients (>18 years) for whom repeat CBC testing was ordered within 14 days of first testing were utilised. Two delta groups were created: Group 1 (n=1000): actual delta derived by pairing of consecutive test results from the same patient; Group 2 (n=85): pseudo-delta derived by pairing of consecutive test results from two different patients at the same delta interval. The latter group is a simulation of the real-time misidentified or mislabeled specimens.
Performance Characteristics
Tests with delta values exceeding the reference delta limits were labeled as positive delta tests (‘delta fail’) and those within as negative delta tests (‘delta pass’). Delta positive results in pseudo-delta group were categorised as ‘true positives’ and those in actual delta group as ‘false positives’. Similarly, delta negative results in actual delta group were ‘true negatives’ and those in pseudo-delta group were ‘false negatives’.
Statistical Analysis
ROC curves and AUC for each of the CBC analytes were derived using MedCalc Version 18.11.6 statistical software [7,8].
Results
[Table/Fig-1] lists the indices of individuality (II) and RCV of CBC analytes [9]. [Table/Fig-2] shows the (AUCs of CBC analytes for delta checking of specimen mix-ups. [Table/Fig-3a,b] depict the ROC curves at different cut-off values.
Area Under Curve (AUC) values for Complete Blood Count (CBC) Tests.
| CBC test | Area Under Curve (AUC) | 95% Confidence Interval (CI) |
|---|
| Haemoglobin (Hb) | 0.75 | 0.69 to 0.82 |
| Mean Corpuscular Haemoglobin (MCH) | 0.87 | 0.82 to 0.92 |
| Mean Corpuscular Haemoglobin Concentration (MCHC) | 0.61 | 0.60 to 0.68 |
| Mean Corpuscular Volume (MCV) | 0.90 | 0.85 to 0.94 |
| Haematocrit (Hct) | 0.76 | 0.69 to 0.82 |
| Red Cell Distribution Width (RDW) | 0.82 | 0.77 to 0.88 |
| Platelet Count | 0.72 | 0.66 to 0.77 |
| Red Blood Cell (RBC) Count | 0.75 | 0.68 to 0.81 |
| Total White Blood Cell (TWBC) Count | 0.64 | 0.58 to 0.70 |
a, b: Receiver Operator Characteristics (ROC) curves for CBC analytes.
Discussion
According to Ilzuka Y et al., among the lab mistakes, “specimen mix-ups” are the most common and serious. Blood specimen collected from one patient occasionally gets mislabeled with identifier of another patient leading to the so-called Wrong Blood in Tube Error (WBIT). Delta check (comparing the previous and current test results) is often the only way to detect such errors [10,11]. This study analysed different CBC tests for delta checking of ‘specimen mix-ups’ using ROC analyses.
The ROC curve depicts the trade-off between the sensitivity (true positive rate) and (1- specificity) (false positive rate) across a series of cut-off points. Area under the ROC curve is considered as an effective measure of inherent validity of a diagnostic test. The larger the AUC value (closer to 1.0), the better will be the test performance [7,8].
We utilised ROC analyses to evaluate the CBC analytes for delta checking of specimen mix-ups. AUCs were determined for each CBC analyte using the non-parametric distribution-free Mann Whitney test [7]. Observed AUC value ranges for CBC analytes were:
0.80-0.90 (excellent discriminator): MCV (0.90), MCH (0.87) and RDW (0.82); ROC curves for these CBC tests were closer to upper left corner implying higher test accuracy.
0.70-0.80 (good discriminator): Haematocrit (0.76), Haemoglobin (0.75), RBC count (0.75) and Platelet count (0.72) and
0.60-0.70 (fair discriminator): MCHC (0.61) and Total leucocyte count (0.64); ROC curves for these CBC tests were closer to diagonal (AUC=0.5) implying least efficiency [7,8].
The published literature on delta checking in haematology is limited for comparative analyses. However, our findings are similar to the other few available studies. Ohara T et al., evaluated delta check method to detect artificial ‘mix-ups’ in haematology. They concluded that MCV is the single best marker and recommended investigation when MCV delta was above 4fl. Using computerised algorithms in haematology delta checking, Houwen B et al., found MCV and MCH to be most useful for random error detection. Similar observations were made by Miller I who preferred MCH over MCV as the latter is influenced by extraneous factors. He formulated composite CBC delta (CCD) using weighted deltas of multiple parameters-Hb, MCH, RDW and platelet count. Stijin J et al., in a simulated study of random sample mix-ups concluded RDW to be the single best CBC parameter followed by MCV and MCH [12-15].
CBC analyte chosen for delta checking of specimen mix-ups should have little intra-subject biological variation and wider inter-subject variation. According to Straseski JA et al., and Randell EW et al., an analyte with low index of individuality (II <0.6) and RCV are ideal candidates for delta checking [16,17]. II is calculated as the ratio of the total intra-individual variation to inter-individual biological variation:
II=(CVA2 + CVI2)1/2/CVG,Where, CVA, CVI, and CVG are analytical, within-subject, and between-subject coefficients of variation respectively [18,19].
Indices of individuality (II) for CBC analytes observed in our study are as follows [Table/Fig-3]: MCH (0.27)< MCV (0.29)< Haemoglobin (0.42)< Haematocrit (0.42) <Platelet count (0.42) <RBC count (0.51) <Total WBC count (0.54) <RDW (0.61) <MCHC (0.88). Whereas, MCH and MCV (with low II) are ideal markers for specimen mix-ups; MCHC (with high II) is not a suitable marker for specimen mix-ups.
Reference change value is the value that must be exceeded before a change in consecutive test results is statistically significant at a predetermined probability. It is calculated using following equation:
RCV=2½ × Z × (CVA2 +CVI2)1/2Where, z=z score=1.96; CVA and CVI are analytical and within-subject variation [18,19].
RCVs for CBC (in increasing order) observed in our study were: MCV (6.73) <MCHC (7.31) <MCH (9.21) <Hct (11.27) <Haemoglobin (11.59) <RDW (12.77) <RBC count (12.89) <Platelet count (35.44) <Total WBC Count (43.84). These findings are consistent with those of other authors [18,19].
Limitation
The present study is a ‘simulation’ study which aimed at detecting the efficiency of CBC tests in delta checking of specimen misidentification and this followed up with larger study in real-time scenario can help in drawing definite conclusions.
Conclusion
MCV and MCH are the most ideal CBC analytes for delta checking of specimen mix-ups/ mis-identification as they have low indices of individuality/ RCV and high AUC values. Other CBC analyte deltas may be useful for detecting significant clinical events or other lab errors. Integration of delta check in the LIS is an effective quality practice that monitors release of erroneous lab results.
CVI and CVG: Intra and Inter-subject biological variation co-efficients from lliterature [9]CVA: Analytical variation co-efficient from internal quality data of lab
[1]. Buttarello M, Plebani M, Automated blood cell counts: State of the art Am J Clin Pathol 2008 130(1):104-16.10.1309/EK3C7CTDKNVPXVTN18550479 [Google Scholar] [CrossRef] [PubMed]
[2]. Bonini P, Plebani M, Ceriotti F, Rubboli F, Errors in laboratory medicine Clin Chem 2002 48(5):691-98. [Google Scholar]
[3]. Gruenberg JM, Stein TA, Karger A, Determining the utility of creatinine delta checks: A large retrospective analysis Clinical Biochemistry 2018 53:139-42.10.1016/j.clinbiochem.2018.01.02329402415 [Google Scholar] [CrossRef] [PubMed]
[4]. CLSI Use of Delta Checks in the Medical Laboratory 2016 1st edWayne, PAClinical and Laboratory Standards InstituteCLSI guideline EP33 [Google Scholar]
[5]. Nosanchuk JS, Gottmann AW, CUMs and delta checks. A systematic approach to quality control Am J Clin Pathol 1974 62(5):707-12.10.1093/ajcp/62.5.7074412704 [Google Scholar] [CrossRef] [PubMed]
[6]. Schifman RB, Talbert M, Souers RJ, Delta check practices and outcomes: a q-probes study involving 49 health care facilities and 6541 delta check alerts Arch Pathol Lab Med 2017 141(6):813-23.10.5858/arpa.2016-0161-CP28402169 [Google Scholar] [CrossRef] [PubMed]
[7]. Hajian-Tilaki K, Receiver Operating Characteristic (ROC) curve analysis for medical diagnostic test evaluation Caspian J Intern Med 2013 4(2):627-35. [Google Scholar]
[8]. Kummar R, Indrayan A, Receiver operating characteristic (ROC) curve for medical researchers Indian Pediatr 2011 48:277-89.10.1007/s13312-011-0055-4 [Google Scholar] [CrossRef]
[9]. Ricos C, Alvarez V, Cava F, Garcia-Lario JV, Hernandez A, Jimenez CV, Current databases on biologic variation: pros, cons and progress Scand J Clin Lab Invest 1999 59:491-500.10.1080/0036551995018522910667686 [Google Scholar] [CrossRef] [PubMed]
[10]. Iizuka Y, Kume H, Kitamura M, Multivariate delta check method for detecting specimen mix-up Clin Chem 1982 28:2244-48. [Google Scholar]
[11]. Rosenbaum MW, Baron JM, Using machine learning-based multianalyte delta checks to detect wrong blood in tube errors Am J Clin Pathol 2018 150(6):555-66.10.1093/ajcp/aqy08530169595 [Google Scholar] [CrossRef] [PubMed]
[12]. Ohara T, Itoh K, Revisited univariate delta check method for hematologic laboratories (I)-Usefulness for detection of specimen mix-up in patients with hematologic disorders Rinsho Byori 2002 50(11):1076-78.(Article in Japanese) [Google Scholar]
[13]. Houwen B, Duffin D, Delta checks for random error detection in hematology tests Laboratory Medicine 1989 20(6):410-17.10.1093/labmed/20.6.410 [Google Scholar] [CrossRef]
[14]. Stijn J, Veronique S, Usefulness of delta checks of standard haematological parameters in rapidly detecting sample mix-ups Conference: Scientific Day at Faculty of Pharmaceutical Sciences 2010 [Google Scholar]
[15]. Miller I, Development and evaluation of a logical delta check for identifying erroneous blood count results in a tertiary care hospital Arch Pathol Lab Med 2015 139(8):1042-47.10.5858/arpa.2014-0494-OA26230597 [Google Scholar] [CrossRef] [PubMed]
[16]. Straseski JA, Strathmann FG, Patient data algorithms Clinics in Laboratory Medicine 2013 33(1):147-60.10.1016/j.cll.2012.11.00923331735 [Google Scholar] [CrossRef] [PubMed]
[17]. Randell EW, Yenice S, Delta Checks in the clinical laboratory Critical Reviews in Clinical Laboratory Sciences 2019 56(2):75-97.10.1080/10408363.2018.154053630632840 [Google Scholar] [CrossRef] [PubMed]
[18]. Ko DH, Park H, Hyun J, Kim HS, Park MJ, Shin DH, Utility of reference change values for delta check limits Am J Clin Pathol 2017 48(4):323-29.10.1093/ajcp/aqx08328967949 [Google Scholar] [CrossRef] [PubMed]
[19]. Zhang P, Tang H, Chen K, Chen Y, Xu D, Biological variations of hematologic parameters determined by UniCel DxH 800hematology analyser Arch Pathol Lab Med 2013 137(8):1106-10.10.5858/arpa.2012-0377-OA23899069 [Google Scholar] [CrossRef] [PubMed]