In this study, the data contains a randomly selected set of 100 patients from a pool of 200 patients, acquired from July 2009 to December 2010. The reason for selecting a limited population is to avoid manual tracings on 200 more images which is expensive, tedious, and time-consuming. These patients went through the carotid US for both left and right neck and were retrospectively analysed. The study was institutional review board approved and the images were anonymised. Written informed consent was also provided by all the patients.

Carotid ultrasonographic examinations were performed using a scanner (Aplio XV, Aplio XG, Xario, Toshiba, Inc., Tokyo, Japan) equipped with a 7.5 MHz linear array transducer. The average calibration factor for carotid B-mode US scans was 0.0529 mm/pixel. All US scans were performed by a skilled sonographer. The B-mode US scans were supplied by the hospital in non-DICOM format having 10 mm white markers on the left sides of the US scan (see, marked as 1 and 2) [Table/Fig-1b].

The study sample contains 100 patients with 75 males; mean age 68±11 years, range 29-88 years. These patients have a mean HbA1c, Low-Density Lipoproteins (LDL), High-Density Lipoproteins (HDL), and total cholesterol of: 6.40±1.2, 104.60±30.4, 51.5±15.9, and 179.40±35.4 (mg/dL), respectively. In the current population, 53 patients had proximal lesion location, 27 at middle and 20 at a distal location, and 39 were smokers. The grayscale wall characteristics of these diabetic patients showed mild 10% stenosis [23-25].

Two different novice operators measured the average cIMT by using the dedicated AtheroCloud software system. A sample view of the AtheroCloud software is shown in [Table/Fig-1b]. Average cIMT was measured in this study since, it is the most effective biomarker for estimating the plaque burden [9,23]. Average cIMT was computed using Suri’s bidirectional polyline distance method, which is a well-established standardised approach [26,27]. This method computes the average of the shortest distances from one interface (say LI wall) to another interface (say MA wall) and vice-versa. This shortest distance is the perpendicular distance from every point on the interface to the opposite polyline on the other interface. The polyline is a segment which joins two neighbouring points on the opposite interface [26,27]. Since, our study population contained mild stenosis, maximum cIMT was not used [28].

While performing the intra and inter-operator reproducibility analysis, the operators manually computed the calibration of the US scans by calculating the pixel to mm resolution. Each of the two novice operators carried out the analysis three times and no manual adjustments on the LI/MA interfaces were performed.

For validation of the reproducibility analysis, one requires the manual or expert readings. All the manual measurements were conducted using the same AtheroCloud software. The observers looked for the gradient changes at the LI and MA interfaces. The starting brightness of the adventitia layer was used as the border between media layer and adventitia layer, so called MA interface. Corresponding, the gradient change between lumen region and intima layer was used for delineating the LI interface. For US scans that had focal thickening, i.e., cIMT greater than 1.5 mm, the observer could zoom into the region of interest for better visualisation [4]. Both the observers manually traced the LI/MA borders at two separate times. Note that to avoid any biasing all the analysis was repeated with a gap of one month with operators and observers blinded to each other.

The following standard approach was adapted for LI/MA interface preparations for ensuring the unbiased results [22]. The automated and manually delineated LI/MA interfaces had a common support (same length). The LI/MA interface was further smoothed using B-spline software and equal distance sampling (100 equal distance interpolated points) was also performed.

Five kinds of statistical analysis were performed using MedCalc 16.8.4 software (Osteen Belgium). Since, we had more than 30 samples, two-tailed z-test was used. Mann-Whitney and chi-square test were performed to identify significance difference and relationship between the variables, respectively. Kolmogorov-Smirnov test and ANOVA tests were also conducted to evaluate the distribution of automated and manual readings as continuous variables. A level of 5% statistical significance was used throughout this study.

The [Table/Fig-2] shows the results of the intra and inter-operator reproducibility of the auto cIMT computed between all the three automated reading pairs. Our observations show that the mean correlation coefficient for intra/inter-operator reproducibility is close to 1.0 (p<0.001), demonstrating high statistical significance. This proves our hypothesis that even a novice operator can yield high reproducibility when computing automated cIMT readings. The regression plots for intra and inter-operator reproducibility are shown in [Table/Fig-3,4 and 5], respectively.

[Table/Fig-3]:

Correlation coefficient for the reproducibility of AtheroCloud system and observer variability of manual tracings.

[Table/Fig-4]:

Regression plots for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, and a6 shows regression plot between (Auto 1a-Auto 2a), (Auto 1a-Auto 2b), (Auto 1a-Auto 2c), (Auto 1b-Auto 2a), (Auto 1b-Auto 2b), and (Auto 1b-Auto 2c) of cIMT measurements using AtheroCloud software.

[Table/Fig-5]:

Regression plots for inter-operator reproducibility. Figure a1, a2, and a3 shows regression plot between (Auto 1c-Auto 2a), (Auto 1c-Auto 2b), and (Auto 1c-Auto 2c) of cIMT measurements using the AtheroCloud software.

The [Table/Fig-3] also shows the results of the intra and inter-observer variability of the manual LI/MA delineations between two manual reading pairs. Even though, the mean correlation coefficient for intra and inter-observer variability is significantly high, one should note that wherever observer one is involved, the inter-observer analysis has slightly lower (up to 5%) correlations. This is because observer one was less experienced as compared to observer two [29,30]. The corresponding regression plots for intra and inter-observer variability is shown in [Table/Fig-6], respectively.

[Table/Fig-6]:

Regression plots for intra and inter-observer variability. Figure a1, a2 shows regression plot for intra-observer variability between (Manual 1a-Manual 1b), (Manual 2a- Manual 2b), and figure a3, a4, a5, and a6 for inter-observer variability between (Manual 1a- Manual 2c), (Manual 1b- Manual 2a), (Manual 1b- Manual 2b), and (Manual 1b- Manual 2c) of cIMT measurements using AtheroCloud software.

We have evaluated both intra and inter-operator reproducibility and intra and inter- observer variability by analysing 21 (15 reproducibilities and 6 variabilities) Bland-Altman plots. The Bland-Altman plots for; a) intra-operator reproducibility are shown in [Table/Fig-7] and b) inter-operator reproducibility are shown in [Table/Fig-8] and [Table/Fig-9], respectively. Similarly, Bland-Altman plot for c) intra-observer variability and d) inter-observer variability of cIMT measurements is shown in [Table/Fig-10], respectively. Results show a high degree of agreement between auto and manual cIMT readings.

[Table/Fig-7]:

Bland-Altman plots for intra-operator reproducibility. Figure a1, a2, a3, a4, a5, and a6 shows Bland-Altman plot between.

[Table/Fig-8]:

Bland-Altman plots for inter-operator reproducibility. Figure a1, a2, a3, a4, a5, and a6 shows Bland-Altman plot between.

[Table/Fig-9]:

Bland-Altman plots for inter-operator reproducibility. Figure a1, a2, and a3 shows Bland-Altman plot between.

[Table/Fig-10]:

Bland-Altman plots for intra and inter-observer variability. Figure a1, a2 shows Bland-Altman plot for intra-observer variability between (Manual 1a-Manual 1b), (Manual 2a- Manual 2b), and figure a3, a4, a5, and a6 for inter-observer variability between (Manual 1a- Manual 2c), (Manual 1b- Manual 2a), (Manual 1b- Manual 2b), and (Manual 1b- Manual 2c) of cIMT measurements using AtheroCloud software.

Four kinds of performance parameters are computed such as: i) mean absolute difference; ii) mean absolute error; iii) Precision-of-Merit, and iv) Figure-of-Merit which are mathematically expressed as shown in Eq. (A3), (A4), (A5) and (A8) (Appendix A) and their corresponding results are shown in [Table/Fig-11]. For all the automated combinations, lowest mean absolute difference and mean absolute error along with highest Precision-of-Merit and Figure-of-Merit are observed for manual 2. This is because manual 2 is more qualified as compared to manual 1.

The receiver operating characteristic analysis was performed on auto against the manual measurements. Since, two operators underwent the cIMT readings three times, there are two sets of the operators’ combinations: (auto 1a, 1b, 1c) and (auto 2a, 2b, 2c). There are four manual readings, taken from two manual observers who perform LI/MA tracings two times (manual 1a, 1b, 2a, 2b). The [Table/Fig-12] shows eight combinations. The first four curves [Table/Fig-12] (a1), (a2), (a3), and (a4) are drawn for first operator combination (auto 1a, 1b, 1c) and next four curves [Table/Fig-12] (a5), (a6), (a7), and (a8) are drawn for the second operator combination (auto 2a, 2b, 2c) taking the four ground truths (manual 1a, 1b, 2a, 2b), respectively. Using the cIMT risk threshold of 0.9 mm, the area under the curve for all the auto combinations can be seen in [Table/Fig-11] [21]. We detected a high area under the curve for all the possible combinations.

[Table/Fig-12]:

Colour image shows eight comparative receiver operating characteristic curves. The first four curves (Figure. 6 (a1), (a2), (a3), and (a4)) are drawn for first operator combination (Auto 1a, 1b, 1c) and next four curves (Fig. 6 (a5), (a6), (a7), and (a8)) are drawn for the second operator combination (Auto 2a, 2b, 2c) taking the four ground truths (Manual 1a, 1b, 2a, 2b), respectively.

The results of three tests (z-test, chi-square test, and Mann-Whitney) are shown in [Table/Fig-13]. In one-way ANOVA test, the f-value is obtained as 1.21 which is lower than the critical value of (f=1.88) confirming the paired samples to pass the one-way ANOVA test. The normal distribution for the entire auto readings and manual readings are greater than 0.05 confirming the paired samples to pass the Kolmogorov-Smirnov test.

The objective of this study was to demonstrate the intra-and inter-operator reproducibility for cloud-based automated cIMT measurement system using novice operators. For validation of reproducibility analysis, we compare the automated readings against the expert reading via manual delineation of LI/MA interfaces. Further, we performed intra and inter-observer variability analysis of the expert readings.

A review of previous studies for intra and inter-operator reproducibility of cIMT detection techniques is summarised in [Table/Fig-14]. Most of the previous studies used semi-automated method for cIMT measurements, unlike our paradigm which was completely automated. Second, our software system was designed for cloud or web-based settings, unlike other techniques which were desktop-based. Third, most of the studies either did reproducibility or variability, while we demonstrated both reproducibility and variability analysis. Lastly, the fundamental bidirectional distance measurement design for cIMT measurement is accurate, reliable and well published.

The Bland-Altman plots show good concordance for intra and inter-observer cIMT reproducibility and variability measurements with a systematic error limit (1.96 standard deviations of the mean difference). The mean bias between the first and the final 50 patients for both manual observers was consistently reduced. The reduction in manual bias was 0.0690 mm to 0.0677 mm for manual 1 and 0.0365 mm to 0.0354 mm for manual 2, respectively. Our results are in concordance with Vanoli D et al.,’s findings that observer’s experience does improve short-term measurement variability [31].

While performing the manual delineations we did not investigate the effect of usage of different computers, time of the day, observer fatigues, and his mood conditions [32]. These are beyond the scope of this pilot study. In future, like a common carotid artery, one can evaluate the internal carotid artery.

AtheroCloud software showed an excellent intra and inter-operator reproducibility agreement towards cIMT measurements. The comprehensive statistical analysis showed consistency and reliability. Our study also showed a decrease in intra-observer variability measurements with an increase in the experience of our novice observers. The encouraging results of this pilot study showed that the system can be adopted in the clinical setting in routine and/or multicenter pharmaceutical trial modes.

Dr. Jasjit S. Suri has a relationship with AtheroPoint™, Roseville, CA, USA which is dedicated to Atherosclerosis Disease Management, including Cerebrovascular and Cardiovascular imaging.

The data used in this work is IRB approved and Jasjit S Suri (PI of this work) will stand as the guarantor of the dataand statistics used.