INTRODUCTION
Carotid intima-media thickness (CIMT) measured by ultrasound has been known as a marker for atherosclerosis progression and regression [1-3]. Increased CIMT and plaque presence are associated with many vascular risk factors and future cardiovascular diseases [4-7]. Ultrasonography is free from radiation exposure or adverse effect, compared to other invasive methods, and operators can get assistance from automated computer-based CIMT and plaque measurement system with real-time image recording [8-11].
However, using CIMT or plaque measurements in multicenter cohort studies may cause various problems. Different centers may have different operators or ultrasound machines. Such operator dependency in ultrasonography has been a well-known problem, and thus, the standardized measurement protocol is needed, especially in multi-center cohort studies [12,13].
There are several previous studies about the inter-rater reliability of CIMT measurement and carotid plaque presence in worldwide [12,14-17]. However, to our knowledge, there are few studies in East Asia. We investigated the reliability on the CIMT and carotid plaque presence in our multi-center cohort study using the predetermined protocol for operators.
METHODS
Participants
The Cardiovascular and Metabolic Disease Etiology Research Center (CMERC) has screened more than 4,800 people during 2013-2015 for a cohort, and plans to recruit 12,000 participants by 2018 at three clinical centers. For the standardized ultrasonographic measurement of CIMT and plaque among the centers, we recruited 10 volunteers each year, and the inter-rater reliability test was done using the same predetermined protocol at all clinical centers. The present analysis is done with 20 people (10 males; 10 females) aged 37-64 years. CIMT was measured for each participant repeatedly at the three centers. This study was approved by the Yonsei university health system, Severance hospital, institutional review board (4-2013-0661, 4-2013-0581), Ajou university hospital institutional review board (AJIRB-BMR-SUR-13-272), and all participants provided written informed consent.
CIMT and other measurements
The CIMT and carotid plaque were measured according to a predetermined protocol. Participants were in a supine position with electrocardiography monitoring patch attached. A participant’s neck was extended, with his/her head turned 30-degree toward the opposite direction of measurement. Then, the CIMT was measured at the participant’s right and left distal common carotid artery (CCA) by long axis view, and an image was automatically taken during R wave in electrocardiography. Ultrasound transducer frequency range was 5-13 MHz, and depth setting of ultrasound machine was 4 cm. After taking an image, an operator selected 1 cm width region between bulb area and CCA, and then maximum and mean values of each CIMT were calculated by automatic methods in ultrasonography machine at the far wall of distal CCA. Each center used different machines for this analysis (Center 1: Accuvix XG, Samsung Medison, Seoul, Korea; Center 2: Logiq S8 ECG module, GE Healthcare, Chalfont St. Giles, UK; Center 3: EKO7 Cardiovascular Ultrasound System, Samsung Medison, Seoul, Korea). There were two operators at each center: a registered nurse and an emergency medical technician at center 1, and two sonographers each at center 2 and 3. All operators have at least three month experience in CIMT measurements. Carotid plaque was measured at both right and left common, external, internal carotid arteries and bulb areas by an operator. The operator determines the presence of carotid plaque and the number of plaques using the predetermined protocol. Plaque presence was assumed when focal thickening was observed, namely in case that CIMT was equal to or greater than 50% of the surrounding CIMT or ≥1.5 mm in its size. For ICA, only ≥1.5 mm in its size was regarded as focal thickening. If a plaque was observed during CIMT measurement, the thickness of plaque was also recorded.
Statistical analysis
The unit of analysis is an individual artery (total = 40). The characteristics of continuous variables were expressed as both the mean with standard deviation (SD) and the median with interquartile range, while those of categorical variables were expressed as the number with percent. Bland Altman plot depicted the maximum and mean values of CIMT between centers. Kolmogorov-Smirnov tests were used to test normality, while the paired t-test and Wilcoxon signed rank test were used for testing significance. The inter-rater reliability of CIMT was measured by an intraclass correlation coefficient (ICC) and Spearman correlation analysis [18], where ICC is defined as a ratio of the between-participant variance divided by the sum of between-participant variance and error in our study. Here, a rater means each clinical center for CMERC study. Out of various versions of the ICC, we used ICC (3,1) in SPSS package since it is considered to be appropriate for our study [19]. To use ICC (3,1), we modeled both participant and center effects, and assumed the random effect for participants and the fixed effect for centers, respectively. However, we also used ICC (2,1) in addition to ICC (3,1) since ICC (3,1) can be used for evaluating consistency only. ICC (2,1) uses the same assumption as ICC (3,1), except for the random effect for centers of ICC (3,1). ICC values were represented separately for right and left carotid arteries. For plaque presence agreement between centers, percent agreement and kappa statistics were used. In this case, the unit of analysis is each segment of carotid artery (total =160). The agreement of the number of carotid plaques was evaluated using weighted kappa statistics. Fleiss kappa and Kendall’s coefficient of concordance were used for overall agreement of plaque presence and the number of plaques by SAS MAGREE macro [20]. All analyses were done by SPSS 21.0 (SPSS Inc., Chicago, IL, USA) and SAS 9.4 (SAS Institute Inc., Cary, NC, USA), and the p-values below 0.05 were considered significant.
RESULTS
General characteristics of participants are shown in Table 1. Center 3 shows higher CIMT value compared to center 1, and center 1 reported more plaque presence than other centers. Figure 1 shows the mean difference and SD between two centers for maximum and mean CIMT values. The difference between two measurements was normally distributed at all possible six pairs. The mean difference was ranged from -0.060 to 0.117 mm, and statistical significant differences existed except for the mean value of CIMT between center 1 and 2. In Bland Altman plot, there was no significant linear relationship between each possible pairs, and most of observed values were distributed within mean difference±1.96 SD ranges.
Table 2 shows the reliability of CIMT measurement. The ICC (3,1) value was 0.647 (95% CI: 0.487-0.779) for CIMT maximum and was 0.758 (0.632-0.854) for CIMT mean. Based on the separated analysis by right and left carotid arteries, the ICC (3,1) of maximum CIMT value is 0.631 for right carotid artery and 0.656 for left. Meanwhile, the ICC (3,1) of mean CIMT is 0.789 for right carotid artery and 0.720 for left, respectively. The value of ICC (2,1) is generally lower than that of ICC (3,1). Spearman correlation coefficients between centers were shown at Table 3. Most of the coefficients were significant and ranged 0.337 to 0.862. The coefficient between center 1 and 2 was lower than that of other pairs, and the coefficients for maximum CIMT values were lower than those for mean CIMT values.
Table 4 shows the reliability of plaque measurement. Percent agreement was ranged from 88.8 to 96.3%. Kappa statistics of plaque presence between two centers were 0.304 (center 1 and 2), 0.507 (center 1 and 3), and 0.606 (center 2 and 3), respectively. Weighed kappa statistics for the number of plaques were 0.363 (center 1 and 2), 0.502 (center 1 and 3), and 0.716 (center 2 and 3). Fleiss kappa of plaque presence and Kendall’s coefficient of concordance of the number of plaques were 0.445 and 0.644, respectively.
DISCUSSION
Based on sample participants and measurement protocols, we assessed the reliability of CIMT and plaque presence between three clinical centers. We regarded an ICC in the range 0.4-0.75 as ‘fair to good’ and > 0.75 as ‘excellent’ [21]. In our results, the mean CIMT values had an ‘excellent’ range of ICC, while the maximum CIMT values had a ‘fair to good’ range. It is probably because the mean CIMT value is less affected by an outlier than the maximum value. We attribute these results to our protocol that allows adding the plaque thickness to CIMT if plaque exists in distal CCA. This tendency was consistent with the results from other articles [14,17]. Spearman correlation showed similar tendency, too. We could not conclude the rating of each center due to the lack of absolute right CIMT value, but from these results, center 3 was likely to provide a highly reliable value. In the separate analysis done by right and left carotid arteries, only the mean CIMT value of right carotid artery had an ‘excellent’ range, while other CIMT values had a ‘fair to good’ range. Still, the mean CIMT values tend to have a higher ICC value. The difference of ICC values between right and left carotid arteries might be caused by hand position of an operator during measuring CIMT. Previous study reported that specific hand positions require practice for reliable measurement, and ICC value of right CIMT has higher value than left carotid artery [9]. We used the ICC (3,1) for evaluating the inter-rater reliability of the CIMT value. Compared to ICC (2,1), there was not much discrepancy, especially in the mean value of right CIMT. From the point of data integration among CMERC clinical centers, the CIMT values from different centers that use different machines need to be integrated using a transformation formula for adjusting values from large number of participants.
The kappa values ranged from 0.21 to 0.40, from 0.41 to 0.60, and from 0.61 to 0.80 were defined as fair, moderate, and substantial agreement, respectively [22]. Kappa statistics of some pairs showed substantial agreement, but the others did not. First, it is because the carotid plaque measurement is determined only by operators in our protocol. Operators in center 2 and 3 are sonographers, while operators in center 1 were a registered nurse and an emergency medical technician in our study. This operator dependency might affect these results. Second, the low frequency of carotid plaque might affect kappa statistics [23]. Since the present study was done with apparently healthy volunteers, the plaque prevalence seems to below. If we considered the simple agreement percent of plaque presence, it ranged from 88.8 to 96.3% in all possible matches, even though it could demonstrate inappropriate results due to not considering the agreement occurred by chance.
Our study has some limitations. First, we did not evaluate within-participant effects. Depending on the time period, the individual CIMT might be changed. There was time difference of several months between measurements. To measure CIMT and carotid plaque, the first ten participants had examinations from January till February, 2014 at center 2 and in February, 2014 at center 1 and 3, while the other ten participants did from September till November, 2014 at center 2 and in January, 2015 at center 1 and 3, respectively. Second, within-center discordance should be evaluated for more desirable results. However, we could not get data to examine within-center discordance, since at each center in our study, two operators measured CIMT by dividing study participants into two groups. Third, the evaluation for each of the center ratings could not be done, since gold standard values of CIMT were not available. Instead, we used the Spearman correlation coefficient between two centers for comparison. The lack of confirmed diagnosis for plaque presence was another limitation, too. Carotid plaque presence needs to be confirmed by radiologists, or be measured by automated methods with morphometric assessment [24]. Finally, the high risk populations were not included in the reliability test. Thus, our results can be applied for the cardiovascular disease-free population only.
CONCLUSION
Inter-rater reliability of CIMT measurements, especially in case of the mean value of CIMT, turned out to be ‘excellent’ for our three clinical centers of CMERC. The results of this study showed that automatic CIMT measurements with a predetermined protocol were reliable in multicenter study, even though the operator dependency existed in ultrasonographic measurements. Recently, a number of large cohort studies are being conducted in different centers. So, our results can be helpful to researchers that work with data integration in the large cohort studies. Meanwhile, the agreement of carotid plaque measurement turned out not to be sufficient, which needs further research in the future.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation 
Print 
