| Home | E-Submission | Sitemap | Editorial Office
top_img
J Health Info Stat > Volume 41(1); 2016 > Article
Lee, Choi, Shim, Kim, Park, and Kim: 다기관 코호트 연구에서 경동맥 내막-중막 두께 측정의 측정자간 신뢰도 평가

Abstract

Objectives:

Carotid intima-media thickness (CIMT) and the presence of carotid artery plaque are widely used as preclinical markers of atherosclerosis. Due to operator dependency in measuring CIMT, it is important to evaluate the reliability of measuring CIMT and plaque between centers in a multicenter study. The purpose of this study is to evaluate the inter-rater reliability of CIMT and plaque presence among three clinical centers of the Cardiovascular and Metabolic Disease Etiology Research Center (CMERC).

Methods:

Twenty people without known cardiovascular disease (age 37-64) were enrolled during 2014-2015, and their left and right carotid arteries were examined repeatedly with ultrasonography for CIMT measurements at three clinical centers according to a predetermined protocol. Maximum and mean values of CIMT at distal common carotid artery were recorded. Plaque presence at a carotid artery was checked by an operator. The reliability of CIMT and carotid plaque presence was assessed using an intraclass correlation coefficient (ICC) and kappa statistics, respectively.

Results:

Calculated ICC was 0.647 (95% CI: 0.487-0.779) for maximum CIMT, and 0.758 (95% CI: 0.632-0.854) for mean CIMT. In Bland Altman plot, most observed values were distributed within mean difference ±1.96 SD ranges. Kappa statistics of plaque presence between two centers were 0.304 (center 1 and 2), 0.507 (center 1 and 3), and 0.606 (center 2 and 3), respectively, while Fleiss kappa for overall agreement was 0.445.

Conclusions:

The inter-rater reliability of CIMT measurements among three clinical centers turned out to be high, and the agreement of measuring carotid plaque presence was fair.

요약

목적:

경동맥 내막-중막 두께와 경동맥 경화반의 존재유무는 죽상동맥경화증의 임상 전단계를 나타내는 지표로 널리 사용되고 있다. 경동맥 내막-중막 두께를 측정할 때의 측정자 의존성 때문에, 다기관 연구에서는 경동맥 내막-중막 두께와 경화반 측정의 기관간 신뢰도를 확인하는 것이 중요하다. 이 연구의 목적은 심뇌혈관 및 대사질환원인 연구센터에 속해 있는 세 임상기관 사이의 경동맥 내막-중막 두께와 경화반 측정의 측정자간 신뢰도를 평가하는 것이다.

방법:

심뇌혈관 질환 과거력이 없는 사람 20명이 2014-2015년 사이에 이 연구에 참여하였고(연령 37-64세), 미리 정해진 프로토콜에 따라 연구참여자의 좌, 우 경동맥을 세 임상기관에서 반복적으로 측정하였다. 총 경동맥의 원위부에서 측정한 경동맥 내막-중막 두께의 최대값과 평균값을 기록하였다. 경동맥에서의 경화반 존재유무는 측정자에 의해 확인되었다. 경동맥 내막-중막 두께와 경화반 존재유무의 신뢰도를 급내상관계수와 카파 통계량을 통해 각각 평가하였다.

결과:

계산된 급내상관계수는 최대 경동맥 내막-중막 두께를 평가하였을 때 0.647이었고 (95% CI: 0.487-0.779), 평균 경동맥 내막-중막 두께를 평가하였을 때 0.758 (95% CI: 0.632-0.854) 이었다. Bland Altman plot에서, 관측치의 대부분은 평균의 차이에서 ±1.96 표준편차 사이에 분포하였다. 각 기관 사이의 경화반 존재유무에 대한 카파 통계량은 0.304 (기관 1과 2), 0.507 (기관 1과 3), 0.606 (기관 2와 3)이었다. 전반적인 일치를 평가하는 Fleiss카파값은 0.445였다.

결론:

세 임상기관 사이의 경동맥 내막-중막 두께의 측정자간 신뢰도는 훌륭하였으며, 경화반 존재유무에 대한 신뢰도는 적정하였다.

INTRODUCTION

Carotid intima-media thickness (CIMT) measured by ultrasound has been known as a marker for atherosclerosis progression and regression [1-3]. Increased CIMT and plaque presence are associated with many vascular risk factors and future cardiovascular diseases [4-7]. Ultrasonography is free from radiation exposure or adverse effect, compared to other invasive methods, and operators can get assistance from automated computer-based CIMT and plaque measurement system with real-time image recording [8-11].
However, using CIMT or plaque measurements in multicenter cohort studies may cause various problems. Different centers may have different operators or ultrasound machines. Such operator dependency in ultrasonography has been a well-known problem, and thus, the standardized measurement protocol is needed, especially in multi-center cohort studies [12,13].
There are several previous studies about the inter-rater reliability of CIMT measurement and carotid plaque presence in worldwide [12,14-17]. However, to our knowledge, there are few studies in East Asia. We investigated the reliability on the CIMT and carotid plaque presence in our multi-center cohort study using the predetermined protocol for operators.

METHODS

Participants

The Cardiovascular and Metabolic Disease Etiology Research Center (CMERC) has screened more than 4,800 people during 2013-2015 for a cohort, and plans to recruit 12,000 participants by 2018 at three clinical centers. For the standardized ultrasonographic measurement of CIMT and plaque among the centers, we recruited 10 volunteers each year, and the inter-rater reliability test was done using the same predetermined protocol at all clinical centers. The present analysis is done with 20 people (10 males; 10 females) aged 37-64 years. CIMT was measured for each participant repeatedly at the three centers. This study was approved by the Yonsei university health system, Severance hospital, institutional review board (4-2013-0661, 4-2013-0581), Ajou university hospital institutional review board (AJIRB-BMR-SUR-13-272), and all participants provided written informed consent.

CIMT and other measurements

The CIMT and carotid plaque were measured according to a predetermined protocol. Participants were in a supine position with electrocardiography monitoring patch attached. A participant’s neck was extended, with his/her head turned 30-degree toward the opposite direction of measurement. Then, the CIMT was measured at the participant’s right and left distal common carotid artery (CCA) by long axis view, and an image was automatically taken during R wave in electrocardiography. Ultrasound transducer frequency range was 5-13 MHz, and depth setting of ultrasound machine was 4 cm. After taking an image, an operator selected 1 cm width region between bulb area and CCA, and then maximum and mean values of each CIMT were calculated by automatic methods in ultrasonography machine at the far wall of distal CCA. Each center used different machines for this analysis (Center 1: Accuvix XG, Samsung Medison, Seoul, Korea; Center 2: Logiq S8 ECG module, GE Healthcare, Chalfont St. Giles, UK; Center 3: EKO7 Cardiovascular Ultrasound System, Samsung Medison, Seoul, Korea). There were two operators at each center: a registered nurse and an emergency medical technician at center 1, and two sonographers each at center 2 and 3. All operators have at least three month experience in CIMT measurements. Carotid plaque was measured at both right and left common, external, internal carotid arteries and bulb areas by an operator. The operator determines the presence of carotid plaque and the number of plaques using the predetermined protocol. Plaque presence was assumed when focal thickening was observed, namely in case that CIMT was equal to or greater than 50% of the surrounding CIMT or ≥1.5 mm in its size. For ICA, only ≥1.5 mm in its size was regarded as focal thickening. If a plaque was observed during CIMT measurement, the thickness of plaque was also recorded.

Statistical analysis

The unit of analysis is an individual artery (total = 40). The characteristics of continuous variables were expressed as both the mean with standard deviation (SD) and the median with interquartile range, while those of categorical variables were expressed as the number with percent. Bland Altman plot depicted the maximum and mean values of CIMT between centers. Kolmogorov-Smirnov tests were used to test normality, while the paired t-test and Wilcoxon signed rank test were used for testing significance. The inter-rater reliability of CIMT was measured by an intraclass correlation coefficient (ICC) and Spearman correlation analysis [18], where ICC is defined as a ratio of the between-participant variance divided by the sum of between-participant variance and error in our study. Here, a rater means each clinical center for CMERC study. Out of various versions of the ICC, we used ICC (3,1) in SPSS package since it is considered to be appropriate for our study [19]. To use ICC (3,1), we modeled both participant and center effects, and assumed the random effect for participants and the fixed effect for centers, respectively. However, we also used ICC (2,1) in addition to ICC (3,1) since ICC (3,1) can be used for evaluating consistency only. ICC (2,1) uses the same assumption as ICC (3,1), except for the random effect for centers of ICC (3,1). ICC values were represented separately for right and left carotid arteries. For plaque presence agreement between centers, percent agreement and kappa statistics were used. In this case, the unit of analysis is each segment of carotid artery (total =160). The agreement of the number of carotid plaques was evaluated using weighted kappa statistics. Fleiss kappa and Kendall’s coefficient of concordance were used for overall agreement of plaque presence and the number of plaques by SAS MAGREE macro [20]. All analyses were done by SPSS 21.0 (SPSS Inc., Chicago, IL, USA) and SAS 9.4 (SAS Institute Inc., Cary, NC, USA), and the p-values below 0.05 were considered significant.

RESULTS

General characteristics of participants are shown in Table 1. Center 3 shows higher CIMT value compared to center 1, and center 1 reported more plaque presence than other centers. Figure 1 shows the mean difference and SD between two centers for maximum and mean CIMT values. The difference between two measurements was normally distributed at all possible six pairs. The mean difference was ranged from -0.060 to 0.117 mm, and statistical significant differences existed except for the mean value of CIMT between center 1 and 2. In Bland Altman plot, there was no significant linear relationship between each possible pairs, and most of observed values were distributed within mean difference±1.96 SD ranges.
Table 2 shows the reliability of CIMT measurement. The ICC (3,1) value was 0.647 (95% CI: 0.487-0.779) for CIMT maximum and was 0.758 (0.632-0.854) for CIMT mean. Based on the separated analysis by right and left carotid arteries, the ICC (3,1) of maximum CIMT value is 0.631 for right carotid artery and 0.656 for left. Meanwhile, the ICC (3,1) of mean CIMT is 0.789 for right carotid artery and 0.720 for left, respectively. The value of ICC (2,1) is generally lower than that of ICC (3,1). Spearman correlation coefficients between centers were shown at Table 3. Most of the coefficients were significant and ranged 0.337 to 0.862. The coefficient between center 1 and 2 was lower than that of other pairs, and the coefficients for maximum CIMT values were lower than those for mean CIMT values.
Table 4 shows the reliability of plaque measurement. Percent agreement was ranged from 88.8 to 96.3%. Kappa statistics of plaque presence between two centers were 0.304 (center 1 and 2), 0.507 (center 1 and 3), and 0.606 (center 2 and 3), respectively. Weighed kappa statistics for the number of plaques were 0.363 (center 1 and 2), 0.502 (center 1 and 3), and 0.716 (center 2 and 3). Fleiss kappa of plaque presence and Kendall’s coefficient of concordance of the number of plaques were 0.445 and 0.644, respectively.

DISCUSSION

Based on sample participants and measurement protocols, we assessed the reliability of CIMT and plaque presence between three clinical centers. We regarded an ICC in the range 0.4-0.75 as ‘fair to good’ and > 0.75 as ‘excellent’ [21]. In our results, the mean CIMT values had an ‘excellent’ range of ICC, while the maximum CIMT values had a ‘fair to good’ range. It is probably because the mean CIMT value is less affected by an outlier than the maximum value. We attribute these results to our protocol that allows adding the plaque thickness to CIMT if plaque exists in distal CCA. This tendency was consistent with the results from other articles [14,17]. Spearman correlation showed similar tendency, too. We could not conclude the rating of each center due to the lack of absolute right CIMT value, but from these results, center 3 was likely to provide a highly reliable value. In the separate analysis done by right and left carotid arteries, only the mean CIMT value of right carotid artery had an ‘excellent’ range, while other CIMT values had a ‘fair to good’ range. Still, the mean CIMT values tend to have a higher ICC value. The difference of ICC values between right and left carotid arteries might be caused by hand position of an operator during measuring CIMT. Previous study reported that specific hand positions require practice for reliable measurement, and ICC value of right CIMT has higher value than left carotid artery [9]. We used the ICC (3,1) for evaluating the inter-rater reliability of the CIMT value. Compared to ICC (2,1), there was not much discrepancy, especially in the mean value of right CIMT. From the point of data integration among CMERC clinical centers, the CIMT values from different centers that use different machines need to be integrated using a transformation formula for adjusting values from large number of participants.
The kappa values ranged from 0.21 to 0.40, from 0.41 to 0.60, and from 0.61 to 0.80 were defined as fair, moderate, and substantial agreement, respectively [22]. Kappa statistics of some pairs showed substantial agreement, but the others did not. First, it is because the carotid plaque measurement is determined only by operators in our protocol. Operators in center 2 and 3 are sonographers, while operators in center 1 were a registered nurse and an emergency medical technician in our study. This operator dependency might affect these results. Second, the low frequency of carotid plaque might affect kappa statistics [23]. Since the present study was done with apparently healthy volunteers, the plaque prevalence seems to below. If we considered the simple agreement percent of plaque presence, it ranged from 88.8 to 96.3% in all possible matches, even though it could demonstrate inappropriate results due to not considering the agreement occurred by chance.
Our study has some limitations. First, we did not evaluate within-participant effects. Depending on the time period, the individual CIMT might be changed. There was time difference of several months between measurements. To measure CIMT and carotid plaque, the first ten participants had examinations from January till February, 2014 at center 2 and in February, 2014 at center 1 and 3, while the other ten participants did from September till November, 2014 at center 2 and in January, 2015 at center 1 and 3, respectively. Second, within-center discordance should be evaluated for more desirable results. However, we could not get data to examine within-center discordance, since at each center in our study, two operators measured CIMT by dividing study participants into two groups. Third, the evaluation for each of the center ratings could not be done, since gold standard values of CIMT were not available. Instead, we used the Spearman correlation coefficient between two centers for comparison. The lack of confirmed diagnosis for plaque presence was another limitation, too. Carotid plaque presence needs to be confirmed by radiologists, or be measured by automated methods with morphometric assessment [24]. Finally, the high risk populations were not included in the reliability test. Thus, our results can be applied for the cardiovascular disease-free population only.

CONCLUSION

Inter-rater reliability of CIMT measurements, especially in case of the mean value of CIMT, turned out to be ‘excellent’ for our three clinical centers of CMERC. The results of this study showed that automatic CIMT measurements with a predetermined protocol were reliable in multicenter study, even though the operator dependency existed in ultrasonographic measurements. Recently, a number of large cohort studies are being conducted in different centers. So, our results can be helpful to researchers that work with data integration in the large cohort studies. Meanwhile, the agreement of carotid plaque measurement turned out not to be sufficient, which needs further research in the future.

ACKNOWLEDGEMENTS

This study was supported by a grant from the Korean Health Technology R&D Project, Ministry of Health & Welfare, Korea (HI13C0715).

NOTES

No potential conflict of interest relevant to this article was reported.

Figure 1.
Bland-Altman plots showing mean difference between center for CIMT measurement. CIMT, carotid intima-media thickness. The difference of maximum and mean CIMT measurements between two centers is presentated according to the mean of CIMT measurement of two center. A solid line represents mean difference while a dotted line represents 1.96 SD limit. (A) The difference of maximum CIMT between center 1 and 2; (B) The difference of mean CIMT between center 1 and 2; (C) The difference of maximum CIMT between center 2 and 3; (D) The difference of mean CIMT between center 2 and 3; (E) The difference of maximum CIMT between center 3 and 1; (F) The difference of mean CIMT between center 3 and 1.
jhis-41-1-49f1.tif
Table 1.
General characteristics of 20 study participants
Characteristics Mean SD Median IQR
Age (y) 46.8 7.5 44.5 11.5
Female sex (number, %) 10 50.0
Disease history
Hypertension (number, %) 3 15.0
Diabetes mellitus (number, %) 0 0.0
Dyslipidemia (number, %) 5 25.0
Maximum CIMT (mm1)
 Center 1 0.675 0.118 0.673 0.163
 Center 2 0.732 0.109 0.720 0.120
 Center 3 0.792 0.132 0.772 0.158
Mean CIMT (mm1)
 Center 1 0.592 0.102 0.582 0.130
 Center 2 0.597 0.092 0.560 0.135
 Center 3 0.627 0.113 0.613 0.172
Carotid plaque2
 Center 1 (number, %) 19 11.9
 Center 2 (number, %) 9 5.6
 Center 3 (number, %) 7 4.4

SD, standard deviation; CIMT, carotid intima-media thickness; HDL, high density lipoprotein.

1 Unit of analysis: each artery (n=40);

2 Unit of analysis: each segment of carotid artery (n=160).

Table 2.
Inter-rater reliability in measuring carotid Intima-media thickness among centers using intraclass correlation
Characteristics Number of arteries Maximum CIMT
Mean CIMT
ICC (3, 1) (95% CI) ICC (2, 1) (95% CI) ICC (3, 1) (95% CI) ICC (2, 1) (95% CI)
Overall 40 0.647 (0.487, 0.779) 0.528 (0.239, 0.728) 0.758 (0.632, 0.854) 0.737 (0.596, 0.842)
Right carotid artery 20 0.631 (0.389, 0.816) 0.525 (0.212, 0.763) 0.789 (0.616, 0.902) 0.789 (0.619, 0.902)
Left carotid artery 20 0.656 (0.422, 0.831) 0.521 (0.179, 0.768) 0.720 (0.511, 0.866) 0.679 (0.441, 0.846)

CIMT, carotid intima-media thickness; ICC, intraclass correlation coefficients; CI, confidence interval.

Table 3.
Inter-rater reliability in measuring carotid Intima-media thickness between centers using spearman correlation
Characteristics Center compared Maximum CIMT
Mean CIMT
Spearman correlation coefficient p-value Spearman correlation coefficient p-value
Overall 1 vs. 2 0.539 < 0.001 0.698 < 0.001
1 vs. 3 0.695 < 0.001 0.801 < 0.001
2 vs. 3 0.686 < 0.001 0.829 < 0.001
Right carotid artery 1 vs. 2 0.337 0.147 0.639 0.002
1 vs. 3 0.705 0.001 0.862 < 0.001
2 vs. 3 0.506 0.023 0.657 0.002
Left carotid artery 1 vs. 2 0.585 0.007 0.677 0.001
1 vs. 3 0.685 0.001 0.682 0.001
2 vs. 3 0.695 0.001 0.828 < 0.001

CIMT, carotid intima-media thickness.

Table 4.
Reliability for plaque measurements among centers
Center compared Percent agreement (concordance/disconcordance pair1) Presence of plaque
Number of plaques
Kappa (95% CI) Weighted Kappa (95% CI)
1 vs. 2 88.75 (142/18) 0.304 (0.07, 0.54) 0.363 (0.09, 0.63)
1 vs. 3 92.50 (148/12) 0.507 (0.27, 0.74) 0.502 (0.25, 0.75)
2 vs. 3 96.25 (154/6) 0.606 (0.32, 0.89) 0.716 (0.46, 0.98)
Overall* 0.445 0.644

CI, confidence interval.

Overall agreement for presence of plaque, Fleiss kappa was used. Overall agreement for the number of plaques, Kendall's coefficients of concordance was used.

1 Unit of analysis is each segment of carotid artery.

REFERENCES

1. Pignoli P, Tremoli E, Poli A, Oreste P, Paoletti R. Intimal plus medial thickness of the arterial wall: a direct measurement with ultrasound imaging. Circulation 1986;74(6):1399-1406.
crossref pmid
2. Salonen JT, Salonen R. Ultrasound B-mode imaging in observational studies of atherosclerotic progression. Circulation 1993;87(3 Suppl):II56-65.
pmid
3. O’Leary DH, Bots ML. Imaging of atherosclerosis: carotid intima-media thickness. Eur Heart J 2010;31(14):1682-1689.
crossref pmid
4. Ebrahim S, Papacosta O, Whincup P, Wannamethee G, Walker M, Nicolaides AN, et al. Carotid plaque, intima media thickness, cardiovascular risk factors, and prevalent cardiovascular disease in men and women: the British Regional Heart Study. Stroke 1999;30(4):841-850.
crossref pmid
5. Lorenz MW, Markus HS, Bots ML, Rosvall M, Sitzer M. Prediction of clinical cardiovascular events with carotid intima-media thickness: a systematic review and meta-analysis. Circulation 2007;115(4):459-467.
crossref pmid
6. Touboul PJ, Labreuche J, Vicaut E, Amarenco P, Investigators G. Carotid intima-media thickness, plaques, and Framingham risk score as independent determinants of stroke risk. Stroke 2005;36(8):1741-1745.
crossref pmid
7. Mathiesen EB, Johnsen SH, Wilsgaard T, Bonaa KH, Lochen ML, Njolstad I. Carotid plaque area and intima-media thickness in prediction of first-ever ischemic stroke: a 10-year follow-up of 6584 men and women: the Tromso Study. Stroke 2011;42(4):972-978.
crossref pmid
8. Vanoli D, Wiklund U, Lindqvist P, Henein M, Naslund U. Successful novice’s training in obtaining accurate assessment of carotid IMT using an automated ultrasound system. Eur Heart J Cardiovasc Imaging 2014;15(6):637-642.
crossref pmid
9. Nichols S, Milner M, Meijer R, Carroll S, Ingle L. Variability of automated carotid intima-media thickness measurements by novice operators. Clin Physiol Funct Imaging 2016;36(1):25-32.
crossref pmid
10. Polak JF, Szklo M, Kronmal RA, Burke GL, Shea S, Zavodni AE, et al. The value of carotid artery plaque and intima-media thickness for incident cardiovascular disease: the multi-ethnic study of atherosclerosis. J Am Heart Assoc 2013;2(2):e000087.
crossref
11. Bar M, Roubec M, Farana R, Ličev L, Tomášková H, Školoudík D. Inter-rater reliability of carotid atherosclerotic plaque quantification by 3-dimensional sonography. J Ultrasound Med 2014;33(7):1273-1278.
crossref pmid
12. Tang R, Hennig M, Thomasson B, Scherz R, Ravinetto R, Catalini R, et al. Baseline reproducibility of B-mode ultrasonic measurement of carotid artery intima-media thickness: the European Lacidipine Study on Atherosclerosis (ELSA). J Hypertens 2000;18(2):197-201.
crossref pmid
13. Touboul PJ, Hennerici MG, Meairs S, Adams H, Amarenco P, Bornstein N, et al. Mannheim carotid intima-media thickness and plaque consensus (2004-2006-2011). An update on behalf of the advisory board of the 3rd, 4th and 5th watching the risk symposia, at the 13th, 15th and 20th European Stroke Conferences, Mannheim, Germany, 2004, Brussels, Belgium, 2006, and Hamburg, Germany, 2011. Cerebrovasc Dis 2012;34(4):290-296.
crossref pmid pmc
14. Plasencia Martinez JM, Garcia Santos JM, Paredes Martinez ML, Pastor AM. Carotid intima-media thickness and hemodynamic parameters: reproducibility of manual measurements with Doppler ultrasound. Med Ultrason 2015;17(2):167-174.
pmid
15. Lundby-Christensen L, Almdal TP, Carstensen B, Tarnow L, Wiinberg N. Carotid intima-media thickness in individuals with and without type 2 diabetes: a reproducibility study. Cardiovasc Diabetol 2010;9:40.
crossref pmid pmc
16. Velazquez F, Berna JD, Abellan JL, Serrano L, Escribano A, Canteras M. Reproducibility of sonographic measurements of carotid intima-media thickness. Acta Radiol 2008;49(10):1162-1166.
crossref pmid
17. Gonzalez J, Wood JC, Dorey FJ, Wren TA, Gilsanz V. Reproducibility of carotid intima-media thickness measurements in young adults. Radiology 2008;247(2):465-471.
crossref pmid
18. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86(2):420-428.
crossref pmid
19. Landers R. Computing Intraclass Correlations (ICC) as Estimates of Interrater Reliability in SPSS. The Winnower 3:e143518.81744.

20. SAS Institute Inc. Compute estimates and tests of agreement among multiple raters. Available at http://support.sas.com/kb/25/006.html [accessed on 25 October 2015].

21. Fleiss JL. Reliability of measurement. Hoboken, NJ: John Wiley & Sons, Inc; 1986. p. 1-32.

22. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-174.
crossref pmid
23. Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J Clin Epidemiol 2000;53(5):499-503.
crossref pmid
24. AlMuhanna K, Hossain MM, Zhao L, Fischell J, Kowalewski G, Dux M, et al. Carotid plaque morphometric assessment with three-dimensional ultrasound imaging. J Vasc Surg 2015;61(3):690-697.
crossref pmid
TOOLS
PDF Links  PDF Links
PubReader  PubReader
ePub Link  ePub Link
Full text via DOI  Full text via DOI
Download Citation  Download Citation
CrossRef TDM  CrossRef TDM
  E-Mail
  Print
Share:      
METRICS
2
Crossref
2,590
View
34
Download
Related article
Editorial Office
The Korean Society of Health Informatics and Statistics
20 Ilsan-ro, Wonju, Gangwon-do, 26426, Korea
E-mail: koshis@hanmail.net
About |  Browse Articles |  Current Issue |  For Authors and Reviewers
Copyright © The Korean Society of Health Informatics and Statistics. All rights reserved.                 Developed in M2Community