Clinical Chemistry 48: 1955-1962, 2002;
(Clinical Chemistry. 2002;48:1955-1962.)
© 2002 American Association for Clinical Chemistry, Inc.
Effect of Analytical Error on the Assessment of Cardiac Risk by the High-Sensitivity C-Reactive Protein and Lipid Screening Model
John Middleton1
1 Department of Clinical Chemistry Development, Beckman Coulter Inc., 200 S. Kraemer, Brea, CA 92821. Fax 714-961-3740; e-mail Jsmiddleton{at}beckman.com.
 |
Abstract
|
|---|
Background: Several prospective epidemiologic studies have demonstrated that high-sensitivity C-reactive protein (hsCRP) is an effective serum marker for cardiac risk assessment. When hsCRP is considered in conjunction with traditional lipid screening, its clinical utility is further increased. In this report, hsCRP, HDL-cholesterol (HDLC), and total cholesterol (TC) assay imprecision is evaluated in terms of the impact on the cardiac risk assessment process.
Methods: Cardiac risk assessment events were simulated using software written in Visual Basic for Applications with Microsoft Excel. Representative sets of analyte concentrations were used for true patient cardiac marker values. Monte Carlo simulations about the marker values were run using assay SD estimates based on College of American Pathologists surveys and journal articles.
Results: Risk distributions for reasonable assay imprecision showed clinically significant variation. Estimated relative risks of cardiovascular disease using the Ridker–Rifai quintile model varied by as much as 2.5-, 3.5-, and 6-fold for conditions of low, medium, and high laboratory test imprecision, respectively. The true relative risks were underestimated by >25% in 3%, 6.7%, and 10.5% of cases under conditions of low, medium, and high laboratory test imprecision, respectively, and overestimated by >25% in 4.4%, 8.5%, and 11.1% of cases under those conditions.
Conclusions: Use of the Monte Carlo simulation method as a tool to assess the impact of analytical variation on the clinical decision-making process is valuable. From this analysis, it is shown that multiple measurements of HDLC may reduce misclassifications that result from assay imprecision. This is most important when using HDLC assays that include sample preparation with 500 000 molecular weight dextran sulfate. For these assays the total assay method SDs are higher than for other assay methods. In addition, as demonstrated by a propagation of error analysis, HDLC has the largest component of overall error in the relative risk estimate. Under conditions of typical TC and hsCRP assay performance, replication of these assays is less important.
|
Introduction
|
|---|
Coronary heart disease is the major cause of death in the industrialized world. Traditionally, cholesterol screening has been used as a tool to identify individuals at increased risk of developing future coronary events (1). However, nearly 50% of the time, these methods have not been able to identify individuals who develop myocardial infarction when cholesterol concentrations are normal moderately increased. Recently the situation has been improved with the discovery that high-sensitivity C-reactive protein (hsCRP)1
is also a good prognostic indicator of future coronary events (2)(3)(4). Ridker and Rifai (5) have published a mathematical/statistical model relating the quintile position of the total cholesterol/HDL-cholesterol (TC/HDLC) ratio and hsCRP to cardiac risk. When hsCRP is considered in conjunction with the TC/HDLC ratio, the clinician’s ability to assess relative risk (RR) of future coronary events is greatly improved. Of course, the issues of biological and analytical variation have to be considered when applying this model.
It has been established that biological variability for the traditional lipids is an important component in the total variation of those analytes (6). Cooper et al. (6) cited intraindividual biological variations for TC, HDLC, LDL-cholesterol, and triglycerides that comprise >50% of the total within-individual variation. For hsCRP, biological variability is also of concern. In a study investigating the effects of Pravastatin on hsCRP concentrations, intraindividual temporal stability of hsCRP was demonstrated (7). In another study evaluating intraindividual variation, investigators suggested that the magnitude of hsCRP intraindividual variation is such that hsCRP assays have limited clinical utility (8). The focus of this effort, however, was on the analytical component of the total variation.
Monte Carlo simulation techniques have been used successfully to assess the impact of analytical variation. Boyd and Bruns (9) used this method to study the variation of analytical performance of glucose meters in the context of insulin dosing. Bennett et al. (10) studied the role that analytical error has on the impact of cholesterol measurements within the National Reference Method Laboratory Network. The simulation technique is used here to explore the impact of analytical variation of TC, HDLC, and hsCRP assay results on the cardiac risk assessment process as defined by the Rifai–Ridker model.
|
Materials and Methods
|
|---|
To simulate the analytical variations about a set of preselected values for TC, HDLC, and hsCRP, Monte Carlo simulation software was written in Visual Basic for Applications within Microsoft Excel. It was assumed that the analyte results were distributed normally. The random normal numbers were generated using Eq. 1
below, where U1 and U2 come from uniform distributions that vary in the range from 0 to 1 (11). For the simulated distributions, normality was confirmed with Pearson 
2 tests:
 | (1) |
The simulation input parameters included analyte target values and their respective SDs. The target values came from a representative sample from the Rifai–Ridker RR model. The quintiles chosen for analysis shown in Fig. 1
have their risk values bolded. Because men and women show only small differences in their quintile ranges for TC/HDLC, the ranges for men were used in the simulations. Total precision estimates, excluding preanalytical variation and contributions from random interferences (12), were made for each of the assays. The estimates for each assay were broken down into low, medium, and high imprecision values, and simulations were performed for each of the three conditions. Extremely low or high imprecision values were not used for simulation because they did not represent realistic scenarios.
View larger version (34K):
[in this window]
[in a new window]
|
Figure 1. RR factor vs quintile position of hsCRP and the TC/HDLC ratio.
The bold values correspond to the quintile positions chosen for simulation. Adapted from Fig. 2
in Rifai and Ridker (16).
|
|
tc precision estimates
For TC, the assay SDs used in the simulations were based on College of American Pathologists (CAP) 2001 C-A survey results. Because the "all method principles all instruments" CVs reported in the survey results were small (
3%), between-manufacturer variation was not an important consideration for this assay. To obtain reasonable estimates of low, medium, and high imprecision, the "within-method within-instrument" SDs were chosen as estimators for total assay SD. For each of the five survey samples, the survey precision results were ranked from smallest to largest. Low, middle, and high (90%, 50%, and 10% ranks, respectively) SDs were selected [supplemental Table 1 (supplemental Tables 1–5 can be accessed in the data supplement posted with the online version of this article atwww.clinchem.org/content/vol48/issue11/)]. These data were then used to construct SD-vs-concentration line equations by linear regressions (Fig. 2
). For the Monte Carlo simulations, a 200 mg/dL TC value was chosen. This is reasonable because what we are exploring is TC/HDLC variation throughout the clinical range of TC/HDLC. Different TC/HDLC targets may be achieved by fixing TC concentrations and varying HDLC concentrations. The 200 mg/dL TC value was chosen at the center of the concentrations spanned by the CAP survey. The lowest survey sample, LP-01, is at 150-mg/dL, whereas the highest (LP-05) is at 250 mg/dL. The TC SD at 200 mg/dL was calculated using the abovementioned line equations.
hdlc precision estimates
For the HDLC SDs, CAP data were also used. Unlike TC, there are several different methods reporting results in the surveys, with "all method principles all instruments" CVs ranging from 6.1% to 8.4%. Because of the relatively large between-manufacturer variation, the (within-method/all instruments) SDs were selected as estimators of total HDLC SD. These SDs were sorted from lowest to highest, and low, middle, and high values were selected (supplemental Table 2). Unlike TC, the middle values for HDLC did not correspond to a 50% ranking. This is because high SDs, corresponding to the HDLC methods that use a 500 000 molecular weight dextran sulfate sample preparation step, skewed the distribution. The selected SD/concentration data were then used to construct line equations by linear regressions (Fig. 3
). For the Monte Carlo simulations, given a HDLC target value, the SD was calculated using these line equations.
crp precision estimates
For hsCRP, unlike TC and HDLC where extensive precision information is available from CAP, the precision values used were based on a comprehensive performance evaluation on many currently fielded hsCRP assays (13). The precision data taken from this study are listed in supplemental Table 3. In addition, the precision estimates used in simulations (supplemental Table 4) were consistent with an independent evaluation of hsCRP assay performance (14). For low hsCRP concentrations (quintiles 1 and 2), the SD was fixed, whereas for higher hsCRP concentrations (quintiles 3–5), the SD was calculated from the %CV.
 | (2) |
monte carlo simulations
Analyte target values were selected based on the center of each of the bolded quintiles in Fig. 1
. With those selections along with low, medium, and high imprecision estimates, 5000 results per analyte per target value set were simulated.
|
Results
|
|---|
The simulation results corresponding to low, medium, and high assay imprecision conditions are shown in Table 1
. Within a precision section (low, medium, or high SDs), the target TC/HDLC and hsCRP quintiles chosen as true values in simulation are listed along with the corresponding target risk values. For given precision conditions and quintile values, simulations were performed. Table entries are of the form (no. of observations/RR value). A typical table entry is shown below.
Here, associated with TC/HDLC quintile 3 and hsCRP quintile 3, the target risk value is 2.9. The table entries indicate the number of observations within the target risk area as well as the number of observations in surrounding areas. In the example, there are 3558 observations at the target 2.9 RR, 20 observations at RR 2.5, and so forth. Another useful metric is percentage of misclassification, defined as the percentage of observations made at two or more quintiles, in either the hsCRP and/or TC/HDLC direction, away from the target quintile. Of course, as the assay imprecision increases, so does the percentage of misclassification. In other words, as assay imprecision increases so does the width of the RR distribution. Fig. 4
contains grids with the (no. of observations/RR) provided for each quintile position of the model. The target TC/HDLC and hsCRP values are based on the middle quintile position (hsCRP = 3, TC/HDLC = 3). For the cells in Fig. 4
that are not shaded, there were no simulated results observed within that region.
View larger version (29K):
[in this window]
[in a new window]
|
Figure 4. RR distributions at low, medium, and high assay imprecision.
The shaded regions correspond to quintile positions for which TC/HDLC and hsCRP values were observed. The value on the left is the number of observations; the value on the right is the RR value. In all instances the target RR value is 2.9.
|
|
An interesting feature of the risk distributions is the nonsymmetry about the target RR value of 2.9. Rather, they are wider in the TC/HDLC direction. To understand the nonsymmetric shape of the RR distribution, the magnitude of the assay precision relative to the magnitude of the quintile width must be considered. For low, medium, and high imprecision, the SDs divided by the quintile widths (expressed as a percentage) are listed in Table 2
. Because the SD relative to the quintile width is approximately two times wider for TC/HDLC than for hsCRP, the distributions are wider in the TC/HDLC direction.
One source of the relatively large TC/HDLC variation is that two components of error rather than one contribute to the total error. Because the individual TC and HDLC results are not considered in the risk analysis, but rather the ratio, the effect of TC and HDLC variation on the variation of the ratio must be considered. This variation can be assessed by the method of propagation of error. In the simulations, because the variable errors are uncorrelated, the expression relating the SD of a function of two variables to the SD of the individual contributions of the individual variables is given by (15):
 | (3) |
Applying this equation to the TC/HDLC ratio provides the following:
 | (4) |
Evaluating Eq. 4
in quintile 3 for TC/HDLC at low imprecision estimates provides the following result:
 | (5) |
The observed precision for the simulation of 0.238 matches the theoretical prediction well.
By putting the variation data in terms of relative SDs (supplemental Table 5), it can be seen that the variation of the TC/HDLC ratio is greater than either the TC or HDLC variations. In fact, the TC/HDLC CV is only slightly greater than the HDLC CV. This can be understood by considering the following equation (also derived from the method of propagation of error). In this expression, the higher of two terms within the square root brackets will overpower the smaller term as the values are squared. In other words, because HDLC has higher CVs, the HDLC term will be more important.
 | (6) |
Simulation results with TC and HDLC duplication and HDLC duplication only are contained in Table 3
. Both scenarios show similar distributions, indicating that HDLC imprecision is more influential than TC imprecision in the RR distributions.
View this table:
[in this window]
[in a new window]
|
Table 3. Simulation results for low, medium, and high assay imprecision conditions measuring TC and HDLC in duplicate and HDLC only in duplicate.1
|
|
|
Discussion
|
|---|
The RR distribution information in Table 1
suggests that individuals within the very low and very high RR groups are not as prone to errors in risk assessment as are individuals in the middle region. Because the lowest and highest quintile widths are the largest relative to assay precision (Table 2
), the chance of obtaining a result outside these quintiles is lower. However, for individuals within the middle risk areas, possibilities for underassigning high risk and overassigning low risk exist because of the wide risk distributions resulting from analytical variation. In Fig. 4
(high imprecision case) it can be seen that for a true RR of 2.9, there are 21 of 5000 instances in which a RR of 1.4 would have been reported. With a risk of 1.4, a clinician may not recommend exercise and or smoking cessation when a risk value of 2.9 may support these courses of action. Conversely, in the same scenario, 71 instances of reporting a RR of 6.0 occurred. With a risk of 6.0, a clinician may recommend drug treatment (4) when lifestyle changes may be a more appropriate course of action. On the basis of these results, it is clear that analytical variation can contribute to misclassification.
According to the Central Limit Theorem of statistics, to effectively reduce assay imprecision multiple measurements can be made. The simulation results indicate that the width of the risk distributions in the hsCRP direction would not benefit significantly from replication. However, in the TC/HDLC direction, assay replication would be desirable. In keeping with this theory, duplicate measurements in the simulations reduced the assay SDs by a factor of the square root of 2. Table 3
contains the results of duplicating TC and/or HDLC in the simulations and shows that errors in risk were mitigated, in part, by replicating the HDLC measurements. Duplication of TC and/or HDLC reduced the number of observed instances of a RR of 1.4 from 21 to 2. In addition, the number of observed instances of a RR of 6.0 was reduced from 71 to 8, and in the medium imprecision case, misclassifications went from 0.2% for the TC/HDLC quintiles 2 and 3 to 0.0% with HDLC-only duplication.
Although the trend of RR distribution width reduction with replication holds, it is important to note here that the effects of replication on the risk assessment process are overestimated in the simulations. This is because it is assumed that measurements are made across all of the variables that contribute to the total imprecision. In practice it is not possible for an individual clinician to replicate measurements over all influential variables. A reasonable variance model for HDLC imprecision would include instrument-to-instrument, calibration-to-calibration, reagent lot-to-reagent lot, sample preparation (for assays using a HDLC separation step only), and within-run components of variation. Because a clinician might replicate only across calibrations and sample preparations, the observed reductions in the RR distributions would be less than predicted by these simulations. In addition, as mentioned previously, the width of the result distributions increases with the inclusion of preanalytical sources of variation and systematic errors such as those that result from interferences.
On the basis of this analysis, it is clear that Monte Carlo simulation is useful in relating analytical variability with diagnostic utility. Varying analytical outcomes based on reasonable performance metrics in the context of clinical decision-making processes is important for improving those processes.
|
Acknowledgments
|
|---|
The thoughtful critiques of this manuscript by Wajdi Abdul-Ahad, Dennis Ashby, Rolf Hinzmann, Phil Martz, Robert Parson, and Jeff Vaks were invaluable.
|
Footnotes
|
|---|
1 Nonstandard abbreviations: hsCRP, high-sensitivity C-reactive protein; TC, total cholesterol; HDLC, HDL-cholesterol; RR, relative risk; and CAP, College of American Pathologists. 
|
References
|
|---|
- Executive Summary of the Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) [Special Communication]. JAMA 2001;285:2486-2497.[Free Full Text]
- Rohde LEP, Hennekens CH, Ridker PM. Survey of C-reactive protein and cardiovascular risk factors in apparently healthy men. Am J Cardiol 1999;84:1018-1022.[ISI][Medline]
[Order article via Infotrieve]
- Ridker PM, Hennekens CH, Buring JE, Rifai N. C-Reactive protein and other markers of inflammation in the prediction of cardiovascular disease in women. N Engl J Med 2000;342:836-843.[Abstract/Free Full Text]
- Ridker PM, Stampfer MJ, Rifai N. Novel risk factors for systemic atherosclerosis. A comparison of C-reactive protein, fibrinogen, homocysteine, lipoprotein(a), and standard cholesterol screening as predictors of peripheral arterial disease. JAMA 2001;285:2481-2485.[Abstract/Free Full Text]
- Rifai N, Ridker PM. High-sensitivity C-reactive protein: a novel and promising marker of coronary heart disease. Clin Chem 2001;47:403-411.[Abstract/Free Full Text]
- Cooper GR, Smith SJ, Myers GL, Sampson EJ, Magid E. Estimating and minimizing effects of biologic sources of variation by relative range when measuring the mean of serum lipids and lipoproteins. Clin Chem 1994;40:227-232.[Abstract/Free Full Text]
- Ridker PM, Rifai N, Pfeffer MA, Sacks F, Braunwald E. Long-term effects of pravastatin on plasma concentration of C-reactive protein. The Cholesterol and Recurrent Events (CARE) Investigators. Circulation 1999;100:230-235.[Abstract/Free Full Text]
- Cambell B, Badrick T, Flatman R, Kanowski D. Limited clinical utility of high-sensitivity plasma C-reactive protein assays [Comment]. Ann Clin Biochem 2002;39:85-88.[ISI][Medline]
[Order article via Infotrieve]
- Boyd JC, Bruns DE. Quality specifications for glucose meters: assessment by simulation modeling of errors in insulin dose. Clin Chem 2001;47:1329-1331.[Free Full Text]
- Bennett ST, Eckfeldt JH, Belcher JD, Connelly DP. Certification of cholesterol measurements by the National Reference Method Laboratory Network with routine clinical specimens: effects of network laboratory bias and imprecision. Clin Chem 1992;38:651-657.[Abstract/Free Full Text]
- Box GEP, Muller ME. A note on the generation of random normal deviates. Ann Math Stat 1958;29:610-611.
- Krouwer JS. Estimating total analytical error and its sources. Arch Pathol Lab Med 1992;116:726-731.[ISI][Medline]
[Order article via Infotrieve]
- Roberts WL, Moulton L, Law TC, Farrow G, Cooper-Anderson M, Savory J, et al. Evaluation of nine automated high-sensitivity C-reactive protein methods: implications for clinical and epidemiological applications. Part 2. Clin Chem 2001;47:418-425.[Abstract/Free Full Text]
- Roberts WL, Sedrick R, Moulton L, Spencer A, Rifai N. Evaluation of four automated high-sensitivity C-reactive protein methods: implications for clinical and epidemiological applications. Clin Chem 2000;46:461-468.[Abstract/Free Full Text]
- Ku HH. Notes on the use of propagation of error formulas. J Res Natl Bur Stand Sect C 1966;70C:263-273.
- Rifai N, Ridker PM. Proposed cardiovascular risk assessment algorithm using high-sensitivity C-reactive protein and lipid screening [Opinion]. Clin Chem 2001;47:28-30.[Free Full Text]
The following articles in journals at HighWire Press have cited this article:
|
|
|
|
 
M. Di Napoli, M. Schwaninger, R. Cappelli, E. Ceccarelli, G. Di Gianfilippo, C. Donati, H. C.A. Emsley, S. Forconi, S. J. Hopkins, L. Masotti, et al.
Evaluation of C-Reactive Protein Measurement for Assessing the Risk and Prognosis in Ischemic Stroke: A Statement for Health Care Professionals From the CRP Pooling Project Members
Stroke,
June 1, 2005;
36(6):
1316 - 1329.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
T. M. Reynolds, P. Twomey, and A. S. Wierzbicki
Accuracy of Cardiovascular Risk Estimation
Clin. Chem.,
April 1, 2003;
49(4):
706 - 707.
[Full Text]
[PDF]
|
|
|
Top
Abstract
Introduction
; R2 = 0.958), medium (
; R2 = 0.982), and high (
; R2 = 0.884) imprecision conditions.