Clinical Chemistry 43: 602-607, 1997;
(Clinical Chemistry. 1997;43:602-607.)
© 1997 American Association for Clinical Chemistry, Inc.
Quality-control (QC) performance measures and the QC planning process
Curtis A. Parvin
 |
Abstract
|
|---|
Numerous outcome measures can be used to characterize and compare the
performance of alternative quality-control (QC) strategies. The
performance measure traditionally used in the QC planning process is
the probability of rejecting an analytical run when a critical
out-of-control error condition exists. Another performance measure that
naturally fits within the total allowable error paradigm is the
probability that a reported test result contains an analytical error
that exceeds the total allowable error specification. In general, the
out-of-control error conditions associated with the greatest chance of
reporting an unacceptable test result are unrelated to the
traditionally defined "critical" error conditions. If the
probability of reporting an unacceptable test result is used as the
primary performance measure, worst-case QC performance can be
determined irrespective of the magnitude of any out-of-control error
condition that may exist, thus eliminating the need for the concept of
a "critical" out-of-control error.
Key Words: indexing terms: quality assurance • laboratory performance • statistics
|
Introduction
|
|---|
The traditional approach to the quality-control (QC) planning
process involves a series of at least five steps(1)(2).1
First, the quality requirement,
usually defined in terms of total allowable error
(TEa), must be specified. Results
that contain analytical errors that exceed TEa are
considered to be of unacceptable quality. Second, the accuracy and
precision of the assay are evaluated. Third, critical size errors are
calculated on the basis of the quality requirement and the assay
accuracy and precision. Next, the performance of alternative QC rules
are assessed in terms of their probability of rejecting an analytical
run when an out-of-control error condition exists
(Ped) and their probability of falsely rejecting
an analytical run that is in control (Pfr).
Finally, control rules and the number of control samples per run (N)
are selected to give a low false-rejection rate
(Pfr 
0.05) and a high error-detection rate
(Ped 
0.90) for critical size errors.
Numerousoutcome measures can be used to characterize and compare the
performance of alternative QC strategies. One performance measure that
naturally fits within the total allowable error paradigm is the
probability of reporting a test result with an analytical error that
exceeds TEa when an out-of-control error condition exists.
The primary performance measure used in the traditional QC planning
process is the probability of rejecting an analytical run when an
out-of-control error condition exists. This paper addresses the
question of how the probability of rejecting an analytical run that
contains a critical size out-of-control error condition relates to the
probability of reporting an unacceptable test result (one with an
analytical error that exceeds the total allowable error specification).
|
Methods
|
|---|
To demonstrate the relation between performance measures, I
evaluated the 1ks rule and the
(c)/R4s rule. The
1ks rule rejects if any of the control
observations in the current analytical run are more than k
analytical SDs from target (2). The
(c)/R4s rule rejects if the
average difference from target of the control observations in the
current analytical run exceeds c SEMs or the range of the
control observations exceeds four analytical SDs (3).
The TEa specification is assumed to be 5.0 analytical SDs.
Out-of-control error conditions that cause a systematic error (SE) or
an increase in analytical imprecision (RE) are evaluated. Analytical
imprecision is assumed to be within-run imprecision. Between-run
imprecision is not considered (3). The critical systematic
error (SEc) is calculated as SEc =
TEa - 1.65 and the critical random error (REc)
is calculated as REc = TEa/1.96 (4).
These critical errors are associated with out-of-control error
conditions that would have a 5% chance of producing a test result with
an analytical error that exceeds TEa.
The probability that a reported test result contains an analytical
error that exceeds TEa will depend on the magnitude of any
out-of-control error condition that may exist and on the probability
(Ped) that the out-of-control error condition is
detected by the QC rules. The probability that a test result contains
an analytical error that exceeds TEa when an out-of-control
error condition exists, but before QC testing has occurred, will be
denoted PE. The probability that a
test result has an analytical error that exceeds TEa after
QC testing has been performed will be denoted
PQE. For a given out-of-control
error condition, PQE is the product
of the probability that a result contains an unacceptable error and the
probability that the out-of-control error condition isn't detected, or
PQE =
PE (1 -
Ped).
All results were obtained by numerical analysis without the use of
simulations. The normal distribution function is required to calculate
PE. Determining
Ped for the 1ks rule and
(c) rule also requires the normal
distribution function. Calculating Ped for the
R4s rule when N = 2 can be accomplished by using the
2 distribution function. When N >2,
Ped for the R4s rule was calculated
by numerical integration (5). Calculations were performed by
using the statistical software package Stata (Stata Corp., College
Station, TX).
|
Results
|
|---|
Figure 1
displays the traditional QC performance measures for the
12.5s and (2.32)/R4s QC rules
with N = 2. The false rejection probability for the
12.5s rule is 0.025 when N = 2. The control limit for
the (c) rule was determined so that the
(c)/R4s rule also had a
false-rejection rate of 0.025. The appropriate control limit rounded to
the nearest hundredth is 2.32 SEM or 2.32/
= 1.52 analytical
SDs. With TEa = 5.0, SEc is 3.35 (Fig. 1A
) and
REc is 2.55 (Fig. 1B
). For a SEc,
Ped = 0.961 for the 12.5s rule and
Ped = 0.992 for the
(2.32)/R4s rule. For a REc,
Ped = 0.547 for the 12.5s rule and
Ped = 0.533 for the
(2.32)/R4s rule.
View larger version (19K):
[in this window]
[in a new window]
|
Figure 1. Traditional power function graphs for detecting SE
(A) and RE (B) for two different QC rules.
SE and RE are in analytical SD units.
|
|
Both QC rules have high probabilities for rejecting a SEc
and would be considered by traditional criteria to provide acceptable
error-detection performance. The
(2.32)/R4s rule has a slightly better
rejection probability at SEc, but the difference doesn't
seem great. Both QC rules have relatively low error-detection
probabilities for detecting a critical increase in analytical
imprecision. A QC rule's ability to detect REc is commonly
observed to be less than its ability to detect SEc(2).
Figure 2A
shows that PE increases as a
function of SE, with the rate of increase depending on the value of
TEa. Fig. 2A
also displays the acceptance probabilities
(1 - Ped) for the 12.5s and
(2.32)/R4s rules. Fig. 2B
gives the
probabilities (PQE) that a result
contains an analytical error that exceeds TEa after QC
testing by each QC rule. These curves result from the product of the
descending acceptance probability curves and the ascending
PE curve. The maximum value is
0.0022 occurring at SE = 3.04 for the 12.5s rule. The
maximum value is 0.0007 occurring at SE = 2.66 for the
(2.32)/R4s rule. In both cases, the SE
condition associated with the highest probability of reporting an
unacceptable test result is less than the traditionally defined
SEc.
View larger version (17K):
[in this window]
[in a new window]
|
Figure 2. Probability that a test result will contain an
unacceptable analytical error as a function of the magnitude of a SE
condition.
PE, along with a QC rule's
probability of accepting the out-of-control error condition (1 -
Ped) (A), determine
PQE (B).
|
|
Figure 3
shows the same performance measures as Fig. 2
, but as a
function of RE. The maximum PQE is
0.047 occurring at RE = 4.49 for the 12.5s rule. The
maximum PQE is 0.049 occurring at
RE = 4.51 for the (2.32)/R4s rule.
In this case, the RE condition associated with the highest probability
of producing a test result that contains an unacceptable analytical
error is substantially greater than the traditionally defined
REc. In general, the out-of-control error conditions that
have the greatest chance of producing an unacceptable test result are
unrelated to the traditionally defined "critical" error conditions.
View larger version (18K):
[in this window]
[in a new window]
|
Figure 3. Probability that a test result will contain an
unacceptable analytical error as a function of the magnitude of a RE
condition.
See Fig. 2
for further details.
|
|
Rather than being specified in terms of the minimum acceptable
probability of rejecting an analytical run that contains a critical
out-of-control error condition, QC performance might better be
specified in terms of the maximum acceptable probability
(Pmax) of reporting a test result with an
analytical error that exceeds total allowable error specifications. The
minimum error detection requirement for every possible error magnitude
can then be easily determined by simply rearranging the inequality
(1 - Ped)
PE
Pmax to give Ped
1 -
Pmax/PE. For
example, Fig. 4A
shows the minimum Ped requirement for a
QC rule to assure that the probability of reporting a test result with
an analytical error >5.0 analytical SDs is never >0.001 for any
possible SE condition. Fig. 4B
shows the minimum
Ped requirement to keep
Pmax <0.01 for any possible RE condition.
View larger version (15K):
[in this window]
[in a new window]
|
Figure 4. Minimum error-detection requirements for a QC rule to
assure that the probability of reporting a test result with an
analytical error that exceeds TEa is never greater than
Pmax for a SE condition (A) and a RE
condition (B).
|
|
Figure 5A
displays the power function curves for the 12.18s
rule (Pfr = 0.058) and the
(2.49)/R4s rule
(Pfr = 0.017) for two control samples. Both
rules meet the quality specification shown in Fig. 4A
and were easily
found by systematically varying the control limits for the
1ks rule and for the
(c) rule until the power function curves
just exceeded the minimum Ped requirement. Fig. 5B
shows the probabilities of reporting an unacceptable test result
when these rules are used. As required, the maximum value for
PQE is just <0.001 and occurs when
SE = 2.84 for the 12.18s rule and when SE = 2.74
for the (2.49)/R4s rule.
Similarly, Fig. 6
gives the performance characteristics of the 12.35s
rule (Pfr = 0.073) and the
(1.91)/R4s rule
(Pfr = 0.079) for four control samples. The
rules produce virtually identical power function curves (Fig. 6A
) that
meet the quality specification shown in Fig. 4B
, and are associated
with a maximum probability of reporting an unacceptable test result
that is just <0.01 occurring when RE = 3.10 (Fig. 6B
).
View larger version (16K):
[in this window]
[in a new window]
|
Figure 6. Power functions (A) for the 12.35s
rule and the (1.91)/R4s rule that meet
the minimum error-detection requirements shown in Fig. 4B
, and the
probability that a reported test result will contain an analytical
error that exceeds TEa (B) for these QC rules.
The power functions for the two rules are virtually identical.
|
|
|
Discussion
|
|---|
The traditional approach to specifying analytical quality goals in
the clinical laboratory has been based on a TEa paradigm.
Westgard and Barry discuss quality goals in terms of the
TEa that can be tolerated in a test result without
compromising its medical usefulness (4). Cembrowski and
Carey refer to quality goals in terms of the maximum clinically
allowable error (Ea) in a test result (6). When
this paradigm is used to evaluate QC strategies, the primary
performance measure of interest should be related to the probability of
reporting a test result that contains an analytical error that exceeds
the TEa specification. However, the traditional performance
measure used in QC evaluation has been the probability of rejecting an
analytical run when a critical out-of-control error condition exists.
Such a condition is usually defined as one that would result in 1% or
5% of test results containing an analytical error that exceeds the
TEa error specification (4)(6). The traditional
approach to performance evaluation only indirectly relates to the
primary performance measure of interest.
Figures 1–3
compare the traditional approach to QC performance
measurement based on the probability of rejecting a run to an approach
that directly calculates the probability of reporting an unacceptable
result. The Ped when a SEc or
REc exists has little to do with the error conditions that
result in the greatest chance of producing unacceptable results. The
probability of reporting an unacceptable result increases as the
magnitude of the out-of-control error condition increases, reaches a
maximum, and then decreases for larger error conditions. This behavior
should not come as a surprise. For very small out-of-control error
conditions, the chance that a QC rule will detect the error condition
is low, but the probability of reporting an unacceptable result even if
the out-of-control error condition isn't detected will still be low.
For very large error conditions, the probability of an unacceptable
result is very high if the out-of-control error condition is not
detected, but the probability that a QC rule will detect the error
condition is also very high, so the ultimate probability of reporting
an unacceptable result is again low. The out-of-control error
conditions that will be associated with the greatest probability of
reporting an unacceptable result will be those whose magnitude is small
enough to still be relatively difficult for a QC rule to detect, but
large enough so that the probability of producing a result with an
unacceptable error is relatively high. The magnitude of the
out-of-control error conditions associated with the highest
probabilities of reporting test results with unacceptable errors will
depend on the TEa specification, the QC rule, and the type
of error.
Traditionally, QC rules are selected to have a high probability
(0.90) for detecting "critical" out-of-control error conditions.
When comparing alternative QC rules that all have error-detection
probabilities that are 0.90 for detecting a critical error,
differences between rules seem relatively small. However, these
apparently small differences between rules can translate into quite
large differences in their probabilities of reporting a laboratory
result with an unacceptable analytical error. In Fig. 1A
, Ped at the traditionally calculated
SEc is 0.961 for the 12.5s rule compared with
0.992 for the (2.32)/R4s rule. However,
in Fig. 2B
the probability of reporting a result with an unacceptable
error is more than three times greater for the 12.5s rule
(0.0022) than for the (2.32)/R4s rule
(0.0007). Thus, differences between QC rules that might be dismissed as
inconsequential when evaluated by traditional performance measures
could be substantial in terms of their probabilities of allowing
results to be reported that contain unacceptable analytical error.
If quality goals are specified in terms of a TEa
specification and a Pmax of reporting a result
that contains an analytical error that exceeds TEa, then
the minimum error detection probability required for every magnitude of
out-of-control error condition can be easily determined (Fig. 4
). While
others have discussed in general terms the optimal power function for a
QC rule (6)(7), this approach provides a precise definition
for an optimal Ped criterion.
The quantity PQE describes the
probability of reporting a test result with an analytical error that
exceeds TEa as a function of the magnitude of an
out-of-control error condition. If an estimate of the overall rate of
reporting unacceptable test results is desired, then the frequency of
occurrence of out-of-control error conditions must be taken into
account. For example, if a SE condition between 1.0 and 4.0 SD exists
less than one in 10 times that QC testing occurs, then for the
situation shown in Fig. 5B
the overall rate of test results containing
unacceptable analytical errors would be considerably less than
(0.1)(0.001) = 0.0001.
Westgard and Barry have described the defect rate of an analytical
process as the portion of test results having medically important
errors (4). However, the quantity that they calculate for
defect rate is f(1 - Ped),
where f denotes the frequency of occurrence of a medically
important out-of-control error condition. This quantity reflects the
fraction of analytical runs containing critical out-of-control error
conditions that are not rejected, which is not the same as the fraction
of test results that have medically important errors.
In summary, when evaluating QC rules, the probability of
reporting a test result that contains an unacceptable analytical error
should be the performance measure of interest, not the probability of
rejecting an analytical run with a SEc or REc.
By using the probability of reporting an unacceptable test result as
the primary performance measure, worst-case QC performance can be
determined irrespective of the magnitude of any out-of-control error
condition that may exist, thus eliminating the need for the concept of
a critical out-of-control error.
|
Footnotes
|
|---|
Division of Laboratory Medicine, Department of Pathology, Washington University School of Medicine, Box 8118, 660 South Euclid Ave., St. Louis, MO 63110. Fax 314-362-3016; e-mail parvin{at}wugcrc.wustl.edu
1 Nonstandard abbreviations: QC, quality control; PE, probability that a test result has an unacceptable analytical error when an out-of-control error condition exists, but before QC testing has occurred; Ped, probability of rejecting an analytical run when an out-of-control error condition exists; Pfr, probability of rejecting an analytical run that is in control; Pmax, maximum acceptable probability of reporting a test with an analytical error that exceeds the total allowable error specification; PQE, probability that a test result has an unacceptable analytical error after QC testing has occurred; RE, an out-of-control error condition that causes an increase in analytic imprecision (random error) in subsequent test results; REc, an out-of-control error condition that causes an increase in analytic imprecision that is considered critical; SE, an out-of-control error condition that results in a constant bias (systematic error) in subsequent test results; SEc, an out-of-control error condition that results in a constant bias that is considered critical; TEa, total allowable error specification; test results that contain analytical errors that exceed TEa are considered unacceptable; (c)/R4s, QC rule that rejects if the average difference from target of the control observations in the current analytical run exceeds c SEMs or the range of the control observations exceeds four analytical SDs; and 1ks, QC rule that rejects if any of the control observations in the current analytical run are more than k analytical SDs from target.
|
References
|
|---|
-
Westgard JO. Selecting appropriate quality-control rules [Letter]. Clin Chem 1994;40:499-500.
[Free Full Text]
-
Koch DD, Oryall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry PL, Westgard JO. Selection of medically useful quality-control procedures for individual tests done in a multitest analytical system. Clin Chem 1990;36:230-233.
[Abstract/Free Full Text]
-
Parvin CA. New insight into the comparative power of quality-control rules that use control observations within a single analytical run. Clin Chem 1993;39:440-447.
[Abstract/Free Full Text]
-
Westgard JO, Barry PL. Cost-effective quality control:
managing the quality and productivity of analytical processes.
Washington, DC: AACC Press, 1986:44–9, 174..
-
Stuart A, Ord JK. Kendall's advanced theory of statistics, Vol 1 1987:464-466 Oxford University Press New York. .
-
Cembrowski GS, Carey RN. Laboratory quality management.
Chicago: ASCP Press, 1989:37–8, 92–5..
-
Hackney JR. Performance characteristics of quality control systems. Lab Med 1989;20:388-392.
The following articles in journals at HighWire Press have cited this article:
|
|
|
|
 
C. A. Parvin
Assessing the Impact of the Frequency of Quality Control Testing on the Quality of Reported Patient Results
Clin. Chem.,
December 1, 2008;
54(12):
2049 - 2054.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
C. A. Parvin and A. M. Gronowski
Effect of analytical run length on quality-control (QC) performance and the QC planning process
Clin. Chem.,
November 1, 1997;
43(11):
2149 - 2154.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Introduction
