| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Articles |
 |
Abstract |
|---|
 Top Abstract Introductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
Key Words: indexing terms: quality control • medical devices
|
Introduction |
|---|
|
TopAbstractIntroductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
In this discussion, a unit test device, or single-use test, is a diagnostic assay in which a result can be obtained from a single sample. Quantitative devices achieve this result without reliance on a calibration curve that is external to the device. Examples of qualitative or semiquantitative unit test devices include Hybritech's ICON HCG Serum and Urine and over-the-counter pregnancy tests. Examples of quantitative unit test devices include Hybritech's ICON QSR CKMB and the i-Stat (i-Stat Corp., Princeton, NJ) point-of-care blood analyzers.
|
description and use of device |
|---|
|
TopAbstractIntroductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
Sample quantitation is achieved when the color development at the test
zones is compared with that at the calibration zones. The amount of
color development at each of the four zones is determined by the
ICON reader, which measures the reflectance of light off the
membrane. A two-point calibration curve is constructed by the responses
of the low and high calibrator zones (Fig. 1
). The average response of the test zones is read off this
curve, and the concentration of CKMB in the patient or control sample
is calculated. The test is designed such that each cylinder is
individually calibrated; in effect, if 10 ICON devices are run, 10
calibration curves are created. Compare this to batch testing, where
replicate test tubes of calibrators and samples (controls and unknowns)
are run and the samples are read off the single calibration curve. This
difference in "location" of the calibration curve is the
essential difference between a unit test device and batch testing. This
also creates special challenges and questions with relation to
laboratory requirements for quality-control (QC) testing of unit test
devices.
|
|
the icon CHALLENGE, OR WHY DO CONTROL TESTING? |
|---|
|
TopAbstractIntroductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
The regulations say that a laboratory must run controls once each day of testing (3). The value of this regulation may be—and has been—questioned. One of the purposes of QC testing is to find mistakes. That is, to identify problems with the product or the assay that would prevent a manufacturer from releasing a bad lot or the laboratory from releasing bad results. With a unit test device, however, running a control solution on one unit does not necessarily ensure the result on another unit because a new calibration curve is created for each unit. Because, by definition, a unit test device can be used for only one sample, whether it is a control or a patient sample, what information is gained about the patient sample run on one by running a control on another cylinder? If controls are run and fail, then immediately repeated and pass, what value does that add to the confidence in the patient results? These are valid questions that manufacturers as well as laboratories have struggled with.
To explore the question of how to find mistakes in a unit test device, it can be helpful to review the different aspects of QC. In general, QC data can be used for three basic purposes: acceptance testing, performance validation, and trend evaluation. Acceptance testing provides a decision about the fitness for use of a lot or of an assay. Performance validation testing provides information about the conditions under which a test is run, including equipment, the technician, and the reagents. Trend evaluation provides information about the long-term performance of the test, including lot-to-lot variation. Each of these is an important component of the complete evaluation of the product, either by the manufacturer or the clinician.
According to NCCLS, "internal quality control procedures monitor analytical performance relative to medical goals and alert analysts to unsatisfactory analytical performance" (4). The IFCC defines QC as "the study of those errors which are the responsibility of the laboratory, and the procedures used to recognize and minimize them. This study includes all errors arising within the laboratory between the receipt of the specimen and the dispatch of the report" (5). These are just two among many documents aimed at providing guidelines related to QC. The idea behind them is that any QC testing should assure the clinician of accurate patient results. Additionally, these documents emphasize that the laboratory must be responsible for this assurance. This applies to manufacturers as well; however, the "result" the manufacturer is providing is a good product.
Regulatory requirements exist to protect the patient. In short, the requirements related to unit test devices require that controls be tested once per assay run. In one respect, a manufacturer or laboratory can prove in this way that it is running a "clean" operation. It is often thought of as non-value-added testing because it requires the laboratory to spend money on tests that are not being used for patient results. However, the regulations are based on the basic QC principles (3), and as such provide a means for a laboratory to improve the quality and productivity of their results. So even if CLIA did not exist, it would be good business, as well as common sense, for a laboratory to apply these principles to their testing.
For unit test devices, the current testing requirements could be examined from a couple of angles. If the daily QC testing is being considered as acceptance testing—of the assay or the lot—then the testing could be considered expensive and non-value-added. As noted above, by definition a single-use device can accommodate only one sample or control, so running a control on one unit will provide no information about the patient sample run on another unit. Additionally, for ICON QSR CKMB, each cylinder contains internal controls to determine whether or not that cylinder is acceptable. If, however, the daily QC testing is being considered as performance and trend testing, then the opportunity is available to monitor product issues and identify and correct process issues.
|
process capability |
|---|
|
TopAbstractIntroductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
Understanding the capability of the process can also help determine what action to take when a control result is out of limits. Of course, if a cause can be assigned to the failure, that decision is easy: The control test can be repeated with confidence that the new result will accurately represent the performance of the process. However, a quantifiable chance exists that a failure will occur randomly. That is, if the control monitored is compared to 99% confidence limits, a 1% probability exists that random failure will occur. If there are more variables being evaluated (two controls, for example), that chance of failure increases according to the calculation 1–0.99n, where 0.99 represents 99% limits and n is the number of variables evaluated in a test. So, if two control results are being evaluated, a 1–0.992 or 0.0199 (1.99%) probability exists that one of the results will fail for no other reason than random chance. If everything was done correctly and the result is still out of limits, a little more investigation is needed. If monitoring procedures are in place, they can be used to determine the severity of the failure: Has it happened before? How often? Is it a "flier"? Is the control in question in a medically important position? If it is determined that additional testing is needed, that testing should be done according to a documented policy. This policy should describe under which conditions retesting is performed, and how much retesting is performed.
So far, this paper has discussed the "ideal" QC testing program and how to use it. A description has been provided of ICON QSR CKMB and why it is different from other quantitative devices. At this point, it may be instructive to review Hybritech's experience in developing the QC testing for release of the ICON QSR CKMB product and how working with this product illustrates the concept of capability.
|
a hybritech case history |
|---|
|
TopAbstractIntroductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
Background
. At Hybritech, samples from each lot of any product
line are tested in the QC laboratory before the lot is released to
ensure that it meets the required specifications. When the QC
release-testing strategy is developed for a new product, this general
scheme is followed. Control amounts are chosen to be tested with each
lot and certain variables are evaluated, such as recovery of controls,
variability, and detection limit. The specification limits are
determined statistically, based on the recommended use of the product.
As previously noted, for a unit test device such as ICON QSR CKMB, the
caveat is that the components are manufactured in batch, but the final
device is a unit, and customers use them individually. So each cylinder
can be considered its own assay, and the QC testing must produce a high
degree of confidence that all of the cylinders in a lot will perform
within specifications.
Correcting testing-related failures
. When Hybritech introduced
ICON QSR CKMB, the QC testing was designed to be consistent with the
other quantitative assays at Hybritech. However, the other quantitative
assays are all batch assays. Batch design includes replicate test tubes
of calibrators and controls, recoveries are read off a single
calibration curve, and the mean result is compared to specification. A
problem occurred because one of the consequences of having multiple
calibration curves, as is the case with ICON QSR CKMB, is that the
variability of results between replicate cylinders is
greater than the variability between replicate tubes in a
batch assay. It should be apparent that reading multiple samples off
one calibration curve is not the same as reading multiple samples off
multiple calibration curves—there will be inherently more variability
in the second scenario.
This variability was manifested by out-of-specification results for ICON QSR CKMB. All failures were thoroughly investigated, and it was discovered that the product performance was not compromised—it consistently met customer expectations. In fact, that was an aspect of ICON QSR CKMB that was quite frustrating: Customer complaints were received, but they were usually not related to the kinds of failures seen in-house. An inordinate amount of additional testing was being performed to show this. This additional testing was considered non-value-added because it wasn't providing any additional information about the product.
At this point, some changes were made to the testing process. Investigation showed that some of the processes in the laboratory were not under tight enough control; for example, frozen controls were left to thaw too long. This was primarily a training issue. Data analysis methods were also reviewed, and it was determined that the current analysis in place did not reflect customer use.
While ensuring more-consistent results, these changes did not have much effect on the rate of additional testing. Before the changes, ~47% of the kits tested had at least one result out of any specification (out of ~20 variables evaluated). After the changes, ~52% did. Compare this to the 18% chance of random occurrence of at least one result out of any specification (based on the number of variables evaluated in kit release testing).
Developing a statistically based sampling plan
. About this
time, a project was launched to evaluate and document the rationale for
in-process and QC testing for all of Hybritech's products and
processes. Each step of the manufacturing process was evaluated using
total quality tools to identify areas for improvement in the testing
process. Each process was put on a flowchart (8), and key
steps were identified. Other tools, such as Pareto charts, contingency
diagrams, and cause-and-effect (fishbone) diagrams
(9)(10)(11) were used to determine whether the
testing performed at that step provided value to the final product. If
not, opportunities were investigated to improve the testing methodology
or delete the testing.
Once each key step was identified and evaluated, it was documented in a control plan. The rationale for testing at each stage of manufacturing was documented: how many units to test and why; how to analyze the data and why. The number of units to test and method of analysis are based on what the product can do—its capability. Samples of the individual device units are taken and tested throughout the manufacturing process to continually ensure that the process is under control. The number of samples taken and method of analysis are based on statistical sampling plans, such as the Military Standard (MIL-STD).
MIL-STD-105E (12) is used in the analysis of control recovery in the final kit release testing for ICON QSR CKMB. This sampling plan uses as its quality index "acceptable quality level," or AQL. AQL for the sampling procedure in use is defined as "the designated value of percent defective for which lots will be accepted" (12). The AQL was determined by evaluating input from customers as well as the intended use of the test. ICON QSR CKMB is intended to be used on serially obtained samples and in conjunction with other clinical and laboratory information, as opposed to a stand-alone test.
Because of the variability in the system, the fact that the testing is
destructive, and the required acceptable quality measurement of 1%, it
was determined that 125 cylinders are required per lot for analysis of
this variable. In general, a lot consists of ~10 000 cylinders. A
double sampling plan is used, which allows for testing a minimal number
of samples initially. The acceptance criteria (Fig. 2
) are that if no more than 2 cylinders are out of QC
specification, no additional testing is required. If 3 or 4 cylinders
are out of specification, additional testing must be performed from
which no more than 6 cylinders (total from first and second testing)
may be out of specification. If 5 or more cylinders are out of
specification on the first test (or 7 or more cylinders combined on the
second test), the lot is considered nonconforming.
|
Another important procedure implemented around this time was a formal retest policy. As has been mentioned before, statistically random failure is expected at a quantifiable rate. The retest policy outlines under what conditions the product can be retested or if the results indicate that the product is truly nonconforming. Hybritech's retest policy as it applies to unit test devices is outlined below. When a control procedure is performed, the analyst is first asked to determine whether a procedural error or instrument malfunction occurred. If neither occurred, the procedure may continue. If there was an error with an assignable cause that occurred before data were generated or evaluated, the procedure must be aborted. In addition, the assignable cause is documented and reviewed by the laboratory supervisor to help identify systematic errors. If the identified error occurred after data were generated, the cause is again documented, and a single repeat assay is performed; the original data are not used. If no identifiable errors occurred, but a control result is out of specification, the analyst is asked to determine whether any outliers in the data exist (13)(14). If so, the data are reevaluated. If no outliers exist or failure still occurs after outlier analysis, three retest assays are performed, and the data from all four assays are combined to evaluate the performance of the kit.
The implementation of these changes did have an impact, reducing the amount of unnecessary additional testing by about 20%. After these changes, ~33% of the kits tested had at least one (out of ~20 variables) result out of any specification, requiring additional investigation. All kits were determined acceptable after investigation.
Aligning specifications with usage
. Although the process and
the statistical aspect of the testing were now well understood, the
individual nature of the device still did not receive enough emphasis.
The third major change to the QC release testing was not only to place
increased emphasis on the individuals, but also to remove
emphasis from the mean result. These changes not only more closely
mimic customer use, but allow a quicker reaction to failures. Also, the
number of superficial failures were reduced. Clinical laboratories
don't use ICON QSR CKMB to obtain a mean result, so it is not adding
any value for the laboratories—or for the manufacturer—to continually
fail a product that doesn't pass a mean specification. Replicate
cylinders are still used in testing so that an appropriate sample of
the lot can be evaluated, but now each individual cylinder must pass
specification.
This change reduced the unnecessary additional testing by another 20% and is now at a rate consistent with the other quantitative assays at Hybritech. At this point, ~13% of the kits tested have at least one result out of any specification. Again, compare this to the 18% chance of any result out of specification occurring randomly.
Where we are now
. As a result of these changes to QC, release
testing now flags true problems with the product, not the testing. The
variability inherent in the design of this device has been recognized
and incorporated into the specifications. Through the use of total
quality tools, and continual monitoring and evaluating of the testing
processes, unnecessary retesting was reduced by about 75%. The quality
of the product has been maintained—the fact that customer complaints
have decreased over that time attest to this. Customer complaints
decreased from ~173 per million tests sold when the product was
launched in 1992 to ~99 per million tests sold currently. Some
failures in QC testing are expected; in fact, a complete lack of
failure would be a problem of its own. But when failures occur they
should indicate a specific problem with the lot or the assay. The
ultimate definition of "acceptable for use" resides with the
customer. But if the product meets the customers' needs and an
inordinate amount of testing is required to release it, a problem with
testing methodology or the specifications is indicated.
The current QC testing system provides a high degree of confidence that the testing represents the entire lot because it is based on statistical sampling principles for unit test devices rather than batches. Also, there is a high degree of confidence that the testing reflects customer use because customer input has been incorporated into the testing design. Confidence in the testing process is now high because of a thorough understanding of the testing methodology. The challenge, then, for clinical laboratories is to assure the quality for the testing process in their hands—the technicians, testing, and storage environment and the monitoring process. Obviously, it is an evolving process. As more is learned about the tools available, such as those suggested by Westgard and Barry (7) and NCCLS (4), and as learning continues about the capability of a process, the QC system can be fine-tuned to make the most of it and improve productivity.
Considering, in retrospect, the changes made to the ICON QSR CKMB QC testing for release at the various stages of development, we learned that both the sampling and specifications must reflect the characteristics of the device being tested. When we started out, our sampling plan was not consistent with the specifications, because the specifications were set by batch assay criteria. Once we understood this about our process and changes were made, we began to see testing results that gave us a high degree of confidence that our testing reflected product performance. Additionally, the testing provides information we can use to continually improve the testing methodology.
To summarize, unit test devices present a unique challenge. To find the mistakes, it is important to use the right methods to understand the process and understand the capability. To fix those mistakes, one must define requirements and standardize methods.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
1 Nonstandard abbreviations: CKMB, creatine kinase MB; QC, quality control; MIL-STD, military standard; AQL, acceptable quality level. 
|
References |
|---|
|
TopAbstractIntroductiondescription and use of...the icon challenge, or...process capabilitya hybritech case historyReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
