| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Laboratory Management |
1 Department of Clinical Physiopathology, University of Florence, Florence, Italy.
2 Operative Unit of Medical Statistics and Biometry, Instituto Nazionale per lo Studio e la Cura dei Tumori, Milan, Italy.
3 Department of Clinical Chemistry, ZiekenhuisGroep Twente, Zilvermeeuw, Almelo, The Netherlands.
4 National Genetics Reference Laboratory, Saint Mary’s Hospital, Manchester, United Kingdom.
5 Institute for Clinical Chemistry, University Hospital Mannheim of the University of Heidelberg, Mannheim, Germany.
6 National Center Rare Diseases, Department of Cell Biology and Neuroscience, Istituto Superiore di Sanità, Rome, Italy.
7 Department of Clinical Chemistry, St. Anna Hospital, Geldrop, The Netherlands.
8 Istituto di Statistica Medica e Biometria, Universitàdegli Studi di Milano, Milan, Italy.
9 Department of Clinical Biology, Institute of Public Health, Brussels, Belgium.
10 Institute for Clinical Chemistry and Diagnostics, School of Medicine, University Hospital Sokolska, Hradec Kralove, Czech Republic.
aAddress correspondence to this author at: Department of Clinical Physiopathology, University of Florence, Florence, Italy. Fax 39-055-4271; e-mail m.pazzagli{at}dfc.unifi.it.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Methods: The EQUAL-qual program provided DNA, blood samples, and primer sets to participant laboratories to assess DNA extraction and PCR amplification. We have developed statistical procedures to identify laboratories performing poorly in DNA extraction (quality and quantity), PCR efficiency, and data interpretation after electrophoresis.
Results: An application to participate was obtained from 213 laboratories (from 25 countries), and 175 (82%) of laboratories submitted results for assessment. Questionable results in terms of quality and/or quantity of DNA derived from blood extractions were returned by 27% of laboratories (46 of 166). PCR efficiency showed high variability, with 3% of laboratories (5 of 163) showing a consistently low rate of amplification and 10% (18 of 175) not reporting the expected number of bands of the amplified targets.
Conclusions: The results showed considerable variability in all phases of the experiment. The approach confirms the validity of EQA as a method for evaluating analytical aspects of PCR-based tests.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The EQUAL Project ["MultiNational External Quality Assay (EQA) Programs in Clinical Molecular Diagnostics Based on Performance and Interpretation of PCR Assay Methods"], proposed under the auspices of the EC4 and funded by the European Commission, was designed to evaluate the implementation of molecular diagnostics relevant to all laboratories. Three methodological EQA programs have been implemented: EQUAL-qual for qualitative PCR assays, EQUAL-quant for quantitative PCR assays, and EQUAL-seq for sequencing-based assays. Results from the EQUAL-quant and the EQUAL-seq programs have been described (5)(6).
This report describes the results of EQUAL-qual, which aims to provide a critical assessment of performance of laboratories in DNA extraction and/or amplification. The program requires participating laboratories to send aliquots of their samples back to a central reference laboratory (EQUAL-Laboratory) for a standardized analysis.
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Each participant laboratory received an express mail package, to be stored at 4 °C, containing 6 vials: 2 contained 1.2 mL of a pool of human whole blood, citrate anticoagulated (HIV negative and hepatitis B virus negative), indicated as sample 1 (female; leukocytes = 3.8 x 109/L) and sample 2 (male; leukocytes = 4.4 x 109/L); sample 3 contained 50 µL pre-extracted DNA prepared by the salting-out procedure (11) from a pool of leukocytes taken from healthy male volunteers and resuspended to a concentration of 20 ng/µL; sample 4 contained 50 µL pre-extracted DNA from a pool of leukocytes taken from healthy female volunteers and resuspended to a concentration of 50 ng/µL. In addition, sample 4 was enriched with 0.04% BSA to produce an artificial decrease of 260:280 ratio to 
1.4. Two complete primer mixes at 20 µmol/L were also included (vials 5 and 6).
primers
Primer set A contained primers to amplify the human growth hormone gene (GH1,
2
also known as HSHGN, GenBank M13438, chromosome 17): forward, GCC TTC CCA ACC ATT CCC TTA, position 893, and reverse, TCA CGG ATT TCT GTT GTG TTT C, position 1319. These primers generate an amplification product of 427 bp (12). Primer set B contained primers to amplify the human amelogenin gene [AMELX, GenBank NM_001142 X-chromosome, and AMELY (also known as HUMAMELY), GenBank NM_001143 Y-chromosome]: forward, TGA CCA GCT TGG TTC TAW(A/T) CCC A, position 534 (X)/545 (Y), and reverse, CAR(A/G) ATG AGR(A/G) AAA CCA GGG TTC CA, position 823C (X)/649C (Y). Primer set B generates the following amplification products: 290 bp on chromosome X and 105 bp on chromosome Y (13). Therefore, PCR performed with the primer set B would generate a single fragment for DNA samples 1 and 4 and 2 fragments for DNA samples 2 and 3.
actions
Participants received detailed instructions for actions to be performed using the samples of the EQA package (see Table 2 in the online Data Supplement), as well as a complete questionnaire (see Table 3 in the online Data Supplement) on the relevant features of laboratory structure.
preamplification phase
Participants were requested to perform DNA extraction by the procedure routinely used in their laboratory. DNA quality and quantity were estimated by the participants in all samples: the 2 blood-derived extracts (samples 1 and 2) and the 2 pre-extracted DNA samples (samples 3 and 4). Participants were asked to provide the following information:
For participants unable to provide the 320-nm absorbance readings, 260 and 280 nm were deemed sufficient.
amplification phase
Participants were asked to set up a PCR with 100 ng DNA (as calculated by participants) from samples 1, 2, 3, and 4 and a water control with primer sets A and B. Participants were instructed to use Taq polymerase, dNTPs, and other reagents commonly in use in their own laboratory at the concentration used for a routine amplification under the following suggested PCR conditions: primer concentration set A, 0.25 µmol/L; primer concentration set B, 0.5 µmol/L; PCR conditions, 94 °C for 30 s, 60 °C for 30 s, 72 °C for 60 s (40 cycles).
data submission
Participants were required to submit the raw data from DNA quantification and post-PCR interpretation (number and size of amplified targets) by means of the dedicated Web site. In addition, in an express mail shipment box provided by the project, participants sent an aliquot of each PCR product and aliquots of DNA extracted from samples 1 and 2 to the EQUAL-Laboratory for further analysis.
reevaluation of samples in equal-laboratory
In EQUAL-Laboratory, the quality (R) and the quantity (Q) of DNA extracts obtained from blood samples 1 and 2 were reevaluated by use of the full-spectrum (220–750 nm) spectrophotometer NanoDrop® ND-1000 (NanoDrop Technologies). The values obtained in the EQUAL-Laboratory are indicated as Re and Qe, respectively. The products of PCR amplification were analyzed in EQUAL-Laboratory using the Agilent 2100 Bioanalyzer (Agilent Technologies) to provide a standardized estimation of the size and quantity of the PCR products (efficiency of amplification = E). The yield of each PCR product was assessed, corrected for PCR volume, and expressed in terms of ng DNA per amplified target.
statistical methods
In the absence of known reference values for each of the factors investigated, we measured the consistency of a given participant’s results against the majority (90%) of the results provided by the other participants. We adopted a statistical distribution–free approach to process the highly positively skewed data arising from this quality control program. We analyzed data according to a 2-step procedure aiming to (a) detect outliers and/or (b) identify laboratories with issues of poor performance. The 1st step involves the computation of the 95th bootstrap centile (14) of the distribution of the absolute value of the M statistic (15). This centile was adopted as the threshold to detect outliers. After removing the outliers (5%) from the analysis, we identified poor-performing laboratories by computing the 2 thresholds: 2.5th and 97.5th bootstrap centile of the original measurements. In each case, the number of bootstrap samples was 1000.
In the preamplification phase, the variables available for statistical analysis were Q (Q1–Q4) and R (R1–R4), provided by each laboratory for all 4 samples, and Qe (Qe1, Qe2) and Re (Re1, Re2), measured by the EQUAL-Laboratory for samples 1 and 2 only. In addition, for samples 1 and 2, we calculated the difference between the individual participant measurements and those obtained by the EQUAL-Laboratory as 
Q = Qe – Q (Q1, Q2) and R = Re – R (R1, R2). Thus a total of 16 variables were processed.
Because samples of poor DNA quality (R) will give unreliable estimates of DNA quantity (Q), we investigated Q only after removing laboratories from the analysis with questionable measurements for R. We processed all the variables according to the 2-step procedure described above.
To assess PCR efficiency (E), 10 measurements yielded by EQUAL-Laboratory were available for analysis. Five of these (E1–E5) were obtained from blood samples and 5 from pre-extracted DNA samples (E6–E10). Considering these 2 data sets individually, we generated a score by incorporating all 5 measurements. We adopted principal component (PC) analysis (16) to evaluate overall laboratory performance. This technique involves the computation of uncorrelated new variables, the PCs, which are ordered so that the 1st retains most of the variation present in all of the original data and the level of importance decreases by moving from the 1st to the last PC. Specifically, the PCs can be thought of as k new variables, obtained as a linear combination of the k original variables. Consequently, for each PC a set of k specific coefficients is defined, and with a small number of original variables, as in this case, the coefficients of the 1st PC are expected to have the same sign. Therefore if, for a given laboratory, all the measurements are lower than the respective means, that laboratory will have a low score for the 1st PC; conversely, if all the measurements are higher than the respective means, it will have a high score for the 1st PC. Therefore the 1st PC identifies laboratories that tend to systematically over- or underestimate the amplification product (quantified by E) with respect to the mean. To assess the overall performance level of each laboratory, we processed the score according to the 2-step approach described above.
We performed statistical analyses with the SAS System (17).
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
preamplification phase
Among the 175 laboratories that completed the survey, 9 did not provide data on DNA quality and quantity for all the samples because this procedure was not routinely used in their laboratories; 1 laboratory did not provide data on DNA quantity and quality for sample 3, and 1 laboratory did not provide these measurements for the 2 pre-extracted DNA samples. Table 1
shows a summary of the preamplification results. Table 5, A–D, in the online Data Supplement reports the identification code (ID-laboratory) of laboratories with outlying measurements or issues of poor performance (laboratories for which at least 1 questionable result was observed).
|
DNA quality.
The median quality (see Table 1
) of extracted DNA in blood samples 1 and 2 (R1 and R2) was constant and similar to that measured in pre-extracted DNA (R3), as confirmed from the overlapping of the interquartile range (IQR; 75th centile to 25th centile). In 21 of 166 laboratories (13%), however, we identified questionable results for either R1 or R2 (see Table 5A in the online Data Supplement), indicating suboptimal extraction protocols and/or incorrect photometric measurements. Moreover, 16 of 164 laboratories (10%) provided questionable results for R3 and 17 of 165 (10%) for R4 (see Table 5B in the online Data Supplement). Because these factors correspond to pre-extracted DNA samples, these results are most likely the result of erroneous photometric measurements. Eight laboratories provided questionable results in both blood and pre-extracted DNA, suggesting a possible technical problem common to both analyses.
DNA quantity.
The median and IQRs for quantities of DNA extracted from blood samples 1 and 2 (Q1 and Q2) were similar and close to those measured in pre-extracted sample 3 (Q3). Among laboratories with results of acceptable quality, some laboratories reported anomalous results for DNA quantity evaluation in blood samples (see Table 5A in the online Data Supplement). In particular, questionable results were identified in 15 laboratories (3 below the 2.5th centile and 12 above the 97.5th centile) for Q1 and in 14 laboratories (3 below the 2.5th centile and 11 above the 97.5th centile) for Q2. Even for these factors, the presence of anomalous results was also evident in pre-extracted DNAs (see Table 5B in the online Data Supplement), with anomalous performances in 13 laboratories (3 below the 2.5th centile and 10 above the 97.5th centile) for Q3 and in 16 laboratories (4 below the 2.5th centile and 12 above the 97.5th centile) for Q4.
Comparison with results of EQUAL-Laboratory.
Participants were asked to return an aliquot of the DNA extracts from blood samples 1 and 2 for evaluation in the central EQUAL-Laboratory. Table 1
shows that the median values for R1 and R2 are closer to 0 (the expected value) than the corresponding values for Q1 and Q2. The results in Table 5, C and D, in the online Data Supplement report the laboratories for which at least 1 questionable measurement was observed during this reevaluation. Twenty-seven laboratories share anomalous results in panels C and D, indicating simultaneous problems for DNA extraction and evaluation. In the 19 laboratories included only in panel C, the presence of questionable performances due to bad DNA extraction from blood samples 1 and 2 can be postulated. Conversely, in the 18 laboratories included only in panel D, the differences with values by EQUAL-Laboratory seem to be due to abnormalities in photometric evaluation.
amplification phase
Efficiency of amplification.
We assessed efficiency of amplification (E) by reanalyzing the PCR products returned by each participant to the EQUAL-Laboratory. With the PC analysis described, only laboratories able to provide a complete data set were considered in the analysis (163 blood samples and 167 DNA pre-extracted samples). Table 2
shows the minimum, median, maximum, and IQR for the 10 measurements.
|
In PC analysis, the 1st PC accounts for 82% and 74% of the total variability for blood and pre-extracted DNA measurements, respectively. Therefore, the 1st component captures almost the total information given by each set of 5 measurements. The pertinent scores were computed according to the following linear combinations:
 |
 |
Because the coefficients of the linear combinations have the same sign and are similar, the load of each measurement in defining the score is almost the same. Consequently, the identification of poor-performing laboratories by means of the 1st PC as suggested in Statistical Methods is valid.
Each panel (A–E) of Figs. 1
and 2
shows a box-plot graph for single measurements of blood samples and pre-extracted DNA samples, respectively. The scores enable us to identify laboratories that report measurements outside of the 10th or 90th centile of the original distribution on at least 3 occasions. Five laboratories showed very low levels of PCR amplification from extracted DNA samples (Fig. 1
), and in 3 of those laboratories this was replicated with the pre-extracted DNA (Fig. 2
), indicating generalized problems of PCR performance. Conversely, some laboratories showed high PCR efficiency, over the limit of the right arm (90th centile); in 5 cases, the data were deemed unusually high, suggesting possible mistakes in protocol execution.
|
|
Number of bands of amplified targets.
As indicated above, primer set A generated an amplification product of 427 bp, and primer set B generated amplification products of 105 bp on chromosome Y and of 290 bp on chromosome X. Therefore, we expected a single product when using primer set A against all 4 samples (C1A, C2A, C3A, C4A), a single product with primer set B against samples 1 and 4 (C1B and C4B), and 2 PCR products in DNA samples 2 and 3 (C2Bl, C2Bu, C3Bl, C3Bu).
Table 3
shows the laboratories (18 of 175; 10%) that did not report the expected number of fragments for primer sets A and/or B. Table 3
also shows the nature of the error (higher or lower number of bands than expected), the bands for which these errors were detected, whether the errors were confirmed by reevaluation of the PCR products at EQUAL-Laboratory, and finally an interpretation of the possible source of the errors. Five laboratories reported the wrong number of bands when the correct number of bands was obtained in the EQUAL-Laboratory, presumably owing to either data transcription errors or mistakes in the electrophoresis interpretation. Eight laboratories reported a lower number of bands than expected, possibly owing to insufficient PCR efficiency. In 6 of those results, the lack of the bands was confirmed in the EQUAL-Laboratory analysis; in the remaining 2 results a low but detectable band was obtained. The other 5 laboratories reported a higher number of bands than expected, probably owing to contamination of the PCR or a nonoptimized PCR protocol. In 1 laboratory, the presence of the contamination was confirmed by the EQUAL-Laboratory analysis, whereas contamination was not confirmed for the remaining 4 laboratories, suggesting possible contamination of the sample during a post-PCR procedure such as loading of the gel.
|
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
The results of the preamplification phase have been evaluated to address 2 major aspects: (a) performance of photometric measurements for DNA quality and quantity and (b) performance of the blood extraction procedure in terms of the amount and quality of the extracted DNA (see Table 1
; see Table 5 in the online Data Supplement). The 2 pre-extracted DNA samples were used to estimate the reliability of conventional photometric measurements.
Twenty-five percent of laboratories (42 of 165) performed poorly in the quantification of at least 1 of the 2 pre-extracted DNA samples, highlighting major concerns in the photometric measurements. This situation weakens the evaluation of the extraction phase by considering the results provided by each participating laboratory for the 2 blood samples included in the EQUAL-qual reagent set. However, because we included the reevaluation of the results of the extracted samples in the central EQUAL-Laboratory, it was possible to examine the values of Q and R by use of an additional independent and standardized analysis. On the basis of these measurements (Re1, Re2, Qe1, Qe2), 27% of laboratories (46 of 166) had questionable results in terms of quality and/or quantity of DNA derived from blood sample extractions. These results illustrate that the extraction phase remains a critical step for a large number of laboratories performing molecular tests in blood.
From Table 2
and Figs. 1
and 2
, we see a high variability among laboratories with regard to PCR efficiency with both pre-extracted DNA and DNA extracted in house. Because 2 of the DNA samples were pre-extracted and the primer sets for the PCR amplification were provided, it seems likely that the high variability of performance among laboratories in this regard is associated with the additional reagents (buffers, Taq polymerases, oligonucleotides) as well as the thermal cyclers (20).
To suggest possible corrective actions, it is useful to consider the final results of the PCR-based assay, i.e., the number of bands for each PCR. Table 3
lists 18 laboratories (10%) that reported at least 1 incorrect result in this regard. By comparing the data reported by the participants with those derived from the reevaluation of the PCR products at the EQUAL-Laboratory, we can deduce possible sources of these errors, including transcription errors, mistakes in the electrophoresis interpretation, sample contamination, and low PCR efficiency.
In conclusion, the results of the EQUAL-qual program demonstrate that in a basic experiment for DNA extraction and amplification, based on a predefined protocol and with the availability of some common reagents, we observed high variability between laboratories, and in some cases performances must be considered unsatisfactory. Subsequent to the EQA survey, EQUAL-qual participants identified as having performance issues were invited to participate in 1 of 3 EQUAL training courses in autumn 2005 in Florence, Rome, and Amsterdam. The methodological skills required to improve analytical performance were reviewed during the courses, and participants were invited to carry out the EQUAL-qual survey for a 2nd time. The results of this 2nd survey showed a significant improvement of the performance and will be presented separately.
|
Acknowledgments |
|---|
Financial disclosures: None declared.
Acknowledgments: We gratefully acknowledge the contribution of all of the partners and participant laboratories (see Table 1 in the online Data Supplement) in the EQUAL project.
|
Footnotes |
|---|
2 Human genes: GH1, growth hormone 1, also known as HSHGN; AMELX, amelogenin, X-linked; AMELY, amelogenin, Y-linked, also known as HUMAMELY.
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
