| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Informatics and Statistics |
1 Division of Metabolic Diseases, Department of General Pediatrics, University Children’s Hospital, Heidelberg, Germany.
2 Department of Pediatrics and Institute of Clinical Chemistry, Hamburg University Medical Center, Hamburg, Germany.
3 Department of Medical Informatics, University of Heidelberg Medical Center, Heidelberg, Germany.
aAddress correspondence to this author at: University Children’s Hospital, Division of Metabolic Diseases, Im Neuenheimer Feld 150, D-69120 Heidelberg, Germany. Fax 49-6221-564069; e-mail Sirikit.Ho{at}med.uni-heidelberg.de.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Methods: We applied a procedure to extract new markers and diagnostic criteria for newborn screening to the data of newborns with confirmed medium-chain acyl-CoA dehydrogenase deficiency (MCADD) and a control group from the newborn screening program, Heidelberg, Germany. We validated the results with external data of the screening center in Hamburg, Germany. We extracted new markers by performing a systematic search for analyte combinations (features) with high discriminatory performance for MCADD. To select feature thresholds, we applied automated procedures to separate controls and cases on the basis of the feature values. Finally, we built classifiers from these new markers to serve as diagnostic criteria in screening for MCADD.
Results: On the basis of 
2 scores, we identified 
800 of >628 000 new analyte combinations with superior discriminatory performance compared with the best published combinations. Classifiers built with the new features achieved diagnostic sensitivities and specificities approaching 100%.
Conclusion: Feature construction methods provide ways to disclose information hidden in the set of measured analytes. Other diagnostic tasks based on high-dimensional metabolic data might also profit from this approach.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Such diagnostic criteria are conventionally derived from knowledge of metabolic pathways. Potentially promising markers are selected manually and then tested against patient and control groups to maximize diagnostic accuracy. For some diseases, it has already been demonstrated that sensitivity and specificity can be improved by taking into account analyte combinations rather than single analytes alone. For example, for medium-chain acyl-CoA dehydrogenase deficiency (MCADD), the ratios of acylcarnitines C8:C10, C8:C6, C8:C2, and C8:C12 have proven useful (1)(2)(3)(4)(5).
The information available for diagnosis of a disease is normally spread over different analytes, each of which may be indicative of different aspects of the disease. Combining the information from different analytes by the computation of ratios, sums, or differences can therefore lead to higher discriminatory performance of the resulting new markers. Other reasons for potential improvement may be hidden interactions or relations between analytes. Such information can be expressed explicitly and made available for diagnosis.
These effects can help to improve classification models constructed by routine data-mining algorithms. The input for such learning algorithms is a data set, in the form of a collection of instances with assigned class labels. In our study the data set was controls vs MCADD. The instances are represented by a set of features, such as patient data or laboratory results.
Finding a good representation of the data for the learning algorithms, in the form of suitable features, is known to have high influence on the performance of the resulting classifiers. This task is addressed by feature construction and feature transformation techniques, which are directed at optimizing the data representation for subsequent data-driven learning algorithms (6)(7)(8). For some algorithms information in data may not be usable when it is not expressed explicitly. For example, in a recent publication on disease classification in newborn screening (9), the studied method failed to identify the analyte tyrosine as relevant feature for diagnosis of phenylketonuria, whereas in Baumgartner et al. (10), decision-tree induction was able to do so. As shown by Chace et al. (11), tyrosine can effectively improve diagnostic accuracy when used in the phenylalanine:tyrosine ratio.
This study demonstrates that systematic, data-driven searches for suitable markers by construction and evaluation of new analyte combinations can further improve diagnostic criteria for newborn screening.
As an example, we investigated this approach for MCADD (12) using acylcarnitine concentrations measured by TMS for newborn screening and their arithmetic combinations. To demonstrate the suitability of feature construction with metabolic data of newborn screening, we compared the results of the study with previously published diagnostic criteria of other working groups and validated them with data from a 3rd-party screening laboratory.
|
Materials and Methods |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
Sample preparation and analysis by electrospray ionization-TMS complied with the protocol described in Schulze et al. (13), with minor variations. The population serviced by the Heidelberg newborn screening center in the southwest of Germany. Beginning December 2003 the Hamad Medical Corporation, the main hospital of Qatar, implemented a statewide neonatal screening program in cooperation with the neonatal screening laboratory situated in Heidelberg. German recommendations for neonatal screening dating from 2002 were applied for this program (14). We also included the data from this cooperation in the study population. In the screened population, the mean age at sampling is 2.9 days (SD 0.86). The data for the study were extracted from neonatal screening data collected from July 2002 to January 2006 for newborns 0 to 5 days old. To our knowledge, no MCADD case was missed in newborn screening during this period.
Patients were divided into 2 classes, the MCADD class, which included the 30 children with proven MCADD identified through neonatal screening. The 2nd class, controls, comprised 2 groups. The 1st group included newborns not suffering from MCADD but showing increased primary marker octanoylcarnitine (C8) concentrations (0.28–1.7 µmol/L; n = 332). The 2nd group represents the normal population with the data of randomly chosen, presumably healthy children (n = 1685). The newborns with increased C8 concentrations, although small in absolute number, were overrepresented within the data set compared with the full screening population. Therefore, to optimize the specificity of resulting classifiers, we intentionally performed the study with many instances likely to be falsely classified as positives.
We randomly divided the selected data into 10 stratified partitions for cross-validation of the study (15). Retests of the same specimen were always assigned to the same partition. In each fold of the cross-validation, 9 partitions were used to create new markers and diagnostic criteria and 1 holdout partition for their validation. Results are given as sums or mean (SD) of the validation results.
methods
For the identification of new markers for MCADD diagnosis, we chose a 3-step data mining approach.
In step 1, we identified promising analyte combinations by a feature construction algorithm. The algorithm generates new numeric features from analytes, and each feature is described by an arithmetic expression composed of 2–4 analytes from Table 1
combined by the arithmetic operators +, –, and /.
|
We investigated 2 types of expressions, sums and divisions. In this context, a sum denoted a combination of analytes with the operators + or – (e.g., C8 + C10 or C8 – C10), and a division was an expression using analytes or sums as divisor and dividend [e.g., (C8 – C10)/C2]. With these restrictions, features were constructed only with operands of the same unit, to achieve a meaningful physical dimension. To avoid undue complexity and decline in the comprehensibility of features, no other operators were used. For example, multiplication was excluded because of the difficulty in interpreting the squared unit of an acylcarnitine concentration.
The algorithm searched >628 000 arithmetic expressions for features showing high discriminatory performance, i.e., the ability to separate the specimens known to be control or MCADD. The discriminatory performance was determined by use of 2 statistics, computed to test the statistical association of class distinction (control vs MCADD) with a feature. Here, however, we used the 2 measures as a numerical method to rank features for their presumed discriminatory performance.
The algorithm applied 3 different filters to any feature to single out those with minor discriminatory performance or features that do not prove superior to possible "predecessor features" composed of a subset of their component analytes. The remaining features were added to the learning set. For further details about the feature construction algorithm, in particular the specification of searched expressions, the 2 measure, and the filters, see Text 1 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol53/issue7.
During step 2, the analyte concentrations and the newly constructed features were converted to binary values by application of a linear threshold function. For example, the analyte C8 might be combined with a threshold of 1 µmol/L. The resulting binary feature C8 > 1 exhibited 2 possible values—true and false—which can be mapped to the classes MCADD and control. Features were by convention arithmetically transformed before applying the threshold, to guarantee that high values indicate MCADD (e.g., C4 – C8 is replaced by C8 – C4). By binarization, an adequate cutoff was embedded in the features, taking into account that false classification of a case as a control (false negative) is a much less tolerable risk than vice versa (false positive).
We investigated 2 variants of automated binarization. The 1st variant selected thresholds based on the range of values in the MCADD group of the learning set. A bootstrapping CI (BCI) for the minimum feature value in the MCADD group was computed, and the lower limit of this interval was used as threshold. Thus the low-value tail distribution of the MCADD cases and the chosen confidence probability determined the threshold and ensured a safety margin, offering protection against false-negative results on unknown cases. Implementation details of the binarization with BCI can be found in Text 2 in the online Data Supplement.
The 2nd binarization variant used the 99% quantile of a presumably healthy reference group as threshold. The baseline values for the quantiles of the features in a normal population were estimated nonparametrically from a random sample of 10 000 healthy newborns extracted from the newborn screening data from July 2003 to December 2005. In the past, high percentiles of the controls have been used as cutoffs in newborn screening to avoid high false-positive rates whenever the small number of available cases did not allow setting cutoffs based on the distribution within the diseased cohort (13).
As a result, each of the binarization approaches provided a learning set for the last data-mining step. Subsequent decision rules—logical combinations of binary features—were built from both learning sets using the rule induction algorithm JRip of the data-mining framework weka (16). The resulting classifiers were easily interpretable and structured like diagnostic criteria published in the literature.
For comparison of the new decision rules with the published diagnostic criteria, the latter were adapted to the data of the newborn screening center at Heidelberg. The markers were adopted as published, but because of missing interlaboratory comparability the reported absolute cutoff values for markers in the diagnostic criteria were replaced with cutoff values analogous to the thresholds in the 2 binarization variants.
We computed sensitivity, specificity, and positive predictive value as assessment criteria on the holdout partition for each cross-validation fold. These evaluation measures included newborns for whom more than 1 data record may be part of the test set. Class assignment for the newborns was therefore done by majority decision (see Text 3 in the online Data Supplement).
Finally, we performed an additional validation of the BCI variant for binarization for all folds with a secondary data set of MCADD cases and controls of the screening center in Hamburg, Germany, with a similar analyte spectrum.
|
Results |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
shows the 2 values of a selection of the features, the primary marker C8 and 3 conventionally established ratios used in Chace et al. (1), Clayton et al. (2), Pourfarzam et al.(3), and Okun et al. (4). C8 reaches a 2 value of 250 (22), whereas the binary features (C8 > threshold99% quantile and C8 > thresholdBCI) reach only 19.2 (0.8) and 7.4 (6.4), respectively. Binarization generally means a loss of information and hence a decline in discriminatory performance. The excessive decrease for the binary feature of variant 99% quantile is caused by the overrepresentation of controls with increased C8 values in both test and learning sets. The 2 value for the best-performing manually selected ratio C8:C10 is 154 (57), and for (C8:C10 > threshold99% quantile) and (C8:C10 > thresholdBCI) it was only 132 (45) and 48 (3), respectively. In contrast, the newly constructed features show mean 2 values between 105 and 264 (binary features of the quantile variant, 0.00–211.03; binary features of the BCI variant, 0.00–244.69).
|
In the following, we list the residual percentage of discriminatory performance for different feature groups achieved through the methods outlined above (see also Table 3
). A residual value of 100% means that according to the 2 values the binary feature separates the classes control and MCADD with the same accuracy as the numeric feature.
|
Binarizations with the 99% quantile variant lead to a mean residual discriminatory performance of 28.10% (mean of the folds). Ninety-nine percent of the resulting binary features exceeded the achieved specificity of the binary feature C8 > threshold99% quantile.
In the features derived from BCI binarization, only 43% achieved higher specificity scores than the respective binary feature of C8. The mean average residual discriminatory performance in the folds was 9.9%, range 0%–97%. In the BCI variant, large declines in discriminatory performance indicated high variation in the MCADD group used to calculate the BCI threshold. Outliers, especially at the lower range of the feature values, can lead to features with poor or even zero discriminatory performance.
The number of evaluated features exceeded the number of instances in the learning set by far, especially in relation to the instances of class MCADD. Therefore, constructed features might fit the learning set only coincidentally, and their discriminatory performance might not be generalizable for future instances, an effect called overfitting. In the present investigation, overfitting does not seem to have been a major problem, as demonstrated by the cross-validation: on average, 78% of all selected numeric features showed equal or higher 2 values than C8, and the lowest 2 value per fold on average reached 79% of the 2 value for C8, i.e., within 21% of the reference.
We performed a visual inspection of the numeric features to determine to what extent overfitting contributed to their discriminatory performance. We focused on the features with the best mean discriminatory performance computed from the numeric feature and the binary variants. The inspection revealed that these constructed features achieved their high discriminatory performance by reducing the number of controls in the proximity of the MCADD cases in comparison to C8. The new analyte combinations significantly reduced the controls in the proximity of the BCI and 99% quantile cutoff compared with C8 (Fig. 1
) and therefore reduced the probability of false-positive classifications. In contrast, the plot of the conventionally established ratio C8:C10 shows much overlapping of MCADD cases and controls.
|
diagnostic criteria and classifiers
For each cutoff variant, we compared the conventionally established diagnostic criteria with the decision rules built from the learning sets. The results of these classifiers and the adapted, conventionally established diagnostic criteria are summarized in Table 4
for the BCI variant and Table 5
for the 99% quantile variant.
|
|
For the BCI variant, the rules generated by the JRip algorithm resulted in 96.67% sensitivity and 99.80% specificity in the cross-validation study. One fold led to the selection of a feature with an unsafe BCI-threshold in rule induction. For the quantile variant, 100% sensitivity and 99.9% specificity were achieved.
The rules generated without cross-validation were as follows: BCI, (C8 – C4 + C5:1 – C10) > 0.004 AND (C8 – C18:1)/(C14OH + C16OH) >–2.012; and quantile, (C8 – C10 – C5DC – C18:1OH) > 0.24. Both achieved 100% sensitivity and a specificity >99.9% when applied to all specimens analyzed in Heidelberg between July 2002 and January 2006 (including the cases from the study).
Two published diagnostic criteria did not reach 100% sensitivity with the quantile variant, because one of the thresholds used for the 99% quantile led to false negatives. Some specificities were very low, because any individual binary feature created with a linear threshold function using the 99% quantile can by definition reach only 99% specificity on a representative sample. For our data set enriched with acylcarnitine profiles with increased C8, the specificity was even lower. In contrast, the use of the BCI thresholds in the adapted conventionally established diagnostic criteria always gave 100% sensitivity but also led to specificities as low as 58%.
evaluation on external data
We acquired a sample of 5 MCADD and 168 controls, including 7 false-positive screened newborns with increased C8 concentrations, from the screening center in Hamburg, Germany. This independent data set was provided blinded, with no information about the final diagnosis, and was used to validate the new BCI approach for automated thresholding and its efficacy in implementing a safety margin.
Of 4443 different binary features created with the BCI variant in the cross-validation study, 1747 could be used in this external validation, given that some of the analytes used for feature construction in this study were not available in the external data. (Additionally, for one of the cases no concentration of C16:1 was available; for that reason, this case was excluded from computation of sensitivity for 21 features with C16:1 as component.) For validation, these 1747 binary features were calculated from the external data, resulting in a class assignment of each instance through each of these features. The data were then unblinded, and sensitivity and specificity were computed.
From the external data sample, 1743 features achieved 100% sensitivity, whereas 4 features misclassified 1 MCADD case and reached only 80% sensitivity. Specificities ranged from 0.00% to 98.81%.
|
Discussion |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
In this investigation, we demonstrated the usefulness of feature construction for improving the data representation for diagnostic criteria in newborn screening for MCADD. The numeric features, analyte combinations created automatically from intermediary metabolites, showed superior discriminatory performance compared with the initial analytes and manually derived analyte ratios. The decision rules built with the improved representation demonstrated the potential of feature construction and overall exceeded the previously reported diagnostic criteria in terms of diagnostic accuracy.
With the use of BCIs for minimum values in the MCADD group, we introduced an approach to automatically select cutoffs with safety margins. According to Carpenter and Bithell (17), the non-Studentized pivotal method for BCIs can be viewed as a suitable and safe method only to find CIs for the median or 50% quantile. We are using this method to compute CIs for the minimum value or 0% quantile and thus cannot expect to identify correct CIs. Nevertheless this approach enables us to select cutoffs based on the number and the estimated distribution of available cases. In this study, the application of BCI cutoffs was empirically reassessed by cross-validation and by validation on external data with good results: of the cutoffs by BCI as derived during cross-validation, only 2.47% led to misclassifications of MCADD cases in the holdout partition, and only 0.23% of the features were affected in the validation with external data.
The safety level for correct classification of future MCADD cases can be controlled by choosing the confidence probability for the minimal value of a feature in the MCADD group. Unfortunately the problem of selecting the right cutoff values resists a solution beyond doubt when data are very sparse. BCIs for the minimum value are applicable only when a sufficiently large representative sample of the relevant cases can be provided.
In the quantile variant, the distribution of the MCADD cases is ignored, so the chosen quantile for binarization will lead to false negatives for features with insufficient initial discriminatory performance. Thus, any thresholds of the quantile variant should be manually reviewed and changed to reasonable cutoffs. This precaution is also recommended for the BCI variant, but here the thresholds should give a safer starting point for setting cutoffs when a sufficient number of cases has been used for learning. Still, the BCI variant shows one disadvantage compared with the quantile variant: safety margins may be too large, and therefore promising features actually may be discarded because of low residual discriminatory performance.
Our study did not aim for automated application of the classifiers or markers for routine newborn screening. We intentionally applied only methods that allow for easy interpretation and comprehensibility of the constructed markers and decision rules, so that they can be used manually in the clinical validation. The use of up to 4 analytes in a feature and 2 features in a discriminator may be problematic because of the increased complexity of comprehending such diagnostic criteria and the increased risk of measurement errors influencing the features. Nevertheless, if a marker is chosen in accordance with the knowledge of the pathway of the metabolites, and the analytical stability of the analytes is guaranteed, there is no reason against using the marker as evidence to formulate a diagnosis. Examination of the reasons for good individual performance of particular analyte combinations may suggest new hypotheses about the secondary effects of deficient pathways and assist in improving the understanding of the diseases under consideration.
In conclusion, feature construction appears to be a promising method to improve diagnostic accuracy in screening for MCADD with TMS. Our results also indicate that the use of feature construction techniques for high-dimensional metabolic data has the potential to improve diagnostic criteria for other diseases screened with TMS and to generate further hypotheses regarding these diseases.
|
Acknowledgments |
|---|
Financial disclosures: None declared.
Acknowledgments: We thank the editor and the 2 reviewers. Their comments and suggestions on the report and the study helped us to clarify and strengthen several arguments.
|
Footnotes |
|---|
|
References |
|---|
|
TopAbstractIntroductionMaterials and MethodsResultsDiscussionReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
