| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Technical Briefs |
Departments of
1
Medical Education,
2
Family Medicine, and
3 Laboratory Medicine, University of Washington School of Medicine, Seattle, WA 98195;
4
University of Kentucky Hospital, Lexington, KY 40536
aaddress correspondence to this author at: Department of Family Medicine, University of Washington School of Medicine, Box 356390, Seattle, WA 98195; fax 206-543-3821, e-mail sarakim{at}u.washington.edu
Under CLIA, clinical laboratories are required to document the competency of all personnel. Competency assessment assures quality laboratory testing through evaluating employees’ competency and providing remedial training to those who need it (1). Competency assessment can (a) identify key training areas, (b) identify processes that need improvement, (c) provide supervisors and managers with data on employee performance, and (d) provide evidence to customers, management, and surveyors that the laboratory assures quality with competent staff (2).
The most common methods of assessing competency in laboratory personnel are direct observation, review of laboratory test and quality-control results, review of instrument preventive maintenance records, and written testing (1). Computer-based exams have not been widely used as competency assessment tools although they have several potential advantages, including automatic exam scoring and error analysis (3), immediate detailed feedback to exam questions, and the ability to simulate laboratory instrumentation.
In this report, we demonstrate Urinalysis-CA (Urinalysis-Competency Assessment, distributed by Medical Training Solutions, Seattle, WA), a web-based competency assessment system for microscopic urinalysis that is a logical and major extension of Urinalysis-ReviewTM, our previous, diskette-based software (4).
Urinalysis-CA is a web-based competency assessment program administered twice annually to registered laboratories. The program runs optimally using Internet Explorer 4 (or higher) or Netscape 4.0 (or higher).
Content experts at the University of Washington Department of Laboratory Medicine constructed the exams. Three exams were administered during the study, which lasted from March 2000 to September 2001. Each exam consists of 10 image-based, multiple choice questions that test the ability to identify 1 or more of 34 urine sediment structures in five categories: (a) cells, (b) casts, (c) normal crystals, (d) abnormal crystals, and (e) organisms/artifacts. The images were collected from urine sediments prepared in the clinical laboratories at the University of Washington and Harborview Medical Centers (Seattle, WA), using methods described previously (4). An example of an exam question can be viewed in the "Supplemental Files" that accompany the online version of this Technical Brief at http://www.clinchem.org/content/vol48/issue9/.
The following administrative features are available to laboratory supervisors: (a) manage examinee information; (b) generate reports of individual or group performance; and (c) view the contents of specific exams. When managing examinee information, supervisors can add or delete examinees from the laboratory record, view exam dates for each examinee, and delete an examinee’s score. An examinee can take an exam only after the supervisor assigns and submits an identification number for that examinee. To protect privacy, the subscribing institution maintains locally the link between the identification numbers and the examinee names. The web site’s database contains only identification numbers.
Four types of reports can be generated: (a) examinee performance by exam; (b) cumulative performance for all exams taken by an examinee; (c) total laboratory performance for an exam; and (d) cumulative laboratory performance for all exams taken by the laboratory.
From the point of view of documenting compliance with CLIA regulations, the most important report is the examinee performance by exam, which is illustrated in Table 1
. For each examinee who took the exam, this report shows the examinee’s identification number, total exam score across all five categories of sediment structures, and the score for each individual category. Scores are presented as both percentages and fractions so that the supervisor can see how many structures were tested. In addition, a supervisor can choose an exam score that represents the competency threshold, e.g., 85%, thus supporting the CLIA guideline that laboratories have local control over competency assessment. Examinee identification numbers are shown in red if the examinee’s total score is below the threshold and in black if the total score is at or above the threshold. By clicking the identification number of an examinee, supervisors can do a more detailed analysis, allowing them to identify sediment structure categories (e.g., casts) and specific structures within a category (e.g., red blood cell casts) that are problematic to an individual.
|
During this study, 369 unique users from 131 institutions used the program. Nine of the institutions were paying subscribers, and the remaining institutions were using the software in response to a free trial that was available throughout the duration of the study. The 131 institutions were in 39 countries, including 65 institutions in the US.
In Fig. 1
, the performance of a particular laboratory (n = 44 technologists) on the Spring 2000 exam is compared with the performance of the 239 others who took the same exam. The first level of detail, shown in the top panel, is a list of results by sediment structure category. The laboratory’s performance on the Spring 2000 exam was 96% correct for cells, 89% for casts, 99% for abnormal crystals, and 96% for organisms/artifacts. For each sediment category, the laboratory scored similarly or higher than the group performance of the 239 other examinees.
|
The second level of detail is a list of results by specific structures within each category. This is illustrated in the bottom panel in Fig. 1
, which shows performance within the "cells" category for the Spring 2000 exam. The three cell types in the exam were oval fat bodies (one example), red blood cells (four examples), and white blood cells (two examples). Epithelial cells were not tested in the Spring 2000 exam. The display shows that the laboratory scored 99% on red blood cells, 97% on white blood cells, and 80% on oval fat bodies. For each specific cell type, the performance of the laboratory was higher than the group performance of the 239 others who took the exam.
The laboratory discussed in Fig. 1
had the following cumulative results on the three exams (Spring 2000, Fall 2000, Spring 2001): 96% for cells, 93% for casts, 96% for normal crystals, 99% for abnormal crystals, and 92% for organisms and artifacts. The mean number of technologists in the laboratory who took each exam was 48 (range, 44–53). For all sediment structure categories, the laboratory’s cumulative performance was above the cumulative performance of the other examinees on the same three exams. The other examinees’ cumulative scores were 93% for cells, 87% for casts, 86% for normal crystals, 82% for abnormal crystals, and 90% for organisms/artifacts. The mean number of other examinees who took each of the three exams was 171 (range, 131–239).
In this report we described the implementation of Urinalysis-CA, a web-based competency assessment system for microscopic urinalysis. In this preliminary study, we showed the group performance of one laboratory on a single exam in comparison with peer laboratories. A complete picture of the laboratory’s competency will emerge over the next few exams, as more sediment structures are tested.
Urinalysis-CA offers the following improvements over our previous diskette-based software:
Future improvements to the system will focus on enhancing intergroup comparisons by developing a feature that calculates competency results for subsets of users. Subsets of interest include laboratory personnel in general, laboratory personnel by country, laboratory personnel by laboratory size, nonlaboratory personnel (e.g., nurses and medical doctors), and students. Subscribing institutions will then be able to select appropriate results for benchmarking their laboratory competency.
In conclusion, we have demonstrated the feasibility and features of a web-based competency assessment system for microscopic urinalysis. The program allows laboratories to routinely monitor and document the competency of individual technologists and the laboratory as a whole. We are currently in the process of using the program, improving it, and expanding the concept to other areas of the clinical laboratory.
Acknowledgments
We thank Debbie Bremner and the technologists of the clinical chemistry laboratory at Harborview Medical Center, Ann Melchior and the technologists of the clinical chemistry laboratory at the University of Washington Medical Center, and Priscilla Sims and the technologists in the core laboratory of the University of Kentucky Medical Center. In addition, we thank Grace Koon for helping to prepare the figures.
References
The following articles in journals at HighWire Press have cited this article:
|
|
 |
|
  S. Kim, M. Reeves, and M. L. Astion Web-Based Method for Establishing National Competency Benchmarks in Fourteen Areas of Clinical Laboratory Services Clin. Chem., April 1, 2004; 50(4): 753 - 755. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
