Six of Eight Hemoglobin A Point-of-Care Instruments Do
Not Meet the General Accepted Analytical Performance Criteria
Erna Lenters-Westra1,2* and Robbert J. Slingerland1,2
BACKGROUND: Hemoglobin A1c (Hb A1c) point-of-care (POC) instruments are widely used to provide rapidturnaround results in diabetic care centers. We investigatedtheconformanceofvariousHbA1cPOCinstru-
ments (In2it from Bio-Rad, DCA Vantage from Siemens, Afinion and Nycocard from Axis-Shield, Clover from Infopia, InnovaStar from DiaSys, A1CNow from Bayer, and Quo-Test from Quotient Diagnostics) with generally accepted performance criteria for Hb A1c.
METHODS: The CLSI protocols EP-10, EP-5, and EP-9 were applied to investigate imprecision, accuracy, and bias. We assessed bias using 3 certified secondary reference measurement procedures and the mean of the 3 reference methods. Assay conformance with the National Glycohemoglobin Standardization Program (NGSP) certification criteria, as calculated from analyseswith2differentreagentlotnumbersforeachHbA1c method, was also evaluated.
RESULTS: Because of disappointing EP-10 results, 2 of the 8 manufacturers decided not to continue the evaluation. The total CVs from EP-5 evaluations for the different instruments with a low and high Hb A1c value were: In2it 4.9% and 3.3%, DCA Vantage 1.8% and 3.7%, Clover 4.0% and 3.5%, InnovaStar 3.2% and 3.9%, Nycocard 4.8% and 5.2%, and Afinion 2.4% and 1.8%. Only the Afinion and the DCA Vantage passed the NGSP criteria with 2 different reagent lot numbers.
CONCLUSIONS: Only the Afinion and the DCA Vantage met the acceptance criteria of having a total CV 3% in the clinically relevant range. The EP-9 results and the calculations of the NGSP certification showed significant differences in analytical performance between different reagent lot numbers for all Hb A1c POC instruments.
Diabetes is one of the most challenging health problems of the 21st century. The International Diabetes Federation estimates that more than 250 million people around the world have diabetes (1). Currently diagnosis and follow-up are usually done in special diabetes care centers. Many patients have their blood drawn a week before they visit the physician to ensure that laboratory results are available for appropriate clinical action. By providing results rapidly following blood collection, point-of-care (POC)1instruments could minimize patient inconvenience and possibly avoid an extra visit to the clinic. Studies have confirmed that immediate feedback of hemoglobin A1c (Hb A1c) results improves glycemic control in type 1 and insulin-treated type 2 diabetic patients (2–4).
Limited information is available regarding the analytical performance of POC instruments that measure Hb A1c and whether National Glycohemoglobin Standardization Program (NGSP) certification ensures the accuracy of every instrument used in the field. The information provided by the manufacturers and the limited published data about the performance of POC Hb A1c instruments suggest that some of these instruments can compete with clinical laboratory methods in terms of analytical performance (5, 6).
Received May 14, 2009; accepted October 7, 2009. Previously published online at DOI: 10.1373/clinchem.2009.130641
The aim of this study was to evaluate all available Hb A1c POC instruments according to CLSI protocols and to check whether the instruments would pass the NGSP criteria with 2 different reagent lot numbers as judged by comparison with 3 certified IFCC and/or NGSP secondary reference measurement procedures. A manufacturer NGSP certification is performed by experienced technologists at the manufacturer’s site under ideal circumstances and may not reflect the analytical performance of the instruments in the field.
Materials and Methods
The 8 POC Hb A1c analyzers evaluated in this study were:
The DCA VantageTM (Siemens Medical Solutions Diagnostics, Tarrytown, NY), which is based on latex agglutination inhibition immunoassay methodology and provides results in 6 min. This is the successor of the DCA 2000TM.
TheAfinionTM(Axis-Shield,Oslo,Norway),whichis based on affinity separation, with results available in 5 min.
The Nycocard (Axis-Shield), which is based on affin-ity separation, with results available in 3 min.
TheClover(Infopia,Kyunggi,Korea),whichisbasedon affinity separation, with results available in 5 min. • The InnovaStar (DiaSys, Holzheim, Germany), which is based on agglutination immunoassay and provides results in 11 min. At the time of this study the InnovaStar was not yet launched on the market andthemanufacturerconsideredtheoutcomeofthis evaluation as a starting point to further improve the method.
The A1CNow(Bayer HealthCare, Sunnyvale, CA), which is an immunoassay with results available in 5 min.
Quo-TestTM (Quotient Diagnostics, Surrey, UK), which is based on affinity separation and the use of fluorescence quenching, with results available in 3 min.
ApartfromtheInnovastarallmethodswereNGSP certified as of May 2009 (7).
We used the CLSI EP-10 protocol to become familiar with the instruments and to get an overall impression of performance (8). The results were sent to the manufacturers for their approval to continue with the evaluation. After we obtained manufacturer’s approval, we used the CLSI EP-5 protocol to further investigate assay imprecision (duplicate measurements twice per day on 2 samples for 20 days) (9). In contrast totheotherinstruments,theAfinionandtheNycocard do not work with hemolyzed material. Therefore, for this purpose with those 2 instruments we used the 2 controls supplied by the manufacturer.
The CLSI EP-9 protocol was performed twice with 2 different reagent lot numbers, and was used to investigate the bias between the POC instruments and the 3 different secondary reference measurement procedures (n 40, 5 days, duplicate measurements) (10). Hb A1c value determination of the samples was performed with 3 certified secondary reference measurement procedures:
Roche Tina-quant Gen.2 Hb A1c on Integra 800, immunoassay, IFCC and NGSP certified (Roche
Primus Ultra2, affinity chromatography HPLC, IFCC and NGSP certified (Primus Diagnostics, a Trinity Biotech Company).
Thesecondaryreferencemeasurementprocedures have documented good results in the IFCC and NGSP monitoring program and were calibrated by using the IFCC secondary reference material with assigned IFCC and derived NGSP values (11–13). To check overall calibration and bias independently of the chosen secondary reference method, the results of the POC instruments in the EP-9 procedure were compared to the mean of the 3 reference measurement procedures. The overall differences in slope and intercept of the regression lines with respect to the 2 reagent lot numbers usedweretestedbyChowstatisticsinSPSSversion16.0 with a univariate general linear model that incorporated an interaction-term (lot number * method). A P value of the interaction-term of 0.05 was considered as statistically significant (14).
The results of the EP-9 protocol were also used to calculate the NGSP certification criteria with 2 reagent lot numbers and 3 different reference measurement procedures. The 95% CI of the differences between methods (test method and reference method) should fall within 0.85% Hb A1c to pass the NGSP criteria. We used the formula: Total error bias 1.96 SD of differences (15).
We performed computations using Microsoft® Excel 2002 (Microsoft Corporation) software. Statistical analyses were also performed with the software package Analyse-It (Analyse-It Software), EP Evaluator Release 8 (David G. Rhoads Associates) (16) and SPSS version 16.0 (SPSS).
Table 1. EP-5 total CV imprecision results from the different POC instruments.
Patient sample 1 4.9% (5.1%)a
Patient sample 2 3.3% (11.2%)
Nycocard normal control
Nycocard abnormal control
Afinion control CI
Afinion control CII
a Hb A1c value of the sample/control are in parentheses.
Two of the 8 manufacturers (local distributor of the A1CNow instrument, and Quotient Diagnostics of the Quo-Test instrument) concluded that the EP-10 outcome data did not warrant progression to the EP-5 and EP-9 protocols and decided to discontinue the study (data not shown). At the time of this study, the QuoTest was a prelaunch instrument and was still in development. The bias found with the EP-10 protocol of the A1CNow was probably due to EDTA interference problems. Normally Hb A1c POC instruments are used to measure Hb A1c directly in capillary blood. Both methods were NGSP certified.
The results of the EP-5 protocol are shown in Table 1. Imprecision ranged from 1.4% CV at an Hb A1c value of 6.3% for the Afinion to 5.3% CV at an Hb A1c value of 6.1% for the Nycocard.
The results of the EP-9 protocol are shown in Table 2, along with the calculations of the NGSP certification criteria and associated P values. The different POC instruments were compared to the 3 reference measurement procedures with 2 different reagent lot numbers. None of the instruments passed the NGSP criteria with 2 lot numbers compared with 3 reference methods. Only the DCA Vantage and the Afinion passed the current NGSP criteria with 2 different lot numbers when compared with just 1 reference method thathadthesamemeasurementprinciple.Basedonthe Chow-statistics testing for differences in regression lines with respect to the lot numbers used, all regression lines except In2it vs Tina-quant were statistically significantly different (Table 2).
The graphs of the comparisons between the different POC instruments with 2 reagent lot numbers and the mean of the 3 reference measurement procedures are shown in Fig. 1. In addition to the Chow-statistics, which demonstrated between-lot differences in the regression lines, the differences in mean bias between the lot numbers of all instruments seen in Fig. 1 reflected lot number instability, and were largest for the Clover (Clover 0.82, DCA Vantage 0.36, Nycocard 0.29, In2it 0.23, Afinion 0.18, InnovaStar 0.15).
There is demonstrated benefit in using POC instruments for the measurement of Hb A1c in certain clinical situations(2–4),butrecentlyconcernshavebeenraisedabout the performance of NGSP-certified POC instruments compared with laboratory-based methods (17). The overall imprecision as determined by means of an EP-5 protocol is very important for interpretation of Hb A1c results (variability in the patient vs analytical variability). The Diabetes Complication Control Trial found that a 10% reduction in Hb A1c levels resulted in a 43%–45% lowering of risk of retinopathy (18). For optimal clinical monitoring and for effective differentiation of an Hb A1c of 7.0% from that of 7.6% an imprecision of 2% CV is required,assuminganintraindividualbiologicalvariation of 2% (19, 20). This criterion is very strict, however, and difficult to meet, even for certain laboratory-based methods (immunoassays). It would therefore seem inappropriatetoimposethisgoalonPOCtestingdevicesmeasuring Hb A1c. Currently, an imprecision of 3% CV is a more realistic, though not optimal goal (21). Only the Afinion and the DCA Vantage were able to meet this criterion in the clinically relevant range (Table 1). The acceptable CVs of these 2 methods make them potentially equivalenttolaboratory-basedmethods,iftheproblemof lot number instability is resolved and assured.
All of the instruments showed statistically significantly different regression lines for the different lot numbers compared to the mean of the 3 reference methods (Fig. 1). The calibration of the In2it is adequate but the variability of the instrument reflected by a high total CV in the EP-5 protocol, and a high standard error of estimates with the first lot number is still amatterofconcern.Thesecondlotnumbergavebetter
Fig. 1. Hb A1c results for 2 different lot numbers from (A) the DCA Vantage, (B) Afinion, (C) In2it, (D) Clover, (E) Nycocard, and (F) InnovaStar POC instruments compared to the mean Hb A1c results from 3 secondary reference measurement (SRM) procedures.
The P values of the regression lines between the 2 lot numbers of all POC instruments were 0.001, which confirmed the statistically significant differences between the regression lines.
Fig. 1. Continued.
results. Unfortunately, it is impossible to predict ber is acceptable because no duplicate measurements whether the precision of a particular reagent lot num- are run routinely with POC-instruments.
Fig. 1. Continued.
The first reagent lot number of the DCA Vantage recent evaluation of the DCA Vantage also showed lower showed slightly lower results in the clinically relevant results compared with the laboratory method; therefore range with a low variability (1.8% CV) and higher results adjustment of the calibration by the manufacturer was in the high range with higher variability (3.7% CV). A justified (22). However, Fig. 1A shows that the manufac-
turer of the DCA Vantage may have overcompensated in adjusting the calibration of the second lot number.
TheAfinionalsodemonstratedacalibrationproblem. The results were consistently lower than the results from the reference methods, independent of the lot number used.
The Nycocard system showed the worst imprecision of all the systems (Table 1) raising questions regarding its suitability for clinical use. The manual nature of this test may possibly explain the poor precision. The CVs presented here were obtained by the work of an experienced technologist and would likely be worse if the method were used by many different inexperienced personnel. However, the Nycocard passed the NGSP criteria with the second lot number compared with 3 reference methods. The bias of the second lot number was very small, which allowed a higher SD of differences.
The lot number dependency of the Clover was unacceptable (Fig. 1D), and the total imprecision was also too high for optimal clinical use. Because of the poor results seen with the first lot number, the software version of the instrument was successfully updated. All results of the controls were within the limits provided by the manufacturer, whereas the patient results of the first lot number proved to be too low. As a possible way to address such problems, manufacturers should be encouraged to narrow the range of acceptable values for provided QC materials sufficiently to enable users to meet the requirements for good clinical test results.
The InnovaStar method was still under development at the time of this study. The manufacturer regarded the outcome of this study as a starting point to further improve the method. In general lower results were obtained compared with the reference methods.
The measurement principle used with 5 of the 8 methods was affinity separation. This measurement principle is well accepted as being free of interference from hemoglobin variants, a very important attribute for use in areas of the world with a high prevalence of hemoglobinopathies.Healthcareprofessionalsmustbe aware of potential interferences of rare hemoglobin variants, especially when they use immunoassay-based POC instruments (23, 24).
TheNGSPuses1comparativesecondaryreference method for certification, which is usually the same method type. The NGSP also states that manufacturer certification is performed only once per year with 1 lot of reagent and it is up to the manufacturer to ensure consistency among different lots (7, 15). Passing or failing outcomes for NGSP certification of the tested POC methods are clearly dependent on lot number and reference method (Table 2). The NGSP criterion (which specifies that the 95% CI of the differences between methods should fall within 0.85% Hb A1c) will be tightened to 0.75% Hb A1c by January 2010 (25). When this criterion is taken into account, only 9 of the 36 comparisons would pass NGSP criteria and only the DCA Vantage would pass it with 2 different lot numbers compared with just 1 reference method.
The reproducibility of the production of the different reagent lots of the POC instruments investigated appears inadequate at this moment for optimal clinical use of the test results. A manufacturer NGSP certification does not guarantee accuracy of a result produced in the field. We often observed significant differences between lots of reagents in this study. The Nycocard instrument data demonstrated that it is possible to pass the NGSP criteria while the total CV is 5%. Adjustments or additions to the criteria might be considered by the NGSP. For example, we believe the SD of differences should not exceed 0.30% Hb A1c. However, a manufacturer NGSP certification is still necessary and is an important tool to prove the optimal analytical performance of a method. In addition users of POC instruments should be required to run daily controls withtightrangesand,aswithanyHbA1cmethod,users should participate in external proficiency-testing schemes.
It is important that the limitations of current POC instruments and laboratory methods be understood by healthcare professionals, because these limitations may have important clinical implications. Clinical chemists can play a valuable role by providing healthcare professionals with the information they need (measurement uncertainty) to properly interpret laboratory and POC Hb A1c results.
AuthorContributions:Allauthorsconfirmedtheyhavecontributedto the intellectual content of this paper and have met the following 3 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; and (c) final approval of the published article.
Authors’ Disclosures of Potential Conflicts of Interest: No authors declared any potential conflicts of interest.
Role of Sponsor: The manufacturers provided all instruments and reagents at no cost. The evaluation protocol was designed by the authors and sent to the manufacturers for comments when they were invited to join the evaluation. None of the manufactures had comments on the design of the study. The manufacturers had the right to discontinue the evaluation after the EP-10 was finished. The manufacturers played no role in the review and interpretation of data or preparation or approval of manuscript, and had no rights to refusal for publication of the data. The funding organizations played no role in the design of study, choice of enrolled patients, review and interpretation of data, or preparation or approval of manuscript.
Acknowledgments: We thank Axis-Shield (Norway), Bio-Rad (the DiaSys (Germany), Quotient (United Kingdom), and EuroMedix Netherlands), Siemens (the Netherlands), Infopia (South Korea), (Belgium) for donating reagent and instruments used in this study.
International Diabetes Federation. Diabetes facts. http://www.idf.org (Accessed November 2009). This is a section on the home page.
Cagliero E, Levina EV, Nathan DM. Immediate feedback of HbA1c levels improves glycemic control in type 1 and insulin-treated type 2 diabetic patients. Diabetes Care 1999;22:1785–9.
Ferenczi A, Reddy K, Lorber DL. Effect of immediate haemoglobin A1c results on treatment decisions in office practice. Endocr Pract 2001;7: 85–8.
Miller CD, Barnes CS, Phillips LS, Ziemer DC, Gallina DL, Cook CB, et al. Rapid A1c availability improves clinical decision-making in an urban primary care clinic. Diabetes Care 2003;26: 1158–63.
Greaves RF, Northfield JT, Cameron FJ. Haemoglobin A1c: evaluation of three point of care analysers for use in a paediatric diabetes clinic. Ann Clin Biochem 2005;42:124–9.
Hawkins RC. Comparison of four point-of-care HbA1c analytical systems against central laboratory analysis. Singapore Med J 2003;44:8–11.
NGSP. List of NGSP certified methods (updated 11/09, listed by date certified). http://www. ngsp.org/prog/methods.pdf (Accessed November 2009).
CLSI. Preliminary evaluation of clinical chemistry methods; approved guideline. CLSI Document EP10-A. Wayne (PA): CLSI; 1998.
CLSI. Evaluation of precision performance of clinical chemistry devices; approved guideline. NCCLS Document EP5-A. Wayne (PA): CLSI; 1992.
CLSI. Method comparison and bias estimation
using patient samples; approved guideline. CLSI Document EP9-A. Wayne (PA): CLSI; 1992.
Little RR, Rohlfing CL, Wiedmeyer H-M, Myers GL, Sacks DB, Goldstein DE. The National Glycohemoglobin Standardization Program (NGSP): a five-year progress report. Clin Chem 2001;47: 1985–92.
Little RR. Glycated haemoglobin standardization. National Glycohemoglobin Standardization Program (NGSP) perspective. Clin Chem Lab Med 2003;41:1191–9.
IFCC Working Group on standardization of HbA1c. Approved laboratories of the IFCC Network Laboratories for HbA1c. http://www.ifcchba1c.net/ IFCC_01.asp?typeN (Accessed November
Chow G. Test of equality between sets of coefficients in two linear regressions. Econometrica 1960;28:591–605.
NGSP. Protocol: C. standardization and certification of methods by manufacturers. http://www. ngsp.org/prog/protocol/prot.html (Accessed November 2009).
Rhoads DG. EP Evaluator Release 8. DG Rhoads Associates Inc. http://www.dgrhoads.com (Accessed April 2009).
Homes EW, Ers¸ahin C, Augustine GJ, Charnogursky GA, Gryzbac M, Murrell JV, et al. Analytic bias among certified methods for the measurement of hemoglobin A1c, a cause for concern? Am J Clin Pathol 2008;129:540–7.
The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complication in insulin-dependent diabetes mellitus. . N Engl J Med 1993;329:977–86.
Goodall, I, Colman PG, Schneider HG, Mclean M, Barker G. Desirable performance standards for HbA1c analysis–precision, accuracy and standardization: Consensus statement of the Australian Association of Clinical Biochemists (AACB), the Australian Diabetes Society (ADS), the Royal College of Pathologists of Australia (RCPA, Endocrine Society of Australia (ESA), and the Australian Diabetes Educators Association (ADEA). Clin Chem Lab Med 2007;45:1083–97.
Westgard QC. A war of words in laboratory medicine, part iv: the quality of glycated hemoglobin (Ghb). http://www.westgard.com/essay120.htm (Accessed November 2009).
Shephard M. Analytical goals for point-of-care testing used for diabetes management in Australian health care settings outside the laboratory. Point Care 2006;5:177–85.
Szymezak J, Leroy N, Lavalard E, Gillery P. Evaluation of the DCA Vantage analyzer for HbA1c assay. Clin Chem Lab Med 2008;46:1195–8.
Roberts WL, Frank EL, Moulton L, Papadea C, Noffsinger JK, Ou CH. Effects of nine hemoglobin variants on five glycohemoglobin methods. Clin Chem 2000;46:560–76.
NGSP. HbA1c test methods and hemoglobin variants (HbS, HbC, HbE and HbD traits). http://www.ngsp.org/prog/index3.html (Accessed November 2009).
NGSP. NGSP news. Changes to NGSP certification criteria beginning in 2010. http://www.ngsp.org/ prog/index3.html (Accessed November 2009).
1 Department of Clinical Chemistry, Isala klinieken, Zwolle, The Netherlands; 2 European Reference Laboratory for Glycohemoglobin, Zwolle, The Netherlands.
* Address correspondence to this author at: Isala klinieken, Department of Clinical Chemistry, Groot Wezenland 20, 8011 JW Zwolle, The Netherlands.