March 16-17, 2010; Gaithersburg, MD Day 1 Full Notes Draft

Executive Highlights

Greetings from sunny Gaithersburg, Maryland—spring is certainly in the air and we are excited this highly anticipated FDA meeting concerning SMBG accuracy is finally underway. The first day of the meeting offered provocative discussions from two sessions, one addressing clinical accuracy requirements for SMBG devices and the other addressing SMBG device performance, interferences, and limitations. For each session, we heard varied perspectives from the FDA, clinicians, and industry representatives in enlightening presentations and panel discussions. We were surprised by the distinctly collegial and cooperative tone at the meeting today. This easily could have become a big battle between the agency and industry, with patients and HCPs caught in the crossfire. We are happy to see all players are interested in working together to find solutions. Three issues had particular resonance with us today. First, all parties seem to agree that there should be a separation of at-home use guidelines from in-hospital use guidelines for SMBG devices. This comes as no big surprise, but we believe this will present its fair share of complex challenges to all parties concerned. Second, the buzz-worthy threshold of the day seemed to be 15% acceptable error for 95% of measurements; however, there is less agreement about this number being appropriate for all measures (in the range of hypoglycemic to hyperglycemic values), and for all populations of users (type 1 v. type 2). Third, and very notably, a very visible theme of the day revolved around improving patient education and communication—both the FDA and industry clearly understand there is room for improving the labeling for these devices and the dissemination of information concerning accuracy to end-users. We salute the “big four” for putting resources into this with great energy; we worry that as more pricing pressure moves ASPs down, fewer resources will be directed here. On a side-note, we were happy to hear CGM come up at this SMBG meeting—any patient using a CGM right now has a vested interest in seeing the accuracy of SMBG devices improve to allow for optimal calibration. Below you will find a draft of our full notes. Tomorrow, we look forward to more specific discussion on tight glycemic control in the hospital setting.

 

 

Highlights

• Jeffrey E. Shuren, MD, JD (Director, FDA Center for Devices and Radiological Health, Silver Spring, MD) opened the two-day meeting by highlighting its main goal: to provide a basic framework upon which the FDA can base its discussions regarding revisions to SMBG accuracy requirements. According to Dr. Shuren, the workshop will focus on the clinical need for accuracy in SMBG devices and the reality of what SMBG devices are capable of achieving. Interestingly, speakers noted throughout the day that these two considerations may not share the same conclusion. The FDA has been challenged to consider higher standards for SMBG devices, but there is no current clear consensus on what the appropriate analytical and clinical accuracy standards are. Dr. Shuren closed his short introduction by persuading the audience to keep two critical questions in mind: 1) How should the FDA address the clinical needs of patients considering certain inherent technological limitations; and 2)What steps should industry take to improve point of care blood glucose monitor safety and accuracy?

• William L. Clarke, MD (University Of Virginia School Of Medicine, Charlottesville, VA), creator of the valuable and influential “Clarke Error Grids” used to evaluate the accuracy of SMBG devices, served as moderator for the first session of the meeting, and kicked off the session with a set of thought-provoking questions and comments. He emphasized that it is important to think carefully about what we mean by “accuracy standards for self blood glucose monitors”, before positing there is a distinction between clinical accuracy and analytical accuracy. Reminding the audience of how much benefit SMBG confers, even with current accuracy standards, he mentioned the very significant results of DCCT. According to Dr. Clarke, analytical, statistical, and clinical accuracy each represent a small portion of overall accuracy—there are several limitations to consider with the devices, particularly in the area of interpretation and response to measurements. He concluded by discussing how changing the target range for accuracy influences the Clarke error grid, suggesting that narrowing the target range increases error significantly, and this inevitably opens the door for greater overtreatment. Dr. Clarke closed his session introduction by stating that he feels there is only one decision that can be safely and reliably made using SMBG test results: to eat something if blood glucose is low. We were impressed that he went on to suggest that we may really need to be talking about continuous systems, rates of change, and trends, in order to be making clinical decisions. We are happy to see CGM obtain clear endorsement at this SMBG meeting, though we know some have expressed the devices are not yet technologically advanced enough – time clearly will change this. Our thanks to Dr. Breton for reminding the audience during one of the panel sessions today that the accuracy of CGM devices depends very heavily on the accuracy of the SMBG device used to calibrate – although most generally assume the SMBG device is the most accurate, it is certainly possible that at some point, CGM may be more accurate.

• Patricia Bernhardt, MT (ASCP) (FDA Center for Devices and Radiological Health, Washington, DC) gave a brief review of medical device regulations and the current FDA procedure for evaluation of SMBG device performance standards. SMBG devices are in a category called in vitro diagnostic devices (IVD) and are classified as Class II devices of moderate risk. As such, new SMBG devices are only required to be substantially equivalent to predicate devices—at least as good as but not better than similar devices already on the market. Each meter-strip combo is considered a separate system that is evaluated separately by the FDA. Each sample type also requires a different FDA review and clearance (i.e., capillary whole finger sticks v. arterial sample v. alternate testing sites). Currently, FDA evaluation of point of care blood glucose monitoring devices is based upon ISO-15197 and CLSI (clinical laboratory standards institute) recommendations. The FDA currently requires SMBG devices used by patients at home to be accurate within ±20%, 95% of the time when measuring glucose over 75 mg/dl (this is in comparison to a laboratory reference method). For readings under 75 mg/dl, the devices must be accurate within ±15 points, 95% of the time. The FDA evaluates the devices based on system accuracy (how well the results agree with truth) and linearity (how well the devices compare to true results conformed to a straight line). The agency also evaluates labeling (user manuals, test strip inserts, box/container labels, quick reference guides—all which cannot be written above an 8th grade reading level). Bernhardt concluded by briefly describing the interference from both endogenous and exogenous substances that creates problems for SMBG device accuracy and how the agency evaluates the risk of these interferences. Her parting words were a key reminder—the user of these devices experiences the cumulative effect of all these factors. We note our understanding that to date, all data is effectively self-reported and accuracy data can vary based on strip lots used. Thus, it is actually difficult to compare safety data for various systems; we expect at some point to see some sort of independent organization conduct such an analysis.

• Marc Breton, PhD (University of Virginia, Charlottesville, VA) described his work in in silico modeling of SMBG accuracy. He spent nearly half of his allotted time explaining why in silico modeling is a valuable tool and illuminating how it was developed and is used today—very valuable background for this audience, in our opinion. Dr. Breton emphasized the in silico modeling is designed to allow scientists to simulate risky situations that patients could not safely be subjected to. An in silico patient represents a complex entity of 26 individual parameters and from these individual with great inter-subject variability, an in silico population is created. In 2008, the simulator created by Dr. Breton and colleagues was accepted to replace preclinical studies in closed loop trials and is now the foundation for preclinical work in that domain. This population represents 300 varied adults, children, and adolescents in equal numbers. Dr. Breton went on to describe a study in which the simulator was used to study the effect of the ISO-15197, looking at the rate of detection of hypoglycemia at different levels of accuracy. Dr. Breton found that the probability of missing a hypoglycemic event was reduced dramatically at tighter standards of accuracy. With a 5% accuracy level, there was almost no chance of missing an event, while the chance of missing a hypoglycemic event rose to 4% for accuracy between 10-15%. Notably, Dr. Breton found that the difference in hypoglycemic detection ability differed by a factor of 10 for error at 10% and error at 20%—this represent a non-linear increase in risk between 10-20% acceptable error. In another experiment, Dr. Breton found that the ability to reach target glycemic values degrades with less accuracy as well and this can have a moderate effect on A1c (on the order of 0.2-0.3%) over a longer period of time. We applaud Dr. Breton for bringing up a consideration largely ignored at the meeting today—hypoglycemia is a major concern and it should be the primary concern for accuracy; however, long term inaccurate SMBG measurements leading to under-treatment can affect overall glucose control and prompt long- term complications and are very important in our view as well. Of course we must address the immediate and very scary problem of hypoglycemia first—but we cannot ignore the long term effects of degraded control.

• Before launching into a provocative discussion suggesting that a blanket requirement is not likely the best solution for SMBG device accuracy regulation, Barry Ginsberg, MD, PhD (Diabetes Technology Consultants, Wyckoff, New Jersey) emphasized that these opinions are solely his own. Dr. Ginsberg was a PI in the DCCT (and as we understand his center had one of the lowest average A1cs of all 29 centers in the trial); he is widely respected so we were happy to see that he was on the agenda today. Speaking exclusively about SMBG use outside of the hospital setting, Dr. Ginsberg challenged the idea that the inaccuracies inherent in SMBG devices ultimately impact diabetes management much more than is introduced with insulin correction factors and carbohydrate counting in dosing; in fact, he suggests the rate of error for that dosing adjustment is much higher than the rate of error with SMBG measures, which from a patient perspective is easily believable. Dr. Ginsberg’s bottom line is that the level of accuracy required for an SMBG device should be dependent upon the type of patient under consideration. According to Dr. Ginsberg, for type 2 patients on diet or oral therapy, an error rate of 20% for 95% of measures would be sufficient, whereas an error rate of 15% may be more appropriate for type 2 patients on insulin. For type 1 patients, an error rate of 10% would be more acceptable, and laboratory testing should have an error rate less than 5%. While Dr. Ginsberg had very strong opinions on just about every aspect of this topic, he openly acknowledged that he did not know what to do about outliers (the rare instances of grossly inaccurate readings), but he agreed that they are very important and should be addressed. Dr. Ginsberg also argued it is time to move beyond internal testing by manufacturers; the standards used to test the accuracy of meters needs to be standard for every device and should be conducted by an external organization. Another key point for Dr. Ginsberg is the label, as he argues a key piece to the puzzle will be creating informed consumers by simple, straight-forward standardization of the information. What we find worrisome about Dr. Ginsberg’s suggestion is that such a practice could essentially demographically stratify and hierarchize blood glucose testing (and there is a consensus it is across the board a necessity in some form or another, at some point in the disease process) in the diabetes community. Presumably, the more accurate devices for type 1 patients would be more expensive than the less accurate devices for type 2 patients only on orals. Some may argue this is fair; however, we have to think about how this could play out with those who hold the purse strings. Payors typically demand evidenced based medicine—if stratified accuracy requirements were put in place without very good clinical data to back it up, we could foresee limited coverage of the more expensive meters and problems with access to these more accurate meters, even for patients who “need” them in Dr. Ginsberg’s opinion. While we agree not one size fits all in this situation, we believe it critically important to stay away from classicizing such a basic tool of diabetes care.

• Mitchell Scott, PhD (Washington University School of Medicine, St. Louis, Missouri) opened with some interesting statistics—there are more than 30 SMBG devices listed for home use, and five listed for hospital use. He went on to discuss several of the recent improvements in meters as well as the known interferences, and user-induced errors, with Dr. Scott highlighting how easy it actually is for patients to disrupt accuracy (just manipulating the strips can alter accuracy). There was a particular focus on the role of SMBG in tight glycemic control in his discussion. At his center at Washington University, the rate of strip use has doubled since tight glycemic control was introduced—almost 300,000 strips are being used per year in his ICU. With almost 600,000 total strips being used per year at his hospital, the 5% of meter readings that can fall outside of the 20% error range amount to over 30,000 tests per year that are more inaccurate than 20% of the true reading. He also discussed personal experience with the variability he sees in repeat testing in his hospital; in 357 repeats within an average of 3.5 minutes of the first test, he saw a mean absolute difference of 84 mg/dl (range of 1-454 mg/dl) between the two readings. (Wow. We found this quite surprising.) What these anecdotal glimpses of a particular center tell us is that the inaccuracy of SMBG devices can be greatly magnified if we consider an entire hospital population, instead of just individual users. Dr. Scott emphasized that meters are in fact, improving in terms of accuracy and tighter goals are achievable. Dr. Scott suggests using biologic variability based criteria, which would generally dictate a total allowable error for glucose of less than 8%. Translating this into achievable goals, Dr. Scott would recommend meters are required to be accurate within ±10% or 10 mg/dl (for less than 100 mg/dl). He also strongly advocates the development of a new tight glycemic control error grid.

• In his discussion, David B. Sacks, MD, MB, ChB (Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts) sought to begin to answer the question if meter performance meets clinical needs. An important consideration raised by Dr. Sacks is that the risk for error increases with therapy directed at near- normoglycemia. In his opinion, under the current ISO/CLSI criteria, SMBG device results cannot reliably detect hypoglycemia. While current meters have superior performance to prior generations and there have been technological advances to decrease operator error, Dr. Sacks suggests the performance of testing by patients is usually inferior to testing performed by medical technologists (Clinical Chemistry 2002)—although we heard a rebuttal to this later in the day. The missing piece of the puzzle, according to Dr. Sacks, is the patient opinion of how accurate these devices need to be. In a study of 201 patients of type 1 diabetes (Clinical Chemistry 2001), at normal and elevated glucose values, patients believe the critical typical error threshold to be about 22-30% for clinical purposes, although the analytical error threshold was calculated to be 6.4-9.7%. Interestingly, these preferences excluded hypoglycemia. When patients were asked what level of error they were comfortable with in the hypoglycemic range of blood glucose values, they responded with a striking conclusion of 3.1%. Clearly, accurate measurements in the hypoglycemic range is critically important to both patients and clinicians. Taking a harder line than Dr. Scott over the lack of details concerning glucose analysis in the NICE SUGAR trial, he provocatively suggested the information from the trial is useless—and it has unfortunately changed clinical practice. Dr. Sack’s made a clear point that current meter accuracy is not sufficient for use in trying to achieve tight glycemic control.

• Steve Brotman, MD, JD ( Advanced Medical Technology Association (AdvaMed), Washington, DC) delivered an industry perspective through the lens of his work with AdvaMed. He emphasized that industry shares the goal of improving meter accuracy, especially through reducing the opportunity for user error and the impact of interferents. He argues that it is important to keep in mind that no single SMBG device will meet the needs of all patients. He described several features that have improved in devices over the past decade, noting that most of the usability improvements are beyond the current standards of ISO 15197. According to Dr. Brotman, AdvaMed strongly suggests consideration of error grid analysis as an important tool for objective assessment of blood glucose monitors for self testing and that the impact of new criteria on current meters should be considered as they could mean increased test time, stability, and cost. Here, Dr. Brotman hints at the trade-offs industry believes will be necessary to improve accuracy substantially beyond current standards. Outside of home use, industry fully supports efforts to develop increased accuracy requirements for monitoring in hospitals and long-term facilities. Dr. Brotman highlighted industry’s strong support of revising accuracy standards to ±15mg/dl and 15% of reference values for 95% of individual glucose values and for consensus recognition that the standards for hospital is CLSI POCT 12 (currently under revision and to be published in the near future) and not ISO 15197. Still, Dr. Brotman suggested that ISO 15197 could be acceptable for home self-testing only.

• Gary L. Myers, PhD (CDC Division of Laboratory Sciences, Atlanta, Georgia), moderator for the second session, discussed issues and questions to consider when thinking about factors and interferences influencing SMBG accuracy. Currently, the investigation of interference effects is the responsibility of manufacturers; however, the information provided in adverse event reports is often too vague to be of any substantial value. Additionally, there is no specific single criterion for delineating the presence of significant interference. There is no current consensus among manufacturers about how to appropriately publish guidelines on how interfering substances affect a particular method. Dr. Myers concluded by listing a set of questions he hoped to be answered during the session: should there be standardized procedures of evaluation for interference; how aware of these factors are end-users; are package inserts enough; is more public opinion needed; and is more analysis of sources of information for end users required?

• Courtney Harper, PhD (Director, FDA Division of Chemistry and Toxicology Devices, Center for Devices and Radiological Health, Silver Spring, MD) offered a window into how the FDA tries to detect problems with devices and how the agency reacts to these problems. From the start, she reminded the audience it is important to contextualize this discussion and to realize that without SMBG, even at its current standards, patients would be much worse off than they are today. She went on to review the different types of meters on the market (glucose oxidase or various kinds of glucose dehydrogenase technologies) and to describe their widespread use outside of the indicated “home use” environment. She spent a significant portion of her time discussing adverse event collection through medical device reporting (MDRs), highlighting that it is often difficult to analyze the data from MDRs and to identify trends to guide action on the part of the agency. While events are assumed to be substantially under-reported, the FDA receives over 12,000 MDRs per year related to SMBG use. Dr. Harper lamented the difficulty in analyzing such a huge body of data, especially considering the data set for each report is often missing several key points. She noted it is easier for the FDA to analyze and track mortality data. Although potentially under-reported as well, 100 deaths were associated with SMBG use between 1992-2009. While zero deaths would be ideal, Dr. Harper noted this statistic is not entirely discouraging considering billions of tests are performed each year. For the final portion of her talk, Dr. Harper discussed the GDH-PQQ non-glucose (maltose) interference issue that recently arose with six deaths occurring since 2008 due to this interference. With this discussion, she raised a critical issue—the FDA’s communication and responses to the GDH-PQQ problem did not appear to have a lasting effect. For Dr. Harper, an important question remains what more can the FDA, industry, and HCPs do to prevent unnecessary deaths due to interferences?

• Ken Ervin, MS (Ken Ervin Consulting Services, Brentwood, California) offered his perspective as a long-time contributor to the evolution of SMBG technology. Ervin has served with Abbott Diagnostics and spent a majority of his career working with LifeScan Director of R&D. According to Ervin, in the early days of innovation, it was initially very important to manufacturers to address user error; however, the devices have evolved considerably and have become far more user-friendly. Ervin argues that the limitations of a product are dependent upon the choice of technology to achieve the design goals. The most critical design goals for developers in recent history have included improving ease of use, reducing cost, increasing the rapidity of testing, reducing the required blood sample size, improving stability at room temperature, and eliminating the necessity for user coding or calibration. He went on to identify several sources of interference and suggest that manufacturers are becoming more and more successful at correcting for interfering factors with various innovations. Ervin particularly emphasized that the design goals inevitably entail trade offs, concluding that the unique goals of a SMBG devices (to achieve a balance of high performance, rapid testing, versatility, and ease of use), will make it unlikely that these devices will ever match a lab based system in accuracy.

• Alan Cariski, MD, JD, FACP, FACE (LifeScan, Inc., New Brunswick, NJ) & Mike Flis, BS (Roche, Nutley, NJ) compared lab instruments to at-home glucose meters on several parameters, and at some length, highlighting the trade offs between accuracy and consumer conveniences. Beyond the environmental barriers to accuracy including various interferants already discussed, Dr. Cariski noted there are other barriers manufacturers must consider such as that release criteria depends on sampling and statistical modeling, patent issues, and variability in raw materials. He concluded by noting that it is a huge challenge to industry to manufacture high volumes of glucose meters and test strips while maintaining flawless control—he estimates individuals with diabetes perform 17-18 billion tests annually. Speaking after Dr. Cariski, Mike Flis, BS (Roche, Nutley, NJ) discussed the work of the CLSI to finalize POCT 12 and ISO revisions, noting the first drafts are still coming together and there is still time for our voices to be heard. He noted it is important to consider that only one-third of these devices are used in the US, the rest are used globally. To this end, he advocates for the next version of the international standards to include study design and data presentation guidelines. He also supports the notion of improving patient education through labeling and suggested that industry, the FDA, and HCPs needs to work together to try to understand how best to convey information on the products to the public.

 

Detailed Commentary and Discussion

Opening Remarks

Jeffrey E. Shuren, MD, JD (Director, FDA Center for Devices and Radiological Health, Silver Spring, MD)

Dr. Shuren opened the two-day meeting by highlighting its main goal: to provide a basic framework upon which the FDA can base its discussions regarding revisions to SMBG accuracy requirements. According to Dr. Shuren, the workshop will focus on the clinical need for accuracy in SMBG devices and the reality of what SMBG devices are capable of achieving. Interestingly, speakers noted throughout the day that these two considerations may not share the same conclusion. The FDA has been challenged to consider higher standards for SMBG devices, but there is no current clear consensus on what the appropriate analytical and clinical accuracy standards are. Dr. Shuren closed his short introduction by persuading the audience to keep two critical questions in mind: 1) How should the FDA address the clinical needs of patients considering certain inherent technological limitations; and 2)What steps should industry take to improve point of care blood glucose monitor safety and accuracy?

 

Session 1: Clinical Accuracy Requirements for Blood Glucose Meters

Moderator's Introduction of Session 1

William L. Clarke, MD (University Of Virginia School Of Medicine, Charlottesville, VA)

Dr. Clarke, creator of the valuable and influential “Clarke Error Grids” used to evaluate the accuracy of SMBG devices, served as moderator for the first session of the meeting, and kicked off the session with a set of thought-provoking questions and comments. He emphasized that it is important to think carefully about what we mean by “accuracy standards for self blood glucose monitors”, before positing there is a distinction between clinical accuracy and analytical accuracy. Reminding the audience of how much benefit SMBG confers, even with current accuracy standards, he mentioned the very significant results of DCCT. According to Dr. Clarke, analytical, statistical, and clinical accuracy each represent a small portion of overall accuracy—there are several limitations to consider with the devices, particularly in the area of interpretation and response to measurements. He concluded by discussing how changing the target range for accuracy influences the Clarke error grid, suggesting that narrowing the target range increases error significantly, this inevitably opens the door for greater overtreatment. Dr. Clarke closed his session introduction by stating that he feels there is only one decision that can be safely and reliably made using SMBG test results: to eat something if blood glucose is low. We are happy to see CGM obtain clear endorsement at this SMBG meeting, though we know some have expressed the devices are not yet technologically advanced enough – time clearly will change this. Our thanks to Dr. Breton for reminding the audience during one of the panel sessions today that the accuracy of CGM devices depends very heavily on the accuracy of the SMBG device used to calibrate – although most generally assume the SMBG device is the most accurate, it is certainly possible that at some point, CGM may be more accurate.

• Dr. Clarke lamented a complete lack of superiority studies comparing one meter to another. He also noted that the devices are not believed to be acceptable for diagnosing diabetes by the American Diabetes Association and are not approved for use outside of the home at this time (despite the fact that they are commonly used in the ICU and major outcomes trials such as NICE SUGAR).

• Dr. Clarke highlighted several areas for concern over accuracy in SMBG devices, including hypoglycemia, glycemic variability, failure to achieve A1c targets, cognitive dysfunction, depression, and new uses (such as in the ICU).

• In suggesting that clinical, statistical, and analytical accuracy only amount to a small portion of overall accuracy concerns, Dr. Clarke brought up limitations these devices face. Of particular importance are the limitations relating to interpretation and response to measurements. To support his argument, he cited a Diabetes Care internet survey study suggesting that over 50% of people with diabetes omit insulin, 30% intentionally, on a fairly common basis. According to Dr. Clarke, this is a significant consideration as Dr. Peter Chase’s research suggests omitting just one insulin injection per week can increase A1c by 0.4-0.5%.

 

FDA Perspective: FDA Evaluation of Point of Care Blood Glucose Meters

Patricia Bernhardt, MT (ASCP) (FDA Center for Devices and Radiological Health, Washington, DC)

Bernhardt gave a brief review of medical device regulations and the current FDA procedure for evaluation of SMBG device performance standards. SMBG devices are in a category called in vitro diagnostic devices (IVD) and are classified as Class II devices of moderate risk. As such, new SMBG devices are only required to be substantially equivalent to predicate devices—at least as good as but not better than similar devices already on the market. Each meter-strip combo is considered a separate system that is evaluated separately by the FDA. Each sample type also requires a different FDA review and clearance (i.e., capillary whole finger sticks v. arterial sample v. alternate testing sites). Currently, FDA evaluation of point of care blood glucose monitoring devices is based upon ISO-15197 and CLSI (clinical laboratory standards institute) recommendations. The FDA currently requires SMBG devices used by patients at home to be accurate within ± 20%, 95% of the time when measuring glucose over 75 mg/dl (this is in comparison to a laboratory reference method). For readings under 75 mg/dl, the devices must be accurate within 15 points, 95% of the time. The FDA evaluates the devices based on system accuracy (how well the results agree with truth) and linearity (how well the devices compare to true results conformed to a straight line). The agency also evaluates labeling (user manuals, test strip inserts, box/container labels, quick reference guides—all which cannot be written above an 8th grade reading level). Bernhardt concluded by briefly describing the interference from both endogenous and exogenous substances that creates problems for SMBG device accuracy and how the agency evaluates the risk of these interferences. Her parting words were a key reminder—the user of these devices experiences the cumulative effect of all these factors. We note our understanding that to date, all data is effectively self-reported and accuracy data can vary based on strip lots used. Thus, it is actually difficult to compare safety data for various systems; we expect at some point to see some sort of independent organization conduct such an analysis.

 

Analytical Performance of Blood Glucose Meters: State of the Art

Mitchell Scott, PhD (Washington University School of Medicine, St. Louis, Missouri)

Dr. Scott opened with some interesting statistics—there are more than 30 SMBG devices listed for home use, and five listed for hospital use. He went on to discuss several of the recent improvements in meters as well as the known interferences, and user-induced errors, with Dr. Scott highlighting how easy it actually is for patients to disrupt accuracy (just manipulating the strips can alter accuracy). There was a particular focus on the role of SMBG in tight glycemic control in his discussion. At his center at Washington University, the rate of strip use has doubled since tight glycemic control was introduced— almost 300,000 strips are being used per year in his ICU. With almost 600,000 total strips being used per year at his hospital, the 5% of meter readings that can fall outside of the 20% error range amount to over 30,000 tests per year that are more inaccurate than 20% of the true reading. He also discussed personal experience with the variability he sees in repeat testing in his hospital; in 357 repeats within an average of 3.5 minutes of the first test, he saw a mean absolute difference of 84 mg/dl (range of 1-454 mg/dl) between the two readings. What these anecdotal glimpses of a particular center tell us is that the inaccuracy of SMBG devices can be greatly magnified if we consider an entire hospital population, instead of just individual users. Dr. Scott emphasized that meters are in fact, improving in terms of accuracy and tighter goals are achievable. Dr. Scott suggests using biologic variability based criteria, which would generally dictate a total allowable error for glucose of less than 8%. Translating this into achievable goals, Dr. Scott would recommend meters are required to be accurate within ±10% or 10 mg/dl (for less than 100 mg/dl). He also strongly advocates for the development of a new tight glycemic control error grid.

• According to Dr. Scott, recent improvements in meters include no wipe strips, sample volume detection, data storage and capture (which he deems the most influential advance to occur in the last 5-6 years), smaller sample size, quality control lockouts, and alternate site testing.

• Important interferences affecting strip accuracy include the hematocrit effect (anemia will give higher glucose and polycythemia will give lower glucose), reducing agents (such as acetaminophen), and the oxygen effect (some strips require specific oxygen exposure for proper enzymatic reactions to occur. Dr. Scott noted that some new meters correct for hematocrit and reducing agents, evidence that meters are improving and that current standards can be surpassed.

• Notably, Dr. Scott emphasized that most of the tight glycemic control trials do not specify which meters or methods were used in the studies, an oversight the NICE SUGAR trial is guilty of. Not only will there be differences between meters used, but the test site can also greatly affect the inter-test reliability as it is known that simultaneous testing of arterial, venous, and capillary samples can all produce different results.

 

Questions and Answers

Q: When you look at the clinical data from virtually every meter out there with 100-500 subjects, you do not see outliers and we rarely see these on the Clarke error grid. However, when you look at data from the manufacturers, with 10,000-50,000 tests, you start to see outliers are very real and it looks like some where between 0.3-1% is out in the range of more than 100-200 mg/dl inaccurate. It is a real issue and we need to start thinking about it.

A: Thank you for your comment.

Q: When you talk about repeatability, you were talking about taking the second sample within 15 minutes. What about taking another sample within one minute? When do we stop sampling?

A: If the test is carefully performed they will likely come within 5-6% with two tests. What we are seeing with the repeated values, in my opinion, are user errors, not an error with the device itself.

Q: I disagree with that. Patients become better than the professionals in our studies, because they are more experienced than the professionals. Whenever you produce large numbers of anything you will have outliers.

A: Thank you for your comment.

Q: We tested five different meters, four currently available and a new four-channel meter. We developed a correction factor for that hematocrit and when applied to the regular meters we were able to achieve less than 5% error in our patients. The technology is there with the four-channel meter, but it can be achieved with the current meters too, if you provide the correction. We reduced our rate of hypoglycemia by doing the correction. We did not feel comfortable going to tight glycemic control without improved devices such as the four-channel meter. I also really agree with you, a lot of what we are seeing is probably user-error, especially in the ICU with busy nurses.

A: Thank you for your comment.

 

Clinical Perspective: Clinical Need for Tighter Performance Requirements

David B. Sacks, MD, MB, ChB (Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts)

In his discussion, Dr. Sacks sought to begin to answer the question if meter performance meets clinical needs. An important consideration raised by Dr. Sacks is that the risk for error increases with therapy directed at near-normoglycemia. In his opinion, under the current ISO/CLSI criteria, SMBG device results cannot reliably detect hypoglycemia. While current meters have superior performance to prior generations and there have been technological advances to decrease operator error, Dr. Sacks suggests the performance of testing by patients is usually inferior to testing performed by medical technologists (Clinical Chemistry 2002)—although we heard a rebuttal to this later in the day. The missing piece of the puzzle, according to Dr. Sacks, is the patient opinion of how accurate these devices need to be. In a study of 201 patients of type 1 diabetes (Clinical Chemistry 2001), at normal and elevated glucose values, patients believe the critical typical error threshold to be about 22-30% for clinical purposes, although the analytical error threshold was calculated to be 6.4-9.7%. Interestingly, these preferences excluded hypoglycemia. When patients were asked what level of error they were comfortable with in the hypoglycemic range of blood glucose values, they responded with a striking conclusion of 3.1%. Clearly, accurate measurements in the hypoglycemic range is critically important to both patients and clinicians. Taking a harder line than Dr. Scott over the lack of details concerning glucose analysis in the NICE SUGAR trial, he provocatively suggested the information from the trial is useless—and it has unfortunately changed clinical practice. Dr. Sack’s made a clear point that current meter accuracy is not sufficient for use in trying to achieve tight glycemic control.

• Discussing the NICE SUGAR trial, Dr. Sack remarked, “It is impossible to obtain details concerning glucose analysis in NICE SUGAR. This is absolutely useless information and it has changed clinical practice”. According to Dr. Scott, the different glucose values produced by diverse methods and sampling sites will lead to clinically relevant differences in insulin dosing, especially in the critical care setting. Considering CAP (College of American Pathologists) proficiency testing data for 17 meters, the coefficient of variation was found to be 12-15% with bias up to 41%. In this case, if true glucose is 144 mg/dl, a bias of 41% could give you a reading difference of 59 mg/dl for the same sample. As the difference in mean glucose between intensive and conventional groups in NICE SUGAR was only 29 mg/dl, this would suggest that the bias introduced just by using different methods of glucose sampling could be twice as much as the difference between the two experimental groups. Dr. Sack’s point here is that these devices are not accurate enough to be used for tight glycemic control. According to Dr. Sacks, if a patient has a true glucose of 95 mg/dl, the acceptable range for the meter’s reading would be 76-114 mg/dl (and that is excluding any of the interfering conditions likely to occur in ICU patients). These values exceed the range for the intensive glucose target of 81-108 mg/dl shown to be affective in Dr. Van den Berghe’s work.

 

Questions and Answers

Q: Given the known difference in the way CAP proficiencies vary, is that an appropriate way to evaluate?

A: It is not whole blood and that is a problem. The bias is exaggerated in those studies because of the matrix effects among the different meters.

Q: Would you suggest that the FDA criteria for testing meters and looking at their accuracy should include a specific percentage of values in the hypoglycemic range? The data we are presented with has very few hypoglycemic values typically, but in the real world we see at least 10% of the values in the hypoglycemic range.

A: I think it is very important that the meters perform as accurately as possible in that range.

Q: How can we take NICE SUGAR and use that for a compelling argument for SMBG accuracy? I’m seeing a lot of mixing of results to get an argument here.

A: The first half of my discussion pertained to self-monitoring, the second part to tight glycemic control. Its clearly not only glucose meters used in NICE SUGAR. I am pointing out the limitations of the currently accepted criteria, they could not be used properly in a study of that nature due to the large variation between the meters. Some of the worst outcomes may be due to undetected hypoglycemia. There is no way to know whether that is true or not.

Q: I’m an ICU director, ICU patients have nothing to do with ambulatory patients. More than 50% of all of my admissions have at least one abnormal ionized calcium. More than 80% have at least one abnormal serum phosphorous. You never see that in an ambulatory patient. We heard at the beginning from the regulatory perspective that this is all about intended use. I would ask the panel what in home care is intended for the ICU? Nothing. I would submit that in the ICU, there currently are not adequate technologies to test the hypotheses of the benefits of tight glycemic control. But to suggest that every ambulatory diabetic should be walking around with an instrument qualified for the care of the critically ill is preposterous. I don’t think we need standards for home care that were established in the critical care unit.

A: Those are clearly very valid points. These people are really sick in the ICU. One of the suggestions that comes up is that glucose meters are not approved for use in the ICUs for tight glycemic control. They are used very widely for this reason though, and not just only in this country. Perhaps there could be two criteria: one in the ICUs and one for home. That is an extensive conversation.

Q: Its my opinion, that the migration of handheld glucose meters into the ICU was the result of a lot of perverse incentives. The turnaround times of labs may not be enough to meet the needs of critical care patients, nurse workload, cost… the fact is meters are easier than the lab. These factors drove meters to the bedside. But, they don’t belong there.

A: No rebuttal to your rebuttal I agree with you.

 

Clinical Perspective: Clinical Needs Relative to Insulin Dosing

Marc Breton, PhD (University of Virginia, Charlottesville, VA)

Dr. Breton described his work in in silico modeling of SMBG accuracy. He spent nearly half of his allotted time explaining why in silico modeling is a valuable tool and illuminating how it was developed and is used today—very valuable background for this audience, in our opinion. Dr. Breton emphasized the in silico modeling is designed to allow scientists to simulate risky situations that patients could not safely be subjected to. An in silico patient represents a complex entity of 26 individual parameters and from these individual with great inter-subject variability , an in silico population is created. In 2008, the simulator created by Dr. Breton and colleagues was accepted to replace preclinical studies in closed loop trials and is now the foundation for preclinical work in that domain. This population represents 300 varied adults, children, and adolescents in equal numbers. Dr. Breton went on to describe a study in which the simulator was used to study the effect of the ISO-15197, looking at the rate of detection of hypoglycemia at different levels of accuracy. Dr. Breton found that the probability of missing a hypoglycemic event was reduced dramatically at tighter standards of accuracy. With a 5% accuracy level, there was almost no chance of missing an event, while the chance of missing a hypoglycemic event rose to 4% for accuracy between 10-15%. Notably, Dr. Breton found that the difference in hypoglycemic detection ability differed by a factor of 10 for error at 10% and error at 20%—this represent a non- linear increase in risk between 10-20% acceptable error. In another experiment, Dr. Breton found that the ability to reach target glycemic values degrades with less accuracy as well and this can have a moderate effect on A1c (on the order of 0.2-0.3%) over a longer period of time. We applaud Dr. Breton for bringing up a consideration largely ignored at the meeting today—hypoglycemia is a major concern and it should be the primary concern for accuracy; however, long term inaccurate SMBG measurements leading to under-treatment can affect overall glucose control and prompt long-term complications and are very important in our view as well. Of course, we must address the immediate and very scary problem of hypoglycemia first- but we cannot ignore the long term effects of degraded control. 

 

Questions and Answers

Q: Its important that you consider all the system errors. How do you do errors with carbohydrate counting and insulin absorption?

A: I wanted to see the sole effect of an error of glucose measurement, I can induce carbohydrate counting error, but I want to see the error only for SMBG. For insulin absorption, the patient has a specific one and it won’t change during the day. We are looking at variability between different subjects that are very different form each other. So the patient will have the same insulin absorption all day, but will be very different from patient two.

Q: I feel generally you can simulate for two reasons: to find the trend or to find an exact number. I think for the first application the simulator is great, but for the second application, the simulator is weak. Any minor adjustment in constants with the differential equation can change the results.

A: Simulations are a limited tool and it should not be used outside of these goals. We claim to replace preclinical data. I disagree that any shift in the constants would shift the curve left or right, that would be right if I was only shifting one set of parameters. So with 300 subjects, we re-present the variability we would see in vivo.

 

Industry Perspective: Tighter Performance Criteria for Blood Glucose Meters; Are They Needed?

Steve Brotman, MD, JD ( Advanced Medical Technology Association (AdvaMed), Washington, DC)

Dr. Brotman delivered an industry perspective through the lens of his work with AdvaMed. He emphasized that industry shares the goal of improving meter accuracy, especially through reducing the opportunity for user error and the impact of interferents. He argues that it is important to keep in mind that no single SMBG device will meet the needs of all patients. He described several features that have improved in devices over the past decade, noting that most of the usability improvements are beyond the current standards of ISO 15197. According to Dr. Brotman, AdvaMed strongly suggests consideration of error grid analysis as an important tool for objective assessment of blood glucose monitors for self testing and that the impact of new criteria on current meters should be considered as they could mean increased test time, stability, and cost. Here, Dr. Brotman hints at the trade-offs industry believes will be necessary to improve accuracy substantially beyond current standards. Outside of home use, industry fully supports efforts to develop increased accuracy requirements for monitoring in hospitals and long-term facilities. Dr. Brotman highlighted industry’s strong support of revising accuracy standards to ±15 mg/dl and 15% of reference values for 95% of individual glucose values and for consensus recognition that the standards for hospital is CLSI POCT 12 (currently under revision and to be published in the near future) and not ISO 15197. Still, Dr. Brotman suggested that ISO 15197 could be acceptable for home self-testing only.

• Dr. Brotman listed several usability improvements to meters in the past decade: faster test times, smaller blood samples, overall increased robustness, enhanced meter displays, ergonomic design, non-user changeable unit measure, no coding/calibration or timing, wider temperature range, improved reagent stability, biosensor in addition to photometric technology, plasma reference results, integrated meter and lancing devices, voice simulators for the visually impaired, flagging of test results, innovative software to organize meter data, and less strip handling to reduce use errors and increase ease of use.

 

Questions and Answers

Q: So we would like to have an ISO standard to improve the home use but not the ICU use, if I am understanding, but I’m sure that industry would not want to forego those 250,000 blood tests done in the ICU each year, where is the disconnect here?

A: I think the it best to address that to members of the blood glucose working group who have been talking about this in some detail.

Q: We’re talking about increasing the accuracy of the monitoring devices, how does that affect the accuracy of the dosing instruments. Is it fair to make the monitors more accurate and not make the dosing instruments more accurate as well?

A: That is valid and these are ongoing discussions.

Q: Could you share with everyone the standard that will be in the new CSLI POCT 12?

A: Not at this time.

 

Industry Perspective: Tighter Performance Criteria Are Achievable and Appropriate

Barry Ginsberg, MD, PhD (Diabetes Technology Consultants, Wyckoff, New Jersey)

Before launching into a provocative discussion suggesting that a blanket requirement is not likely the best solution for SMBG device accuracy regulation, Dr. Ginsberg emphasized that these opinions are solely his own. Dr. Ginsberg was a PI in the DCCT (and as we understand his center had one of the lowest average A1cs of all 29 centers in the trial); he is widely respected so we were happy to see that he was on the agenda today. Speaking exclusively about SMBG use outside of the hospital setting, Dr. Ginsberg challenged the idea that the inaccuracies inherent in SMBG devices ultimately impact diabetes management much more than is introduced with insulin correction factors and carbohydrate counting in dosing (in fact, he suggests the rate of error for that dosing adjustment is much higher than the rate of error with SMBG measures, which from a patient perspective is easily believable). Dr. Ginsberg’s bottom line is that the level of accuracy required for an SMBG device should be dependent upon the type of patient under consideration. According to Dr. Ginsberg, for type 2 patients on diet or oral therapy, an error rate of 20% for 95% of measures would be sufficient, whereas an error rate of 15% may be more appropriate for type 2 patients on insulin. For type 1 patients, an error rate of 10% would be more acceptable, and laboratory testing should have an error rate less than 5%. While Dr. Ginsberg had very strong opinions on just about every aspect of this topic, he openly acknowledged that he did not know what to do about outliers (the rare instances of grossly inaccurate readings), but he agreed that they are very important and we should do something about them. Dr. Ginsberg also argues it is time to move beyond internal testing by manufacturers; the standards used to test the accuracy of meters needs to be standard for every device and should be conducted by an external organization. Another key point for Dr. Ginsberg is the label, as he argues a key piece to the puzzle will be creating informed consumers by simple, straight-forward standardization of the information. What we find worrisome about Dr. Ginsberg’s suggestion is that such a practice could essentially demographically stratify and hierarchize blood glucose testing (and there is a consensus it is across the board a necessity in some form or another, at some point in the disease process) in the diabetes community. Presumably, the more accurate devices for type 1 patients would be more expensive than the less accurate devices for type 2 patients only on orals. Some may argue this is fair; however, we have to think about how this could play out with those who hold the purse strings—payors. Payors typically demand evidenced based medicine—if stratified accuracy requirements were put in place without very good clinical data to back it up, we could foresee limited coverage of the more expensive meters and problems with access to these more accurate meters. While we agree not one size fits all in this situation, we believe it critically important to stay away from classicizing the basic tool of diabetes care.

• According to Dr. Ginsberg, information is king in the consumer product industry and he advocates for industry to help patients become informed consumers. He calls for standardized labeling (such as the energy star labeling on appliances or nutrition labeling on food packaging) at a fifth grade reading level and for the addition of a “batting average” type statistic that is easily digestible to consumers and can help them differentiate between different meters.

• Dr. Ginsberg believes clinical testing should be performed externally, initially, and then every 6-12 months on random lots. In his opinion, reasonable failures should require a correction and not a recall. Outlier testing should be continuously performed and likely internally (due to the huge number of tests needed to achieve an “event”).

 

Questions and Answers

Q: You talked about labeling and pitching at a fifth grade level, but it seems to me that the labeling you were talking about putting on the strip bottle could not be interpreted by someone with a fifth grade education?

A: I don’t think that is necessarily true. If you have an insert that explains what the label means, its not that hard to explain or interpret.

Q: I think that your data suggests we are in a much better position than we think, correct?

A: Yes, I would agree with that.

Q: I think that education of the consumer is really important and you have made that point. I don’t think our patients and the mothers of our patients don’t expect error at all. They expect a number to be a number. They call us when they do measurements and they are different. There is a huge opportunity for education here with patients.

A: I agree. Let me point out that in terms of accuracy, a number of people talked about DCCT. The average error in DCCT was between 13-15%.

 

Session 1 Panel Discussion

Patricia Bernhardt, MT (ASCP) (FDA Center for Devices and Radiological Health, Washington, DC), Marc Breton, PhD (University of Virginia, Charlottesville, Virginia), David B. Sacks, MD, MB, ChB (Harvard Medical School and Brigham and Women’s Hospital, Boston, Massachusetts) Steve Brotman, MD, JD ( Advanced Medical Technology Association (AdvaMed), Barry Ginsberg, MD, PhD (Diabetes Technology Consultants, Wyckoff, New Jersey), Mitchell Scott, PhD (Washington University School of Medicine, St. Louis, Missouri), David C. Klonoff, MD, FACP (Mills-Peninsula Health Services, San Mateo, CA), Ellen H. Ullman, MSW, (Children with Diabetes, Boca Raton, Florida), Alberto Gutierrez, PhD (FDA, Washington, DC), & Courtney Harper, PhD (Director, FDA Division of Chemistry and Toxicology Devices, Center for Devices and Radiological Health, Silver Spring, MD)

Questions and Answers

Q: You have tested in children a lot, I can empathize with that as a pediatrician, do you feel that you need a SMBG which is significantly more accurate than you had, and would you be willing to give up something to get more accuracy?

A:

Ullman: My son was one year old when he was diagnosed. When you have a one year old and you see a 360 on a meter, you’re going to dose. If that was really a 240, your kid is going to go hypoglycemic and you’re going to have to get out the glucagon. I would sacrifice several of the convenience factors for accuracy.

Q: Do you believe parents realize that the SMBG devices have inherent inaccuracy?

A:

Ullman: In a recent survey, approximately 40% thought it was accurate within 15%. I bet a lot of people have no idea. I definitely believe clearly labeling would be a great improvement.

Q: I think we should fix what we can right now?

A:

Dr. Ginsberg: The biggest error in the home setting is hand washing. At least four out of five patients don’t wash hands before testing. There is nothing you can do in the meter to protect against that. The average meter is running 0.3-0.5 microliters of blood. That is no more than the dot of a period on the page and it doesn’t take much glucose to raise that drop of blood substantially. I’m not suggesting we don’t need better accuracy of meters, just that not everyone needs a better accuracy of meters. Its appropriate to help people understand what they need.

Dr. Harper: I agree, user error and unawareness of things that might impact a glucose result contribute to error a lot. But that error is not captured in the number in the requirements. That error is above and beyond what the FDA sees when we evaluate a device. We can try to increase education and awareness. I would like to focus on the error we can address. You have the 20% inherent in a system plus any other user error on top of that.

Q: Is there some way for us to inform patients that they should retest?

A:

Dr. Harper: The home user does not have another method to rely upon to check. A false negative would not be as likely to be caught at home.

Dr. Ginsberg: We’re dealing with two separate items together and that is inappropriate. We are talking about outliers, they occur at 1%, but they are not that common. When we talk about 20% error, those that don’t make it the extra 5%, its not going to be a reading of 400 mg/dl, they’re going to be off by 21% or 22%. The change in insulin dosing would not be that great. They will be half a unit to two units off.

Ullman: One unit in a two year old is huge.

Q: From the public health perspective, meter accuracy is critical for patients testing for management and hospital use, but its also very important in the screening arena. Your thoughts on screening using SMBG?

A:

Bernhardt: We do not clear meters for that purpose. We are well aware it happens but they are not approved for that use currently.

Dr. Clarke: I have parents who are worried and screening their kids and they are getting inaccurate readings. If we had something more accurate in terms of screening I would think it would be a major asset.

Dr. Ginsberg: If you had a more accurate meter, moving from 20% to 10% lets say, you’d go from 144 mg/dl to 122 mg/dl, that’s not going to make a major difference in the amount of calls you receive from scared parents. What happens if you test lower than the reality? Someone two hours after a meal, glucose value of 160 mg/dl, you should be looking at them anyway. I think current meters meet these needs.

Dr. Klonoff: I feel that this group has already accomplished a lot. When this meeting was announced there were many who thought this meeting would be confrontational between FDA and industry. I’m seeing this as very collegial and for the most part, the groups are coming together very well. We are all generally agreeing that it would be good to go down to a more tight accuracy (something near 15%). We want something that is achievable for regulatory needs; the clinical needs may not be achievable. Industry is responding to this need. The change in requirements is a natural evolution as the technology improves. It is a gradual process and every once in a while things get tighter and tighter. I think there is a lot of really good work behind the scenes going on by AdvaMed, industry, and the FDA.

Q: If we get a value we like, we accept it, if we get a number we don’t like, we retest. How many of those numbers that we like are actually accurate?

A:

Dr. Clarke: That is an exceedingly important question.

Q: Has anyone done theoretic calculations on the analytic capability of measuring smaller and smaller samples of blood?

A:

Dr. Ginsberg: I think a drop of blood is not an appropriate thing to use in the ICU. I think capillary blood is a bad way to go. I note I am not a hospital expert though.

Dr. Harper: Since there is a need there, how do you design a study to see if there is a device that performs adequately in that ICU population?

Dr. Scott: It’s a very difficult kind of study to do. Your outcomes are going to be fairly rare. Ideally you would compare meters to something that is far more accurate. The number needed would be quite large.

Dr. Harper: Analytically how would you determine for the range of patients seen in the hospital, which of these patients is a capillary sample good for?

Dr. Scott: At mayo, they simultaneously drew three types of samples and I was surprised that capillary seemed the best.

Dr. Klonoff: One feature is to make sure there are enough data points in the hypoglycemic range. When we did our CSLI guidance, we required a certain amount of hypoglycemic points.

Dr. Ginsberg: The 100 patients that you use for an SMBG study is the assumption that out at home, most patients are fairly healthy and similar. You must have a wide variety of races involved of course, but by and large they are similar. Once you go to the ICU, 100 patients is completely insufficient.

Q: Everyone talks about meter accuracy, but isn’t it really the test strip that is the issue. They deteriorate over time. No one mentions glucose control solutions. I’ll run my glucose up and down on purpose to test glucose meters. If I open a vial of test strips, test the first time and it reads 175 mg/dl, then I test again 200 mg/dl, then 225 mg/dl. If I had taken any one of those individually, you realize it is probably confusing to the patient. Hand washing is an interesting point. If you wash your hands but you don’t dry them really well, you get low readings. If you take a strip out of a pack with wet hands you can ruin the whole pack. No one teaches their patients about these issues.

A:

Dr. Harper: Labeling isn’t the end all be all, but it is really important and can help.

Q: The accuracy standards are for populations, correct? The variability that any individual patient will see will be a lot less than 20%.

A:

Dr. Harper: Some of that depends on if you are comparing results or not. It actually could be 20% for a patient with a low hematocrit or something, or more.

Q: The standards say to test on normal subjects. There is a big disconnect in the providers understanding the variability within the meters and what the accuracy of the meters are supposed to be. I believe there should be a requirement within the ICU to test the device on the patients you intend to use it with, not on normal controls. An outcome study is clearly not feasible to do, although you can look at the rates of hypoglycemia using different meters.

A:

Dr. Ginsberg: Hematocrit is much more complicated than that. No meter is a pure whole blood glucose meter and no meter is a full plasma meter. If the hematocrit is not correct, you need to correct the correction factor. It also turns out that the change in the volume of red blood cells can affect the reading in other ways. The surface area of the electrode is key for the current passing, the more red blood cells the more they clog the current. The correction could be up to 50%. If you are correcting using empirical data that is best.

Dr. Klonoff: I think what you’re doing makes sense in a certain group of patients, but I think we need more modeling like Dr. Breton is doing. What are the clinical consequences of error? The modeling studies do require assumptions and I think it would be very nice if others in this room could try to do a study like that. It goes beyond insulin dose accuracy and addresses blood glucose inaccuracy. If we see more of this type of information brought forward we will all have a better idea of what we are dealing with. It is hard for us to get an accurate handle on what we really need.

Q: In the first 25 years of my life, I checked my blood sugar 118,000 times. We’ve come a long way, but we can keep moving the needle forward. In the last three years with CGM, I’ve checked my blood sugar 109,000 times. If the person only checks their blood glucose two to three times a day they do not really know where their blood sugar is for most of the day. Why are we not pushing for CGM more in type 1 patients? I feel that this would bring down user error enormously. That curve telling us where we are going is enormously helpful.

A:

Dr. Harper: We are talking to the CGM manufacturing community to try to get the devices accurate enough in the future where people can use them in more settings.

Dr. Klonoff: I think something should be done to address that 5% outlier group more.

Dr. Breton: What we haven’t discussed is that these devices are used to calibrate the CGM. The 10% error can be enormously magnified if it is used to calibrate the CGM and it could result in as much as 50-100% inaccuracy in the CGM.

Dr. Ginsberg: There are different patient segments. For those who are looking to calibrate their CGM, they may want 5% accuracy. It is not appropriate to suggest that everyone needs that. We ought to publish the accuracy so that the person can select the meter that is right for them.

Dr. Harper: How do we get good enough data to make that decision? What we would like to hear from people is if you have ideas on how you can actually tell how they would work in the real use environment.

Q: CGM would be ideal for intensive IV insulin therapy in the ICU, if we had a reliable device. Interstitial is not good enough. But intravenous sampling would be better—what is the status of these devices?

A:

Dr. Harper: There are some blood gas indwelling devices cleared, but none of continuous nature have been cleared yet—there are some in development.

Q: We have heard 15% over and over again. Is 15% good enough for the clinicians? How many mg/dl should it be below 75 mg/dl. How would you apply those to home vs. clinical settings?

A:

Dr. Ginsberg: I think it depends upon who you are. If you are calibrating CGM, then 15% is not good enough. For tight glycemic control patients, 15% is not good enough. If you are a type 2 on insulin 15% is likely good. If you are a type 2 not on insulin you don’t need 15%. Dr. Breton and I are talking about the same problem, its key to know when you are hypoglycemic.

Dr. Clarke: I have a little problem with the 15% under 75 mg/dl. I think 10% under 75 mg/dl would be much more acceptable, especially for children. You need to be clear in the labeling to say “it will say this above 75 mg/dl”.

Dr. Breton: For hypoglycemia detection, for values in the low range, 15% still will give you the opportunity to miss hypoglycemic events in quite a large number of cases. At 10% you are starting to reduce the # of missed events in quite a dramatic way. Also, I’m wondering where that fixed rate of error below 75 mg/dl came from. It seems to me that even though it is technically challenging I want to be more precise at 50 mg/dl than I am at 75 mg/dl.

Q: What about human input/user error?

A:

Dr. Breton: We’ve noticed CGM calibrated with a YSI is tremendously more accurate than a CGM calibrated by a very good meter. A perfect calibration makes all the difference in a closed loop system.

Dr. Ginsberg: Its critical to realize the YSI is not perfect. It has a 2.5% error. We need something between 5-10% for a calibration device. We’re not too far from that goal.

Q: What would need to happen from a manufacturing point of view to allow meters to become more accurate?

A:

Dr. Brotman: You reach a level where you are going to have to have trade offs. Accuracy v. user errors and interferences. If you are looking for consistency across the board, as you get closer to 10% accuracy, as you get closer and closer to a non-tolerant level of any errors, you need to consider how much these improvements can be sustained across the board?

Dr. Harper: what is hard about it? How can scientists help?

Dr. Klonoff: It strikes me that there are a lot of nice features of monitors that could be sacrificed if it could lead to more accurate values.

Comment: My suggestion would not be to lower the standard, my suggestion would be to enforce the standard you have in the user environment. I know that any of the large strip companies can perform a large clinical trial and hit that mark under certain conditions, but that might not happen in the real world. No monitor in my view is worth anything unless it is safe. We don’t need a one-size fits all. I think that from an FDA point of view, two things have been lost. What is the intended use and what is the indication for use.

 

Session 2: Blood Glucose Meter Performance, Interferences and Limitations

Moderator's Introduction of Session 2

Gary L. Myers, PhD (CDC Division of Laboratory Sciences, Atlanta, Georgia)

Dr. Myers, moderator for the second session, discussed issues and questions to consider when thinking about factors and interferences influencing SMBG accuracy. Currently, the investigation of interference effects is the responsibility of manufacturers; however, the information provided in adverse event reports is often too vague to be of any substantial value. Additionally, there is no specific single criterion for delineating the presence of significant interference. There is no current consensus among manufacturers about how to appropriately publish guidelines on how interfering substances affect a particular method. Dr. Myers concluded by listing a set of questions he hoped to be answered during the session: should there be standardized procedures of evaluation for interference; how aware of these factors are end-users; are package inserts enough; is more public opinion needed; and is more analysis of sources of information for end users required?

 

FDA Perspective: Public Health Notification: Potentially Fatal Errors With GDH-PQQ Glucose Monitoring Technology

Courtney Harper, PhD (Director, FDA Division of Chemistry and Toxicology Devices, Center for Devices and Radiological Health, Silver Spring, MD)

Dr. Harper offered a window into how the FDA tries to detect problems with devices and how the agency reacts to these problems. From the start, she reminded the audience it is important to contextualize this discussion and to realize that without SMBG, even at its current standards, patients would be much worse off than they are today. She went on to review the different types of meters on the market (glucose oxidase or various kinds of glucose dehydrogenase technologies) and to describe their widespread use outside of the indicated “home use” environment. She spent a significant portion of her time discussing adverse event collection through medical device reporting (MDRs), highlighting that it is often difficult to analyze the data from MDRs and to identify trends to guide action on the part of the agency. While events are assumed to be substantially under-reported, the FDA receives over 12,000 MDRs per year related to SMBG use. Dr. Harper lamented the difficulty in analyzing such a huge body of data, especially considering the data set for each report is often missing several key points. She noted it is easier for the FDA to analyze and track mortality data. Although potentially under-reported as well, 100 deaths were associated with SMBG use between 1992-2009. While zero deaths would be ideal, Dr. Harper noted this statistic is not entirely discouraging considering billions of tests are performed each year. For the final portion of her talk, Dr. Harper discussed the GDH-PQQ non-glucose (maltose) interference issue that recently arose with six deaths occurring since 2008 due to this interference. With this discussion, she raised a critical issue—the FDA’s communication and responses to the GDH-PQQ problem did not appear to have a lasting effect. For Dr. Harper, an important question remains what more can the FDA, industry, and HCPs do to prevent unnecessary deaths due to interferences?

• According to Dr. Harper, medical device reporting is sometimes difficult to analyze and it can be difficult to find trends suggesting common failure issues. The data can be misleading because different reporters will disclose more or less data (and rarely the same data points). This lack of consistent data reporting limits the agency’s ability to do analyses. With 12,672 MDRs between 2004-2008 fulfilling the criteria of “serious injuries reported related to SMBG”, the reported failures resulted in: treatment with medications (44.7%), hospitalization (42.7%), therapy/non-surgical treatment (40%), low blood glucose (25%), and hypoglycemia (23.3%). Clearly, this data provides very little information from which the agency can extract trends. As discussed in many other settings, the way questions are asked and how the reporting is conducted may require major revisions (should HCPs be brought in the loop, etc.). Dr. Harper suggested the agency is in the process of trying to improve the reporting methods and analyses so they can identify trends more effectively.

• There were 100 deaths reported to be associated with SMBG failure between 1992- 2009. The largest category was failure of unknown cause in 34 cases. Meter malfunctions (for instance the meter inexplicably switching from mg/dl to mmol/l) occurred in 11 cases. False high results were cited in 11 cases while diabetic keto-acidosis was cited in eight cases. Maltose non- glucose interference in GDH-PQQ meters occurred in 13 cases, six of which occurred after 2008.

• The rise in GDH-PQQ related deaths between 2008-2009 represents the challenges the FDA, manufacturers, and HCPs face in disseminating information regarding interferences affecting SMBG device accuracy. As a reminder, GDH-PQQ enzyme technology is nonselective for glucose and also detects maltose, xylose, and galactose. Certain drugs have been approved containing these sugars, such as extraneal (icodextrin) and intravenous immunoglobins. These drugs were approved with clear warning labels alerting patients and HCPs to the contraindication with GDH-PQQ meters. In patients taking these drugs, GDH-PQQ meters give inappropriately high false results and death can result from over-insulin dosing and severe hypoglycemia. The devices, in addition to the drugs, contain clear labeling of the contraindication. Dr. Harper highlighted previous FDA actions included a MedWatch safety alert in 2005, an FDA patient safety news publication in 2006, two articles put out by CDER and CBER in 2008, and a 2009 black box warning addition for the drug extraneal. Despite the extended effort to clearly communicate the risk to patients and HCPs, six of the deaths related to this contraindication occurred between 2008-2009. Dr. Harper emphasized there are competing interests involved in a situation such as this one for the agency—they want to disseminate safety warnings without scaring patients from testing regularly.

Questions and Answers

Q: Maltose stays in the body for two weeks after extraneal use and the patient may not be getting it in the hospital setting as it is a chronic use medicine. These issues could be affecting the FDA’s success in communicating the issue effectively.

A: Thank you!

 

Analytical Interferences And Physiological Limitations of Blood Glucose Meters

Ken Ervin, MS (Ken Ervin Consulting Services, Brentwood, California)

Ervin offered his perspective as a long-time contributor to the evolution of SMBG technology. Ervin has served with Abbott Diagnostics and spent a majority of his career working with LifeScan Director of R&D. According to Ervin, in the early days of innovation, it was initially very important to manufacturers to address user error; however, the devices have evolved considerably and have become far more user-friendly. Ervin argues that the limitations of a product are dependent upon the choice of technology to achieve the design goals. The most critical design goals for developers in recent history have included improving ease of use, reducing cost, increasing the rapidity of testing, reducing the required blood sample size, improving stability at room temperature, and eliminating the necessity for user coding or calibration. He went on to identify several sources of interference and suggest that manufacturers are becoming more and more successful at correcting for interfering factors with various innovations. Ervin particularly emphasized that the design goals inevitably entail trade offs, concluding that the unique goals of a SMBG devices (to achieve a balance of high performance, rapid testing, versatility, and ease of use), will make it unlikely that these devices will ever match a lab based system in accuracy.

• Design goals over the last decade for manufacturers have included accuracy, precision, specificity, stability at room temperature, rapidity of testing, ease of use, required blood volume, cost, and eliminating user calibration. More recently, manufacturers have begun to focus more on interferences such as oxygen exposure, maltose interference, and the hematocrit effect.

• Interfering substances or conditions that can effect SMBG accuracy include:

  • Endogenous substances and factors such as uric acid, bilirubin, lipidemia, and hemolysis can affect SMBG accuracy.

  • Exogenous substances and factors such as acetominophen, ascorbate, maltose, icodextrin metabolites, mannitol, dopamine, DKA, sample pH and/or viscosity can affect SMBG accuracy.

• Physiological limitations affecting SMBG accuracy include: sample choice (e.g., capillary, venous, or arterial sampling), and hematocrit abnormalities. Another physiological problem was related to the reliance of certain enzymatic chemistry (glucose oxidase) on oxygen—which could become a problem in certain conditions such as high altitude. The GDH-PQQ systems were originally introduced to alleviate this issue, as this chemistry does not need oxygen to react (but the trade-off is that it is not as specific!). According to Ervin, manufacturers are working to develop other enzymatic chemistries to be both stable and specific (such as GDH-NAD and GDH-FAD) and these products are beginning to appear on the market. In terms of the effects of hematocrit abnormalities, most system are calibrated at normal hematocrit. However, whole blood sample hematocrits can vary significantly (up to five fold). A higher hematocrit (65) can produce a -6% bias in the SMBG measurement while a lower hematocrit (25) can result in a +6% bias. This occurs because the hematocrit may influence the access of plasma, or diffusion of glucose to the measurement system, thus suppressing results.

 

Questions and Answers

Q: How many different methods for calibration are there?

A: YSI used to be a popular reference point. Also initially all the testing was intended to be done on capillary blood. Started seeing other reference systems. I don’t have info, a lot is proprietary, there are difference schemes in the way people approach calibration of their devices, some will use equations built into the system to actually describe the behavior, others will make sure that their product behaves in a linear fashion.

 

Industry Perspective: Barriers to Overcoming Interferences and Limitations

Alan Cariski, MD, JD, FACP, FACE (LifeScan, Inc., New Brunswick, NJ) & Mike Flis, BS (Roche, Nutley, NJ)

Dr. Cariski extensively compared lab instruments to at-home glucose meters on several parameters, highlighting the trade offs between accuracy and consumer conveniences. Beyond the environmental barriers to accuracy including various interferents already discussed, Dr. Cariski noted there are other barriers manufacturers must consider such as that release criteria depends on sampling and statistical modeling, patent issues, and variability in raw materials. He concluded by noting that it is a huge challenge to industry to manufacture high volumes of glucose meters and test strips while maintaining flawless control—he estimates individuals with diabetes perform 17-18 billion tests annually. Speaking after Dr. Cariski, Flis discussed the work of the CLSI to finalize POCT 12 and ISO revisions, noting the first drafts are still coming together and there is still time for our voices to be heard. He noted it is important to consider that only one-third of these devices are used in the US, the rest are used globally. To this end, he advocates for the next version of the international standards to include study design and data presentation guidelines. He also supports the notion of improving patient education through labeling and suggested that industry, the FDA, and HCPs needs to work together to try to understand how best to convey information on the products to the public.

 

Questions and Answers

Q: With respect to the strips, a lot of people are getting their strips delivered through mail services, are those strips still accurate?

A: The supply chain and transport is taken into account when we manufacture the strips. What isn’t taken into account is that someone can leave the vial in the sun baking for days.

 

Session 2 Panel Discussion

Gary L. Myers, PhD (CDC Division of Laboratory Sciences, Atlanta, Georgia), Courtney Harper, PhD (Director, FDA Division of Chemistry and Toxicology Devices, Center for Devices and Radiological Health, Silver Spring, MD), Ken Ervin, MS (Ken Ervin Consulting Services, Brentwood, California), Alan Cariski, MD, JD, FACP, FACE (LifeScan, Inc., New Brunswick, NJ) & Mike Flis, BS (Roche, Nutley, NJ)

Questions and Answers

Q: I think it is safe to say more than 13 people have died from this use of GDH-PQQ enzyme. Why have these been allowed to stay on the market?

A:

Dr. Harper: We may not know the extent of the problem, any information you may have discovered we would be happy to review. Any information we get is really important.

Q: What can we do to further mitigate the interference issue? For extraneal we have a global risk management program in place to make sure we have the tools to respond and prevent. This will shortly be approved as a REMS through CDER. What are device companies doing to mitigate risk and train users? What can device manufacturers do?

A:

Dr. Harper: Our recommendations are in the public health records. Healthcare facilities should avoid using GDH-PQQ at all, if they choose to use them we have clear guidance in place. How effective has it been?

Flis: We want to make the best design to avoid issues, but if you cannot achieve that through design, then you must do it through patient education. For instance, prompts could be built into the system that ask, “Is the patient currently or recently (within 2 weeks) on extraneal?”

Q: Where we put the bar for our own standards at OptiScan is we met the ISO even with all the interference you might encounter in the ICU. Can meters meet current ISO standards in an ICU study?

A:

Dr. Cariski: I think this meeting has distinguish between meters for consumers and meters for hospital. The ISO standard by its very statement addresses consumer meters. When you start stacking interferences it becomes a real problem. I don’t know enough about the examples.

Q: Should interference be a part of the total allowable error, or in addition to it?

A:

Flis: It is a challenge when you are trying to stack multiple errors t0 evaluate what their risk might be. You can conduct method comparison studies. I think you would have to look at the regression equation coming out of this study—it should have whatever interfering compounds that are relevant. We are not aware if it would be a broad enough selection of patients to get an idea of the range of interferences possible.

Dr. Harper: We are curious how to asses, I don’t think we are confident we are getting a good sample that is fully representative. I don’t know how much current study designs actually evaluate that. We would all like to see this communicated.

Q: In the lay population, you may have interferences that are not identifiable by the patient. How do we improve reporting?

Ervin: Interferences generally are not sporadic. They tend to reflect a system-wide effect. I also would like to suggest that rather than including interference in the total error, go on and add them. There are things out there that scientists can use to change what they do.

Fils: Ascorbic acid has been showing up in the label, but would the lay person know what that means in their lives?

Dr. Harper: At the FDA we tend to believe that all sources of error should be included in the allowable error.

Q: I expected the barriers to be described in detail enough for us to start working on them from this panel, can we ever get closer to the performance that we need in terms of eliminating the physiology or do we need to look at a different technology?

A:

Ervin: There are at least two companies out there showing this can be done. I don’t know if we can get this technology into the home use situation. I don’t see any long-term barrier to it. As most things when you scale down, it becomes more challenging. I know some are actually targeting home use now.

Comment: As far as interference goes its not so easy in the central lab on analyzers either. It varies over time. I give the device manufacturers their due, it is not easy.

Q: One of the limitations of the technology is the time. I can understand size, small sample, but why this needs to be a five second test instead of a 60 second test I don’t understand.

A:

Dr. Cariski: Clearly the patient preference is for shorter, people have published this before.

Flis: People are living more active lives, they want to be able to discreetly perform the test, fast is important to some patients for this reason—they can test quickly and discreetly.

Q: I think if the patients don’t understand the importance and you try to market a meter that took a minute next to a meter that took five seconds they wouldn’t go for the longer one. They would sacrifice speed for accuracy if we convinced them that they needed it, the problem is they are not convinced yet.

A:

Dr. Cariski: For some technology, more time would not make a difference.

Q: A label can be more transient for devices than for drugs, correct? Can the manufacturer change the label without approval from the FDA?

A:

Dr. Harper: When a drug is approved, the labeling is online and it cannot be modified without the FDA’s approval. This is not the case for class 2 devices. We look at a draft of the labeling and it can be modified by the manufacturer without review by the FDA. There isn’t consistently any resource to obtain device labeling. Because of the nature of the 510-k program, it certainly is possible.

Flis: We cannot remove a warning or expand an indication without approval.

Q: What would it take to make it a standards based comparison?

A:

Dr. Harper I think where we have verification we could do that for safety. We will be looking into 510ks in general.

Q: I think there needs to be more effort put into preventing user errors. Need the device to predict or help mitigate user error.

A:

Flis: We see progress, the meters are now smart enough to tell you if the blood sample is too small.

Q: Need to have different performance criteria in different settings. The problem is, it doesn’t exist. The technology interferences are spread across home care and hospital use. The hypothetical question becomes do you tell those of us in the hospital to keep using what we’re using and promise to get us a better meter in three to five years OR do you tell us to stop what you are doing and go use a blood gas meter or YSI in the hospital. We will have a lot of issues with costs.

A:

Dr. Kariski: Isn’t this a question the hospital can answer for itself?

Q: You talked about the oxygen issue and the difference between arterial and venous blood, but we are often trying to keep blood glucose at a steady level (clamp etc.) in our research and we often arterialize the blood, is that an analytic or physiologic difference (we are warming blood to arterialize).

A:

Ervin: That will have an analytical effect.

Dr. Ginsberg: You essentially increase the oxygen. Not going to create a major oxygen problem but I’m not positive. Blood glucose is not the same in the three sites, partly because there is some glucose extracted by the muscles as you go along. Arterial and capillary blood are close, but venous blood is lower, particularly around a meal.

Q: Is the result that we need to have different requirements for different types of patients settings, neonatal v. ICU for instance?

A:

Dr. Harper: Part of the reason we hosted this meeting is to hear from this community of people about what steps we need to take. There needs to be some enforcement of FDA’s point of view on intended use. These are often used off label and we have feedback that may not be safe while others say we need them in our hospital. How do we make these safe for all patients? We already ask special questions about certain populations.

Q: There is some technology that does address the issue of technology and precision. NKEDP. Looking at creatinine problems in the lab. We have an initiative to standardize creatinine analysis so we can better assess the issues at hand. We should standardize glucose testing. We need harmonization of this approach with ethos. Glucose is glucose, we really need to give accurate and precise glucose measurements to our clinicians so they can make accurate decisions about dosing a lethal drug.

A:

Dr. Myers: One of the things that we’ve looked at the CDC in the past is that very issue of a standardization program for blood glucose meters. One of the challenges we have is we do not have a good reference material that simulate what the blood glucose meters are actually measuring. How do you come up with a good reference material?

Q: We have already heard that there are some meters meeting a more stringent standard. What incentives can we provide to get more manufacturers to catch up? Should there be something in place like a sunset law, if you still haven’t met the new criteria in five years, we pull it?

A:

Ervin: I would be surprised if every manufacturer is not already trying to get to ±10%. I know there are programs in development that have that as their target. I think that is on the near horizon. It is an important goal of all these companies to get there.

Flis: My sense is when individuals become accustomed to using a system they want that to continue to be available to them if it is reliable to that individual.

Dr. Cariski: It should be based on risk as one can assess it. For example, it may be that one wants to look at different risk categories. It may not be appropriate to require people who don’t need that accuracy to get a more accurate meter. May have to make that distinction in the labeling. Of course, the sunset law policy is what they do in Europe.

--by Jessica Swienckowski & Kelly Close