Today’s blog will discuss regulatory trends in reviewing benefit/harm assessments in medical interventions. The important question here is: how do we balance the two? When we use the term “benefits” we are referring to “the good” that actually happens to patients by prescribing medical interventions; more specifically, the ability of medical intervention to improve outcomes for patients in terms of decreasing symptoms, improving function, or improving survival. It could also mean fewer adverse effects compared to other interventions. Benefit is also termed “efficacy” or “effectiveness.”
Harms, on the other hand, are “the bad.” They are the adverse unwanted consequences reasonably associated with use of medical interventions. These can include things like signs, symptoms, lab values, vital signs, ECG, etc. Harms are often erroneously termed “safety.” However, no intervention is completely “safe” in terms of absence of all harms. Safety is really the balance of benefits vs. harms, not just harms alone. This concept was recognized early on in the history of FDA, even before the 1962 requirement for effectiveness.
Prior to 1938, there was no premarket review, only response to crises. However, based on the sulfanilamide tragedy that year, drugs had to be shown as “safe” prior to marketing and there was recognition that efficacy was important in this consideration. “If the drug that killed one person in ten thousand was of only minor use therapeutically, it might still be judged to be unsafe, whereas the drug that killed one in a thousand persons, if it had marked and undisputed therapeutic value it would still be a safe and valuable drug.” [J.J. Durett, Chief, Drug Division, FDA, December 1938] In other words, “safety” depends upon context of use – the magnitude of benefit, in what patient population, for what disease and at what dose/exposure.
In 1962, the requirement to demonstrate efficacy (benefits) to justify any potential harms, prior to marketing, was established. The standard of efficacy is “substantial evidence” from “adequate and well-controlled studies.” Efficacy is not based on p-values alone. You need to show clinically meaningful differences as well as statistical significance. [Warner-Lambert v Heckler 1986] This entails judgment of what is considered “clinically meaningful.” A better way to frame that judgment is to have actual evidence from patients about what is meaningful for them through the development and use of patient reported outcomes (PROs).
The standard for evaluating harms is not very clear in terms of law and regulations. In the Federal Food, Drug, Cosmetic Act, section 505 it states that “adequate tests by all methods reasonably applicable to show whether or not such drug is safe for use under the conditions prescribed, recommended, or suggested in the proposed labeling.” “Adequate tests” are very contextual depending upon what kind of patients and medical interventions you are studying. During a recent court case [Matrixx vs. Siracusano, March 22, 2011], Justice Sotomayor stated statistical significance is not necessary to show harm – “Because adverse event reports can take many forms, assessing their materiality is a fact-specific inquiry…Something more than the mere existence of adverse event reports is needed to satisfy that standard, but that something more is not limited to statistical significance and can come from the source, content and context of the reports.” To address the need for improvement in the clarity and transparency of the FDA’s benefit-risk assessment in human drug review, the FDA has made recent efforts to move toward more structured assessments, entailing both quantitative and qualitative types of decision making.
Current FDA Thinking:
As part of Prescription Drug User Fee Act (PDUFA) V negotiations, the FDA was tasked to develop structured benefit-risk assessment to serve as a template in product reviews. In February 2013, a document entitled “Structured Approach to Benefit-Risk Assessment in Drug Regulatory Decision Making: Draft PDUFA V Implementation Plan” was published to lay out the thinking on the approach to the assessment of benefits and harms when reviewing medical interventions.
One of the things people have been pushing the FDA to do is to use a more “quantitative” approach to benefit-harm assessments. On page four of this document it states: “The term ‘quantitative benefit-risk assessment’ can have various meanings depending on who is asked. Some hold the view that a quantitative benefit-risk assessment encompasses approaches that seek to quantify benefits and risks, as well as the weight that is placed on each of the components such that the entire benefit-risk assessment is quantitative.” “This approach is typical of quantitative decision modeling. It usually requires assigning numerical weights to benefit and risk considerations in a process involving numerous judgments that are at best debatable and at worst arbitrary. The subjective judgments and assumptions that would inevitably be embodied in such quantitative decision modeling would be much less transparent, if not obscured, to those who wish to understand a regulator’s thinking.” In other words, if you reduce this to only a number, the meaning behind that number will be lost. So, much like efficacy assessments, you cannot boil down assessments of harm to a “p value” which is independent of its clinical meaningfulness.
This document provides a very helpful way to think about doing benefit/harm assessments including the suggestion to divide up the considerations into two major areas: therapeutic and product specific. Therapeutic area considerations cover the disease and the types of patients being studied including an analysis of the condition as well as current treatment options. It lays out the problem but does not address whether the new intervention provides a solution to the problem. Product specific considerations cover the benefits of the particular intervention (what are they, the magnitude, and in whom) and the harms (risk), including how these harms can be mitigated (risk management). In essence, you want to briefly frame the therapeutic area considerations and then spend your time on how your product actually addresses the problem. Below is the provided matrix for FDA Benefit-Risk Framework.
“Uncertainty,” as presented in this matrix, refers to decisions based on assumptions that are not verifiable based on available information. The best way to de-risk development and decrease uncertainties regarding the nature and magnitude of benefits (to balance against harms) is through better measurement of benefits – using improved measurement tools, such as PROs. Oftentimes, sponsors are very hesitant to study additional potential benefits due to “regulatory uncertainty.” However, if we do not move beyond that thinking, FDA approval will become nothing more than a rubber stamp.
The FDA regulatory standard for most interventions, except for life threatening or contagious diseases, is that you only have to be better than nothing. However, patients, clinicians, and payers have no interest in whether your product is slightly better than placebo when there are other interventions out there for that particular disease. People will begin to look beyond FDA approval and compare how you have improved patients’ lives against other interventions in order to decide which they will pay for and ultimately use. Improved and additional information on benefits and harms through the use of PROs leads to more informed decision making. This will help you rely less on assumptions to address uncertainties and get you into more of a fact-based assessment that will actually help patients, clinicians, regulators, and payers understand which medical interventions are the better treatments options.