Editor’s note: The following column was originally published online in The ASCO Post, December 10, 2019. It has been edited for Bulletin style and is reprinted with permission from The ASCO Post © 2019, Harborside Press.
About four decades ago, as a young physician, I observed that most surgeons were numerator physicians; they remembered their successes and their failures, but they did not remember the frequency of either. No denominator entered the equation. Worse, the approach to any specific surgical problem was always the same. It depended less on critical observation than on where the surgeon was trained. We believed what we were doing was right because “that is the way we always do it.” Was it true that failure to close the abdomen with interrupted number 2 silk would result in incisional hernia or even evisceration and possible death? I thought I knew that was false, but would I risk my residency to challenge it? As a resident, I awoke at night, fearful that any transgression from “the way we always do it” would cost me my residency and some patients their lives.
In fact, there are facts. We just didn’t have them then. The old approach sometimes worked and sometimes did not. Now we know that by recording in real time what we are doing, and the consequences of those actions, we can create databases and begin to ask what factors influence outcomes for better or for worse. Where the change is small, we can discuss what benefit is enough to justify surgery. Facts need to precede opinion, not the other way around.
Having convinced myself and others that we were making a difference—the ultimate confirmation bias—I had to prove that what I believed was true. On most occasions, accomplishing this goal meant conducting a randomized controlled trial. My early attempts at single-institution randomized trials worked well, albeit most of them showed that what I believed to be true was not! Facts overwhelmed opinions.
A denominator problem remains
Time moved on, and we no longer practice numerator medicine, but we still have a denominator problem. Now that we know the facts, how can we decide what constitutes a significant, clinically relevant benefit? Determining clinical relevance is a great struggle.
For example, a 20 percent benefit in most trials of metastatic solid tumors would be of the order of six months of life. Is that enough? How was I to make progress if I was to demand unrealistic improvement? As difficult as it was, if we were to improve some endpoint (local recurrence, disease-free survival, or even overall survival for patients who already had a 90 percent favorable outcome), was getting to 93 percent really important? If I had to enter my patients into a 2,000-patient trial to show a 2 percent or 3 percent benefit, was that justifiable? Even if the predicted result was to be statistically real, did I start the consent process by explaining to the patient, “If this trial succeeds, you have three chances in 100 of being helped?” Or, “If this works, we would have to treat 100 patients like you for three to benefit?”
Scenarios in melanoma and pancreatic cancer
I am ethnically at risk of melanoma, and my childhood environment compounded the risk, so with my melanoma, should I have a sentinel lymph node biopsy (SLN)? I read the trials for intermediate-risk melanoma: I have a 15 percent risk of the SLN biopsy being positive, and if I then had a completion lymph node dissection, a maximal 20 percent improvement in melanoma-specific survival, but no overall survival benefit. My optimistic—but unproven—benefit to initiate the SLN biopsy is 20 percent of 15 percent, or 3 percent. Should I proceed? I am saved the decision, only because my thin melanoma and my age (older than age 65) preclude me from being in the trial. The results of a clinical trial only apply to those entering the trial.
Do we really need a randomized trial in resected pancreatic cancer, with a five-year mortality of 90 percent? What I need to see is easy—if you throw a brick out the window, and it goes up, you do not need a trial. So far, for that disease in those circumstances, no brick has ever failed to fall. As a surgeon, can I justify a placebo operation? The incision would need to be minimal, although it has been done in other countries. We do trials against regimens that previously have shown to be marginally effective (that is, a clinical placebo, with side effects).
Statistical manipulations
Of greater concern, when do we address statistical manipulations? Should I be upset to see survival, particularly progression-free survival, being measured to two decimal places of a month? If 0.1 of a month is three days, 0.01 of a month is seven hours! How ludicrous that appears, but statistical improvement is declared—and published—and the U.S. Food and Drug Administration approves the adjuvant.
Are we correct to declare a positive clinical trial as the new standard of care? How often must we remind our trainees that the benefits of a positive clinical trial accrue only to the patients who enter the trial? The results may or may not have applicability to the population. How carefully we must screen the entrance criteria.
But allowing math to kill innovation is not the answer. We can enrich populations most likely to benefit by intervention, using mathematical models, such as nomograms, prognostic indicators, molecular diagnosis, and, dare I suggest it, validated recorded clinician experience.
We need to maximize the positive numerator and minimize the denominator. We need to apply the math before, not after, the medicine.
