It is a truth universally acknowledged that a pharmaceutical company in want of a profit will use relative risk. It will do this because presenting the results of clinical trials as relative risk reductions tends to make their new wonder drug appear dramatically effective. But relative risk reduction is only one way of presenting the data. We can also look at the absolute risk reduction, and a useful number derived from the absolute risk reduction, the number needed to treat. This tells us the number of people we need to treat to prevent one adverse outcome. As we shall see, absolute risk reductions and numbers needed to treat tend to dramatically lower the apparent, and so perceived, benefits of treatment. These observations, we shall further see, can be applied to all treatments, including covid vaccines.
Imagine a trial with 2,000 patients, 1,000 in the treatment group and 1,000 in the control (placebo) group. At the end of the trial, there have been 20 adverse outcomes in the treatment group, and 40 in the control group. This gives us a relative risk reduction (RRR) of 50% (40/1000 in the control group vs 20/1000 in the treatment group, 4% vs 2%, RRR = the difference between the groups divided by the rate in the control group = (4% – 2%)/4% = 50%). Seems good. But if we look at the absolute risk reduction (ARR), and so the number needed to treat (NNT), we get a rather less impressive result. The ARR is the rate in the untreated group minus the rate in the treated group, 4% – 2% = 2%. Seems considerably less impressive than the 50% RRR. The NNT is the inverse of the ARR, so in this example, we need to treat 100/2, or 50 people, to prevent one adverse outcome (the numerator is 100 rather than 1 because we are using percentages; if you use proportions, the NNT is 1/0.02). The other 49 people get no benefit, yet may suffer harm for their pains.
This is the paradox of much of modern medicine: most of the time, for most people, the drugs don’t work. For statins (cholesterol lowering drugs) used for people at low risk of cardiovascular disease to prevent heart attacks and strokes, NNT estimates range from around 50 to several hundred or more. Putting aside the disturbing fact we still don’t really know what the NNTs are for one of the most commonly prescribed group of drugs known to man, the stark and uncomfortable reality is that doctors have to hoodwink tens if not hundreds of people to take a drug so that one may benefit. None, not even one, of the other people — poor lambs — gets any benefit. Yet they have — poor lambs — to take their drugs dutifully, and all the while may suffer from side effects, so that the one, a stranger they will never know, can get the benefit. Never in the field of human medicine have so few owed so much to so many.
We can of course apply the same maths to covid vaccines, but there are, as we shall see, some important caveats. Let’s do the numbers for a well publicised paper on the Pfizer/Biotech vaccine, which reported a headline efficacy (efficacy is effectiveness under ideal, or trial, conditions) of 95%. The paper uses a marginally more complex calculation based on surveillance time, but we can use the same method as the one used above in the imaginary trial to get the same result. There were 8 cases of lab confirmed covid–19 in the 18,198 treatment group, and 162 cases in the 18,325 control group. That works out as 8/18,198 or 0.044% in the treatment group, and 162/18,325, or 0.884% in the control group, giving us a RRR of (0.884% – 0.044%)/0.884%, or 95%. Seems good, but what about the ARR, and so the NNT? The ARR is 0.884 – 0.044% = 0.840% (rather less striking than 95%) and the NNT is 100/0.840% or 119. Well over 100 people need to be vaccinated for one person to be prevented from getting lab confirmed covid–19. To put it bluntly, it is hard to square the optimism of 95% efficacy (none of these numbers are cooked, they are just different ways of presenting the same data) with an ARR of 0.84%, and an NNT of 119, where 118 of those 119 people gain no benefit. Never in the field of human pandemics have so few…
By way of benchmarks, NNTs for flu vaccines — not generally considered the world’s most effective vaccines — to prevent culture confirmed influenza in healthy individuals range from around 71 overall, to 29 in the elderly, and 5 in children. These figures come from Cochrane reviews, and are based on meta-analyses of a number of studies of varying quality, meaning they are at best ball park figures, but they do provide some context.
What about the caveats? These may work in the vaccines favour, but not dramatically so, if indeed at all. Although the NNT is a crisp single number, it is in fact tied to a particular context, the study which generated the numbers used to calculate the NNT, because the NNT is the inverse of the ARR, which in turn depends solely on the risk of the adverse event in the treated and control groups. One consequence of this is that when the risk in the control group is low (the condition is rare), then the ARR will always be low (if the baseline risk is 1%, the maximum the ARR can be is 1%), and so the NNT will always be high (the NNT is the inverse of the ARR, so the smaller the ARR, the bigger the NNT). Covid vaccine apologists make use of this to say you should not judge a covid vaccine on its NNT when the rates are low, but this is to miss the essential point made by the NNT: when the baseline risk is low, the NNT will always be high. Never in the field of human pandemics…
The second caveat is that NNTs are always over a particular time frame. Generally, the longer the follow up, the more benefit you will see, even if the RRR remains constant. We can see this if we imagine the Pfizer/Biontech trial follow up time was doubled, with everything else remaining the same. We now have 16 cases in the treatment group, and 324 cases in the control group. This gives as the same RRR, (1.768% – 0.088%)/1.768%, or 95%, but because the extra time has meant more cases have accrued, the ARR has increased to 1.680% (1.768% — 0.088%), and so the NNT becomes 60 (100/1.680%). If we take the median follow up given in the Pfizer/Biontech paper of two months, and extrapolate the results to one year, the NNT then becomes a more respectable 20. The idea that covid vaccine trials, which are necessarily to date of short duration, will underestimate vaccine benefits is fair comment, but this caveat needs its own caveat. It assumes everything else, from baseline infection rates to vaccine effectiveness, remains constant over time. If any of these change — vaccine effectiveness declines over time, background covid rates increase or decrease — the NNT will change.
The third caveat is that NNTs are outcome specific, which is to say they only tell us the NNT for the particular outcome(s) covered by the study. In the Pzizer/Biontech trial, the outcome was lab confirmed covid–19, which means the particular trial tells us nothing about the NNT to prevent hospitalisation, serious disease or death. By the same token, and because NNTs are study specific, the results can only be extrapolated with caution, if at all, to other populations.
What does this canter round the snakes and ladders of covid vaccine treatment effects tell us? It tells us that in the study population, which appears to have been weighted towards fat fair-skinned fifty year old Americans (known in the trade as F3 Yankees, Table 1 in the paper), the Pfizer/Biontech vaccine has rather more modest benefits than the headline 95% relative risk reduction reported by in the paper suggests. It does appear to work, but there are as many snakes as ladders. It is more a mule train doggedly making its way along the valley floor, than a triumphant cavalry cresting the brow of Mount Vaccinia.