There has been some conflation and confusion recently over ONS’s initially intriguing but ultimately infuriating deaths by vaccination status reports. They intrigue because they offer a possible route to the paired holy grails of vaccine effectiveness and vaccine safety. A number of bloggers, including Dr No, have gleefully thrown eye of newt and toe of frog into the bubbling cauldron, only to be rewarded with a poke in the eye from a howlet’s wing. A couple of tyros have even added time warps to try and make sense of the data, only to reach uncertain conclusions. Throughout the melee, the focus has at times shifted from numbers to rates and back to numbers again, at others, from covid deaths to non-covid deaths and even all deaths, but the data has work-hardened as it has been worked back and forth, and cracks have begun to appear. Perhaps the cracks were always there, we just couldn’t see them.
One of the more baffling mysteries which might now be solved is how can ramping up vaccinations in a particular age band trigger a sharp and striking rise in mortality in the unvaccinated in that age band, as seen in Figures 1–3 in Dr No’s last post. On the face of it, the unvaccinated just sit there, untouched, and yet their mortality soars as others get vaccinated. It makes no sense until you consider the possibility that, in each age band, the very frail with a short life expectancy may be excused the trauma of vaccination. This, Dr No should add, is not doctors playing God, it is doctors exercising compassion in the face of imminent death. Nature then takes its course, and as the others get vaccinated, the excused start falling off their perches. Over time, things level out, and mortality in the vaccinated and unvaccinated converge, exactly as seen in the charts.
The key thing going on here is the denominator is changing. We tend to focus on the number of events, in this case deaths, and so the numerator in any rates, but in this example, the numerator remains the same — the very frail with a short life expectancy do what they were always going to do — but the denominator gets rapidly smaller, as more and more people get vaccinated. The striking rise in mortality rate happens not because the number of deaths goes up, but because the denominator has plummeted, as can be seen in Figure 1. Note how the rate rises as the number falls, which can only happen because the population at risk, those remaining unvaccinated, falls even faster than the number of deaths.
Figure 1: numbers vs rates as the denominator for rates falls even faster than the numbers. Note that to get all three lines on the same chart involved a bit of jiggery pokery with the axes. The actual population at risk falls from over 4 million in week 1 to less than 400,000 in week 8, a ten fold reduction, while the number fall from just under 3,200 to just over 800, a four fold reduction. Source: ONS
The lesson here is that denominators matter as well, but are all too often overlooked in the frenzy over numerators. We can just as easily double a rate by halving a denominator — 10/100 = 10%, 10/50 = 20% — as by doubling the numerator — 20/100 is also 20% — or indeed any combination of the two that gives the same result. This raises an intriguing question: if ONS’s death by vaccination status rates are being driven as much by denominator changes as numerator changes, could these changes make, for example, the vaccines appear more, or indeed less, effective than they really are? What if, for example, the vaccines have no effect on outcomes themselves, but the shifting sands of numerators and denominators caused by changing vaccination patterns over time make it appear that they do?
Regular readers will know that as a rule Dr No is not a great fan of whatiffery. There are times, however, when it has a place, and this one of them, given that ONS’s data aren’t sufficient to answer the question directly themselves. Instead, we can get a partial answer by approaching the problem from the other end, and say what if the vaccines are ineffective — there is no difference in mortality — can we still nonetheless see an apparent, but spurious, effect, driven by shifting numerators and denominators?
The answer is yes. Let us consider three age bands, the 60 year olds, the 70 year olds and the 80 year olds. Each age band has a fixed mortality, unaffected by vaccination, that rises with age band. At the same time, vaccination proceeds in stages, starting with the 80 year olds, and then progressing to the younger age bands. This means that in the early stages, there will be more people at risk of dying in the vaccinated group that the unvaccinated group. We can use a formula in a spreadsheet (cumulative normal distribution) with various inputs (mean, standard deviation) to generate vaccination curves. The table looks like Figure 2 (a screen grab, actual spreadsheet can be downloaded here). Vaccination starts in week 1, after a lead in period indicated by negative week numbers.
Figure 2: distributions of vaccinated and unvaccinated persons over time for three age bands. Source: whatiffery
Note how the denominators, the population at risk in the Unvac and Vac columns change over time. Charting cumulative curves for the vaccinated groups gives us Figure 3, which looks rather like UKHSA’s cumulative vaccine uptake charts, but with only three age bands shown. So far, so good.
Figure 3: cumulative vaccine uptake curves for each age bands. 80 year olds get it first and fast, the younger age bands progressively less aggressively. Source: data from Figure 2
The risk (rate) of death, shown in row 7 of Figure 2 for each age band, does not change over time, or with vaccination status. It is fixed throughout at the rate shown for each age band. Using these fixed rates, which are not a million miles away from reality, we can calculate the number of deaths in each age band, by vaccination status, for each week. Using these numbers, we can then calculate the overall death rate for each group, vaccinated and unvaccinated, for each week. These calculations are shown in Figure 4. Note that rounding means that some nonsense numbers (0.0002 deaths etc) disappear in the number columns, but they still apply, and so appear, in the rate calculations.
Figure 4: death numbers by age group by week and then overall rates by week. Source: data from Figure 2
Note first the Total column for death numbers, and the All column for death rates, which confirm that overall, each week had identical numbers and rates. Nothing happened: there was no epidemic, just a steady number and rate of deaths each week. Then look at the death numbers and rates for the all the unvaccinated and all the vaccinated. Note how, in the early stages of vaccination, around weeks 4 and 5, the deaths are dominated by the 80 and over group, with its relatively high death rate, but the total number vaccinated is still relatively small. The overall death rate soars. In the later stages, when almost all are vaccinated, the overall rate trends towards the overall rate. Figure 5 shows this ‘epidemic’ in chart form. The rates in the unvaccinated decline over time, as only the group moves towards small numbers of only 60 year olds.
Figure 5: a pseudo-epidemic of deaths in the vaccinated caused by differential vaccination rates in different age bands. Source: more whatiffery
Now the curious thing is, we have seen something similar before. It follows the same pattern as the 10–59 year old curve from Dr No’s last post, the one that caused so much consternation. Here it is again, in Figure 6. Note the early peak, and then decline, in the vaccinated, and the overall decline in the unvaccinated rates. They are not identical, but they do follow the same pattern, which begs the question: is the ONS data also the product of shifting numerators and denominators?
Figure 6: weekly 10–59 year old death rates by vaccine status. Source: ONS
What can we conclude from this whatiffery? Nothing about the actual data, because this whatiffery is an independent exercise carried out on hypothetical data. What we have learnt, though, is that when things are changing rapidly, for example during the early stages of a vaccine rollout, the shifting sands of numerators and denominators can cause apparent epidemics to appear, even when there is no underlying epidemic. The takeaway for Dr No is to be even more cautious in coming to any conclusions based on ONS’s deaths by vaccination status data; and, with that, to lament again the scandal that is the missed opportunity to collect substantive data on covid vaccine effectiveness and safety.