Another day, and another tractor production is up paper from the University of AstraZeneca sitting at Oxford. The headline finding reported is that the covid vaccines do cause rare serious complications, but at a lesser rate than covid–19 itself causes the same complications. The mainstream media, including Twiddle and Tweedle on the Today programme, predictably reported the story uncritically, creating what the Chinese call a “trustworthy and glorious public opinion atmosphere“. In reality, the paper has more holes in it than a tramp’s trousers, and stinks about as much, but as is often the case these days, the methodology is wrapped in a statistical miasma, making teasing out what was actually done a challenge. Undeterred, Dr No will take you on a journey that weaves through worm holes, and show why this paper isn’t even fit to make a contribution to a tramp’s bedding, let alone appear in a peer reviewed medical journal.
The key to understanding the paper’s failing lies chiefly in the methodology, but before we look at that, we might note a number of other significant failings. The first, and one of the most important, is that although the authors commendably published a pre-specified protocol for the study, they then, rather less commendably, failed to follow the protocol. This is a strong indication that the paper contains results based either on data-dredging or post hoc analysis, techniques that can uncover hot results when the pre-specified protocol results are lukewarm. In this paper’s case, the hot finding is that the vaccines cause less serious complications than the disease itself. Yet there is no mention whatsoever of this arm of the study in the protocol, meaning that authors added it after the event, no doubt to nurture a trustworthy and glorious public opinion atmosphere. In at least one parish in the land, the atmosphere nurtured was more odious than glorious.
It gets worse. It turns out the lower adverse event rates for following covid vaccine compared to covid the disease, which is the finding widely reported in the media, and even present in the paper’s Visual Abstract, are not what they seem. The groups are not vaccinated, and covid–19 in the definitely unvaccinated; they are instead vaccinated compared to those testing positive for SARS-CoV-2 test in the same population. This, as we shall see when we consider the methodology, makes no sense, because, in a self-controlled case series study, there is, by definition, no population; instead, there is just a series of cases, all of whom experience the adverse event. Even if the phrase ‘in the same population’ were to mean ‘in the 30 million or so who received a vaccine’, the vast majority of whom did not experience side effects, and so are not in the case series, there is no way of knowing whether the positive SARS-CoV-2 test happened before, around or after vaccination, making the interpretation of the results hopelessly confused.
The protocol has another problem. It’s final version was published on the 4th April 2021, shortly before the study period ended on 24th April, and months after it started in December 2020, making a mockery of pre-specification. Other date related problems are present too: at one point, in the method section, the authors refer to an adjustment in their method of analysis, based on a paper published by JAMA on 29th July 2021, yet their own paper was accepted by the BMJ on 2nd August 2021. While it might be possible to re-jig the analysis and rewrite the paper over of four days, such speeds are normally within the habits of academic authors. Again, one is left with a sense of alterations after the event. Other oddities, or rather annoyances, are the omission of many of the figures and tables given in the text, or even links to them, meaning readers have to take it on trust — the authors no doubt assume the paper will be read in a trustworthy and glorious public opinion atmosphere — that the figures and tables show what the authors say they do.
And so to the methodology, the self-controlled case series. Originally conceived as a way of assessing rare vaccine side-effects — so it is at least being used in the right setting, but that doesn’t necessarily mean its headline conclusion is valid — the idea is simple enough. The researchers identify cases, that is, those who have had the adverse event, and then compare the risk of the event in a tightly defined post exposure at risk period after being vaccinated, with the risk in the same individual during a baseline period, normally immediately before, after or both before and after, the exposure risk period. If patient X is vaccinated on 1st February, the at risk period might be all February, with the baseline periods being January and/or March. If it turns out there is a higher relative incidence — absolute risks cannot be calculated, because we do not know the denominator, the number of people at risk — of the side effect in the at risk period in February, compared to the baseline risk in January and/or March, then that suggests, but doesn’t prove — post hoc ergo propter hoc applies at least to some extent — that the adverse event may be a side effect of the vaccine. The key thing about self-controlled case series studies is that the cases act as their own controls.
The idea is simple enough, and even has some theoretical advantages, but it needs a lot of numerology to make it work. But let us accept the premise might work, and accept that the numerology might work. The first thing to note, as we did above, is that a case series only includes cases. If the cases are relatively rare serious side effects, then the numbers will be relatively small, perhaps up to tens of thousands for each adverse event in a national study, and this is indeed the case. For all three main adverse outcomes, there were just over 122,000 cases from 1st December 2020 to 24th April 2021. Yet the paper breezily talks of 29,121,633 participants, but the vast majority of these ‘participants’ — around 29,000,000 of them — never got into the case series, because they never experienced a relevant adverse event. This ‘participant’ inflation is isn’t just over egging the pudding, it’s over egging it to the point where it explodes and starts oozing out round the sides of the oven door.
In reality, the researchers’ soufflé fails to rise at all. Recall that the headline conclusion is that you get less of these rare adverse events after vaccination than you do after getting covid. Recall too that for a self-controlled case series study to work, you need accurate times for the exposure, to establish the exposure and baseline risk periods. For vaccination, the exposure date is straightforward: it is the day of vaccination. For exposure to covid–19, it is anything but straightforward, because the study uses a positive SARS-CoV-2 test as the definition of exposure to covid, and as we all know, finding a broken needle in a haystack doesn’t mean you’ve got a working sewing machine. This means the researchers don’t really know whether the cases had covid, and if they did, when they had covid, and this in turn means they have no way of defining the exposure risk period. The study fails to achieve an essential requirement for a self-controlled case series, knowing precisely when the exposure event occurred.
That alone is enough to damn the study, but there is an even more audacious failing. Cast your mind back to the essence of a self-controlled case series: it compares incidence rates during the post exposure risk period to that in the baseline period in the same individual, and produces a relative incidence ratio which describes the relative incidence compared to the baseline incidence. Two things follow. The first is that we have no idea of absolute incidence rates, because we have no idea of the denominator; all we know is that relatively, the incidence is, or maybe isn’t, higher in the post exposure risk period. The second thing is that, because the cases act as their own controls, the absolute baseline risk, even though we don’t know what it is, must be specific to that particular group of cases, because they, as well as being cases, are also the controls.
This means the only valid comparison is within a particular self-controlled case series, between baseline risk and that in the exposure period. As there is no way of establishing the absolute baseline incidence rates, we cannot compare one self-controlled case series with another. We might, for example, know that adverse event incidence rate ratio of exposure X (say being vaccinated) is 2, and that for exposure Y (say being exposed to covid) it is 4, but that doesn’t mean we can say Y has twice the incidence of adverse events compared to X, because we don’t know the underlying absolute incidence rates. If, for example, the underlying absolute baseline incidence in case series X was one in fifty (2%), and in case series Y it was one in a hundred (1%), then each series would in fact have the same absolute exposure period risk of 4% (2% x 2 for X, and 1% x 4 for Y). Because we have no way of verifying the underlying baseline risks, we cannot compare exposure X with exposure Y, only within X and within Y. If we do compare exposure X with exposure Y, it is a classic oranges and apples comparison, and that is why this paper fails.