Another day, and another tractor production is up paper from the University of AstraZeneca sitting at Oxford. The headline finding reported is that the covid vaccines do cause rare serious complications, but at a lesser rate than covid–19 itself causes the same complications. The mainstream media, including Twiddle and Tweedle on the Today programme, predictably reported the story uncritically, creating what the Chinese call a “trustworthy and glorious public opinion atmosphere“. In reality, the paper has more holes in it than a tramp’s trousers, and stinks about as much, but as is often the case these days, the methodology is wrapped in a statistical miasma, making teasing out what was actually done a challenge. Undeterred, Dr No will take you on a journey that weaves through worm holes, and show why this paper isn’t even fit to make a contribution to a tramp’s bedding, let alone appear in a peer reviewed medical journal.

The key to understanding the paper’s failing lies chiefly in the methodology, but before we look at that, we might note a number of other significant failings. The first, and one of the most important, is that although the authors commendably published a pre-specified protocol for the study, they then, rather less commendably, failed to follow the protocol. This is a strong indication that the paper contains results based either on data-dredging or post hoc analysis, techniques that can uncover hot results when the pre-specified protocol results are lukewarm. In this paper’s case, the hot finding is that the vaccines cause less serious complications than the disease itself. Yet there is no mention whatsoever of this arm of the study in the protocol, meaning that authors added it after the event, no doubt to nurture a trustworthy and glorious public opinion atmosphere. In at least one parish in the land, the atmosphere nurtured was more odious than glorious.

It gets worse. It turns out the lower adverse event rates for following covid vaccine compared to covid the disease, which is the finding widely reported in the media, and even present in the paper’s Visual Abstract, are not what they seem. The groups are not vaccinated, and covid–19 in the definitely unvaccinated; they are instead vaccinated compared to those testing positive for SARS-CoV-2 test in the same population. This, as we shall see when we consider the methodology, makes no sense, because, in a self-controlled case series study, there is, by definition, no population; instead, there is just a series of cases, all of whom experience the adverse event. Even if the phrase ‘in the same population’ were to mean ‘in the 30 million or so who received a vaccine’, the vast majority of whom did not experience side effects, and so are not in the case series, there is no way of knowing whether the positive SARS-CoV-2 test happened before, around or after vaccination, making the interpretation of the results hopelessly confused. 

The protocol has another problem. It’s final version was published on the 4th April 2021, shortly before the study period ended on 24th April, and months after it started in December 2020, making a mockery of pre-specification. Other date related problems are present too: at one point, in the method section, the authors refer to an adjustment in their method of analysis, based on a paper published by JAMA on 29th July 2021, yet their own paper was accepted by the BMJ on 2nd August 2021. While it might be possible to re-jig the analysis and rewrite the paper over of four days, such speeds are normally within the habits of academic authors. Again, one is left with a sense of alterations after the event. Other oddities, or rather annoyances, are the omission of many of the figures and tables given in the text, or even links to them, meaning readers have to take it on trust — the authors no doubt assume the paper will be read in a trustworthy and glorious public opinion atmosphere — that the figures and tables show what the authors say they do.

And so to the methodology, the self-controlled case series. Originally conceived as a way of assessing rare vaccine side-effects — so it is at least being used in the right setting, but that doesn’t necessarily mean its headline conclusion is valid — the idea is simple enough. The researchers identify cases, that is, those who have had the adverse event, and then compare the risk of the event in a tightly defined post exposure at risk period after being vaccinated, with the risk in the same individual during a baseline period, normally immediately before, after or both before and after, the exposure risk period. If patient X is vaccinated on 1st February, the at risk period might be all February, with the baseline periods being January and/or March. If it turns out there is a higher relative incidence — absolute risks cannot be calculated, because we do not know the denominator, the number of people at risk — of the side effect in the at risk period in February, compared to the baseline risk in January and/or March, then that suggests, but doesn’t prove — post hoc ergo propter hoc applies at least to some extent — that the adverse event may be a side effect of the vaccine. The key thing about self-controlled case series studies is that the cases act as their own controls.

The idea is simple enough, and even has some theoretical advantages, but it needs a lot of numerology to make it work. But let us accept the premise might work, and accept that the numerology might work. The first thing to note, as we did above, is that a case series only includes cases. If the cases are relatively rare serious side effects, then the numbers will be relatively small, perhaps up to tens of thousands for each adverse event in a national study, and this is indeed the case. For all three main adverse outcomes, there were just over 122,000 cases from 1st December 2020 to 24th April 2021. Yet the paper breezily talks of 29,121,633 participants, but the vast majority of these ‘participants’ — around 29,000,000 of them — never got into the case series, because they never experienced a relevant adverse event. This ‘participant’ inflation is isn’t just over egging the pudding, it’s over egging it to the point where it explodes and starts oozing out round the sides of the oven door.

In reality, the researchers’ soufflé fails to rise at all. Recall that the headline conclusion is that you get less of these rare adverse events after vaccination than you do after getting covid. Recall too that for a self-controlled case series study to work, you need accurate times for the exposure, to establish the exposure and baseline risk periods. For vaccination, the exposure date is straightforward: it is the day of vaccination. For exposure to covid–19, it is anything but straightforward, because the study uses a positive SARS-CoV-2 test as the definition of exposure to covid, and as we all know, finding a broken needle in a haystack doesn’t mean you’ve got a working sewing machine. This means the researchers don’t really know whether the cases had covid, and if they did, when they had covid, and this in turn means they have no way of defining the exposure risk period. The study fails to achieve an essential requirement for a self-controlled case series, knowing precisely when the exposure event occurred.

That alone is enough to damn the study, but there is an even more audacious failing. Cast your mind back to the essence of a self-controlled case series: it compares incidence rates during the post exposure risk period to that in the baseline period in the same individual, and produces a relative incidence ratio which describes the relative incidence compared to the baseline incidence. Two things follow. The first is that we have no idea of absolute incidence rates, because we have no idea of the denominator; all we know is that relatively, the incidence is, or maybe isn’t, higher in the post exposure risk period. The second thing is that, because the cases act as their own controls, the absolute baseline risk, even though we don’t know what it is, must be specific to that particular group of cases, because they, as well as being cases, are also the controls.

This means the only valid comparison is within a particular self-controlled case series, between baseline risk and that in the exposure period. As there is no way of establishing the absolute baseline incidence rates, we cannot compare one self-controlled case series with another. We might, for example, know that adverse event incidence rate ratio of exposure X (say being vaccinated) is 2, and that for exposure Y (say being exposed to covid) it is 4, but that doesn’t mean we can say Y has twice the incidence of adverse events compared to X, because we don’t know the underlying absolute incidence rates. If, for example, the underlying absolute baseline incidence in case series X was one in fifty (2%), and in case series Y it was one in a hundred (1%), then each series would in fact have the same absolute exposure period risk of 4% (2% x 2 for X, and 1% x 4 for Y). Because we have no way of verifying the underlying baseline risks, we cannot compare exposure X with exposure Y, only within X and within Y. If we do compare exposure X with exposure Y, it is a classic oranges and apples comparison, and that is why this paper fails.         

Comments

  1. DevonshireDozer Reply

    I absolutely love what you do.

    My own (non-medical) background has ample maths & science in it to do the same things, but I will confess to suffering something akin to combat fatigue. It has been many months (years?) since I could be bothered to make my own investigations into such studies, having reached a point long ago where I just don’t believe any of them. As Malcolm Kendrick once put it, we are being bombarded with chaff, so I simply assume establishment output, such as you have dissected here, to be yet more chaff.

    Your effort is not wasted. It helps keep me going. I am, however, left with the constant question; “What do we have to do to get our moronic leaders to wake up?”. Dragging them back to school to do some basic remedial maths & science isn’t an option – and the effort was obviously wasted on them their first time around. They were too busy doing media studies & PP&E I guess.

    The infection of drivel mongering has reached local authorities (they must love that word). My own county council sent me this in an email – https://content.govdelivery.com/accounts/UKDEVONCC/bulletins/2ee6f51 .

    It makes me want to punch things – and I am not by nature, a violent person. You’ve probably had something similar – but ‘engaging’ with these people is pointless. I’m an old coot but, one day, things are going to go very nasty when younger, smarter, more energetic people decide that enough is enough. I’d be happy to look after their coats.

    • Tom Welsh Reply

      “I will confess to suffering something akin to combat fatigue”.

      Likewise. In the first weeks of the Covid panic I busied myself collecting stories and data to support analysis. At one point Hector Drummond was kind enough to react to a comment I wrote on his blog by suggesting I expand it into an article. I replied positively and set to work. Soon it became apparent that I had set out to eat the Moon. Everything I wrote led to several other important facts, figures, inferences or absurdities. The preliminary notes soon expanded to 40 pages or so, at which point I gave up.

      Ever since then I have looked appreciatively at articles like Dr No’s latest, while feeling deep down that it doesn’t really matter much.

      If the mere existence of such a virus as Covid-19 is in doubt – and if it does exist, no one knows whether or not it is a biological weapon or a prototype of one – and if the alleged virus has no unique symptoms and there is no test for it… what is the point in studying government figures (which may well be falsified)?

      However I also agree with you that “Your effort is not wasted. It helps keep me going”.

    • Tom Welsh Reply

      “What do we have to do to get our moronic leaders to wake up?”

      I am very much afraid that they are wide awake, and know exactly what they are doing.

  2. Rick Reply

    Another great insight into the murky world of research and dodgy papers. The complicit nature of the press to bolster the positions of those in power is something to behold. Hanging heads in shame is not good enough.

  3. Tish Farrell Reply

    Many thanks for all the mathematical toil, Dr. No. It makes you think that this whole miserable fiasco relies on the majority of us being innumerate. And even when someone explains the discrepancies it tends not to shift calcified mind-sets.

    I found another study this week – a pre-print Lancet effort wherein it appears vaccinated hospital staff were a considerable infection vector because (if I’ve understood it) the vaccine made them less aware of their Delta symptoms/or masked them/or some such:
    Transmission of SARS-CoV-2 Delta Variant Among Vaccinated Healthcare Workers, Vietnam
    https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3897733

    Off topic, but it’s infuriating me: I can’t believe the Junta now wants to vax 12 year olds who are not at risk from SARS CoV2. What kind of government does this to their children, the country’s future, without rigorous long-term safety studies first. As if we did not have the appalling results of the Pandemrix nightmare still unresolved.

    • DevonshireDozer Reply

      I’m absolutely with you. It is a disgrace but, as Mark Twain said; “It is easier to fool someone than to convince them that they have been fooled.” That now applies to 7/8 of the entire population. Where & how to start? Perhaps if some celebrity bimbo or pretty boy suddenly sees the light then they might carry a few muppets with them, but I’m not holding my breath. Said boys & bimbos are pretty much all in the pocket of the establishment & its media chums.

  4. Helen McArdle Reply

    Though this study promises much, after reading the paper and your critique I have more questions than answers. Sadly I won’t be using the visual abstract for risk communication, even though as a GPs I do love a nice infographic.

    There is a lot to be commended about the study. Through linking different UK health databases here we have big data, specific to our population, and it overcomes the reporting biases of the Yellow Card MHRA system. Also it is the first UK study I have seen to explicitly acknowledge a pro-thrombotic vaccine-risk above the now well recognised VITT risk.

    I’ll declare I do have a bias in favour of Prof Hippisley-Cox, who has had a profoundly positive impact on risk communication in General Practice. I did have high hopes for her https://www.qcovid.org risk calculator, but unfortunately this is not licensed or intended for individual clinical risk communication.

    Covid-19 risk factors and complications have been known since March 2020. Knowledge about vaccine risks are evolving. I get weary of mantras parroting vaccines are ‘safe and effective’, ‘very safe and very effective’, and even ‘100% safe and 100% effective’. Some media medics have made statements about vaccine safety that are simply untrue. This undermine trust.

    We have seen certain vaccine risks described as ‘rare’, ‘very rare’, ‘extremely rare’, ’vanishingly rare’ and even ‘extraordinarily rare’. In July we were told the risk of myocarditis from Pfizer and Moderna is ‘extremely rare’. For some sub-groups of children one Israeli study suggests the risk may be about 1 in 3000-6000.

    Regarding VITT, on 14th March we were told (and some GPs were telling patients) ‘AstraZeneca finds no evidence of increased blood clot risk from the vaccine’, a statement reiterated by the WHO and European Medicines Agency (EMA), though other sources at that time were quoting one serious clot for every 500,000 people given the AZ vaccine. By 2nd April there were early reports of a potential link on the BBC but we were told ’it remains entirely possible the risk is zero as vaccines are not proven to cause the brain clots’. Only 5 days later the JCVI announced that people under 30 should be offered other vaccines. Chris Whitty stated, ‘The closer you get to someone who’s right down at 20, and otherwise blameless in their health, the more you have to think these really very rare side effects – the risks/benefits might get closer to parity’. The following day the MHRA quoted a risk of one in 250000, whilst the EMA put it at nearer 1 in 100000. Only 2 weeks later, the MHRA recommended further restrictions as the risk appeared to have ‘jumped’ from 1 in 250000 to 1 in 126000 in 2 weeks . To my knowledge, the current stats for this specific risk are 1 in 100000 in the under 40s and 1 in 60000 in under 30s. Of course the risk didn’t jump from ‘no risk’ to 1 in 60000 for under 30s over 6 weeks. We just discovered an unknown risk.

    In June the MHRA described the risk of VITT from Astra Zeneca as ‘extremely rare’. So are we to understand that a 10-fold difference in risk from 1 in 3000-6000 (Pfizer induced myocarditis in 16-24y old boys) to 1 in 60000 (AZ induced VITT in under 30s) are both ‘extremely rare’? Quoting pooled risk/benefit trade-off to a person who may have a significantly higher or lower individual risk/benefit is dishonest, as is the loose language of relative rarity.

    Getting back to the paper, I find there are too many confounders to have confidence that this gives us a true comparison between vaccine risk and infection risk. As such, I’m mainly interested for now in whether it furthers understanding and communication of individual age-adjusted vaccine-risk to facilitate informed consent.

    I may be missing something obvious (please help out if so) but I have the following questions/observations:

    1. Why do they exclude people who have had an admission for the same event in the the 2 years before the study period? Does this mean that people who have had hospital admissions with VTE, arterial thromboses and thrombocytopenia are excluded from the analysis? Surely these are the very people who will be most concerned about these specific risks?

    2. The period of the study is 1/12/2020 – 24/4/21 and only vaccinated people are included. Under 50s, without co-morbidities, weren’t offered the vaccine in the UK until after 13/4/21 (with the exception of healthcare workers and carers, who were offered vaccine early). This study therefore excludes the majority of people who would be at lower risk of infection-induced thromboses, at increased risk of vaccine-induced adverse reactions and who have a less clear risk-benefit trade-off. The mean age of the people in the study is 55.5y (AZ) and 61.5y (Pfizer). How does this study help me communicate vaccine risk/benefit to a 20 year old healthy man? Or a child?

    3. The subgroup analysis compares Incidence Risk Rate in the under and over 50s. Though the confidence intervals overlap, after a single dose of the vaccine, all the primary vaccine outcomes (hospitalisation/death with low platelets; VTE; CVST and other rare arterial thromboses) appear to be (sometimes only marginally) higher in the under 50s than the over 50s. And yet the most of the now vaccinated under 50s are not included in this study. I want to see this under-50 group broken down further and include a more representative lower risk population. For instance, we know that the risk of vaccine-induced myocarditis in teenage boys can be lost in pooled data of under 50s. The same may apply here.

    4. Why does the study only include outcomes within 28 days of the FIRST dose of either vaccine? Perhaps there is an assumption that because the risk of VITT is highest with the first dose, the same applies to all risks? Perhaps the authors were keen to get the paper out, and it is true that some data is better than no data. Having personally seen and Yellow-carded some significant events occurring only after second dose AZ, I want to see a follow-up report now that we have a majority fully vaccinated adult population.

    5. The usual incidence of VTE in the UK is 1-2 per 1000 per year (2/3 DVT 1/3 PE) and this varies by age (1 in 10000 in under 40s, 1 in 100 in over 80s). In the BBC article they helpfully quote that with the AZ vaccine out of 10 million vaccinated, an extra 66 would be hospitalised or die from blood clots (about 1 in 151,000). I can’t remember the last time I admitted someone to hospital with a DVT – they are almost always managed in primary care. These days we seldom admit people with PEs either. The study only tells us about the very severe end of the spectrum of VTE (hospitalisation/death). Could this increased IRR be the tip of a much bigger clot-berg?

    6. I’m also interested in knowing if there is a signal for acute unexpected suspected CV death at home. I tried to work out from the study protocol whether those deaths would be included. It looks like the MHRA database is interrogated for sudden deaths. MHRA doesn’t just rely on Yellow Card notifications, but I’m wondering if their systems will pick up all the acute suspected cardiac/thromboembolic deaths that occur at home or whether it may be influenced by the quality of GP coding. Perhaps you can figure it out? https://www.gov.uk/government/publications/report-of-the-commission-on-human-medicines-expert-working-group-on-covid-19-vaccine-safety-surveillance/report-of-the-commission-on-human-medicines-expert-working-group-on-covid-19-vaccine-safety-surveillance

    This paper does suggest that we have the tools to answer questions about individual risk-benefit trade-off, and it is possible that the Oxford team will come up trumps with a follow-up study. My worry is that there is little appetite to give people this information, in case they make the ‘wrong’ decisions. Until then I expect we will be left comparing apples and pears.

  5. Tom Welsh Reply

    ‘AstraZeneca finds no evidence of increased blood clot risk from the vaccine’.

    I am sure that the late lamented Robert Maxwell found no evidence of fraud on his part.

    Mainly because he was careful not to look for it.

  6. Tom Welsh Reply

    “Another day, and another tractor production is up paper from the University of AstraZeneca sitting at Oxford”.

    Dr No, may I award you another palm for the finest opening line? (“Palm” as in “wreath” or “handshake” – or both).

    Quite apart from its excellent and thought-provoking content, your writing is consistently excellent – witty and erudite. Offhand I can only compare it with that of Dr Theodore Dalrymple, C.J. Hopkins, and Fred Reed – all of whom make one laugh out loud, then shake one’s head in appreciation, and finally think deep and hard.

  7. dr-no Reply

    Tish – the Vietnamese study has been doing the rounds, including getting some coverage on anti-vax sites, which means the Fact Checkers have tried to pop the balloon. It’s a typical mess of mis-interpretation on all sides. The anti-vax headline is vaccinated individuals infected with the delta scariant had 251 times the viral load compared to unvaccinated infected individuals, ergo vaccinated individual + delta infection = detonating covid thermonuclear device. The fact checkers then pointed out, correctly, that the comparison as described in the paper was in fact not to unvaccinated individuals, but to individuals infected in Mar-Apr 2020, with the original non-scariant. But what the Fact Checkers failed to point out was that there were no vaccines in Mar-Apr 2020, so the comparison group was in fact unvaccinated after all.

    Both are right, but none shall have prizes, because the method of calculation viral load is dodgy. The full paper (behind an email registration wall, Dr No uses a disposable email address for such things) reveals the viral loads were determined by using a numerological manipulation of the Ct values from PCR tests: “PCR Ct values were converted to RNA loads using an in-house established formula (y = -0.3092x + 12.553, R² = 0.9963, where y is viral load and x is Ct value) based on 10-fold dilution series of in-vitro transcribed RNA”. The “in-house established formula” suggests a likely trustworthy and glorious in-house opinion atmosphere, with no explanation or reference given for how the formula was established. The 251 number comes from a bit more numerology: the (wildly) guesstimated number of viral copies (based on the Ct value) is expressed as the log to the base 10 (a way of dealing with the vast ranges) of the number of copies per ml: 6.7 for the Mar-Apr 2020 PCR positives, and 9.1 for the delta infected patients. If we convert these back to real numbers, we get 5,011,872.3 and 1,258,925,411.8 respectively, and the ratio between these numbers is indeed 251. But it’s based on Ct values from always dodgy quantitative PCR tests done at very different stages on the pandemic…

    The observation that the Mar-Apr 2020 cases must also have been unvaccinated is an after the event observation, and some might say that on a strict ‘intention to analyse’ basis it is no better than any other post hoc analysis. Be that as it may, the paper methodology means there is no way of teasing out the extent to which being
    either vaccinated or unvaccinated and/or either original variant or delta scariant infected, or indeed something else entirely separate, contributed to the apparently different viral loads. Bottom line: perhaps of some interest, but not more than that unless and until some better, more relevant findings appear.

    dearieme – that looks like the lawyers doing their usual black magic of turning apples into oranges. As Dr No reads it the situation is:

    (1) the FDA have jumped the gun and given full marketing approval to P-BT’s branded vaccine, Comirnaty, for adults. This means related two things: P-BT can market, advertise, promote etc their branded product, but there is a price: product liability falls on the manufacturer, which means those claiming vaccine harms can sue P-BT.

    (2) As it happens, there isn’t a lot of Comirnaty about, just at the time when the American authorities want to vaccinate just about anyone and everyone, with many of the vaccinations being mandatory.

    (3) However, there is a lot of unbranded P-BT vaccine lying about, but it can only be used under the EUA, which means two important things: (a) it’s experimental (so potential vaccinees have the right to refuse it) and (b) in effect there is a blanket removal of all liability from manufacturer to person sticking the needle in and pushing the plunger. Neither sit well with mandatory vaccination programs.

    (4) The fudge, so far as Dr No can make it out, is for m’learned friends to say the gloop in both the branded and unbranded vials is the same, so they can in effect be used interchangeably. One one level (pharmaceutical) this is not unreasonable, but on another, informed consent and coercion, it is totally unacceptable. If a mandated vaccinee is given the unbranded EUA vaccine, believing it is the branded one, then they inadvertently (a) don’t get the opportunity to refuse an experimental treatment and (b) forego any meaningful opportunity to sue if they suffer harm. Dr No suspects his readers will take an extremely dim view of such dishonest practices.

    PS just seen the new comments before posting this, will read and respond later today or tomorrow.

  8. Annie Davenport Turner Reply

    Tom and Devonshire Dozer, this: ‘“What do we have to do to get our moronic leaders to wake up?” I am very much afraid that they are wide awake, and know exactly what they are doing.’ Yes. Just, yes, because, yes, they do, and despite knowing things will have to get a whole lot worse before many people’s eyes open in/through shock, I am deeply concerned at the sheer amount of irreparable damage that will have occurred en route to that point. I fear what we’ve seen is nothing, yet… I also feel now there’s little point in much else now than the constant honing of the ‘mark one eyeball’ around the far bigger picture; ‘they’ are loving how ‘both sides’ – much as I dislike ‘sides’ – are spending 39 hours a day and 10 days a week considering bat-passenger vs bio-weapon, and maskless-granny-killers vs shiny-virtue-badges, and thinking ‘Children are our futures and we must care for them’ vs thinking ‘Children are super-spreaders and must be finalised’, because, thus, ‘they’ believe very, very few will notice what’s actually going on. But there are way more than a few who can see what’s going on, and who wont rest until the detail is clear. I think many of us wrote 40 pages which have expanded into something the size of Encyclopedia Brittanica…. And what important pages they are; someone has to document the culmination of decades (plus) of planning to deal with the ‘useless eaters’. May we all be around to read the pages in our dotage, and may the children be here to read them, too. For that we must – must – stop the seemingly inexorable sinking into a repeat of horribly recent history – one people seem not to remember, or to have ever fully understood how it happened.

    Dr No, I am with all those who quote “Your effort is not wasted. It helps keep me going”. Thank you.

  9. dr-no Reply

    Thank you all for the positive feedback – it is very welcome, especially when sitting in a bomb shelter waiting for Jenny Harries or Heaven forbid the Milk Curdler to lob one in through the door.

    Helen – you are absolutely right about more questions than answers. Opportunities for confounding and bias are everywhere. The study design is complex and multi-faceted and in places difficult or even impossible to unravel. Many of the supplementary data mentions don’t have links, meaning the data can’t be checked. But in a post like this Dr No always (happily) bears in mind he is writing for a very intelligent but mostly lay readership. He also has, and has always had, a one post, one topic rule which he doesn’t always follow, but the intention is there and its reasoning straightforward: overloaded posts make bad posts. The same goes for length: overlong posts make bad posts (but note, no comment meant or implied on your comment! Or indeed on this comment, which could be said to be overlong…).

    That is why the original post zooms in on the major failing: there is nothing to establish the vaccine groups and the positive PCR test groups are comparable. In fact there are many reasons to suspect they may not be, meaning there is no justification for the headline finding, and no justification for the chart in the infographic, which visually invites us to imagine that the baseline risk is the same for all groups, by plotting the relative risk ratios on a single chart, with ‘1’ as the baseline — but we don’t know what ‘1’ is, because a self-controlled case series (SCCS) doesn’t have, let alone use, denominator data. The method parallels a conventional case control study, which also doesn’t have denominator data, and the only real difference is the cases act as their own controls in their baseline not exposed to the risk periods. A diagram for a very simple hypothetical SCCS with three cases:

    Figure 1: Dr No’s Sketchbook. A self controlled case series with three cases. In two, the adverse event occurs in the at risk period, in the other it occurs during a baseline period. The incidence rate ratio is therefore 2 (one adverse event in the baseline period to two in the at risk period). Note that no denominator data is included: the underlying, and invisible, absolute rates could be 1 in 3, 1 in 3,000 or 1 in 30,000,000. This means two very important things: (a) you cannot guesstimate the underlying incidence and (b) you cannot compare one self-controlled case series with another.

    The hypothetical study in Figure 1 is an over-simplification of the method — there is a lot of numerology used to enable each case to be his or her own control — but it indicates the general principle of the method. And the bottom line remains the same: you can’t compare apples with oranges, full stop.

    But let us for a moment consider potential biases. They key question is whether cases are more likely to have had a PCT test around the time of their adverse event than they are to have been vaccinated. If they are, then the incidence rate ratio will be artificially inflated, ie biased. They will appear to be an increased incidence in the at risk period, but it is spurious. It seems very possible that the cases, who are by definition, ill and in hospital, might be more prone to have a peri-adverse event PCR test. The authors claim to have fixed this potential bias, by careful allocation of the baseline and at risk periods, and using only the date of the first positive PCR test, but still the major problem remains: a positive PCR test (finding a broken needle in a haystack) does not mean concurrent active infection (a working sewing machine). SCCS studies need very accurate exposure times, and this study simply doesn’t have them. Another important concern is the identification of cases, which used HES (Hospital Episode Statistics) data. Though perhaps better that it used to be, this is still a notoriously dodgy set of data, and the opportunities for biased cases finding and subsequent exposure identification are rife.

    You are right about the exclusions, and the generalisability of the findings, given the limited rollout of vaccination during the study period, and the restriction to first dose. Exclusion of pre-existing conditions is common enough, to have a ‘clean’ dataset, but you are right, in a pragmatic world, they are some of the people we are most interested in. Likewise the de facto exclusion of younger people, who perhaps are at the greatest risk of harm for the least benefit, is unfortunate to say the least. The demographic and medical characteristics (Tables 1 and 2) of the various groups are similar, but not the same, and there is at least one notable co-morbidity omission: obesity.

    The shifting sands of and the use of relative language to describe the varying risks reflect both vaccine fanaticism and poor communication of emerging knowledge. But there is perhaps another more general problem with conveying risk, which Dr No calls the Collapse of the Probability Function. He wrote about it in one of his very early posts, and the general idea is that while risk makes sense at the group level, it’s meaning starts to collapse for the individual. A GP might know that her elderly male patients have a 10% risk of having a fatal heart attack within ten years, and so in her waiting room filled with ten elderly male patients, one will have a fatal heart attack, but for the individual, on one level, a 10% risk is meaningless. You can’t have a 10% fatal heart attack, it’s all or nothing, a binary outcome: you are either dead or alive. There is some utility in saying the patient belongs to a group with a high risk, and they can move into a group with a lower risk by doing this or that, but at the individual level, the outcome is still a binary outcome: dead or alive.

    6. I’m also interested in knowing if there is a signal for acute unexpected suspected CV death at home…Perhaps you can figure it out? The short answer is no! The MHRA uses, according to the link you include, the Yellow Card system, a Yellow Card Plus system, active monitoring of large GP based datasets and ad hoc formal studies of varying epidemiological rigour. None of the active surveillance methods explicitly mentions monitoring for any death, let alone unexpected home deaths. In the examples linked to on the page, in the pre-defined adverse events of interest, death was not an option. One has to commend the MHRA’s optimistic view. One thing to look at might be ONS weekly deaths, which sometimes include data on place of death, but that is a global figure, with no immediately obvious way to link to vaccination data.

    The only way we are going to get reliable data on rare serious side effects is through proper formal controlled cohort studies, and the tragedy is that, as more and more of the population get vaccinated (as of yesterday, 88% have had their first dose, and 78% two doses), the more impossible it becomes to find the controls. The window of opportunity to do these studies is still just about open, but is closing fast. The curious thing is the authors chose not to do such a study while they still can*, and instead used methods that are far less robust, and, in the case headline results in this paper, downright misleading.

    * A so-called retrospective cohort study for a recent, and so more inclusive, vaccination period should be possible: national vaccine register and GP records to identify those who have been vaccinated and those who have not, and then follow-up via GP records and HES. By no means perfect, non-randomised, overflowing with potential biases and confounders, dodgy follow up data, especially when HES are used, but nonetheless, still doable. But they didn’t do it…

  10. Helen McArdle Reply

    Thank you Dr No, for your thought-provoking posts. I always enjoy the comments of your readers and am grateful for you taking the time to respond.

    In light of everything you have drawn our attention to in this study, I’m concerned that the misleading ‘visual abstract’ will be used to communicate something about relative risk to ‘vaccine hesitant’ people, that is simply not possible to conclude from the actual study.

    That 2009 post of yours that you link to is spot on. One only has to read ‘Risk Savvy’, by Gerd Gigerenzer, to confirm your statement, ‘Cosying up to numbers is all very well, as long as you understand them. Most doctors do not’. When I started working in Scotland I was intrigued to see that here we have a different CV risk calculator (ASSIGN), tailored for the Scottish population, and that the ASSIGN CV risk score is often quite different to the QRISK2 calculator used in England. I did a bit of research at the time into the different risk scores and stumbled on this:
    https://researchonline.lshtm.ac.uk/id/eprint/1987637/1/pone.0106455.pdf
    The majority of CVD cases occurred in people deemed to be at ‘low’ CVD risk (‘low risk’ then taken to be <20%, though now is <10%). The bottom line being, as you say, that using population averages to advise on individual risk is problematic. That being said, these risk scores can be used as a trigger to discuss lifestyle changes (as a visual aid/prompt, one could as easily get a tape measure out to measure abdominal girth, though that seems to get people's backs up, and can't be done over the phone). The problem is that many people who use these risk scores regularly to communicate risk, don't understand them.

    On the same subject of risk communication, the nearest thing we have had to a useable tool for the purpose of relative risk of vaccine vs infection is the Winton Centre tool. I am sure you can see all the limitations in even these (now out of date) charts, when it comes to communicating individual risk. Over 60% of adults in the UK are now overweight or obese. Clearly even the Winton charts aren't much use in helping communicate relative risk to a healthy 55 year old with a normal BMI and no co-morbidities. I agree that as a bare minimum any Covid risk communication tool that doesn't stratify by age or obesity is pretty unhelpful.

    Dr No note: link tweaked to stop margin overflow, hope that’s OK

  11. dr-no Reply

    A number of the Rapid Responses to this paper have pointed out another flaw with serious implications in this paper, which Dr No only briefly alluded to in the original post: “[The groups] in the paper’s Visual Abstract, are not what they seem”. The Visual Abstract suggests three groups were compared: the Pfizer Vaccinated, the AstraZeneca vaccinated, and those with positive SARS-CoV-2 tests. This is seriously misleading, because the positive SARS-CoV-2 tests were “among [the] vaccinated population” (see table captions and column headings) which, if we read this literally, means that the positive tests were breakthrough positives (the ambiguity is does “among [the] vaccinated population” mean “among the already vaccinated population” or does it mean “among the population who will by the end of the study be vaccinated” so there is a question of timing: were there some who tested positive and then got vaccinated, or were they all vaccinated and then tested positive? The wording suggests the latter).

    If they were breakthrough cases, as seems most likely, then this means the study specifically excluded the group the Visual Abstract suggests the study included: the unvaccinated who tested positive. In other words, they excluded the comparison group we are most interested in. The Visual Abstract Summary wrongly suggests “The risks of most of these [adverse] events were substantially higher and more prolonged after SARS-CoV-2 infection” when if fact the paper – flawed methodology notwithstanding – suggests that “The risks of most of these [adverse] events were substantially higher and more prolonged after breakthrough (post vaccination) SARS-CoV-2 infection”.

    For individuals considering whether to accept the vaccine, the question isn’t what are the risk if I have the vaccine and don’t subsequently test positive vs have the vaccine and do subsequently test positive, but what are the risk from the vaccine vs what are the risks from SARS-CoV-2 if I haven’t been vaccinated? The paper can’t even begin to answer that question, because the study didn’t include any such individuals.

    None of this alters the fatal flaws described in the original post. Rather, it reveals new depths of inadequacy in the research, and shows just how desperate the vaccine fanatics are to use extensive publicity conducted through the news media to create a trustworthy and glorious public opinion atmosphere in favour of mass vaccination, wherein model individuals will be commended in accordance with the regulations.

  12. dearieme Reply

    We should put the authors of the papers in the stocks and hurl tomatoes at them. Our sturdy ancestors wold doubtless have thrown harder or fouler objects.

Leave a Reply to Tom Welsh Cancel reply

Your email address will not be published. Required fields are marked *