This image shows the climax of one of two animated charts produced by the FT’s data journalist. Currently doing the rounds on twitter, it purports to show the strain on, and so by implication the imminent collapse of, the NHS. It matters, not least because it has been viewed for getting on for 2 million times, and so is likely to have had something of an impact on the current political, media and public perception of where we are with covid. At first glance, it is alarming. But is it a fair representation of the facts? Dr No is pretty sure it is not, though proving that is easier said than done. Nonetheless, nothing ventured, nothing gained, so what can be found out about this chart?
The chart is a plot of selected weekly ICU admissions for recent winters. The red line is this winter’s covid admissions, and the other beige lines each represents a recent past winter’s flu admissions. On the face of it, this winter is exceptional. Although the period from August to the start of November is unremarkable, being just a normal flu curve happening earlier in the year, once we get to early November, the curve takes off, quite literally dwarfing the curves from previous years. To get the full effect, you need to see the animated version.
The animation, and even the still image above, work so well because of the visual difference between the beige flu lines and the red covid line. After some fairly deep digging, Dr No has confirmed the lines do indeed represent the data held by PHE, though some minor details have been changed, for instance the plot is per million rather than per 100,000. The denominator, the per million/per 100,000 are per trust catchment population of the trusts reporting the data, so the denominators are comparable. Can the same be said for the numerator, the number of cases, per denominator?
Dr No dropped a big hint a couple of paragraphs ago, by italicising covid and flu. This is a red flag, because the data represent two different conditions. For the rates to be comparable, the reliability of case counting, or case ascertainment, has to be the same for each condition. If, for example, you were to count all covid cases, and then some, but were for whatever reason less rigorous in counting flu cases, then you cannot reliably compare covid cases with flu cases. The chart then becomes at best meaningless, at worst, a damned lie likely to instil fear and despondency in a demoralised population.
Given the crux of the matter is how cases were counted, what do we know about this? For both flu and covid, the case definition is effectively the same, an ICU admission with a laboratory confirmed diagnosis of the relevant condition. But that is where any semblance of comparability ends. For covid, we know there has been testing on a reckless and colossal scale, and furthermore that testing, particularly in hospital, is done using a remarkably sensitive test, the PCR test, that is more than capable of detecting past, now no longer relevant infection. Almost all, if not all patients admitted to hospital will get a covid test, and if positive, will become a covid patient, even if their primary diagnosis is something unrelated. Any of these patients then admitted to ICU will be counted in PHE’s data as covid ICU admissions, and so will contribute to the red line on the chart.
For flu, the situation is entirely different. Most patients don’t get tested for flu, because flu isn’t part of their diagnosis. Even in patients admitted with an influenza like illness (ILI), there is no obligation to test for flu. Despite his best efforts, through both formal and informal enquiries*, Dr No has not been able to get an estimate of how many patients admitted with ILI would routinely have been tested for flu over the last decade, the period covering the beige lines on the chart. Nor has he been able to discover much , apart from a distant hint that early data might not be that complete, on the quality of data collected by the UK Severe Influenza Surveillance System (USISS), the routine surveillance system used over the last decade to collect the data that directly gives rise to the beige lines on the chart.
What we do know, however, is that routine data surveillance, not exactly the most glamorous corner of the medical world, tends to under-perform. Under-reporting is rife. A 2015 study looking at a similar flu surveillance system used in America, which is the closest we are going to get to an indirect feel for the quality of USISS data, found that less than half of patients aged 18-64, falling to less than a third of patients aged 65 and over, admitted with a respiratory infection to the study hospitals in the surveillance network were tested for influenza. The study concluded that, overall, surveillance detected laboratory confirmed cases of influenza under-reported true 18-65 year old cases by a factor of three, and rising to a factor of five in those aged 65 and over.
There is a world of difference between sleepily recorded routine data, and that collected in the urgency of a full blown pandemic. There is no reason not to suppose that the recent historical routine data from England, the beige lines, have not suffered a throttling similar to that seen in the American flu surveillance system data. Recall, no test means no lab confirmed case means no show in the PHE surveillance data, and so no show in the beige lines in the chart. If — Dr No accepts it is if — similar throttling of flu testing happened here over the last decade, causing significant under-reporting, then we need to give those beige lines a big kick up the backside, a kick almost certainly large enough to put them on a par with, if not above, perhaps significantly above, current covid levels.
*Edit 08:40 11th Jan 2021: Dr No has now had an answer to an informal enquiry, and turns out that, at least in one trust, all admissions with suspected flu are tested. This clearly weakens his argument, but does not rule it out, since it doesn’t rule out the hot stuff bias/diagnostic drift elements of the argument, ie everyone gets tested for covid in a climate of heightened covid awareness vs sleepy normal routine surveillance. Those who know how medicine works on the ground will Dr No is sure concur — as Shawn does in the comments — that idealised policy on paperwork and reporting rarely translates into actual practice. And then there is that American paper, and that distant hint in a British paper: “[USISS] data are available by age group and influenza type/subtype. However, when stratified by both, as well as week, many zero counts are observed.”