The Fatal Flaw in the UKHSA’s Inflated Vaccine Effectiveness Estimates

In last week’s post I mentioned that prior to the vaccines’ release there were calls for there to be rigorous monitoring of the vaccines’ safety and effectiveness.  Thus I was pleased to read last Wednesday of the latest attempts by the the Health Advisory and Recovery Team (HART), an independent group of medics, scientists and academics, to get the U.K.’s medical products regulator, the MHRA, to explain why it hasn’t kept its promises to monitor the safety and effectiveness of the Covid vaccines in the time since they were rolled out back in December 2020. I’d add that this situation has been mirrored worldwide, with remarkably little in the way of the type of monitoring that would be expected of a new medical product.

The importance of these post-approval monitoring studies can’t be understated – pre-approval testing can definitely give an indication of safety and effectiveness, but a full understanding of the impact of medical products, including longer term effects and interactions with other drugs, is generally only obtained through rigorous monitoring on release. It is for this reason that initial roll-out of new medical products is generally limited to those with the greatest risk, with the product being offered to those at a lower risk once the results from the initial roll-out show effectiveness and safety in the ‘real-world’. I have found it troubling how medical authorities worldwide didn’t explain this need for rigorous monitoring of new medical products – instead they merely repeated the mantra that the pre-approval studies showed the Covid vaccines to be ‘safe and effective’, as though science had found the truth and that there was no more discussion to be had.

Indeed, the Covid vaccines had more need than usual for very rigorous post-approval monitoring:

• Their testing was rushed. This was sold to the population as being ‘all the parts of the testing being done in parallel’, but it really doesn’t work this way – by the time a medical product is ready for approval there will be many years’ worth of data from those first given the product. For the Covid vaccines there were only data from clinical trials on the impact of the vaccines for a few months post vaccination. It would have helped if the pharma companies had given regular updates on the fate of their human guineapigs during 2021. However, there has been little update of the continued safety and effectiveness in those individuals beyond the six month point post vaccination.  
• The product was a vaccine. By design this means that it will be given to individuals who aren’t, at that point, under any particular risk (as opposed to a treatment for an individual who has already caught a given disease). This is doubly true for healthy younger individuals who weren’t at risk from Covid in the first place. I was expecting the vaccine rollout to stop after those aged over 65 had been vaccinated, to allow for the impact of the vaccines to be studied without risking those who weren’t at risk. I still don’t understand why our medical authorities proceeded with a universal vaccination policy at that point.
• The experimental nature of the vaccines. All of the Covid vaccines used in the West were based on novel technologies for which there were limited data on the longer term impact of the technological approach. The viral-vector vaccines (AstraZeneca, Janssen) were not quite so novel as that technology has been used for a number of candidate vaccines for a number of diseases, though never for very large numbers of people. Prior to Covid the only approved viral-vector vaccines were for Ebola, which have only been given to relatively small numbers at very high risk, as ‘ring vaccination’. The mRNA based vaccines were much more novel, and before the Covid vaccines only very limited numbers of people had ever been given this type of medical product. It is truly bizarre that our medical regulators decided that it would be appropriate to inject these mRNA vaccines into a large proportion of the population, most of whom were at little risk from the disease itself.
• The outcomes from prior attempts to make a coronavirus vaccine. Scientists have been trying to make a vaccine to protect against coronavirus infection (‘colds’) for decades. They all failed – not simply because ‘they didn’t work’, but also because some candidate vaccines resulted in increased risk from disease, potentially coming to light some time after the vaccines were given.

The undertaking of rigorous vaccine effectiveness and safety studies isn’t trivial. The chief difficulty is that different people have different vulnerabilities to a given disease (particularly noticeable as an age effect), different risks of catching the disease and different health-seeking behaviours. These introduce a source of bias in any efforts to identify the effectiveness of a vaccine (or, indeed, any medical treatment). The traditional way of dealing with this bias is to split the group (vaccinated vs unvaccinated) into multiple risk and behaviour groups and undertake the analysis such that differences between groups are (almost) compensated for (e.g. the outcomes for vaccinated elderly male individuals with no co-morbidities who don’t wear face-masks will be compared only with unvaccinated elderly male individuals with no co-morbidities who don’t wear face-masks). The best studies are prospective, meaning individuals are put into the different risk and behaviour groups before the medical products are given; this is what should have been done for the Covid vaccines. Nearly, but not quite as good, are retrospective trials where the individuals are put into different groups after the medical products were given. It is still possible to do these; however, few have actually been done. 

These prospective and retrospective matched cohort trials are effective and accurate, if done properly. But because of the effort involved in grouping the different individuals appropriately they are quite expensive and effortful to undertake. Ever mindful of the need to spend taxpayers’ money wisely (or perhaps from some other motive), governments worldwide chose to use other, cheaper and easier, methods instead.

At the end of each Vaccine Surveillance Report is a long list of references for the studies providing the supporting data for the claims made in the report. The vast majority of these studies that relate to vaccine safety and effectiveness used the Test-Negative Case-Control (TNCC) method as the basis of their design.  This is quite a smart way to undertake vaccine effectiveness studies; it is certainly a far easier and cheaper approach than other methods, primarily those that use the matched cohort approach. However, it isn’t completely clear that this method is appropriate for use with the Covid vaccines.

TNCC is quite simple in concept – it compares the positive rate of infection with the incidence of disease that ‘looks like’ the disease in question but which isn’t.   For example, for influenza you might investigate the proportion of people with typical flu like symptoms that have a positive test for influenza with people who also have typical flu like symptoms but that test negative for influenza. This method automatically takes into account multiple potential differences between different vulnerabilities and behaviours, and eases the analysis process. TNCC is, in principle, an excellent and accurate method to measure the effectiveness of a vaccine. However, it makes a fundamental assumption – that all factors that might change the probability of an individual testing positive affect equally the probability of an individual testing negative. Thus TNCC assumes that:

• If a proportion of the population were more vulnerable to catching the specific disease being vaccinated against they’d be equally vulnerable to catching a similar-but-different disease.
• If a proportion of the population exhibited risky behaviours that resulted in them being more likely to catch the specific disease they’d also be more likely to catch a similar-but-different disease.
• If a proportion of the population were more likely to get themselves tested with only minor symptoms they’d also be more likely to get themselves tested if they had minor symptoms after catching a similar-but-different disease.

The problem with the TNCC should now be clear – it assumes that the vaccine itself has no impact other than in the risk of the disease being vaccinated against.  This is often a fair assumption; however, it is by no means guaranteed. In particular, if the vaccine happens to increase the risk of catching a similar-but-different disease then this can result in the TNCC giving misleading results.

To illustrate this point, consider a theoretical vaccine that halves the risk of a certain disease, but where individuals were ten times more likely to have a ‘similar but different’ disease before vaccination started and where the vaccines didn’t alter the chances of catching this ‘similar but different’ disease. For a similar number of vaccinated and unvaccinated (let’s say 10,000 and a 1% chance of catching the disease in question) the numbers tested might be:

• Unvaccinated: 100 positive tests (1% actually had the disease) and 1,000 negative tests (10 times more likely to catch the ‘similar but different’ disease).
• Vaccinated: 50 positive tests (the vaccine halves the chance of catching the disease to 0.5%) and 1,000 negative tests (same incidence as the unvaccinated as the vaccine doesn’t impact on the ‘similar but different’ disease).
• The TNCC estimate of vaccine effectiveness is given by 1 – (50/1,000)/(100/1,000) = 0.5, i.e., 50% VE.

Thus TNCC estimates that the theoretical vaccine halves the risk of catching the disease, which means it agrees with our initial definitions of how well the vaccine performs. Great – the TNCC approach works.

However, consider an alternative scenario, where the vaccine increases the risk of both catching the target disease and a similar-but-different disease. To continue the example, consider that the risks of both are doubled:

• Unvaccinated: 100 positive tests (actually had the disease) and 1,000 negative tests (10 times more likely to catch the disease).  Again, this is the same as the previous example.
• Vaccinated: 200 positive tests (twice as likely to catch the disease ) and 2,000 negative tests (twice as likely to catch the ‘similar but different’ disease).
• The TNCC estimate of vaccine effectiveness is given by 1 – (200/2,000)/(100/1,000) = 0%. 

In this case the vaccine appears to have no impact (perhaps the interpretation is that its effectiveness has waned with time). However in reality it is actually increasing the risks both of the disease in question and of similar-but-different diseases. Because the assumptions of the TNCC method aren’t valid the estimate of VE is incorrect. Note that a matched cohort study would have identified the increased risks of vaccination in the above example, and should also have identified the increased risks of catching the similar but different disease.

Obviously this is an oversimplified example, but the fundamental is the same. If vaccination results in an increase in the incidence of a ‘similar but different’ disease then the TNCC method will give highly misleading results.

But how can vaccination cause a change in the risk of a disease different to the one vaccinated against? This goes against our idea of how vaccines work – they’re a marvel of science that in some cases have resulted in significant benefits to global health. However, there are instances where vaccination does change risk from other similar diseases. These cases are unusual in vaccines actually in use, but that’s because the years of clinical trials (in animals first, and then humans) generally identify such a risk before they are used in the general population. Nevertheless, before TNCC is used it is necessary to test whether there is any change in the ‘test-negative’ condition. This hasn’t been done for the Covid vaccines. However, I note that there is anecdotal evidence that there might be changes in infection risk for other diseases following vaccination: the arrival of ‘the worst cold ever’ last autumn; monkeypox (a disease not normally associated with human-to-human transmission); meningitis in Florida; an increase in cases of norovirus in the U.K. this spring; a nasty influenza outbreak in Australia and New Zealand during their recent winter. I’m sure there’ll be a repeat of last year’s ‘worst cold ever’ this autumn, and we’re already hearing shouts of the imminent danger posed by influenza. It is important to note that an increased risk of infection with other diseases isn’t only faced by the vaccinated. There might be an increased risk of disease in the vaccinated, but as they will have some prior immunity to most common viruses this might not have much impact on the individuals themselves beyond a mild cold. However, young children will often have no prior immunity and there’s a risk of them getting quite ill even with common viruses, and the elderly have impaired immunity in general and can also get rather ill with these viruses. I believe that this is what we’re seeing in the U.K. (earlier this year) and USA (now) with increases in hospitalisations in the very young and very old due to RSV infection. Again, note that just because the hospitalisations are in the young and old it doesn’t mean that only the young and old are getting infected, just that they’re the ones that seek healthcare support; the majority of those not old nor young will simply moan about having a cold.

One interesting piece of circumstantial evidence pointing towards viral interference is the incidence rates of ‘other respiratory infections’ compared with Covid infections. These data are seldom gathered because ‘colds’ are usually rather inconsequential and the assumption is there would be no link with Covid. However, data from the Zoe Symptom Tracker might suggest otherwise.

These data certainly suggest that there’s some kind of connection between risk of infection with Covid (blue line) and risk of infection with other respiratory viruses (orange line); this appears to start to emerge last autumn and become prominent with the arrival of the Omicron variant last winter.

Is there any other evidence? One of the fundamentals of any measure of vaccine effectiveness is that whatever the method it should give a similar result. While most estimates of the Covid vaccines’ effectiveness use TNCC there are a few instances where a study has included the results of both a matched-cohort analysis and a TNCC analysis. There have been hints of this in many studies over the months; my favourite was a study undertaken last year by Imperial College. The results?

The estimate of vaccine effectiveness (VE) using the matched cohort method is consistently much lower than the TNCC method, falling below zero for some data points. This isn’t definitive proof that TNCC is giving misleading results. But it very much is evidence to support looking much harder at the assumptions of TNCC and whether they’re introducing bias and thus an overestimate of VE. My favourite part about this study was how Imperial released the data in one of their occasional reports on Covid – and the data were never seen again. I wonder if this is because the data were inconvenient.

The study suggests that the vaccines offer negative effectiveness – that is, that they make it more likely to catch the disease. Can this even be possible?  Interestingly, it isn’t that unusual to find vaccine candidates that increase risk of disease. Of course, we don’t see this in our experience of vaccines because those vaccine candidates that did increase disease risk are usually filtered out by the clinical trials that are needed to obtain regulatory approval (with some infamous exceptions, such as RSV and measles vaccines in the 1960s, and more recently a Dengue virus vaccine). What’s more, prior coronavirus vaccine candidates did often show a propensity to increase infection risk or disease severity – (for example, see Jaume et al.). Thus there should have been some expectation that the Covid vaccines might have led to negative vaccine effectiveness.

There have been other studies that have suggested that the vaccines might be increasing the risk of infection with Covid. Just to consider recent publications, Oxford University published the results of a matched cohort study that showed a negative vaccine effectiveness – that the vaccinated have a higher risk of infection than the unvaccinated. This has been discussed in the Daily Sceptic, as have two other recent studies that show negative vaccine effectiveness. I’ll note again a point that I made earlier – if matched-cohort and TNCC studies give different results it is a huge red-flag that the assumptions necessary to use TNCC aren’t valid, and that we should disregard these data and focus on using more robust methods to measure vaccine effectiveness.

One final point on the impact of the vaccines on reducing (or not) Covid infections: there has never been any serious mention of the impact of natural immunity in the UKHSA Vaccine Surveillance Reports. I suppose this might be excused (as it is a vaccine surveillance report). However, people are using these documents (and similar) to decide whether it would be worthwhile taking the vaccines. Without information on the protection offered after natural infection, how could individuals give fully informed consent? The lack of any real information on the protection gained after natural infection has been a fairly reliable theme from various authorities worldwide. It was almost as if these authorities were mainly interested in getting everyone vaccinated, and underplayed any information that might make an individual more likely to choose to forgo vaccination even if this was what was supported by the science.

In my post next week I’ll discuss an aspect surprisingly not actually covered by the Vaccine Surveillance Reports but that should have been – the side-effects of the vaccines.

Amanuensis is an ex-academic and senior Government scientist. He blogs at Bartram’s Folly.