We Don’t Have Accurate and Reliable Data on How Effective the Covid Vaccines Actually Are

  • a group of vaccinated people
  • a similar group of unvaccinated people
  • transmission
  • symptomatic infection in the medium and long-term
  • severe Covid disease
  • Covid mortality
  • all-cause mortality
  • observational studies
  • cases/hospitalizations/deaths numbers released by public health institutions
  • experimental data showing various types of immune responses
  • governments and media reporting on such data

Biases in the Processing and Presentation of Data

When comparing the number of cases, hospitalizations and deaths that occur in vaccinated and unvaccinated populations, scientists, public health institutions and media will ideally:

  • present results as daily rates relative to population size instead of as absolute numbers and/or totals over a longer period: When comparisons are made over an extended period in which at least some people are getting vaccinated, the comparison should involve calculating for each person how many days of that period they were in the vaccinated category and how many in the unvaccinated category. For the two denominators of the relative to population rate calculations all vaccinated person-days should be combined in one group and all unvaccinated person-days in another group. For the numerator all cases (or hospitalizations or deaths) that occurred in people who were vaccinated at the time of getting infected should be combined in one group, and all cases that occurred in unvaccinated people combined in the other.
  • use reliable numbers for total population sizes of the groups of vaccinated and unvaccinated people, and for the number of cases, hospitalizations or deaths that occurred in each group: Health insurance databases are ideal for this purpose as they offer complete and known population data sets that also have — or can be used in combination with other datasets that have — reliable data on the number of cases/hospitalizations/deaths in each group.
  • continually adjust the population sizes of the group of vaccinated and unvaccinated: The population of vaccinated and unvaccinated people constantly changes as a result not just of people getting newly vaccinated, but also because people die or move in or out of an area. The denominators in the calculations should be continually adjusted to account for these changes.
  • match people in one group to those people in the other group who are similar in terms of age, comorbidities and (absence of) confirmed prior infection: Vaccinated and unvaccinated populations may differ from each other in respects that are relevant for meaningful comparisons. If the elderly are overrepresented in the vaccinated populations, then this will bias the data against vaccination. To adjust for this a selection needs to be made so that the two groups being compared are and remain similar.
  • make adjustments for variability in incidence throughout a period: If, for example, in Canada the period under observation is January — August 2021 then there will be an overrepresentation of cases, hospitalizations and deaths among the unvaccinated because as the vaccination campaign progressed the percentage of unvaccinated people was 1) high during the winter and early spring period in which respiratory virus activity is typically high, 2) low in the summer period in which such virus activity is typically low.
  • not add people who are within two weeks of a dose to the group who has not yet had that dose: It can take up to two weeks before a dose will start to have a protective effect. This is why cases that occur in people within two weeks of e.g. their first dose are typically not included in the cases for the group of people who have been vaccinated with that first dose. Sometimes they are put in a category of their own (‘vaccinated but not yet protected’) and sometimes they are added to the cases in the ‘unvaccinated’ category. If they are added to the cases in the unvaccinated category and the population size of that group is not similarly adjusted, this will bias the data in favor of vaccine effectiveness. The distortive effect can be surprisingly large, as professor Norman Fenton explains in this short clip (full version, research paper, accessible explanation):
  • account for the temporarily increased susceptibility to infection in the period directly following vaccination: The distortive effect just mentioned is significantly amplified as a result of the empirical fact that in those first two weeks after vaccination people are not just not yet protected but actually more likely to get infected (and hence subsequently being hospitalized or dying) than unvaccinated people are. So adding those people to the group of unvaccinated instead of to the vaccinated group or in a group of their own, further increases the case/hospitalization/death rate in the unvaccinated group and decreases it in the vaccinated group.
  • account for the effect of reporting delays: as Norman Fenton explained in the clip above, a similar distortive effect can be achieved as a result of failure to account for reporting delays.
  • adjust for differences in testing willingness and testing requirements: Vaccinated and unvaccinated people may be subject to different testing requirements in society, or they may differ in their willingness or readiness to get tested. All else being equal, if one group gets tested more often than the other, there will be more cases in that group. Studies should take this into account.
  • account for differences in risk-seeking behavior between vaccinated and unvaccinated people: Getting vaccinated may embolden people to engage in riskier behavior than they did when they were still unvaccinated. Alternatively, unvaccinated people may be less concerned about the virus and on average engage in riskier behavior than unvaccinated people. These differences need to be taken into account.
  • look not just at Covid hospitalizations and deaths but all-cause hospitalizations and deaths as well: If the vaccines reduce Covid hospitalizations and deaths but themselves cause adverse events that result in hospitalizations and deaths then this is important information when evaluating the effectiveness of the vaccines. Moreover, it is also at least a theoretical possibility that the vaccine reduces the likelihood of testing positive for Covid when experiencing Covid-like disease but not to the same extent the likelihood of Covid-like disease itself, or the hospitalizations or deaths that result from it. If all-cause hospitalization and death data are not taken into account, then the vaccines may appear more effective at preventing hospitalization or death than they in fact are.
  • correctly interpret differences in all-cause hospitalizations and deaths: When observational studies match people in the vaccinated group with people in the unvaccinated group who are relevantly similar with regard to age, sex, comorbidities and other factors, and there are significant differences not just in Covid hospitalization and death rates but in all-cause hospitalization and death rates as well, this likely does not indicate that the Covid vaccine protects against all-cause hospitalization and death. Instead it suggests there are relevant behavioral differences that independently explain 1) the willingness to get vaccinated, 2) Covid infection, hospitalization and death, 3) all-cause hospitalization and death. One such possible explanation is how responsibly people behave with regard to their own healthcare, such as taking a vaccine that is promoted as safe and effective, avoiding situations that are high-risk for SARS-CoV-2 transmission and seeking medical care if sick, and reliably taking the medication prescribed for an existing condition.
  • distinguish between hospitalizations and deaths ‘with Covid’ and ‘due to Covid’: If, for example, the vaccines are effective at preventing severe Covid disease but not Covid infection then not distinguishing between on the one hand hospitalizations and deaths that were due to other causes but accompanied by a Covid infection, and on the other hand hospitalizations and deaths that were due to Covid, will appear to make the vaccines less effective against Covid hospitalization and death than they in fact are. Or if fully vaccinated people without Covid symptoms are not routinely tested upon hospital admission for non-Covid reasons while unvaccinated people are, then not distinguishing between hospitalizations for and hospitalizations with Covid will make the vaccine seem more effective at preventing hospitalization than they in fact are as incidental hospitalizations of unvaccinated people will be counted but incidental hospitalizations of fully vaccinated people will not be.

Biases in the Generation and Collection of Data

When it comes to vaccine effectiveness data biases creep in not just in how data are handled but in how data are generated and collected as well. For example, case rates may be influenced by differences in testing behaviors:

  • Demands from employers or family could mean that unvaccinated people are tested more frequently than unvaccinated people.
  • Alternatively, unvaccinated people may be more opposed to the restrictions, and as a result less willing to get tested than vaccinated people.
  • Similarly, vaccinated people may be less likely to seek medical attention because they assume that their vaccine protects them. Or people who are more anxious may be both more likely to get vaccinated and more likely to seek medical attention if they experience symptoms than unvaccinated people are.
  • Hospitals may routinely test unvaccinated people without Covid symptoms but not fully vaccinated people without symptoms, under the assumption that fully vaccinated people are much less likely to have and transmit Covid.
  • To the extent that in their reporting hospitals distinguish between hospitalizations with and for, doctors may be more inclined to think a person’s condition is not due to the Covid infection but the result of other factors simply because they assume the vaccine is protecting the patient against severe Covid disease.
  • Similarly, when patients present to ER with Covid-like symptoms, hospitals may be more likely to admit them if they are unvaccinated than if they are vaccinated because they assume unvaccinated people are much more likely to become severely ill than vaccinated people are.
  • For a variety of reasons hospitals may sometimes code patients for whom upon admission vaccination status cannot be determined as ‘unvaccinated’ rather than ‘unknown’, let alone as ‘vaccinated’.
  • When doctors determine the cause of death of a patient they may be more likely to put ‘Covid’ as that cause if the person was unvaccinated than if they were vaccinated because they assume the vaccine protected against severe Covid disease.

Biases Add Up

The individual net effect of each or most of the biases just mentioned will be small but when several such biases are at play they can add up. Moreover, there may be a self-fulfilling prophecy effect. For example, the assumption that vaccines are highly effective at preventing severe disease can result in several biases that in turn make the vaccines seem more effective at preventing severe disease. Over time the realization that the vaccines are not as effective as previously thought may also result in changes in the protocols and practices that then in turn result in data that further confirm this. It is possible that part of the appearance of waning vaccine effectiveness in the data is in fact the result of protocols and practices being changed in response to indications that the vaccines are less effective than previously thought.

No Excuse

In conclusion, with all the data purporting to tell us something about how effective the vaccines are, we should keep in mind that those data are very much imperfect and should be treated with considerable caution. Even more so when the people involved in the generation, collection, processing and presentation of the data have a strong belief in the effectiveness of the vaccines, such that these assumptions can unintentionally introduce biases that distort the data. Let alone when they may have incentives to intentionally distort and misrepresent.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store