Now this is actual lying with statistics, as opposed to how people take numbers from a study and misrepresent them in ads. The authors could say they weren't deliberately untruthful and that could be true. A model is a what-if calculation and only as good as its assumptions. Models can be dreamed up all day, like story plots--the best stories aren't lies, but they aren't reality either. In this one the assumptions have too many holes, they don't jibe with reality as observed (lack of face validity), and imo all models deserve massive skepticism, many minds picking over all the possible alternative assumptions, pilot testing, and not ever used for policy without such checks.
I hate to even call it statistics. There isn't any real data to analyze. They just made up a scenario where "IF* certain things were true, and other things never happened, THEN there would be a certain result. The only real numbers are known like population and the rest might as well be magic.This is a combination of both "dazzling with [fictitious] data" and "baffling with bullshit." It works because not only is math hard but the caveats take work to figure out.
Edit: to clarify, it was the original model that was BS. The Lancet article points out all the ways that it went wrong, and the best part of the article is bringing in the all-cause deaths, imo, because that not only brings attention to the side effects of the vaccines, but ought to bring about a more reasonable way of evaluating vaccines in general, their benefit to harm ratio.
Abstract
A recent study published in The Lancet Infectious Diseases applied mathematical modeling to estimate that mass COVID-19 vaccinations saved between 14-20 million lives worldwide during the first year of the COVID-19 vaccination program. This commentary provides a brief overview of the study, and then identifies potential sources of model misspecification including inaccurate assumptions about vaccine-derived immunity and ignoring additional contributors of pandemic-related excess deaths. We describe how these factors could lead to grossly inflated estimates of deaths averted due to mass vaccinations, which may explain the study's lack of face validity and internal consistency.
(Body of study)
Models ignore excess death due to factors other than COVID-19. The fitted models and their counterfactuals assume that excess deaths in each country are explainedsolely by a naturally evolving COVID-19 virus and its (fitted model-inferred) time-varying transmissibility. The models do not attempt to account for excess deaths caused by other pandemic-related factors, for example the vaccines themselves as well as other non-pharmaceutical compulsory interventions. The CDC reports a vaccine-induced death risk of 0.0026% on average per dose basedon the Vaccine Adverse Events Reporting System, or VAERS3. VAERS is a passive reporting systemand it is well known it suffers from under ascertainment in that it only captures ~1% of all vaccine-related side effects [6]. Recent independent lines of evidence suggest that VAERS may only capture 35% of all vaccine-induced deaths [7,8]. In addition, the models do not account for excess deathsresulting from other factors such as lockdown-induced “deaths of despair”.4 By ignoring other potentialsources of pandemic-related excess deaths in their models, the fitted models will over- and/ormisestimate the effects of natural, time-varying virus transmissibility in order to achieve a good model fitwith reported excess deaths, which in turn would lead to inflated excess death counts in theircounterfactual simulations.
Lack of face validity
According to the authors’ country level estimates 1.9 Million deaths were averted in the US assuming a61% vaccine coverage (see Supplementary Table 3 in original study). In the first year of the pandemicwhen no vaccines were available (2020), there were 351,039 confirmed COVID deaths in the US5. Theauthors’ models thus suggest that 1.9M / 350k = ~5.5x as many COVID deaths in the US would haveoccurred in 2021 (vs. 2020) had no vaccines been introduced. This is highly implausible as there is verylittle reason to believe the virus would have naturally evolved to be that much more transmissible, infective and lethal. The authors allude to higher transmissibility in 2021 due to the relaxing and/or lifting of public healthmeasures and restrictions (lockdowns, travel restrictions, mask mandates etc.). However, theassumption that this could account for a >5-fold increase in COVID deaths in 2021 contradicts >400studies that have concluded there were very little to no public health benefits of these measures inreducing COVID outcomes6. Moreover, in 2021 (after vaccinationwas introduced), there were 474,890confirmed COVID deaths in the USaccording5. This suggests crudeevidence that mass vaccination mayhave worsened COVID outcomesoverall, consistent with observationsof increased infectivity before vaccineprotection kicks in (see 1st pointabove) as well as previouspredictions and observations ofenhanced respiratory disease viaantibody dependent enhancement based on preclinical studies [9,10].
Summary
While generative models are a usefultool to simulate scenarios that havenot occurred, not accounting properlyfor relevant variables in the modelmay lead to model misspecification.In such cases, counterfactuals maygrossly inflate estimates of deaths averted due to mass vaccinations. Rather than rely on simulationswhich may be sensitive to input parameters, prone to overfitting, and that are difficult, if not impossibleto validate, more accurate and reliable approaches to inform public health vaccination policies arequantitative risk-benefit ratio analyses for specific outcomes using clinical trial or real-world data[7,11,12].4 See https://www.aier.org/article/study-indicates-lockdowns-have-increased-deaths-of-despair5 See https://ourworldindata.org/coronavirus/country/united-states6 See https://brownstone.org/articles/more-than-400-studies-on-the-failure-of-compulsory-covid-interventionsFigure 2. Two left bars show total confirmed COVID-19 deaths in the US in 2020 and 2021 from Our World in Data. Rightmost bar (*) indicates predicted number of averted US deaths due to vaccines in the Watson et al. article (Supplementary Table 3).
Now this is actual lying with statistics, as opposed to how people take numbers from a study and misrepresent them in ads. The authors could say they weren't deliberately untruthful and that could be true. A model is a what-if calculation and only as good as its assumptions. Models can be dreamed up all day, like story plots--the best stories aren't lies, but they aren't reality either. In this one the assumptions have too many holes, they don't jibe with reality as observed (lack of face validity), and imo all models deserve massive skepticism, many minds picking over all the possible alternative assumptions, pilot testing, and not ever used for policy without such checks.
Isn’t that the jist of this article, they played magic #’s with the jab?
I hate to even call it statistics. There isn't any real data to analyze. They just made up a scenario where "IF* certain things were true, and other things never happened, THEN there would be a certain result. The only real numbers are known like population and the rest might as well be magic.This is a combination of both "dazzling with [fictitious] data" and "baffling with bullshit." It works because not only is math hard but the caveats take work to figure out.
Edit: to clarify, it was the original model that was BS. The Lancet article points out all the ways that it went wrong, and the best part of the article is bringing in the all-cause deaths, imo, because that not only brings attention to the side effects of the vaccines, but ought to bring about a more reasonable way of evaluating vaccines in general, their benefit to harm ratio.
Well stated!
Abstract A recent study published in The Lancet Infectious Diseases applied mathematical modeling to estimate that mass COVID-19 vaccinations saved between 14-20 million lives worldwide during the first year of the COVID-19 vaccination program. This commentary provides a brief overview of the study, and then identifies potential sources of model misspecification including inaccurate assumptions about vaccine-derived immunity and ignoring additional contributors of pandemic-related excess deaths. We describe how these factors could lead to grossly inflated estimates of deaths averted due to mass vaccinations, which may explain the study's lack of face validity and internal consistency.
(Body of study)
Models ignore excess death due to factors other than COVID-19. The fitted models and their counterfactuals assume that excess deaths in each country are explainedsolely by a naturally evolving COVID-19 virus and its (fitted model-inferred) time-varying transmissibility. The models do not attempt to account for excess deaths caused by other pandemic-related factors, for example the vaccines themselves as well as other non-pharmaceutical compulsory interventions. The CDC reports a vaccine-induced death risk of 0.0026% on average per dose basedon the Vaccine Adverse Events Reporting System, or VAERS3. VAERS is a passive reporting systemand it is well known it suffers from under ascertainment in that it only captures ~1% of all vaccine-related side effects [6]. Recent independent lines of evidence suggest that VAERS may only capture 35% of all vaccine-induced deaths [7,8]. In addition, the models do not account for excess deathsresulting from other factors such as lockdown-induced “deaths of despair”.4 By ignoring other potentialsources of pandemic-related excess deaths in their models, the fitted models will over- and/ormisestimate the effects of natural, time-varying virus transmissibility in order to achieve a good model fitwith reported excess deaths, which in turn would lead to inflated excess death counts in theircounterfactual simulations.
Lack of face validity
According to the authors’ country level estimates 1.9 Million deaths were averted in the US assuming a61% vaccine coverage (see Supplementary Table 3 in original study). In the first year of the pandemicwhen no vaccines were available (2020), there were 351,039 confirmed COVID deaths in the US5. Theauthors’ models thus suggest that 1.9M / 350k = ~5.5x as many COVID deaths in the US would haveoccurred in 2021 (vs. 2020) had no vaccines been introduced. This is highly implausible as there is verylittle reason to believe the virus would have naturally evolved to be that much more transmissible, infective and lethal. The authors allude to higher transmissibility in 2021 due to the relaxing and/or lifting of public healthmeasures and restrictions (lockdowns, travel restrictions, mask mandates etc.). However, theassumption that this could account for a >5-fold increase in COVID deaths in 2021 contradicts >400studies that have concluded there were very little to no public health benefits of these measures inreducing COVID outcomes6. Moreover, in 2021 (after vaccinationwas introduced), there were 474,890confirmed COVID deaths in the USaccording5. This suggests crudeevidence that mass vaccination mayhave worsened COVID outcomesoverall, consistent with observationsof increased infectivity before vaccineprotection kicks in (see 1st pointabove) as well as previouspredictions and observations ofenhanced respiratory disease viaantibody dependent enhancement based on preclinical studies [9,10].
Summary
While generative models are a usefultool to simulate scenarios that havenot occurred, not accounting properlyfor relevant variables in the modelmay lead to model misspecification.In such cases, counterfactuals maygrossly inflate estimates of deaths averted due to mass vaccinations. Rather than rely on simulationswhich may be sensitive to input parameters, prone to overfitting, and that are difficult, if not impossibleto validate, more accurate and reliable approaches to inform public health vaccination policies arequantitative risk-benefit ratio analyses for specific outcomes using clinical trial or real-world data[7,11,12].4 See https://www.aier.org/article/study-indicates-lockdowns-have-increased-deaths-of-despair5 See https://ourworldindata.org/coronavirus/country/united-states6 See https://brownstone.org/articles/more-than-400-studies-on-the-failure-of-compulsory-covid-interventionsFigure 2. Two left bars show total confirmed COVID-19 deaths in the US in 2020 and 2021 from Our World in Data. Rightmost bar (*) indicates predicted number of averted US deaths due to vaccines in the Watson et al. article (Supplementary Table 3).