Efficacy vs Effectiveness of the COVID Vaccines…”tomato, tomahto”?

You like potato and I like potahto
You like tomato and I like tomahto
Potato, potahto, tomato, tomahto
Let’s call the whole thing off

But oh, if we call the whole thing off
Then we must part
And oh, if we ever part
then that might break my heart

—Ira Gershwin

The eye-popping efficacy rates reported for the Moderna (94%), Pfizer (95%) and, to a lesser extent, the Johnson & Johnson (66%) COVID-19 vaccines have undoubtedly not escaped your attention.

But what is vaccine efficacy and how is it calculated? And how does it differ from vaccine effectiveness?

Moderna vaccine efficacy

First, consider efficacy. Using Moderna’s reported clinical trial results as an example, we see that it is a straightforward calculation.

Moderna reported results from it’s COVID-19 vaccine trial in November 2020. The results are shown below in a 2×2 “contingency” or “cross-tabulation” table. The columns show the number of subjects who were infected (or not); the rows show the number who received the vaccine (or the placebo). And the cells show the intersection of those two events.

Relative risk

The strength of the association, or the effect size, between receiving the vaccine and not getting infected is measured by the relative risk.

The probability or risk of a vaccinated subject being infected is 0.08%. That is, (11 / 14,134) or the expected number of events / sum of events and non-events. For a subject receiving the placebo, the probability of infection is higher at 1.31% (i.e., 185 / 14,073).

So, using the placebo group as the reference group, the relative risk is (11 / 14,134) / (185 / 14,073) or 0.059.[1]

In other words, the risk of a vaccinated person being infected is 94.1% lower compared to a subject who received the placebo (i.e., (1 – 0.059) * 100)).

It is this calculation of 94.1% that was reported by Moderna as the vaccine’s efficacy rate.[2]

Vaccine effectiveness

So, what about vaccine effectiveness? The term effectiveness refers to how the vaccine performs in the real world.  Efficacy refers to how the vaccine performs under “optimal” conditions of a clinical trial.

Clinical trials are based on a sample of subjects who may not be fully representative of the general population (e.g., all comorbidities are not controlled for). In addition, the COVID strain that existed in the population during the clinical trial period may not be the same that occurs when the vaccine is released. Also, vaccine transportation, storage and delivery may differ from the more controlled environment of the clinical trial. Thus, the effectiveness of the vaccine may be different from what was found during the clinical trial.

Studies on COVID vaccine effectiveness

So, do we have any data yet on the real-world effectiveness of the COVID vaccines? It takes time to collect data, but we do have some indication that vaccine effectiveness is very high.

An early study appeared February 24, 2021 in the New England Journal of Medicine.  The study examined the Pfizer vaccine performance in Israel. The sample was matched data from over 1 million people, half who were vaccinated between December 2020 to February 2021 and half who were not. The results of the study suggest a symptomatic infection effectiveness rate of 94% 7+ days after the second dose.

A more recent study released by the CDC on April 2 examined both the Pfizer and Moderna vaccines.  This study used US data from December 2020 to March 2021. The sample consisted of 3,950 health care personnel, first responders, and other front-line workers.  The study found that the vaccines were 90% effective against COVID infection 14+ days after the second dose. Even 14+ days after the first dose the vaccines were 80% effective.

As a point of comparison, according to the CDC, effectiveness of the annual flu vaccination ranges between 40 and 60%.[3]

So, the effectiveness rate, after 2 doses of the Pfizer and Moderna vaccines, appears to be very close in magnitude to the efficacy rate.

Very good news indeed!

Tomato, tomahto?

[1] A relative risk ratio of 1.0 would mean no difference in effect between the treatment types.

[2] A summary of efficacy rates across the range of current COVID vaccines can be found here.

[3] One reason for the range is that the flu strain that is in circulation can differ from what was predicted when the annual flu vaccine was developed earlier in the year.