After a study in England found that the Pfizer/BioNTech Coronavirus vaccine is 88% effective against the Delta variant, data from Israel suggests just 41% effectiveness.
The contradictory results has created confusion in the media. For example, a Forbes article claimed “Israel’s findings [are] the most up to date and accurate assessment of the Pfizer shot against Delta,” which implies that Israel’s report is more reliable because it’s more recent. That’s not necessarily true.
Which is the correct figure for how well Pfizer’s vaccine works? And why is there such a big difference in the effectiveness estimated by the two countries?
Comparing the English and Israeli results is challenging because there are at least three potential sources of uncertainty that might influence the figures for efficacy.
1. Waning Immunity
A vaccine’s power to trigger an immune response can wane over time, so one explanation for the data discrepancy is that England and Israel estimated Pfizer’s effectiveness based on people with differing levels of immunity.
The design of existing Coronavirus vaccines is modelled on the original strain of SARS-CoV-2, which means they can remain potent against that variant while also being less effective against new variants.
If Pfizer’s vaccine provides less protection against specific variants, it’s less powerful against ‘the’ virus generally. As a consequence, you would expect some fully-vaccinated people (who received both doses of the drug) to be infected by a variant, known as ‘breakthrough’ infection.
The report from Israel’s Health Ministry supports the possibility that Pfizer has lost potency, leading to waning immunity: the country’s rate of breakthrough Covid cases is higher among people who had been double-jabbed a few months ago, long before they were exposed to Delta.
Over a 4-week period (20 June to 17 July 2021), Pfizer’s general effectiveness decreased as the time before exposure increased: efficacy was 79% for those who received their second dose in April, 69% for March, 44% for February, and 16% for people who were fully-vaccinated back in January.
The average for those four vaccination months gives Pfizer’s vaccine an overall figure of 41% effective.
According to sampling of the Israeli population since mid-June, the prevalence of Delta is 93%, which suggests that Pfizer’s figure largely reflects effectiveness against that variant.
By contrast, the study led by Public Health England reported efficacy based on everyone who tested positive for Delta over a 7-month timeframe (26 October 2020 and 16 May 2021) and doesn’t mention when those 122 people had been vaccinated.
At first glance then, Israel’s data looks more accurate as it shows Pfizer’s effectiveness in finer detail — but other factors undermine that conclusion.
2. Age Groups
Another possible reason for the discrepancy is the impact of age, which could introduce errors into the statistics used to estimate the vaccine’s efficacy.
A person’s susceptibility to Covid is influenced by age, with the elderly being more vulnerable to developing disease and children being less susceptible — in part thanks to their stronger immune systems. And because a vaccine works by prompting an immune response, its effectiveness will vary among different age groups.
One big difference between the data is that Pfizer’s drug wasn’t distributed equally in England and Israel.
Although vaccination was rolled-out by age group in both countries, in England almost all adults over 40 years received the Oxford/AstraZeneca vaccine, while many younger people were given the option of Pfizer. In Israel, everyone received Pfizer’s vaccine.
The unequal distribution could affect estimates, as a drug’s effectiveness is usually adjusted to account for the variation in efficacy among age groups.
For example, England’s narrow range for Pfizer-vaccinated adults potentially means less confidence in how the country’s data applies to older people.
On the other hand, Israel’s report compared breakthrough cases in two groups — aged 16-59 and 60+ — and when you cross-reference that with Pfizer’s efficacy, it becomes clear that there are relatively few elderly people in the dataset.
Broadly speaking, the more data points, the more confident you can be in your results — a large sample size reduces the likelihood that a result is down to a statistical fluke.
As Eran Segal, a biologist at the Weizmann Institute of Science in Rehovot and vaccine consultant to the Israeli government, told The New York Times, “I think that data should be taken very cautiously because of small numbers.”
Another source of statistical error is not collecting a random or representative sample of the population.
Vaccine efficacy is expressed as a percentage and often calculated by checking the vaccination status of people who develop disease and multiplying the total number (including unvaccinated individuals) by the proportion vaccinated.
The Health Ministry’s report doesn’t include samples from Israeli regions (or populations) with low vaccination rates, or haven’t yet experienced a surge in infections due to the Delta outbreak. According to another biologist, Dvir Aran, the missing numbers could affect calculations and produce an underestimate of Pfizer’s effectiveness.
3. Testing Differences
The third reason why a direct comparison might be unreliable is due to technical differences in Covid testing.
Both countries use PCR (polymerase chain reaction) to detect whether a person is infected with the virus. PCR tests try to detect tiny amounts of genetic material by repeatedly copying or ‘amplifying’ what they find — more cycles of amplification provide a better chance of detecting the material and therefore catching cases of Covid.
According to The Jerusalem Post, Israel’s PCR tests have greater sensitivity through more amplification cycles.
As Cyrille Cohen, Health Ministry advisor on clinical trials of SARS-CoV-2 vaccines, explained, “If the PCR testing is less sensitive, England may miss some cases — or Israel may catch more cases — and that could play a role in the numbers.”
Both countries tested for Coronavirus through PCR as well as assessed whether a person showed signs of Covid — such as respiratory symptoms or fever — which allowed estimates of Pfizer’s effectiveness at either preventing infection or protecting against disease.
But at least a third of Coronavirus infections are asymptomatic (infected people don’t show clear symptoms).
In principle then, a vaccine’s efficacy, as tested via PCR, could differ from its efficacy based on diagnosis of symptomatic Covid. In practice, however, those two figures for effectiveness tend to be similar.
Israel’s report showed that Pfizer’s vaccine is 39% effective against infection and 41% effective against symptomatic disease. Meanwhile, a study in Scotland that compared fully-vaccinated and unvaccinated people found that Pfizer provided 92% protection against infection — close to neighboring England’s 88% figure for preventing Covid.
So is the effectiveness 41% or 88%?
Trying to nail-down the right number is tricky for many reasons, including the fact that the word ‘effective’ can have several meanings, making the very concept of a vaccine’s effectiveness confusing.
It’s also important to remember that figures should be treated as approximate — that includes my charts showing whether current vaccines are effective against Coronavirus variants.
Finally, the three sources of uncertainty I’ve covered above aren’t mutually exclusive — the correct figure for the Pfizer vaccine’s effectiveness may be unclear because of statistical issues from age groups and waning immunity, for instance.
If immunity starts to wane after about 6 months, it’s possible that Pfizer is 88% effective if you’ve only recently been fully vaccinated, but the efficacy is closer to 41% if it’s been a while since you received your second dose.
In fact, Pfizer and BioNTech believe a third booster shot may be needed to help protect against variants like Delta. Their argument is based on the assumption that you need a response that generates neutralizing antibodies, however, which may not be necessary if other parts of the immune system — notably T cells — are able to effectively fight-off infection.
More importantly, regardless of the time between vaccination and exposure to any variant, Pfizer’s drug — like most Covid-19 vaccines — has proven to be over 90% effective at preventing severe disease leading to hospitalization or death.