Meta-analysis in a time of pandemic

Christian Lederer, Martin Daumer, Romain-Daniel Gosselin, Ijaz S. Jamall, and Björn L.D.M. Brücher 1 Sylvia Lawry Centre for Multiple Sclerosis Research e.V. – The Human Motion Institute, Munich, Germany Theodor-Billroth-Akademie , Germany, USA 3 INCORE, International Consortium of Research Excellence of the Theodor-Billroth-Academy , Germany, USA Precision Medicine Unit, Lausanne University Hospital, Chemin des Roches 1a/1b CH-1010 Lausanne, Switzerland Risk-Based Decisions Inc., Sacramento, CA, USA Department of Surgery, Carl-Thiem-Klinikum, Cottbus, Germany


Editorial
There is a perception that meta-analyses can provide neat and concise conclusions which can then be used for clinical treatment guidelines or in public health as in the on-going Coronavirus disease 2019 (COVID-19) pandemic. The widespread popularity of meta-analyses also presents a risk in that inconclusive results in the original individual studies, evaluated under the rubric of a meta-analysis, become accepted based on flawed methodologies and thus be used inappropriately for decision-making, especially for easy to implement, but potentially incorrect, patient treatments or public health measures. Recently a meta-analysis was published in 4open as a means to obtain a better estimate of the serial interval (SI) for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) during the COVID-19 pandemic [1]. The author aimed to reduce the uncertainty around the SI for different COVID-19 variants by performing a meta-analysis of publications between 1st December 2019 and 15 February 2022 and concluded: "The meta-analysis was unable to provide a suitable estimate of serial intervals for Covid-19 modeling purposes although its uncertainty was reduced. Furthermore, serial intervals estimate for alpha variant was close to earlier reports and lower than previous publications, respectively. Another limitation is, that meta-analysis of COVID pandemic studies in principle contains and produces itself a significant source of heterogeneity." The author addresses the heterogeneity of SI estimates using a random effects model. However, one should distinguish between two sources of heterogeneity: Firstly, the studies underlying the meta-analysis used different methods to estimate SI, e.g. selection of infector-infectee pairs. For this source of variability, the use of a random effects model seems appropriate although one may question if these pairs can be considered as typical, as other plausible transmission modes, e.g. overcrowded living conditions or public transportation, do not show up in the data. Secondly, the source of heterogeneity includes studies that address different stages of the pandemic which could result in different SI estimates.
The basic reproduction number R0 (expected number of infections generated by one case in a fully susceptible population) and the generation interval (average time interval between primary and secondary infection, approximated) are key parameters in the modeling of the spread of COVID-19. For obvious reasons, the public discussion on control measures was focused on R0, which determines if the epidemic will spread or subside. The generation interval, approximated by the SI (time between symptom onset between infector and infected) is the time for the infection to be observed, which is necessary in order to estimate R0 from incidence data. Since all control measures were aiming to decrease the contact rate and hence R0, it would be absurd to perform a meta-analysis in order to estimate a universal R0. The SI, however, is linked to the infectious period and, in fact, if control measures would randomly filter out a proportion of infectious contacts, the SI would be unaffected. However, control measures, such as early isolation, can have a considerable effect on the SI (decreased by a factor of 3) [2]. Therefore, the term "effective serial interval" was introduced to distinguish it from the biological period of infectiousness of a subject.
Jusot [1] included [2] in his meta-analysis and referred to this change of the SI estimate. He also noted that control measures in the studies underlying the meta-analysis were not sufficiently described in the underlying literature to include this factor into his meta-analysis. However, he did not draw the conclusion that estimating a universal SI is as meaningless as estimating a universal R0, but instead investigated the underlying publication bias. The rationale for the use of the Egger's test is that studies showing no statistically significant effect are more likely to remain unpublished. This can result in a bias towards larger effects for small studies (skewness and asymmetry in funnel plots). But would there be a reason for a publisher to be biased towards small SI estimates since they are likely to have a small standard error?
Concerning decision making, forecasting of the time course of the COVID-19 epidemic is desirable and since the beginning attempts have been made to make long-term forecasts of various COVID-19 consequences to better plan intensive care beds and to decide on control measures (lockdowns). Increasing intensive care capacity however requires a time scale which is far beyond the forecasting horizon. When comparing forecasts for the second wave over a four week time frame from 19th Oct until 19th Nov 2020 ( [3] and see also [4]), report forecasts ranging from a return to the lower incidence of previous weeks to exponential growth. No validated model was available at that time, and the model that performed best for the 19th Oct 2020 forecast did not necessarily perform well for 19th Nov 2020 forecast. Notably, in regard to planning of intensive care unit requirements, statistical modeling is not helpful beyond worst-case scenarios.
With regard to short-term forecasting for decisions on control measures, various phenomenological models have been proposed which are not based on SI or R0. For example, [5] presented a model that gave reliable short-term estimates for bed occupancy in various hospitals, although it was applied during the highly variable dynamics of the first wave in 2020. Here, the hospital admission rate (arrival rate as term in accordance to the queuing theory) was ad hoc modeled by fitting a Richard's curve based on data available at the hospital. Models like this are exactly what are needed for short-term decisions on the need for lockdowns. The only disadvantage is the lack of interpretable parameters like R0 for public communication of decisions.
In summary, the author correctly questioned and illustrated what can be gleaned from his meta-analysis [1]. The dynamics of the pandemic makes it complicated as the historical SI values are no longer valid given the continuous change in control measures, vaccination status, and vaccination efficacy.