COVID precise: Living systematic review of diagnostic and prognostic prediction and machine learning models for COVID-19
In March 2020, the COVID-19 pandemic plunged the world into turmoil. Doctors everywhere found themselves confronted with questions. Which patients were at risk of severe illness? And what were an individual’s chances of survival in the intensive care unit? Laure Wynants immediately realised that she, as a statistician, could help them find the answers by identifying accurate prediction models. Her initiative was eagerly embraced by experts in the Netherlands and abroad. Together, they developed a plan to systematically review models and papers being published on the topic. Through high-quality methodological research, Wynants and her team were able to provide doctors and policymakers on the front lines with a useful tool. The living systematic review was even adopted as the standard by the WHO. It received praise for its speed, relevance, and methodological quality. And that wasn’t all. More generally, Wynants’s approach has served as a wake-up call for the quality of prediction models.
Precise Risk Estimation to optimise COVID-19 Care for Infected or Suspected patients in diverse sEttings
- Start project:
- Project website: www.covprecise.org
With the pandemic threatening to overwhelm all national health services, it was essential to make early predictions of disease progression to guide preventive, diagnostic and therapeutic decision-making. Studies had already shown that prediction models could be used for various diseases. The same must be true for COVID, thought Wynants. Predictions could then help inform all kinds of important decisions. For example, when testing capacity was still limited: does this person need to be tested? Or, at the GP surgery: can this patient recover at home or do they require hospitalisation? In the hospital: can this patient be admitted to the COVID ward or do they need intensive care? And in the ICU: what is the prognosis for this patient? How long will we continue to provide care?
The gold standard
From the moment Wynants began her search for useful prediction models, the initiative took off. Some 50 researchers and 40 institutes in the Netherlands and abroad were keen to work with her. The BMJ, a renowned journal, invited Wynants and her colleagues to conduct its first-ever living review. In the absence of tools available for use in clinical practice, the WHO adopted their recommendations. The team continuously updated the review, identifying accurate tools and offering suggestions. They reviewed tools ranging from traditional prediction models based on regression models and machine learning to automatic artificial intelligence-based CT image analysis. At each stage of the pandemic, they selected the best state-of-the-art tools. Wynants’s review became the gold standard for practitioners, researchers and policymakers alike. By bridging the gaps between these stakeholders, it had a significant positive impact on society.
Laure Wynants, PhD
But the review was not just a powerful weapon in the fight against coronavirus. It has also highlighted the importance of conducting sound methodological research. Most models were found to be of poor quality, especially those published in the initial phase of the pandemic. Several models turned out to be even worse at predicting disease progression than using age as the only predictor. The number of high-quality, representative and large datasets did grow over time, albeit slowly. But the vast majority of studies continued to be at high risk of bias due to poor methodologies and statistics – a problem that could easily be prevented or solved by focusing on adequate methods in teaching, research funding and publications.
Wynants’s international team ended up screening 126,978 titles and including 412 studies describing 731 new prediction models or validations. They were able to identify just a handful of reliable tools. The Qcovid models, the PRIEST score, Carr’s model, the ISARIC4C Deterioration model, and the Xie model were the only models to show adequate performance in predicting COVID-19 disease progression in studies at low risk of bias.
What’s next? Three main issues
The pandemic is over, and the living review has been closed. But Wynants’s work is far from done. She is turning her attention to three main issues. Firstly, she would like to draw up recommendations for model developers – judging from the sheer number of citations of the living review, there is a knowledge gap to be filled. Secondly, she plans to use the data for international validations to see if specific models perform equally well in other circumstances. One such validation study has already been published in the BMJ. Most of the models reviewed were developed using data and variables from wealthy countries; in collaboration with the WHO, a validation study has now been launched in low- and middle-income countries. There are lessons to be drawn from this. It is necessary to develop versions of models for low- and middle-income countries using only patient characteristics and widely available clinical tests as input variables; in addition, it is important to support data collection in those settings and to develop tools to combat the continued existence and growth of inequality. Finally, a model has been developed for general practitioners. GPs were largely left out in the cold during the pandemic, when it was they who often had to make important decisions – such as which patients needed to go to hospital or receive antiviral treatment.
Text: Eline Dekker
The COVID PRECISE consortium consists of clinical and prediction model experts across the globe. It now includes over 50 investigators who actively collaborate with the (meta-)data of hospitals and practices of numerous countries, among which, Belgium, Sweden, England, Italy, Spain, Denmark, Ireland, Turkey, Switzerland, Germany, Brazil, United States and Mexico.
The steering group of COVID PRECISE is:
- Laure Wynants, PhD, Care and Public Health Research Institute Maastricht, Netherlands
- Maarten van Smeden, PhD, Julius Center for Health Sciences and Primary Care Utrecht, Netherlands
- Prof.dr. Carl Moons, Julius Center for Health Sciences and Primary Care Utrecht, Netherlands
- Ben Van Calster, PhD, KU Leuven / LUMC Leiden, Belgium / The Netherlands
Involved research line
- Optimising Patient Care
Our most important publications in scientific journals
- Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JAA, Debray TPA, de Jong VMT, De Vos M, Dhiman P, Haller MC, Harhay MO, Henckaerts L, Heus P, Kammer M, Kreuzberger N, Lohmann A, Luijken K, Ma J, Martin GP, McLernon DJ, Andaur Navarro CL, Reitsma JB, Sergeant JC, Shi C, Skoetz N, Smits LJM, Snell KIE, Sperrin M, Spijker R, Steyerberg EW, Takada T, Tzoulaki I, van Kuijk SMJ, van Bussel B, van der Horst ICC, van Royen FS, Verbakel JY, Wallisch C, Wilkinson J, Wolff R, Hooft L, Moons KGM, van Smeden M. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020 Apr 7;369:m1328. doi: 10.1136/bmj.m1328.
- Valentijn M T de Jong, Rebecca Z Rousset, Neftalí Eduardo Antonio-Villa, Arnoldus G Buenen, Ben Van Calster, Omar Yaxmehen Bello-Chavolla, Nigel J Brunskill, Vasa Curcin, Johanna A A Damen, Carlos A Fermín-Martínez, Luisa Fernández-Chirino, Davide Ferrari, Robert C Free, Rishi K Gupta, Pranabashis Haldar, Pontus Hedberg, Steven Kwasi Korang, Steef Kurstjens, Ron Kusters, Rupert W Major, Lauren Maxwell, Rajeshwari Nair, Pontus Naucler, Tri-Long Nguyen, Mahdad Noursadeghi, Rossana Rosa, Felipe Soares, Toshihiko Takada, Florien S van Royen, Maarten van Smeden, Laure Wynants, Martin Modrák; CovidRetro collaboration; Folkert W Asselbergs, Marijke Linschoten; CAPACITY-COVID consortium; Karel G M Moons, Thomas P A Debray 4. Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis. BMJ. 2022 Jul 12;378:e069881. doi: 10.1136/bmj-2021-069881.
Several invited talks (>20) at national and international conferences and meetings
In the media
The study attracted a lot of media attention (in national and internatiol media).
- NRC newspaper (May 2020): Zet er corona op en je onderzoek krijgt aandacht
- UM website (April 2020): Good prediction models for COVID-19 are urgently needed
- Het Belang van Limburg newspaper (August 2020): “Coronastudies blijken al te vaak haastwerk”
- Observant (March 2021): Which model can predict how COVID-19 will progress?
- MIT Technology review (July 2021). Hundreds of AI tools have been built to catch covid. None of them helped.
- Volkskrant newspaper (August 2021): Nut van inzet kunstmatige intelligentie bij diagnose covid twijfelachtig
- Tagblatt Swiss newspaper (August 2021). Doktor Frankenstein verwirrte die Künstliche Intelligenz
Our most important output for societal target groups
Several citations in policy documents. A selection:
- Cited by the EU Department for Economic, Scientific and Quality of Life Policies (European Parliament), “Impacts of the COVID-19 pandemic on EU industries”
- Cited by the WHO "COVID-19 Clinical management Living Guidance document"
- Citied by the OECD, "OECD Health Working Papers No. 128: Laying the foundations for artificial intelligence in health"