Right now there are hundreds if not thousands of Covid-19 models floating out there. Some are better than others and some have much more influence than others and the ones that have the most influence are not necessarily the best. There must be a better way of doing this. The world’s greatest minds convened in Los Alamos in WWII and built two atomic bombs. Metaphors get thrown around with reckless abandon but if there ever was a time for a Manhattan project, we need one now. Currently, the world’s scientific community has mobilized to come up with models to predict the spread and effectiveness of mitigation efforts, to produce new therapeutics and to develop new vaccines. But this is mostly going on independently.
Would it be better if we were to coordinate all of this activity. Right now at the NIH, there is an intense effort to compile all the research that is being done in the NIH Intramural Program and to develop a system where people can share reagents and materials. There are other coordination efforts going on worldwide as well. This website contains a list of open source computational resources. This link gives a list of data scientists who have banded together. But I think we need a world wide plan if we are ever to return to normal activity. Even if some nation has eliminated the virus completely within its borders there is always a chance of reinfection from outside.
In terms of models, they seem to have very weak predictive ability. This is probably because they are all over fit. We don’t really understand all the mechanisms of SARS-CoV-2 propagation. The case or death curves are pretty simple and as Von Neumann or Ulam or someone once said, “give me 4 parameters and I can fit an elephant, give me 5 and I can make it’s tail wiggle.” Almost any model can fit the curve but to make a projection into the future, you need to get the dynamics correct and this I claim, we have not done. What I am thinking of proposing is to have the equivalent of FDA approval for predictive models. However, instead of a binary decision of approval non-approval, people could submit there models for a predictive score based on some cross validation scheme or prediction on a held out set. You could also submit as many times as you wish to have your score updated. We could then pool all the models and produce a global Bayesian model averaged prediction and see if that does better. Let me know if you wish to participate or ideas on how to do this better.
Scientific Clearing House 
Thanks for writing this, Carson. I just had a conversation with a hospital administrator where he had completely given up on models because they weren’t consistently helpful. He also was very skeptical of the CDC data because there was such a time lag in reporting, and that the dates reported were actually result dates rather than test collection dates (an issue I am dealing with in my data). If there isn’t already some editorial for a national pandemic modeling group and associated weekly or biweekly online meeting, it would be great if you or someone could write it – contact all the major modeling groups and get them onboard for a multi-author editorial in a widely read periodical. I think the modeling community needs to speak in a unified way to prevent the models from being weaponized, politicized, and misinterpreted. It’s also really frustrating how fragmented the effort is – for example I just found out that the local city forecasting group for their hospital admissions, but I only randomly heard about it by joining a local-area webinar. Most hospitals, counties, and local groups don’t have enough experience to interpret models, to estimate the R0 locally, or to set policy based on that (although how good an idea that is… don’t know how reliably these models are forecasting, how they are validated, and how well they are doing). I wish I could easily run my data through ALL the major models once I have it ready to go, but it’s crazy that I probably would have to individually go through Github to do so. There should be a portal where people can upload standardized data into the models and get projections easily. Of course getting clean data is the biggest problem. I’m working on data quality more than even modeling! It is hard to be able to interpret and I am basically having biweekly meetings with hospital administrators to understand what policy changes are happening as they change all the time, so it’s hard to pin down the right set of assumptions (or the timeframes in which assumptions are appropriate).
LikeLike
@Karen Well, I’m kind of trying but it is hard to get any traction. First of all, I’m an interloper in the field. The flu community has tried to come up with a standardized way to evaluate models and Nick Reich at Amherst is the person providing the predictions for the CDC site. See here https://github.com/reichlab/covid19-forecast-hub. The flu groups are loosely collaborating on Covid-19 too. Data quality is a big issue. We can start to build the tools you need at NIH. I have a connection to FDA and have tried to reach out to Reich but it’s slow.
LikeLike
I think the Los Alamos group is doing stuff too – I reached out but haven’t heard back. I interviewed with the UW group as well. I am tempted to pull the MD card here (“my patients are dying, please hurry up!”) because we have to make and are making decisions with or without information. Unfortunately right now it is often without information, and I know that we do not have a good handle on our local reproduction number. I think we are about 1-2 weeks away from having a very high quality data source ready internally though. I have MDs and biostatisticians ready to check the data assumptions and am working with admin to automate the data source as much as possible – and even collect and collate the data we need for dynamical systems models. I have a physicist working on connecting data to the models. If I didn’t know I was headed back to the ICU in a week, I would be emailing people all over the country right now to organize this. As far as I know I am one of very few translational mathematical biologists (in any field?) who are seeing COVID patients AND building models, but if there is anyone else I want to meet them! I hate that we are going pretty blind into the future when we could do better than that.
LikeLike
A very thorough article written by the author on corona virus , the author knows his audience very well, this article shows the dedication and ingenuity of the author. Hats off
LikeLike