The web-based dashboard also lets users compare their predictions to professional models, giving users insight into how to build better forecasts.

The problem with forecasting

Accurate forecasting models are vital to pandemic safety because they can influence public policy and healthcare resource allocation. Yet Covid-19 has been especially hard to predict because its spread is affected by region, season, and other factors.

The idea of combining forecasts comes from forecasting competitions traditionally used to determine the outcome of elections, or the success of businesses, where rewards are given for accurate predictions.

“Competitive forecast mechanisms can be very useful for making predictions in uncertain environments,” College of Computing undergraduate researcher Gayeon Yoo said. “A number of studies have shown the benefits of trusting the ‘wisdom of the crowd.’”

Aggregating the data

The website combines various Covid-19 databases and predictions from the Institute for Health Metrics and Evaluation (IHME), Columbia University, and Georgia Tech for case and death totals. The site determines their accuracy by calculating the mean squared error of the models by prediction time: a week, two weeks, four weeks, and overall. The most accurate forecasts are ranked on the Top Forecasters page.

Users can also provide their own forecasts and learn later how their forecasts performed relative to other users and professional modelers.

“Seeing how your predictions compare to others on different outcomes such as daily cases or daily deaths is what intrigues users,” New York University undergraduate research Rachel Ombok said. “Being able to see the shift in your own predictions and see what your forecasts over time can be useful to researchers, and also helps to engage users who may have private information about the pandemic’s spread in their region.”

The team believes the ranking system encourages users to develop better prediction models by creating a competitive environment around forecast accuracy.

Future iterations of the site will include more data to forecast on hospitalizations and infections, as well segment by country, state, and other regions. The team also plans to make the interface more customizable so users can write their own programs and use statistics or machine learning for automated submissions.

“We want people to see what forecast modeling is like first-hand, and we also want to use our evaluation metrics to quantify the accuracy of user and institution forecasts,” College undergraduate Aniruddha Murali said.

The website was created by College of Computing undergraduates Muraliand Yoo and New York University undergraduate Ombok.

The project is part of a Covid-19 Research Experience for Undergraduates (REU) started by School of Computer Science Assistant Professor Jake Abernethy with School of Computational Science and Engineering Associate Professor Aditya Prakash, University of Michigan Assistant Professor Eric Schwartz, and University of Colorado Assistant Professor Bo Waggoner. The project was funded by the National Science Foundation and Georgia Tech’s Institute for Data Engineering and Science (IDEaS).