Human Forest vs. Random Forest in Time-Sensitive COVID-19 Clinical Trial Prediction

How do we generate accurate predictions about the timely advancement of clinical trials? We report the methods and results of the first study comparing machine modeling and human crowdsourcing methods in clinical trial prediction. The study spans two 6-month tournaments that focus on trial phase transition for vaccines and treatments for COVID-19 and other infectious diseases. The machine approach is represented by a time-specific random survival forest (RSF) model. The human crowdsourcing approach focuses on the Human Forest process and software, which enables forecasters to define custom reference classes, query a historical database and review base rates specific to their selections. These base rates, and adjusted probabilistic estimates, are then aggregated. Forecasters receive proper scoring feedback and accuracy incentives. We compare forecast accuracy across 60 forecasting questions in two tournaments. We show that the crowdsourcing methods significantly outperform the RSF model, registering mean Brier scores between 36% and 52% lower than those for the model. The differences in accuracy among human crowdsourcing methods are smaller. Including Human Forest-derived base rate estimates at the aggregation stage improves overall performance among new and elite forecasters. Our results show that human forecaster crowds with access to historical information, appropriate elicitation and aggregation tools can provide more accurate predictions than models