Forecast performance in the ECB SPF: Ability or chance?

In this paper, we consider whether differences in the forecast performance of ECB SPF respondents reflect ability or chance. Although differences in performance metrics sometimes appear substantial, it is challenging to determine whether they reflect ex ante skill or other factors impacting ex post sampling variation such as the nature of economic shocks that materialised or simply which rounds participants responded in. We apply and adapt an approach developed by D'Agostino et al. (2012) who used US SPF data. They developed a test of a null hypothesis that all forecasters have equal ability. Their statistic reflects both the absolute and relative performance of each forecaster and they used bootstrap techniques to compare the empirical results with the equivalents obtained under the null hypothesis of equal forecaster ability. Our results, at a first pass, suggest that there would appear to be evidence of good/bad forecasters. However once we control for the autocorrelation that is caused by the overlapping rolling horizons, we find, like D'Agostino et al. (2012), that the best forecasters are not statistically significantly better than others. Unlike D'Agostino et al. (2012), however, we do not find evidence of forecasters that perform very significantly worse than others. Controlling for autocorrelation is a key feature of this paper relative to previous work. Our results hold considering the whole sample period of the ECB SPF (1999-2018) as well as the pre- and post-global financial crisis samples. We also find that when assessed across all variables and horizons, the aggregate (consensus) SPF forecast performs best.