Testing the predictive power : A comparative study of current default probability validation tests

Credit ratings are expert systems which assess the likelihood of a borrower to default. The Basel Accord allows banks to base regulatory capital requirements on the default probability of a rating. Banks must prove that the employed default probabilities are valid estimates. Since in credit risk observations are not independent – default rates fluctuate cyclically – the substantial literature on testing a binomial proportion is not applicable without reservations. To overcome the limitations of traditional binomial testing special tests were developed for the area of credit risk. The aim of these newly developed tests is to detect default probabilities which are too low. At present it is unclear how some of these new tests perform and how much power they possess. The paper presents a simulation study which evaluates and compares several of the newly proposed statistical tests used for out-of-time validation. The results of the simulation study show that many tests lack the power to detect default probabilities which are too low. A second important finding of the study is that for more cyclical portfolios the significance level of a test should be increased