Using Beaton Fit Indices to Assess Goodness-of-fit of IRT Models

The purpose of this study was to investigate the performance of Beaton~{!/~}s MR and MSR fit indices for assessing goodness-of-fit of IRT models. These statistics are based on a standardized residual calculated from an expected and observed response. The investigation was conducted using a Monte Carlo simulation study that varied conditions relevant to testing applications.This research had three objectives: 1) To identify the sampling distribution of the fit statistics; 2) To assess the Type I error rates under different combinations of manipulated factors; and 3) To investigate the empirical power under different combinations of manipulated factors by introducing different types of model misfit.The sampling distribution of Beaton~{!/~}s MR and MSR statistics belonged to the family of normal distribution. However, there was no basis for a theoretical normal distribution to test the hypothesis of model-data-fit. Therefore, Monte Carlo resampling methods were required to test the hypothesis of model-data-fit for Beaton~{!/~}s fit statistics.Using Monte Carlo resampling methods for hypothesis testing, nominal Type I error rates were observed in this study regardless of test length, sample size, Monte Carlo resample size and number of replications. With regard to empirical power, higher power was observed for Beaton~{!/~}s MR statistic than MSR statistic under the condition that H0 was false for the entire test. Under the condition that H0 was false for a subset of test items, higher power for the misfitting item and more false rejections than expected for all the other items were obtained for Beaton~{!/~}s MR statistic. In contrast, reasonable empirical power for the misfitting item and nominal Type I error rates for all the other items were observed for Beaton~{!/~}s MSR statistic. Based on the results of this study, Beaton~{!/~}s MSR fit statistics can be used to assess goodness-of-fit for both shorter (12 items) and longer test (36 items). The recommended sample size is 500 or more, and a Monte Carlo resample size of 100 should be adequate for hypothesis testing.