Using mathematical graphs for questionnaire testing in large-scale surveys

In this article, we present an automated test procedure for examining the filter structure and instructions implemented in electronic questionnaires, and for checking the fit of a questionnaire to the targeted sample. With our approach, we can represent and describe questionnaires using mathematical graphs and specify questionnaire properties in a formal and standardised way. It also allows us deriving mathematical graphs from empirical data. We can then compare the questionnaires (mathematical graphs) with the survey data in an automatable process. Our procedure also includes a test plan we developed for automatic testing. Our approach is complete, portable, and scalable: It is complete in that graphs are used to describe the questionnaires and questionnaire data. It is portable as a result of its generic structure, which is not limited to a specific questionnaire (type), and the provision of free, extensible open-source software. It is scalable through the use of a modular test structure and efficient, up-to-date graph representation and manipulation and display algorithms. We illustrate the functionality and broad applicability of our approach using a hypothetical example and two real examples from two large well-established survey studies (the German National Educational Panel Study, NEPS, and the German Socio-Economic Panel, SOEP).