A Field Evaluation of Natural Language for Data Retrieval

Although a large number of natural language database interfaceshave been developed, there have been few empirical studies of theirpractical usefulness. This paper presents the design and results of afield evaluation of a natural language system - NLS - used for dataretrieval .A balanced, multifactorial design comparing NLS with a referenceretrieval language, SQL, is described. The data are analyzed on twolevels: work task (n=87) and query (n=1081). SQL performed betterthan NLS on a variety of measures, but NLS required less effort touse. Subjects performed much poorer than expected based on theresults of laboratory studies. This finding is attributed to thecomplexity of the field setting and to optimism in grading laboratoryexperiments.The methodology developed for studying computer languages in realwork settings was successful in consistently measuring differences intreatments over a variety of conditions