Gauss' 1809 discussion of least squares, which can be viewed as the beginning of mathematical statistics, is reviewed. The general consensus seems to be that Gauss' arguments are at fault, but we show that his reasoning is in fact correct, given his self-imposed restrictions, and persuasive...