The Bayesian ``evidence'' approximation, which is closely related to generalized maximum likelihood, has recently been employed to determine the noise and weight-penalty terms for training neural nets. This paper shows that it is far simpler to perform the exact calculation than it is to set up the evidence approximation. Moreover, unlike that approximation, the exact result does not have to be re-calculated for every new data set. Nor does it require the running of complex numerical computer code (the exact result is closed form). In addition, it turns out that for neural nets, the evidence procedure's MAP estimate is {\it in toto} approximation error. Another advantage of the exact analysis is that it does not lead to incorrect intuition, like the claim that one can ``evaluate different priors in light of the data.'' This paper ends by discussing sufficiency conditions for the evidence approximation to hold, along with the implications of those conditions. Although couched in terms of neural nets, the anlaysis of this paper holds for any Bayesian interpolation problem.
