A fundamental problem with the mathematics of models ensures we’ll always get unreliable predictions
From my article on the Scientific American Website, posted Oct. 26, 2011 (A companion piece to my feature article on economic models in the Nov. 2011 print edition, posted just below)
simpler models that perfectly reflected reality? And what if we had perfect financial data to plug into them?
From my article on the Scientific American Website, posted Oct. 26, 2011 (A companion piece to my feature article on economic models in the Nov. 2011 print edition, posted just below)
When it comes to assigning blame for the current economic doldrums, the quants who build the complicated financial risk models, and the traders who rely on them, deserve their share of the blame. But what if there were a way to come up with
simpler models that perfectly reflected reality? And what if we had perfect financial data to plug into them?
Incredibly, even under those utterly unrealizable conditions, we’d still get bad predictions from models.
The reason is that current methods used to “calibrate” models often render them inaccurate….read more
David H. Freedman 
I would love to see the original study by Jonathan Carter. Can you point us in he right direction?
Mr Freedman, your description of Mr Carter's methods as "current" is inaccurate. In the field of Machine Learning, which among other things is useful in financial modeling, it is a standard method to save some of the historical data to use for testing after the calibration step. Indeed, an even better way is to successively apply this by choosing testing subsets randomly over a series of calibrations and trials. Yes this is computationally expensive. Yet I suspect that the financial modelers may be referring to this methodical rather than ad-hoc successive recalibration. Furthermore, your analysis could be taken by climate skeptics to indicate that their predictions are false. In the matter of Hurricanes as well, although imperfect, predictions have improved greatly over the last few decades and have helped to save lives.