I’m starting a new thread for this topic, since interest in the previous thread has re-generated owing to this AGU abstract by Steve Easterbrook, entitled Do Over or Make Do? Climate Models as a Software Development Challenge.
Dan Hughes writes:
My reading of these posts at Professor Easterbrook’s site seems to me to be saying that all is sweetness and light within the climate science software development community:
Verification and Validation of Earth System Models
Do Climate Models need Independent Verification and Validation?
Should science models be separate from production models?
The AGU abstract, on the other hand, seems to indicate that there are significant problems within that community:
The testing processes are effective at removing software errors prior to release, but the code is hard to understand and hard to change. Software errors and model configuration problems are common during model development, and appear to have a serious impact on scientific productivity. These problems have grown dramatically in recent years with the growth in size and complexity of earth system models. Much of the success in obtaining valid simulations from the models depends on the scientists developing their own code, experimenting with alternatives, running frequent full system tests, and exploring patterns in the results. Blind application of generic software engineering processes is unlikely to work well.
My direct experience with application of recent engineering and scientific software development methodologies has shown that many of the problems noted above can easily be avoided. Documentation of the specifications, for one example, provide an excellent starting point for avoiding these problems. Coding guidelines including careful specifications for interfaces is another example.
This last sentence in the above quote:
Blind application of generic software engineering processes is unlikely to work well.
is a strawman in that ‘generic software engineering processes’ are not the only processes employed for engineering and scientific software.
The following statement from this post The difference between Verification and Validation:
For climate models, the definitions that focus on specifications don’t make much sense, because there are no detailed specifications of climate models (nor can there be – they’re built by iterative refinement like agile software development).
is especially troubling.
Professor Easterbrook has also cited the infamous paper by Oreskes Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. A paper that has been refuted several times within the engineering and scientific software development community. A paper that says, in effect, that we might as well not even think about starting development of software for complex natural physical phenomena and processes. And yet at the same time, some such software is readily accepted, climate science software for example, while others are simply rejected based solely on that paper.