by Judith Curry
In my original essay on this topic October 2010, my short answer to this question was “I’m not sure.” My current thinking on this topic reframes the question in the context of fitness for purpose of climate models across a range of uses and applications.
This post will eventually get to a talk that I gave a few weeks ago, but first some context and background for the venue and audience.
Some of the presentions of the DOE Office of Science administrators should be of general interest:
- William Brinkman, Comments from the Director, Office of Science
- Sharlene Weatherwax, State of BER Report
- Adam Arkin, Knowledgebase Discussion
- Harriett Kung, Basic Energy Sciences
- Gary Geernaert, Climate and Environmental Sciences Division Update
- Susan Gregurick, Biological Systems Science Division Update
I have to say that I came away from this meeting very impressed with DOE science.
As a new member of BERAC, I was invited to give a one hour science presentation, my talk “What can we learn from climate models?” can be found [BERAC curry talk].
I’ve taken the text from my slides and put some narrative around it, and embedded some links to references and my previous posts for background.
What can we learn from climate models?
The Strategic plan for the DOE Climate Change Research Program (2009) provides the following overarching statements:
- “Priorities for climate change research have moved beyond determining if Earth’s climate is changing and if there is a human cause. The focus is now on understanding how quickly the climate is changing, where the key changes will occur, and what their impacts might be. Climate models are the best available tool for projecting likely climate changes, but they still contain some significant weaknesses.”
- “Projections of future climate change . . are the basis of national and international policies concerning greenhouse gas emission reductions.”
In the 2008 Report CCSP SAP 3.1 Climate Models: An Assessment of Strengths & Weaknesses, the path of climate modeling in the U.S. is articulated as:
The science of climate modeling has matured through finer spatial resolution, the inclusion of a greater number of physical processes, and comparison to a rapidly expanding array of observations.
With increasing computer power and observational understanding, future models will include both higher resolution and more processes.
Strategic planning regarding climate modeling is currently underway in the context of a NRC Study in Progress on A National Strategy for Advancing Climate Modeling. From what I can glean from meeting agendas and committee membership and other information provided about the project, the following topics seem to be the main focus:
- Increasing resolution and adding complexity
- Fully interactive earth system models (chemical, biogeochemical, land cryosphere); interface with human systems models
- Seamless prediction across timescales; data assimilation and initialization
- Downscaling for regional applications
- Communication of climate model results (including uncertainty, credibility); engagement with stakeholders: usefulness for decision making
So the U.S. is continuing on a path that builds upon the current climate modeling paradigm by adding more complex physical processes (e.g. carbon cycle) and increasing the model horizontal resolution.
I think there are some deep and important issues that aren’t receiving sufficient discussion and investigation. Some issues of concern:
- There is misguided confidence and “comfort” with the current GCMs and projected developments that are not consonant with understanding and best practices from other fields (e.g. nonlinear dynamics, engineering, regulatory science, computer science).
- GCMs may not be the most useful option to support many decision-making applications related to climate change
- Is the power and authority that is accumulating around GCMs, and the expended resources, deserved? Is it possibly detrimental, both to scientific progress and policy applications?
Why do climate scientists have confidence in climate models?
With regards to the question in the last bullet, this quote from Heymann (2010) poses the questions surrounding this issue in the following way:
The authority with which climate simulation and other fields of the atmospheric science towards the close of the twentieth century was furnished has raised new questions.
- How did it come about that extensive disputes about uncertainties did not compromise the authority and cultural impact of climate simulation?
- How did scientists manage to reach conceptual consensus in spite of persisting scientific gaps, imminent uncertainties and limited means of model validation?
- Why, to put the question differently, did scientists develop trust in their delicate model constructions?
I wrote a previous essay entitled The culture of building confidence in climate models. There are two basic approaches for building confidence in models:
- Formal approach: verification & validation; explicit analysis of model errors, including a detailed analysis of model interactions
- Informal approach: GCM modelers personal judgment as to the complexity and adequacy of the models
For GCMs, the informal approach has dominated, because model complexity limits the extent to which model processes, interactions and uncertainties can be understood and evaluated.
So why do climate modelers have confidence in their climate models?
Confidence derives from the theoretical physical basis of the models, and the ability of the models to reproduce the observed mean state and some elements of variability. Climate models inherit some measure of confidence from successes of numerical weather prediction.
‘Comfort’ relates to the sense that the model developers themselves have about their model, which includes the history of model development and the individuals that contributed, the reputations of the various modeling groups, and consistency of the simulated responses among different model modeling groups and different model versions. This kind of comfort is arguably model truthiness.
Knutti (2007) states: “So the best we can hope for is to demonstrate that the model does not violate our theoretical understanding of the system and that it is consistent with the available data within the observational uncertainty.”
Based upon the interactions with modelers from other fields, that I have engaged with mostly in the blogoshere (here and also ClimateAudit), there is ‘rising discomfort’ about climate models. The following concerns have been raised:
- Predictions can’t be rigorously evaluated for order of a century
- Insufficient exploration of model & simulation uncertainty
- Impenetrability of the model and formulation process; extremely large number of modeler degrees of freedom
- Lack of formal model verification & validation, which is the norm for engineering and regulatory science
- Circularity in arguments validating climate models against observations, owing to tuning & prescribed boundary conditions
- Concerns about fundamental lack of predictability in a complex nonlinear system characterized by spatio-temporal chaos with changing boundary conditions
- Concerns about the epistemology of models of open, complex systems
Verification and Validation
Verification and validation (V&V) is the process of checking and documenting that a model is built correctly and meets specifications and is an accurate representation of the real world from the perspective of the intended uses of the model. Previous posts on climate model V&V are [here and here].
Arguments for climate model V&V:
- V&V promotes and documents model robustness, which is important for both scientific and political reasons.
- Lack of V&V is major source of discomfort for engineers
Arguments against climate model V&V:
- V&V is overkill for a research tool; inhibits agile software development
- A tension exists between spending time and resources on V&V, versus improving the model.
V&V for climate models is gaining traction (more on this next week), mainly because climate models are now so complex that no one person can really wrap their head around the code, and much more documentation is needed for the users of climate models, especially community climate models such as NCAR.
So, what kind of V&V makes sense for climate models? The same V&V used for a NASA space launch isn’t necessarily appropriate for climate models. Several papers have been written on V&V for scientific models with recommended approaches:
From Roy and Oberkampf (2011) A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing:
“The framework is comprehensive in the sense that it treats both types of uncertainty (aleatory and epistemic), incorporates uncertainty due to the mathematical form of the model, and it provides a procedure for including estimates of numerical error in the predictive uncertainty.”
From Sargent (1998) Verification and validation of simulation models
“The different approaches to deciding model validity are presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; and a recommended procedure is presented.”
From Pope and Davies (2011) Testing and Evaluating Climate Models:
Range of techniques for validating atmosphere models given that the atmosphere is chaotic and incompletely observed:
- Simplified tests: against analytical or reference solutions
- Single column tests for physics components
- Dynamical core tests e.g. numerical convergence, aquaplanet simulations
- Realistic climate regimes e.g. compare observations, multiple models
- Double call tests to assess the impact of model changes
- Spin up tendencies to evaluate model biases
- tests of climate models operating as numerical weather prediction models
WGNE Climate Model Metrics Panel Gleckler et al. (2008):
Identify a limited set of basic climate model performance metrics based on observations
Climate Models: Fit for (what?) purpose
Verification of a model occurs in the context of the intended purpose of the model. Consider the following different applications of climate models:
- Numerical experiments to understand how the climate system works; sensitivity studies
- Simulation of present and past states to understand planetary energetics and other complex interactions
- Attribution of past climate variability and change
- Simulation of future states, from decades to centuries
- Prediction and attribution of extreme weather events
- Projections of future regional climate variation for use in model-based decision support systems
- Guidance for emissions reduction policies
- Projections of future risks of black swans & dragon kings
These applications are by no means exhaustive, but cover a range of territory from purely scientific applications to purely policy applications. Lets assess how well climate models that are General Circulation Models (GCMs) are fit for each of these applications
• Explore scientific understanding of the climate system
GCMs are certainly used to explore scientific understanding of the climate system. However a major challenge is that nearly all of the resources ($$ and personnel) are being spent primarily on IPCC production runs, with little time and $$ left over for innovations and scientific studies. It seems that the main beneficiaries of the IPCC production runs are the impacts area and surrounding sciences (e.g. ecosystems).
What is needed to further our scientific understanding of the climate system is a plurality of models with different structural forms and different levels of complexity in physical processes. This is difficult if $$ and personnel are focused on GCMs and IPCC production runs.
• Attribution of past climate variability and change
• Simulation of plausible future states
• Support for emissions reduction policies
These applications are at the science-policy interface and are the main applications for the IPCC. The challenges of using GCMs for these applications include:
- require adequate simulation of natural internal variability on multi-decadal to century time scales, which has not been achived
- solar forcing: better understanding of historical 20th century forcing; solar sensitivity studies conducted as part of attribution assessments; investigation of solar indirect effects; development of scenarios of 21st century solar forcing
For these applications, GCMs may be less effective than intermediate models (with lower resolution and a much greater ensemble size) in developing an understanding of climate sensitivity and attribution.
•Projections of future regional climate variation for use in model-based decision support systems
GCMs currently have little skill in simulating regional climate variations; it is unclear how much increased resolution will help. Dynamical & statistical downscaling adds little value, beyond model output statistics to account for local effects on surface variables.
GCM’s are probably not the best approach for supporting regional decision making. Of equal or greater usefulness for such applications are to
- improve understanding of historical/paleo regional climate dynamics and black swan events
- inclusion of a broader range of future scenarios of natural forcing changes (e.g. solar, volcanoes) and natural internal variability
- creative, regional approach to scenario development, including population and land use changes and alternative policy scenarios
• Prediction and attribution of extreme weather events
Challenges for GCMs:
- Climate models do not currently predict explicitly many types of extreme weather events (e.g. hurricanes, flash floods, tornadoes)
- Much higher resolution climate models with much larger ensemble sizes are necessary (but probably not sufficient)
- Greatest short term contribution would come from regional historical and paleo analyses of extreme events
- Climate dynamics approach, interpreting past extreme events in context of teleconnection and climate regimes, blocking patterns
• Projections of future risks of black swans & dragon kings
The possibility of future black swans and dragon kings are not only the most interesting from a scientific perspective, but arguably the most important from the policy perspective. GCMs are currently incapable of predicting emergent phenomena, e.g. abrupt climate change. It is not at all clear that GCMs will be able to generate counterintuitive, unexpected surprises. The current GCM’s have become ‘too stiff.’
Other possible approaches include synchronization in spatio-temporal chaos and other theoretical developments from nonlinear dynamics, network theory.
Usefulness for policy making
A conclusion that I draw from this analysis is that a completely general, all encompassing climate model that is accepted by all scientists and is fit for all purposes seems to be an idealistic fantasy. Instead, we need a plurality of climate models that are developed and utilized in different ways for different purposes. For decision support, the GCM centric approach may not be the best approach. Given the compromises made for multiple purposes, GCMs may not be the optimal solution for any of these purposes.
A very provocative paper by Shackley et al. addresses the following issues:
“In then addressing the question of how GCMs have come to occupy their dominant position, we argue that the development of global climate change science and global environmental ‘management’ frameworks occurs concurrently and in a mutually supportive fashion, so uniting GCMs and environmental policy developments in certain industrialised nations and international organisations. The more basic questions about what kinds of commitments to theories of knowledge underpin different models of ‘complexity’ as a normative principle of ‘good science’ are concealed in this mutual reinforcement. Additionally, a rather technocratic policy orientation to climate change may be supported by such science, even though it involves political choices which deserve to be more widely debated.”
So, are GCMs especially policy useful?
- potential for providing regional climate change scenarios (this potential is currently unrealized)
- perception that complexity = scientific credibility; sheer complexity and impenetrability, so not easily challenged by critics
- demands massive computing and personnel resources; creates dependency on a few centers and their experts
- slow to incorporate new scientific insights or understanding
- precludes conducting extensive sensitivity and uncertainty analyses
- precludes rapid exploration of different model assumptions and policy scenarios
- not user friendly for advisory scientists or policy makers
So lets consider the key policy issues, and assess how useful GCMs are.
CO2 mitigation policies:
- GCMs have a role to play, but large ensembles from lower order models with interactive carbon cycle may be the best solution for determining sensitivity
Regional climate change and extreme events:
- natural climate variability is at least as important as AGW, particularly on decadal time scales
- much to be learned from the climate dynamics of past and paleo regional climates and extreme events
- regional impact models can be forced by wide range of creatively produced scenarios
Understanding and representing uncertainty
- Uncertainty and ignorance assessment is a critical element for decision making strategies
- Parameter and parameterization uncertainty is inadequately assessed for individual models or multi model ensembles
- Ensemble size for initial condition uncertainty is far too small
- Uncertainty associated with model structural form is rarely assessed
Other approaches for assessing uncertainty:
- Stochastic models; stochastic parameterizations
- Monte Carlo techniques and sensitivity analysis
- Uncertainty management approaches such as NUSAP
In the climate community, ‘uncertainty’ is something that is regarded as something that needs to be communicated to decision makers. However, uncertainty quantification, assessment, and/or management is central to scientific understanding and to the overall accountability and usefulness of the models for decision making.
- A completely general, all encompassing climate model that is accepted by all scientists and is fit for all purposes seems to be an idealistic fantasy
- Increasing complexity (adding additional sub models) is less important for many applications than ensemble size
- We need a plurality of climate models that are developed and utilized in different ways for different purposes.
- For many issues of decision support, the GCM centric approach may not be the best approach
- Given the compromises made for multiple purposes, current GCMs may not be the optimal solution for any of these purposes
JC comment: I realize that this thread is pretty technical, including jargon, etc., and may not be easily understood by denizens who aren’t modelers or who haven’t been following along on the earlier climate modeling threads. Tamsin Edward’s new blog should help in this regard.