Site icon Climate Etc.

What can we learn from climate models? Part II

curryja

by Judith Curry

In my original essay on this topic October 2010, my short answer to this question was “I’m not sure.”   My current thinking on this topic reframes the question in the context of fitness for purpose of climate models across a range of uses and applications.

This post will eventually get to a talk that I gave a few weeks ago, but first some context and background for the venue and audience.

DOE BERAC

I gave a talk on this topic several weeks ago at the DOE BERAC    meeting.  The agenda and presentations of the meeting are [here].

Some of the presentions of the DOE Office of Science administrators should be of general interest:

I have to say that I came away from this meeting very impressed with DOE science.

JC’s presentation

As a new member of BERAC, I was invited to give a one hour science presentation, my talk “What can we learn from climate models?” can be found [BERAC curry talk].

I’ve taken the text from my slides and put some narrative around it, and embedded some links to references and my previous posts for background.

What can we learn from climate models? 

The Strategic plan for the DOE Climate Change Research Program (2009) provides the following overarching statements:

In the 2008 Report CCSP SAP 3.1 Climate Models: An Assessment of Strengths & Weaknesses, the path of climate modeling in the U.S. is articulated as:

The science of climate modeling has matured through finer spatial resolution, the inclusion of a greater number of physical processes, and comparison to a rapidly expanding array of observations.

With increasing computer power and observational understanding, future models will include both higher resolution and more processes.

Strategic planning regarding climate modeling is currently underway in the context of a NRC Study in Progress on A National Strategy for Advancing Climate Modeling.   From what I can glean from meeting agendas and committee membership and other information provided about the project, the following topics seem to be the main focus:

So the U.S. is continuing on a path that builds upon the current climate modeling paradigm by adding more complex physical processes (e.g. carbon cycle) and increasing the model horizontal resolution.

I think there are some deep and important issues that aren’t receiving sufficient discussion and investigation.  Some issues of concern:

Why do climate scientists have confidence in climate models?

With regards to the question in the last bullet, this quote from Heymann (2010) poses the questions surrounding this issue in the following way:

The authority with which climate simulation and other fields of the atmospheric science towards the close of the twentieth century was furnished has raised new questions.

I wrote a previous essay entitled The culture of building confidence in climate models.   There are two basic approaches for building confidence in models:

For GCMs, the informal approach has dominated, because model complexity limits the extent to which model processes, interactions and uncertainties can be understood and evaluated.

So why do climate modelers have confidence in their climate models?

Confidence derives from the theoretical physical basis of the models, and the ability of the models to reproduce the observed mean state and some elements of variability.  Climate models inherit some measure of confidence from successes of numerical weather prediction.

‘Comfort’ relates to the sense that the model developers themselves have about their model, which includes the history of model development and the individuals that contributed, the reputations of the various modeling groups, and consistency of the simulated responses among different model modeling groups and different model versions.  This kind of comfort is arguably model truthiness.

Knutti (2007) states:  “So the best we can hope for is to demonstrate that the model does not violate our theoretical understanding of the system and that it is consistent with the available data within the observational uncertainty.”

Based upon the interactions with modelers from other fields, that I have engaged with mostly in the blogoshere (here and also ClimateAudit), there is ‘rising discomfort’ about climate models.  The following concerns have been raised:

Verification and Validation

Verification and validation (V&V) is the process of checking and documenting that a model is built correctly and meets specifications and is an accurate representation of the real world from the perspective of the intended uses of the model. Previous posts on climate model V&V are [here and here].

Arguments for climate model V&V:

Arguments against climate model V&V:

V&V for climate models is gaining traction (more on this next week), mainly because climate models are now so complex that no one person can really wrap their head around the code, and much more documentation is needed for the users of climate models, especially community climate models such as NCAR.

So, what kind of V&V makes sense for climate models?  The same V&V used for a NASA space launch isn’t necessarily appropriate for climate models.  Several papers have been written on V&V for scientific models with recommended approaches:

From Roy and Oberkampf (2011) A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing:

“The framework is comprehensive in the sense that it treats both types of uncertainty (aleatory and epistemic), incorporates uncertainty due to the mathematical form of the model, and it provides a procedure for including estimates of numerical error in the predictive uncertainty.”

From Sargent (1998) Verification and validation of simulation models

“The different approaches to deciding model validity are presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; and a recommended procedure is presented.”

From Pope and Davies (2011) Testing and Evaluating Climate Models:

Range of techniques for validating atmosphere models given that the atmosphere is chaotic and incompletely observed:

WGNE Climate Model Metrics Panel Gleckler et al. (2008):

Identify a limited set of basic climate model performance metrics based on observations

Climate Models: Fit for (what?) purpose  

Verification of a model occurs in the context of the intended purpose of the model.  Consider the following different applications of climate models:

These applications are by no means exhaustive, but cover a range of territory from purely scientific applications to purely policy applications.  Lets assess how well climate models that are General Circulation Models (GCMs) are fit for each of these applications

• Explore scientific understanding of the climate system

GCMs are certainly used to explore scientific understanding of the climate system.  However a major challenge is that nearly all of the resources ($$ and personnel) are being spent primarily on IPCC production runs, with little time and $$ left over for innovations and scientific studies.  It seems that the main beneficiaries of the IPCC production runs are the impacts area and surrounding sciences (e.g. ecosystems).

What is needed to further our scientific understanding of the climate system is a plurality of models with different structural forms and different  levels of complexity in physical processes.  This is difficult if $$ and personnel are focused on GCMs and IPCC production runs.

•  Attribution of past climate variability and change

•  Simulation of plausible future states

•  Support for emissions reduction policies

These applications are at the science-policy interface and are the main applications for the IPCC.  The challenges of using GCMs for these applications include:

For these applications, GCMs may be less effective than intermediate models (with lower resolution and a much greater ensemble size) in developing an understanding of climate sensitivity and attribution.

•Projections of future regional climate variation for use in model-based decision support systems

GCMs currently have little skill in simulating regional climate variations; it is unclear how much increased resolution will help.  Dynamical & statistical downscaling adds little value, beyond model output statistics to account for local effects on surface variables.

GCM’s are probably not the best approach for supporting regional decision making.  Of equal or greater usefulness for such applications are to

•  Prediction and attribution of extreme weather events

Challenges for GCMs:

Other approaches:

• Projections of future risks of black swans & dragon kings

The possibility of future black swans and dragon kings are not only the most interesting from a scientific perspective, but arguably the most important from the policy perspective.  GCMs are currently incapable of predicting emergent phenomena, e.g. abrupt climate change.  It is not at all clear that GCMs will be able to generate counterintuitive, unexpected surprises.  The current GCM’s have become ‘too stiff.’

Other possible approaches include synchronization in spatio-temporal chaos and other theoretical developments from nonlinear dynamics, network theory.

Usefulness for policy making

A conclusion that I draw from this analysis is that a completely general, all encompassing climate model that is accepted by all scientists and is fit for all purposes seems to be an idealistic fantasy.  Instead, we need a plurality of climate models that are developed and utilized in different ways for different purposes.  For decision support, the GCM centric approach may not be the best approach. Given the compromises made for multiple purposes, GCMs may not be the optimal solution for any of these purposes.

A very provocative paper by Shackley et al. addresses the following issues:

“In then addressing the question of how GCMs have come to occupy their dominant position, we argue that the development of global climate change science and global environmental ‘management’ frameworks occurs concurrently and in a mutually supportive fashion, so uniting GCMs and environmental policy developments in certain industrialised nations and international organisations. The more basic questions about what kinds of commitments to theories of knowledge underpin different models of ‘complexity’ as a normative principle of ‘good science’ are concealed in this mutual reinforcement. Additionally, a rather technocratic policy orientation to climate change may be supported by such science, even though it involves political choices which deserve to be more widely debated.”

So, are GCMs especially policy useful?

Main advantages:

Disadvantages:

So lets consider the key policy issues, and assess how useful GCMs are.

CO2 mitigation policies:

Regional climate change and extreme events:

Understanding and representing uncertainty

The challenges:

Other approaches for assessing uncertainty:

In the climate community, ‘uncertainty’ is something that is regarded as something that needs to be communicated to decision makers.  However, uncertainty quantification, assessment, and/or management is central to scientific understanding and to the overall accountability and usefulness of the models for decision making.

Summary

JC comment:  I realize that this thread is pretty technical, including jargon, etc., and may not be easily understood by denizens who aren’t modelers or who haven’t been following along on the earlier climate modeling threads.  Tamsin Edward’s new blog should help in this regard.

Exit mobile version