by Judith Curry
To solve these pressing problems, there needs to be much better recognition of the importance of probability models in climate science and a more integrated view of climate modelling whereby climate prediction involves the fusion of numerical climate models and statistical models. – Stephenson et al.
Over the last two weeks, there have been some interesting exchanges in the blogosphere on the topic of interpreting an ensemble of models. See this recent post at Climate Etc. for background: How should we interpret an ensemble of models? Part I: Weather Models.
Excerpts from the IPCC AR4, section 10.5.4:
Uncertainty in the response of an AOGCM arises from the effects of internal variability, which can be sampled in isolation by creating ensembles of simulations of a single model using alternative initial conditions, and from modelling uncertainties, which arise from errors introduced by the discretization of the equations of motion on a finite resolution grid, and the parametrization of sub-grid scale processes (radiative transfer, cloud formation, convection, etc).
While ensemble projections carried out to date give a wide range of responses, they do not sample all possible sources of modelling uncertainty. More generally, the set of available models may share fundamental inadequacies, the effects of which cannot be quantified.
Non-informative prior distributions for regional temperature and precipitation are updated using observations and results from AOGCM ensembles to produce probability distributions of future changes. Key assumptions are that each model and the observations differ randomly and independently from the true climate.
Recent IPCC Report
The IPCC presents a more mature perspective on interpretation on the multi-model ensemble in Report from the IPCC Expert Meeting on Assessing and Combining Multi-Model Climate Projections. Excerpts:
Climate model results provide the basis for projections of future climate change. Previous assessment reports included model evaluation but avoided weighting or ranking models. Projections and uncertainties were based mostly on a ‘one model, one vote’ approach, despite the fact that models differed in terms of resolution, processes included, forcings and agreement with observations.
The reliability of projections might be improved if models are weighted according to some measure of skill and if their interdependencies are taken into account, or if only subsets of models are considered. Since there is little opportunity to verify climate forecasts on timescales of decades to centuries (except for a realization of the 20th century), the skill or performance of the models needs to be defined.
Statistical frameworks in published methods using ensembles to quantify uncertainty may assume (perhaps implicitly):
a. that each ensemble member is sampled from a distribution centered around the truth (‘truth plus error’ view). In this case, perfect independent models in an ensemble would be random draws from a distribution centered on observations.
Alternatively, a method may assume:
b. that each of the members is considered to be ‘exchangeable’ with the other members and with the real system . In this case, observations are viewed as a single random draw from an imagined distribution of the space of all possible but equally credible climate models and all possible outcomes ofEarth’s chaotic processes. A ‘perfect’ independent model in this case is also a random draw from the same distribution, and so is ‘indistinguishable’ from the observations in the statistical model.
With the assumption of statistical model (a), uncertainties in predictions should tend to zero as more models are included, whereas with (b), we anticipate uncertainties to converge to a value related to the size of the distribution of all outcomes. While both approaches are common in published literature, the relationship between the method of ensemble generation and statistical model is rarely explicitly stated.
When analyzing results from multi-model ensembles, the following points should be taken into account:
• Consideration needs to be given to cases where the number of ensemble members or simulations differs between contributing models. The single model’s ensemble size should not inappropriately determine the weight given to any individual model in the multi-model ensemble. In some cases ensemble members may need to be averaged first before combining different models, while in other cases only one member may be used for each model.
• Ensemble members may not represent estimates of the climate system behaviour (trajectory) entirely independent of one another. This is likely true of members that simply represent different versions of the same model or use the same initial conditions. But even different models may share components and choices of parameterizations of processes and may have been calibrated using the same data sets. There is currently no ‘best practice’ approach to the characterization and combination of inter-dependent ensemble members, in fact there is no straightforward or unique way to characterize model dependence.
Statistical problems in the probabilistic prediction of climate change
David Stephenson, Matthew Collins, Jonathan Rougier, Richard Chandler
Abstract. Future climate change projections are constructed from simulated numerical output from a small set of global climate models—samples of opportunity known as multi-model ensembles. Climate models do not produce probabilities, nor are they perfect representations of the real climate, and there are complex inter-relationships due to shared model features. This creates interesting statistical challenges for making inference about the real climate. These issues were the focus of discussions at an Isaac Newton Institute workshop on probabilistic prediction of climate change held at the University of Exeter on 20–23 September 2010. This article presents a summary of the issues discussed between the statisticians, mathematicians, and climate scientists present at the workshop. In addition, we also report the discussion that took place on how to define the concept of climate.
Despite their increasing complexity and seductive realism, it is important to remember that climate models are not the real world. Climate models are numerical approximations to fluid dynamical equations forced by parameterisations of physical and unresolved sub-grid scale processes. Climate models are inadequate in a rich diversity of ways, but it is the hope that these physically motivated models can still inform us about various aspects of future observable climate. A major challenge is how we should use climate models to construct credible probabilistic forecasts of future climate. Because climate models do not themselves produce probabilities, an additional level of explicit probabilistic modelling is necessary to achieve this.
To make reliable probabilistic predictions of future climate, we need probability models based on credible and defensible assumptions. If climate is considered to be a probability distribution of all observable features of weather, then one needs well-specified probability models to define climate from the single realisation of weather provided by nature. In an increasingly non-stationary climate forced by accelerating rates of global warming, the time average over a fixed period provides only an incomplete description perhaps better suited to a more stationary pre-industrial world.
We also need probability models (or more generally a statistical framework) to make inference about future observable climate based on ensembles of simulated data from numerical climate models. Such frameworks involve making simplifying, yet transparent and defensible, assumptions about ensembles of climate models and their relationship to the real world. These frameworks not only have to model dependency between different climate models, but also need to account for model discrepancies (biases) and how these might evolve in the future. Development and testing of such frameworks are a pressing interdisciplinary challenge in statistical and climate science. Furthermore, it is likely that good statistical frameworks might be useful in the sequential design of climate model experiments, which take increasingly large amounts of time on the world’s fastest supercomputers.
To solve these pressing problems, there needs to be much better recognition of the importance of probability models in climate science and a more integrated view of climate modelling whereby climate prediction involves the fusion of numerical climate models and statistical models. This challenge will require greater collaboration and understanding between climate scientists and statisticians. Details of project areas are given in the final programme report (http://www.newton.ac.uk/reports/1011/clpdraft.pdf).
I wrote about this issue in my paper Climate Science and the Uncertainty Monster. Excerpts:
Given the inadequacies of current climate models, how should we interpret the multi-model ensemble simulations of the 21st century climate used in the IPCC assessment reports? This ensemble-of-opportunity is comprised of models with generally similar structures but different parameter choices and calibration histories. McWilliams (2007) and Parker (2010) argue that current climate model ensembles are not designed to sample representational uncertainty in a thorough or strategic way. Stainforth et al. (2007) argue that model inadequacy and an inadequate number of simulations in the ensemble preclude producing meaningful probability density functions (PDFs) from the frequency of model outcomes of future climate. Nevertheless, as summarized by Parker (2010), it is becoming increasingly common for results from individual multi-model and perturbed-physics simulations to be transformed into probabilistic projections of future climate, using Bayesian and other techniques. Parker argues that the reliability of these probabilistic projections is unknown, and in many cases they lack robustness. Knutti et al. (2008) argues that the real challenge lies more in how to interpret the PDFs rather whether they should be constructed in the first place. Stainforth et al. (2007) warns against over interpreting current model results since they could be contradicted by the next generation of models, undermining the credibility of the new generation of model simulations.
Stainforth et al. (2007) emphasize that models can provide useful insights without being able to provide probabilities, by providing a lower bound on the maximum range of uncertainty and a range of possibilities to be considered. Kandlikar et al. (2005) argue that when sources of uncertainty are well understood, it can be appropriate to convey uncertainty via full PDFs, but in other cases it will be more appropriate to offer only a range in which one expects the value of a predictive variable to fall with some specified probability, or to indicate the expected sign of a change without assigning a magnitude. They argue that uncertainty should be expressed using the most precise means that can be justified, but unjustified more precise means should not be used.
And from my paper Reasoning About Climate Uncertainty:
Following Walker et al. (2003), statistical uncertainty is distinguished from scenario uncertainty, whereby scenario uncertainty implies that it is not possible to formulate the probability of occurrence particular outcomes. A scenario is a plausible but unverifiable description of how the system and/or its driving forces may develop in the future. Scenarios may be regarded as a range of discrete possibilities with no a priori allocation of likelihood. Whereas the IPCC reserves the term “scenario” for emissions scenarios, Betz (2009) argues for the logical necessity of considering each climate model simulation as a modal statement of possibility, stating what is possibly true about the future climate system, which is consistent with scenario uncertainty.
Stainforth et al. (2007) argue that model inadequacy and an insufficient number of simulations in the ensemble preclude producing meaningful probability distributions from the frequency of model outcomes of future climate. Stainforth et al. state: “[G]iven nonlinear models with large systematic errors under current conditions, no connection has been even remotely established for relating the distribution of model states under altered conditions to decision-relevant probability distributions. . . Furthermore, they are liable to be misleading because the conclusions, usually in the form of PDFs, imply much greater confidence than the underlying assumptions justify.”
Stainforth et al. make a statement that is equivalent to Betz’s modal statement of possibility: “Each model run is of value as it presents a ‘what if’ scenario from which we may learn about the model or the Earth system.” Insufficiently large initial condition ensembles combined with model parameter and structural uncertainty preclude forming a PDF from climate model simulations that has much meaning in terms of establishing a mean value or confidence intervals. In the presence of scenario uncertainty, which characterizes climate model simulations, attempts to produce a PDF for climate sensitivity are arguably misguided and misleading.
JC summary: Interpretation of an ensemble of models remains an open question, and it is good to see this issue being discussed by mathematicians and statisticians. In the meantime, take the multi-model ensemble mean with a grain of salt.
Moderation note: this is a technical thread, comments will be moderated for relevance.