by Judith Curry
Short answer: I’m not sure.
I spent the 1990’s attempting to exorcise the climate model uncertainty monster: I thought the answer to improving climate models lay in improving parameterizations of physical processes such as clouds and sea ice (following Randall and Wielicki), combined with increasing model resolution. Circa 2002, my thinking became heavily influenced by Leonard Smith, who introduced me to the complexity and inadequacies of climate models and also ways of extracting useful information from weather and climate model simulations. I began thinking about climate model uncertainty and how it was (or rather, wasn’t) characterized and accounted for in assessments such as the IPCC. A seminal event in the evolution of my thinking on this subject was a challenge I received at Climate Audit to host a thread related to climate models, which increased my understanding of why scientists and engineers from other fields find climate models unconvincing. The Royal Society Workshop on Handling Uncertainty in Science motivated me to become a serious monster detective on the topic of climate models. So far, it seems that the biggest climate model uncertainty monsters are spawned by the complexity monster.
This post provides my perspective on some of the challenges and uncertainties associated with climate models and their applications. I am by no means a major player in the climate modeling community; my expertise and experience is on the topic of physical process parameterization, challenging climate models with observations, and extracting useful information from climate model simulations. My perspective is not in the mainstream among the climate community (see this assessment). But I think there are some deep and important issues that aren’t receiving sufficient discussion and investigation, particularly given the high levels of confidence that the IPCC gives to conclusions derived from climate models regarding the attribution of 20th century climate change and climate sensitivity.
I don’t think we can answer the question of what we can learn from climate models without deep consideration of the subject by experts in dynamical systems and nonlinear dynamics, artificial intelligence, mechanical engineers, philosophy of science, and probably others. I look forward to such perspectives from the Climate Etc. community.
This discussion is somewhat esoteric, and written for an audience that has some familiarity with computer simulation models. For background on global climate models, I suggest the following references:
For the latest thinking on the topic, I recommend:
- Series on Mathematical and Statistical Approaches to Climate Modeling hosted by the Isaac Newton Institute for Mathematical Sciences
- Special issue in Studies in History and Philosophy of Modern Physics
- Special issue in the Journal of Computational Physics on Predicting Weather, Climate and Extreme Events
- WIRE’s Interdisciplinary Reviews on Climate Modeling
Modeling the complex global climate system
The target of global climate models is the Earth’s climate system, consisting of the physical (and increasingly, chemical) components of the atmosphere, ocean, land surface, and cryosphere (sea ice and glaciers). When simulating climate, the objective is to correctly simulate the spatial variation of climate conditions in some average sense. There is a hierarchy of different climate models, ranging from simple energy balance models to the very complex global circulation models such as those used by the IPCC. It is this latter category of climate models that provides the focus for this analysis. Such models attempt to account for as many processes as possible to simulate the detailed evolution of the atmosphere, ocean, cryosphere, and land system, at a horizontal resolution that is typically 100s of km.
Climate model complexity arises from the nonlinearity of the equations, high dimensionality (millions of degrees of freedom), and the linking of multiple subsystems. Solution of the complex system of equations that comprise a climate model is made possible by the computer. Computer simulations of the climate system can be used to represent aspects of climate that are extremely difficult to observe, experiment with theories in a new way by enabling hitherto infeasible calculations, understand a system of equations that would otherwise be impenetrable, and explore the system to identify unexpected outcomes.
At the heart of climate model complexity lies the nonlinear dynamics of the atmosphere and oceans, which is described by the Navier-Stokes equations. The solution of Navier-Stokes equations is one of the most vexing problems in all of mathematics: the Clay Mathematics Institute has declared this to be one of the top 7 problems in all of mathematics and is offering a $1M prize for its solution (Millenium Prize Problems).
Chaos and pandemonium
Weather has been characterized as being in state of deterministic chaos, owing to its sensitivity to initial conditions. The source of the chaos is nonlinearities in the Navier-Stokes equations. A consequence of sensitivity to initial conditions is that beyond a certain time the system will no longer be predictable; for weather this predictability time scale is weeks.
One way of interpreting climate is as the distribution of states on some ‘attractor’ of weather. Annan and Connelly make the case that weather chaos averages out over time in climate simulations, and so does not preclude predictability in climate simulations. However, climate model simulations are also sensitive to initial conditions (even in an average sense). Coupling of a nonlinear, chaotic atmospheric model to a nonlinear, chaotic ocean model gives rise to something much more complex than the deterministic chaos of the weather model, particularly under conditions of transient forcing.
Coupled atmosphere/ocean modes of internal variability arise on timescales of weeks (e.g. the Madden Julian Oscillation), years (e.g. ENSO), decades (e.g. NAO, AMO, PDO), centuries and millenia (global thermohaline circulation), plus abrupt climate change. These coupled modes give rise to bifurcation, instability and chaos. How to characterize such phenomena arising from transient forcing of the coupled atmosphere/ocean system defies classification by current theories of nonlinear dynamical systems, where definitions of chaos and attractor cannot be invoked in situations involving transient changes of parameter values. Stainforth et al. (2007) refer to this situation as “pandemonium.” I’m not sure what this means in the context of nonlinear dynamics, but pandemonium seems like a very apt word to describe this situation.
Confidence in climate models
Particularly for a model of a complex system, the notion of a correct or incorrect model is not well defined, and falsification is not a relevant issue. The relevant issue is how well the model reproduces reality, i.e. whether the model “works” and is fit for its intended purpose.
In the absence of model verification or falsification, Stainforth et al. (2007) describes the challenges of building confidence in climate models. Owing to the long time scales present, particularly in the ocean component, there is no possibility of a true cycle of model improvement and confirmation, particularly since the life cycle of an individual model version (order of a few years) is substantially less than the simulation period (order of centuries). Model projections of future climate states relate to a state of the system that has not been previously experienced. Hence it is impossible to calibrate the model for the future climate state or confirm the usefulness of the forecast.
Confidence in climate models relies on tests of internal consistency and physical understanding of the processes involved, plus comparison of simulations of past climate states with observations. Issues surrounding climate model verification and validation and challenges of model evaluating climate model simulations with observations will be the subject of a future post. Failure to reproduce past observations highlights model inadequacies and motivates model improvement, but success in reproducing past states provides only a limited kind of confidence in simulation of future states.
Climate model imperfections
This discussion of model imperfections follows Stainforth et al. (2007). Model imperfection is a general term that describes our limited ability to simulate climate and is categorized here in terms of model inadequacy and model uncertainty. Model inadequacy reflects our limited understanding of the climate system, inadequacies of numerical solutions employed in computer models, and the fact that no model can be exactly isomorphic to the actual system. Model uncertainty is associated with uncertainty in model parameters and subgrid parameterizations, and also uncertainty in in initial conditions. As such, model uncertainty is a combination of epistemic and ontic uncertainties.
Model inadequacy
Model structural form is the conceptual modeling of the physical system (e.g. dynamical equations, initial and boundary conditions). In addition to insufficient understanding of the system, uncertainties are introduced as a pragmatic compromise between numerical stability and fidelity to the underlying theories, credibility of results, and available computational resources. One issue in the structural form of a complex system is the selection of subsystems to include, e.g. whether or not to include stratospheric chemistry and ice sheet dynamics.
Issues related to the structural form of the atmospheric dynamical core are of paramount importance in climate models. Staniforth and Wood 2008 give a lucid overview of the construction of the atmospheric dynamical core in global weather and climate models. The dynamical core solves the governing for fluid dynamics and thermodynamics on resolved scales, and parameterizations represent subgrid scale processes and other processes not included in the dynamical core (e.g. radiative transfer).
Thuburn (2010) (oops 2008) articulates the challenges for a dynamical core to possess discrete analogues of the conservation properties of the continuous equations. Because a numerical model can have only a small finite number of analogous conservative properties, some choice must be made in the design of a numerical scheme as to which conservation properties are most desirable, e.g. mass, momentum and angular momentum, tracer variance and potential enstrophy, energy, potential vorticity. Other aspects of a numerical scheme may be incompatible with certain conservation properties, such as good wave dispersion properties and computational efficiency. The relative importance of different conservation properties depends on the timescales for their corresponding physical sources and sinks. This particular assessment of Thuburn caught my attention:
“Moist processes are strongly nonlinear and are likely to be particularly sensitive to imperfections in conservation of water. Thus there is a very strong argument for requiring a dynamical core to conserve mass of air, water, and long-lived tracers, particularly for climate simulation. Currently most if not all atmospheric models fail to make proper allowance for the change in mass of an air parcel when water vapour condenses and precipitates out. . . However, the approximation will not lead to a systematic long term drift in the atmospheric mass in climate simulations provided there is no long term drift in the mean water content of the atmosphere.”
Given the importance of water vapor and cloud feedbacks in the simulated climate sensitivity, Thuburn’s analysis sounds a warning bell that that mass conservation schemes that seem adequate for numerical weather prediction may be the source of substantial error when applied in climate models.
Characterizing model uncertainty
Model uncertainty arises from uncertainty in model structure, model parameters and parameterizations, and initial conditions. Uncertainties in parameter values include uncertain constants and other parameters, subgridscale parameterizations (e.g. boundary layer turbulence, cloud microphysics), and ad hoc modeling to compensate for the absence of neglected factors.
Calibration is necessary to address parameters that are unknown or inapplicable at the model resolution, and also in the linking of submodels. As the complexity, dimensionality, and modularity of a model grows, model calibration becomes unavoidable and an increasingly important issue. Model calibration is accomplished by kludging (or tuning), which is “an inelegant, botched together piece of program; something functional but somehow messy and unsatisfying, a piece of program or machinery which works up to a point” (cited by Winsberg and Lenhard 2010; draft). A kludge required in one model may not be required in another model that has greater structural adequacy or higher resolution.
Continual ad hoc adjustments of the model (calibration) can mask underlying deficiencies in model structural form; Occam’s razor presupposes that the model least dependent on continual ad hoc modification is to be preferred. It should be noted that in a climate model with millions of degrees of freedom, it is impossible to tune the model to provide a correct 4D solution of many variables. This post at realclimate.org addresses some of these issues in more detail.
Ensemble methods are a brute force approach to representing model uncertainty. Rather than conducting a single simulation, multiple simulations are run that sample some combination of different initial conditions, model parameters and parameterizations, and model structural forms. While the ensemble method used in weather and climate predictions is inspired by Monte Carlo approaches, application of a traditional Monte Carlo approach far outstrips computational capacity owing to the very large number of possible combinations required to fully represent climate model uncertainty. A high level of model complexity and high model resolution precludes large ensembles. Stochastic parameterization methods are being introduced (see this presentation by Tim Palmer) to characterize parameter and parameterization uncertainty, reducing the need to conduct ensemble simulations to explore parameter uncertainty.
Fit for purpose?
George Box has famously stated that: “All models are wrong, but some are useful.” Some of the purposes that climate scientists use climate models are for:
- Hypothesis testing, numerical experiments, to understand how the climate system works, including its sensitivity to altered forcing.
- Simulation of present and past states to understand planetary energetics and other complex interactions
- Atmosphere and ocean state reanalysis using 4D data assimilation into climate models
- Attribution of past climate variability and change
- Attribution of extreme weather events
- Simulation of plausible and dynamically consistent future states, on timescales ranging from months to decades to centuries
- Projections of future regional climate variation for use in model-based decision support systems
- Projections of future risks of extreme weather events
The same climate model configurations are used for this plurality of applications. Choices about model structural form, level of detail in the physical parameterizations, parameter choices, horizontal resolutions, and experimental design (e.g. ensemble configuration and size) have to be made within the constraints of available computer resources. There is continual tension among climate modeling groups about allocation of computer resources to higher model resolutions versus more complex physical parameterizations versus simulation of large ensembles. One alternative is to devote all of the resources to a single best model with increased resolution, improved model parameterizations, and greater model complexity. Another alternative is to use simpler models and conduct a large ensemble simulation with varying initial conditions, varying parameter values and parameterizations, and different model structures. Different applications would be optimized by different choices.
Wendy Parker nails it in this statement:
Lloyd (2009) contends that climate models are confirmed by various instances of fit between their output and observational data. The present paper argues that what these instances of fit might confirm are not climate models themselves, but rather hypotheses about the adequacy of climate models for particular purposes. This required shift in thinking—from confirming climate models to confirming their adequacy-for-purpose—may sound trivial, but it is shown to complicate the evaluation of climate models considerably, both in principle and in practice.
My personal opinion is that at this stage of the game, #1 is paramount. This implies the need for a range of model structural forms and parameter/parameterization choices in the context of a large ensemble of simulations. The fitness for task #1 of current climate models is suboptimal owing to compromises that have been made to optimize for the full range of applications. The design of climate models and their experiments needs renewed consideration for #1 in light of model inadequacy and uncertainties. Once we learn more about the climate models and how to design climate model experiments, we can possibly have a realistic chance of effectively tackling regional climate variation and extreme weather events. However, there may be no regional predictability owing to the ontic uncertainty associated with internal modes of natural climate variability. The challenge of predicting emergent extreme weather events seems overwhelming to me, I have no idea whether this is possible.
Conclusions
Atmospheric science has played a leading role in the development and use of computer simulation in scientific endeavors. Computer simulations have transformed the climate sciences, and simulations of future states of weather and climate have important societal applications. Computer simulations have come to dominate the field of climate science and its related fields, often at the expense of the traditional knowledge sources of theoretical analysis and challenging theory with observations. The climate community should be cautioned against over reliance on simulation models in climate research, particularly in view of uncertainties about model structural form. However, given the complexity of the climate problem, climate models are an essential tool for climate research, and are becoming an increasingly valuable tool for a range of societal applications.
Returning to the question raised in the title of this post, we have learned much from climate models about how the climate system works. But I think the climate modeling enterprise is putting the cart before the horse in terms of attempting a broad range of applications that include prediction of regional climate change, largely driven by needs of policy makers. Before attempting such applications, we need a much more thorough exploration of how we should configure climate models and test their fitness for purpose. An equally important issue is how we should design climate model experiments in the context of using climate models to test hypotheses about how the climate system works, which is a nontrivial issue particularly given the ontic uncertainties. Until we have achieved such an improved understanding, the other applications are premature and are detracting resources (computer and personnel) from focusing on these more fundamental issues.
And finally, we should ponder this statement by Heymann (2010):
Computer simulation in the atmospheric sciences has caused a host of epistemic problems, which scientists acknowledge and with which philosophers and historians are grappling with. But historically practice overruled the problems of epistemiology. Atmospheric scientists found and created their proper audiences, which furnished them with legitimacy and authority. Whatever these scientists do, it does not only tell us something about science, it tells us something about the politics and culture within which they thrive. . . The authority with which scientific modeling in climatology in the eighteenth century or numerical weather prediction, climate simulation and other fields of the atmospheric science towards the close of the twentieth century was furnished has raised new questions. How did scientists translate or transform established practices and standards in order to fit to shifted epistemic conditions? . . . How did they manage to reach conceptual consensus in spite of persisting scientific gaps, imminent uncertainties and limited means of model validation? Why, to put the question differently, did scientists develop trust in their delicate model constructions?
This post is envisioned as the first in a series on climate modeling, and I hope to attract several guests to lead threads on this topic. Future topics that I am currently planning include:
- How should we interpret simulations of 21st century climate?
- Assessing climate model attribution of 20th century climate change
- How should we assess and evaluate climate models?
- The challenge of climate model parameterizations
- The value of simple climate models
- Seasonal climate forecasts
- Complexity (guest post)
Moderation note: this is a technical thread. The thread will be tightly moderated for topicality.
I guess throwing in biological processes would be darn near impossible and just pile on the pandemonium.
Well the good (?) news is that the community is going towards Earth Systems Models that includes human systems and biology, driven by “policy” needs. IMO this is not the optimal way to use resources to get to the heart of the climate modeling problem
Question: Would it be an oversimplification to retain the shape of the solid portions of the Earth and model primarily the oceans? Land mass could be modeled as a perfect insulator and perfect reflector of solar energy. Once that was working, the other 30% of the surface could be addressed.
Judy,
Excellent summary.
This non-technical essay I wrote for Anthony Watts’ might be useful to your non-technical readers: http://wattsupwiththat.com/2009/06/12/common-sense-and-the-perils-of-predictions/
Mike
Including the atmosphere, of course.
Judith, Supurb. I’m drawn to this issue of climate modelling to reconnect to my experience many years ago at university and at my first job where we attempted to do this sort of modelling but applied to physical response of ships and structures in shallow water. Same equations. Same challenges. Pretty much same results; but we gave up after a few years instead of investing a career in it. I would recommend another bullet point for future topics: what “new” physical experiments/measurements should be designed and planned for model verification. For example, amongst other things the climate models attempt to forecast future global temperatures; but do we have adequate global temperature measurement systems commensurate with the model forecasts?
Much of my career was involved in developing and testing financial models for projects. We found that Monte Carlo simulation provided a powerful method for evaluating the reliability of a model. We would select the ten most significant independent input variable, apply engineering estimates of the uncertainty attaching to those input variables (for example, Normal Distribution, 1 standard deviation = 10% of the input variable) and run a Monte Carlo simulation using the @risk program produced by Palisades.
The results of the model were presented with the Net Present Value of the project as the result. @risk provides a distribution of the output NPV particularly with 1 standard deviation of the output NPV after running, say, 1000 iterations of the model.
In some very few cases, we would find that the result was that 1 Std Deviation was +/- 5% of the Mean of the output which meant that in that instance the model was very reliable, and could be relied upon for financing purposes.
More often however, 1 Std Deviation was very much wider. One example was a Mean output NPV of $33m, but 1 SD was +/- $75m (more than 200% of the Mean) which obviously meant a pretty much equal probability of any outcome. In the latter cases, financing was by definition very difficult.
Financial models for projects are not really very complex and quite a lot is known about each of the key input factors, with generally robust engineering reports available for each factor. Climate models, by contrast, are much more complex, and much less is known about the key input factors or their relationships with each other. The uncertainty spread for input factors must necessarily be very wide, with the result that in nearly all cases, the result of the climate model must be very unreliable, or, to put it another way, there would be a pretty much equal probability of any outcome. In statistics speak, the kurtosis of the result curve is flat.
Monte Carlo simulation has its critics, but what is unchallengeable is that it provides a very effective method for evaluating the reliability or unreliability of a model.
In the financial world, we still undertook detailed model analysis, but it was designed to explore and better understand the characteristics of the project. We learned to be very careful about models, particularly since their nature meant that any analyst who did not provide a Monte Carlo simulation could “drive” the model towards his preferred outcome, simply by biasing each of the input factors by 2-3 % in the same direction. The understanding provided by Monte Carlo simulation meant that the team was much more careful in ensuring a proper understanding of all of the factors pertinent to the project and its financial outcomes.
It would be very instructive indeed if climate modellers were to undertake Monte Carlo simulation runs of their models, and so provide a direct measure of the reliability of their models.
Hi Mondo,
At the European Centre for Medium Range Weather Forecasts (they also do climate modeling), probably the preeminent modeling group in the world today, they have adopted techniques something like you suggest. It is well known that errors in the initial conditions can grow nonlinearly, hence the use of multiple ensembles of model runs, each with slightly different initial conditions supposedly encompassing observational error. But Tim Palmer, one of the gurus of modeling, noted that there are many other uncertainties. especially in our knowledge of clouds, radiation, boundary layer processes and etc. These are often sub-grid scale and so to reach closure these are parameterized in terms of larger scale variables that are modeled explicitly. These parameterizations often have assumed constants. Palmer’s idea has been to run multiple ensembles with small variations in these parameters noting that they are often not known well. I think this is a similar technique to what you suggest where one does Monte Carlo experiments, so to speak, with
those parameters that are least well known.
Regards
Peter W
Peter,
Tim’s recent talk on this ( the issac newton workshop) was a fantastic introduction to the problem)
In the modelling I used to work on we also had to use parameterizations for certain physical processes that were not realizable in real time on the hardware we had. and we just took the approach that Palmer was suggesting. I kinda assumed it was already being done that way.
One of the more interesting attributes of the weather atmospheric forecasting community is its evolutionary skill. As we have previously seen the “idea” of increased computer power and model integration is seen as the solution of the open problem of forecast, or predictive capabilities.
At first glance this would be seen as logically correct, however the atmosphere (weather –climate system) is a complex system not in thermodynamic equilibrium(often far from TDE) and in a perpetual state of reorganization .Here the rules of the game, where idealistic assumptions for the instantaneous state of the atmosphere are not valid when the equations of motion(transformation) to a future state are presently applied.
This is evident when seen in the evolution of the weather forecast model ability of the ECMWF.A widely used model producing forecasts in the range for a few days to a number of weeks. The preparation base is a n-day forecast with n= 10 days of the global atmospheric state.
In any forecast there is an error dependent on initial conditions (due to arbitrary assumptions/estimates of unknown qualities) with the ECMWF model over the last 20 or so years in a paradox the model error has increased.
eg Nicolis and Nicolis Foundations of complex systems page 223.
In 1982 in a seminal paper in which ECMWF data was first used ,to measure predictive ability. Edward Lorenz found the mean error evolution (doubling time of initial error) was two days, presently has dropped to 1.2 days.
This suggest that there is a limiting of predictive capabilities for long range weather forecasting with models of increasing sophistication ,owing to interconnected complexity in the atmospheric dynamics.
Sensitivity to the initial conditions-the principle signature of deterministic chaos-is thus not an artifact arising from when lower order models are used but is, rather, deeply rooted in the physics of the atmosphere
I’ve done plenty of acquisition and project valuations. some of them very detailed and complex. I think the casual reader will get the wrong impression of accuracy about project investment decisions from mondo’s remarks. Yes, a spread of NPVs is nicer than tjha a single one. And ther are interesting things to be learned (sensitivity analysis, whether upsides or downsides are more prominent, etc.) But when he says that there are good enginerring estimates for what happens in the makretplace…bwahahaha!
Note: I’mnot saying it’s futile. But if anything, the process is more imporatant than the excel analyzer. Like in war, it’s not the war plan, but the process of war planning (if you’ve ever done a war plan, or started an acquisitions synergy estimation from a blank sheet of paper, with no info in a conference room….and then blown it out with multutiple people and iterations of qualitative and quantiative info/estimates, with special info gathering on the key issues, etc….youll see how the process of discovery is MORE than just the expected value, a 10, 50, 90 and some sensitivities.
Oh….and you can drive an acquisitiong with uncertanties, just like you can drive an acquisition with a single NPV. I could kill any deal or justify any deal based on how I push the synergies. Yet, I still think that NPV-DCF style estimates (with or without uncertainties, preferred with) are superior to simple multiples estimates or the like. There;s something about that process how it forces you to really take a company apart and consider a variety of internal, external and own company factors.
Thanks JC.
Ok lots of questions!
1) I dislike the simple skeptical argument thate goes “Junk in, junk out!” because it seems to contain the worst anti-science assumptions but it’s hard to not find it on some level seductive. While scientist will strive to accurately portray the climate in their models there is certainly room to just re-confirm one’s assumptions and prejudices. Surely the outputs are non-surprising to the scientist that develops the model? Say if you assume the climate reacts to increases in CO2 with a trend of X. Then you set the equations to do that and hey presto your model tells you CO2 gives a trend of X. Sorry this sounds crude but aren’t models just reconfirma tion of our unlying knowledge and assumptions about the climate? also if you assume internal variability etc can’t affect long term trends then why bother including them in models? I don’t know what’s new to learn from the application of models?
2) I’m curious about the process of developing a model? Have you ever done this? I guess it goes you input all your knowledge and assumptions, run it and then find it doesn’t fit that well to observations so you spend time tinkering. Is the tinkering generally the bulk of the process?
3) My final question is contained in the Heymann quote “Why, to put the question differently, did scientists develop trust in their delicate model constructions?” Judith (or others) why do you think the IPCC and model scientist have such confidence in the outputs of models, you didn’t actually say this in the article. I’m really looking for scientific answers to this question not “thier livelihoods depend on it” or “it’s part of the global communist conspiracy fool” ;)
Thanks
HR, the issue of why scientists trust their model construction is a key one. I find that climate modeling groups in the UK (at UK Met Office, Hadley Centre, ECMWF) are far more active at challenging the atmospheric dynamical core and assessing uncertainty than the big modeling groups in the U.S. The success of weather forecasting, and the understanding of its limitations, transferred to climate modelling (in the climate modelers minds), which is a much more complex undertaking. The focus has been on adding more processes to the model and conducting the comprehensive production runs for the IPCC. People at the global climate modelling centers tell me that more than half their total time is spent on the execution and interpretation of the production runs for the IPCC. This is time that is not spent on fundamental thinking and model development, assessment and understanding. In the U.S., a lot of $$ are thrown at climate modelling (to the modeling centers, people at universities developing and testing parameterizations, some field observations to test parameterizations). Further, much of the motivation for the big satellite systems is for climate model evaluation; unfortunately the valuable satellite data sets are not used in this way at anywhere near their potential.
In terms of scientists involved in writing the actual IPCC reports, very few of them are actual climate model developers, and few of these are actually involved in issues related to the dynamical core and predictability. People outside this core group (the users of climate model results) seem to have more confidence in the models than the model developers themselves.
An interesting issue for philosophers and sociologists of science to explore.
Dr Curry
Ref your comment ..”People outside this core group (the users of climate model results) seem to have more confidence in the models than the model developers themselves.”..
Would this include the media, in your opinion, or do you think that in their case they merely trumpet the results (and what the ‘users say’) without question -in most cases?
Well, I would be surprised if anyone in the media would have the scientific sophistication to actually question the models in a serious way, but they certainly respond to the general rumblings of skepticism and look to find an opposing viewpoint to make their story more interesting. The so called “balance is bias,” I think it is essential that opposing viewpoints be covered by the media. In many ways, i think the blogosphere could be a good venue to sort all this out, get all the questions and issues on the table, and have climate researchers respond. I’ll give it a shot in my own little corner of the blogosphere.
A couple of pieces of sociology on this – but it certainly deserves more attention:
http://precisrcm.com/multimodel/fulltext%285%29.pdf
Shackley, S., Young, P., Parkinson, S., & Wynne, B. (1998). Uncertainty, complexity, and concepts of good science in climate change modeling: Are GCMs the best tools?. Climatic Change, 38, 159-205.
Shackley, S. (2001). Epistemic lifestyles in climate change modeling. In C. A. Miller & P. N. Edwards (Eds.), Changing the atmosphere: Expert knowledge and environmental governance (pp. 107-133). Cambridge, MA: MIT Press.
I read the Shackley et al. journal article, it is brilliant! hard to imagine that it was written over a decade ago. I checked for later pubs by Shackley, seems he has moved on to energy policy.
HR,
Please explain how it is anti-science to believe GIGO (garbage in, garbage out).
AFAIK, that originated in the field of computer science, or at least IT. The climate models are programs, so I would say GIGO applies – perhaps in spades!
Hunter I just think that ‘GIGO’ fails to acknowledge that climate science (and modelling) ultimately is a genuine attempt to get greater understanding of the systems of our planet. The problem seems to me to be the attempt to fit this learning process into the demands of the political process.
What I really want is to try and step outside the political process to evaluate what the science is saying but I’m not sure that’s possible anymore. I actually think that the science is now being driven by the politics even down to the most basic level of what questions are being asked.
So on a political level the ‘GIGO’ argument is very seductive. My instinct says on a scientific level it’s probably too simplistic and probably at it heart is an attempt to reject the science as a whole.
This might be a completely irrelevant analogy but I got thinking that climate science is like soccer. The IPCC is Wayne Ronney and Cristiano Ronaldo. You’d think soccer was all about WR and CR based on volume that is written about these characters but there are 100’s of millions of people involved in the ‘world game’. People playing, coaching and supporting at all levels. In community football, school football, lower leagues and playing on the streets all keeping the game alive. Are WR and CR representative of soccer as a whole?
Us, as consumers of climate science, only ever here about the WR’s and CR’s of climate science through the IPCC and blockbuster Nature and Science papers. There is no doubt plenty of great science being done that never comes close to informing the IPCC or being picked up by journalists and which will never inform or challenge us.
HR,
Thanks. Do you think the old saying, ‘the road to hades is paved with good intentions’ is also anti-science?
Who gives a flip if the efforts are ‘genuine’? How do you quantify ‘genuine intentions’?
The problem is not political. The problem is that claims are being made about the models that do not work, and the ones making those claims refuse to deal with it except, in many cases, by seeking to silence those who disagree.
GIGO is the heart of all discussions. Occasionally someone can do GIGO and win big- think of Columbus sailing west to the orient.
But mostly GIGO means things break. Think BP.
..The problem is not political. The problem is that claims are being made about the models that do not work…
If the claims and models being made are politically funded and motivated, then ultimately, the problem is political.
@ HR October 4, 2010 at 2:27 am
I spent the early 70s and the early 80s modelling. The former period had as desired output a) coal calorific value and b) chlorine breakpoint of sea-water. The latter was an attempt to model nationwide electricity demand. In all cases, the quality of the input data and the calibration (known output data) was impeccable. So forget GIGO. They all failed – abysmally. (Historical note: by Moore’s Law, although I was using what, at the time, were super-computers, they had, in the former case, less power than the microwave oven in your kitchen).
‘Tinkering’ with the parameters is all that it was.
I gave up in disgust, and found another way of earning a crust.
Steven Mosher – October 4, 2010 at 2:53 pm summarises well (as do many others), far beyond my then understanding.
The GI portion includes the algorithms and and the code that operates on the data. Was it impeccable as well?
My reply is on October 5, 2010 at 11:21 am.
Sorry!
BTW, the quality of the discussion ghasts my flabber, leaving me sadly contemplating the depths of my ignorance. And I thought I knew something about modelling and time series!
Absolutely brilliant idea, Judith.
Hamish, I agree, I am learning a lot here!
Hamish,
I did not mean to sound snooty – sorry about that. But it is a big (and overlooked) part of the GIGO statement.
@Mondo,
While at two recent Palisade User Conferences (New York and London), I asked around widely if there was any known use of @Risk for climate modelling. None knew of it having been attempted (doesn’t mean it wasn’t!), but those who were familiar with the Climate modelling debate were supportive of the idea. Like me, the view was “if I only had the time, I’d give it a shot.”
For background on global climate models, RC also has a lot of useful references (one of which is mentioned later on in Judith’s post):
http://www.realclimate.org/index.php/archives/2004/12/index/#ClimateModelling
Particularly the two FAQ’s:
http://www.realclimate.org/index.php/archives/2008/11/faq-on-climate-models/
http://www.realclimate.org/index.php/archives/2009/01/faq-on-climate-models-part-ii/
and Gavin’s Physics Today backgrounder:
http://scitation.aip.org/journals/doc/PHTOAD-ft/vol_60/iss_1/72_1.shtml
Also, let’s keep in mind that climate models are used for much more than “just” making projections of future climate. E.g. as discussed at http://dotearth.blogs.nytimes.com/2010/09/22/a-sharp-ocean-chill-and-20th-century-climate/
Peter Stott:
“Climate models capture much of the complexity and richness of this behaviour – behaviour that is not programed into the climate models but which – actually a remarkable success for the science of climate modelling – but which emerges from these models when we make
simulations of climate over the last century.”
Gabrielle Hegerl:
“This simplistic way of arguing [physical reasoning related to the SH-NH temp difference] shows you why models are so useful – you can actually look at more sophisticated climate physics than what I can do with hand waving”
And without wanting to wave away the large uncertainties that exist, perhaps it’s good to also look at the various successes of climate modeling, eg this list:
http://bartonpaullevenson.com/ModelsReliable.html
“Multi-model Ensemble – a set of simulations from multiple models. Surprisingly, an average over these simulations gives a better match to climatological observations than any single model. ”
This quote from the Real Climate faq#1 kind of epitomizes the problems that Dr. Curry has posed. In order to model some process the relevant variables need to be understood. So the real surprise here is that more than one climate model exists. Multiple models indicate major disagreements as to the relevant variables and model configurations. Running multiple models and averaging the results is in effect yet another model with no rationale behind it. The mention that a multi-model ensemble results in “better match” with observations gives the false impression that something useful is happening,
The climate of the earth is chaotic. The local weather can vary wildly from time to time and place to place, but overall it varies within fairly narrow limits and exhibits the presence of attractors- for example the oscillation from ice age to interglacial periods with a fairly regular time behavior. This kind of climate behavior needs to be the real basis of study.
Actually, multiple models sort of addresses the issue of characterizing the uncertainty associated with model structural form and parameter choices. So multiple models are arguably necessary given the uncertainties.
Bart, I glanced at you references. I could not see a reference that had a quantitative prediction. Do you have a prediction which gives a quantitative answer, with error bars, and where the prediction has come true, within those error bars. The only one I know of is Smith et al Science AUGUST 2007. but we wont know if this is correct until 2014.
Hansen 1988 ( http://www.skepticalscience.com/A-detailed-look-at-Hansens-1988-projections.html )
The cooling effect of Pinatubo was predicted to be in the ballpark that was later confirmed by measurements.
Lots of hindcasting that can only successfully match the obs with a climate sensitivity in the IPCC ballpark, even more so when combining several of such constraints in a Bayesian framework (Annan and Hargreaves, 2006)
The models do have a series of successful predictions.
However, since they make millions of predictions, they will always get something right.
This has to do with the technical question of V&V.
Let me give you an example. In doing the V&V for a model that predicted the radar cross section of a airplane, we would be given a spec that would specify what we had to measure, how we had to measure it, and how we had to validate our model.
So, if I built a model that got the return from a flat surface on the plane right, but screwed up the result from the tiny crack in the surface, I could not point to my success on the former and settle the matter and the latter.
What people from my back ground would like to see is this:
1. we intend the models to predict X, Y and Z. (MOMs)
2. we will test them using proceedure A, B and C
3. we will publish the following performance metrics
4. X will constitute a failing grade for the model
of course people can go beyond that and dig for more stuff, but at least there should be some kind of performance metric minimum. the democracy of models, seems to make no sense. just sayin
“Also, let’s keep in mind that climate models are used for much more than “just” making projections of future climate. E.g. as discussed at http://dotearth.blogs.nytimes.com/2010/09/22/a-sharp-ocean-chill-and-20th-century-climate/”
Bart my understanding is that this paper was largely based on a long grind through the observational data.
A good choice of discussion topic … a lot of my questions about climate models are contained here and I really hope for guest-post, lucid replies and counter-replies
Judith, you write “Particularly for a model of a complex system, the notion of a correct or incorrect model is not well defined, and falsification is not a relevant issue. The relevant issue is how well the model reproduces reality, i.e. whether the model “works” and is fit for its intended purpose”
Many years ago, I developed and used computer models. One the the issues that bedevilled my work is contained in the last words of the quote “i.e. whether the model “works” and is fit for its intended purpose” How does one do this successfully, and how could I satisfy my superiors that I had, indeed, achieved this. This aspect always received attention in any of the reports I wrote.
My major complaint with the use of models in AGW is the failure of authors to address this issue. Most papers just seem to say “We have developed a model, and have run it. Here are the answers”. To quote just one specific example, here is what Myhre et al say in GRL July 15 1998 Vol 25 No 14. “We have performed new calculations of the radiative forcing due to changes in the concentrations of the most important well mixed greenhouse gases since pre-industrial times, Three radiative transfer models are used”. I have read and re-read this document, and nowhere is there a discussion of WHY radiative transfer models are suitable to estimate radiative forcing. My limited knowledge indicates such models are not suitable, but my point is that Myhre makes no attempt to explain why such models are, indeed, suitable.
In the absence of this sort of discussion in ANY of the papers I have read on the use of models for AGW, I am highly suspicious of any of the results. The reliance of the IPCC on the output of such models, IMHO, makes it impossible to rely on any of the conclusions which are stated.
Jim the issue you raise is the very devil in all this. I am working on a post on this topic, and I am currently working on digesting the two Elisabeth Lloyd papers.
Thank you, Judith, for your response. Do you agree with me that the reports which use models and which support AGW, are very deficient simply because they do not discuss this issue?
Well, the key issue is for the modelers to do the work and provide the documentation. Then people that run the models or analyze model results don’t have to retread this ground. The modeling groups need to do a much better job of this IMO.
Judith writes “Well, the key issue is for the modelers to do the work and provide the documentation.”
Sorry to persist with this, but I dont agree. I dont believe there are generalized models that can solve any problem within their scope. My experience tells me that the model and the problem to be solved are inseperable. One needs to show that the particular model used is, indeed, suitable to solve the problem at hand. Just because a model can solve one problem, does not mean it can, necessarily, solve a similar problem.
In the example I gave of Myhre et al. I am sure radiative transfer models are quite capable of solving a range of problems. What I am querying is whether these models can properly estimate a change in radiative forcing.
I am convinced that it is the job of the people who run the models, and use them to produce answers to questions, to show that the model chosen is, indeed, capable of solving the particular problem at hand. If the people who use the models dont understand, in complete detail, how they work, how can they possibly show that they are suitable to solve the problem under consideration?
Jim – I’m with you – I don’t agree either. Judith you write of the models “working”. Wouldn’t it be clearer to talk of them having “demonstrable predictive skill”? And if we all agree that unless they have such skill they ought not to be used for predictive purposes, then surely it becomes vital (not to say routine, outside climate “science”) to declare what constitutes skill, as Mosher outlines? And if so, when asked, can’t you just say so, or rebut? Your equivocation on this simple point is disappointing.
Two years ago, I blithely assumed that no scientist in possession of his senses would consider touting models as the grounds for economic disruption without being able to join all the dots in the way Jim and Steve Mosher describe. Post-Climategate, I now see that this is routine in climate “science” (hence my use of quotes – I barely deem it a science at all, for this and many other reasons).
Lastly, are you aware of any climate models with demonstrable predictive skill – no assistance, just initial conditions, and repeated success – and can you point me to them? Although as has been pointed out, ensembles are really just another model, but I would accept them if they prove skilful.
Judy – this is an excellent much needed summary of climate modeling issues!
To add to the discussion on this thread, I wrote this post
What Are Climate Models? What Do They Do? [http://pielkeclimatesci.wordpress.com/2005/07/15/what-are-climate-models-what-do-they-do/]
where I recommended three categories of modeling –
“Process studies: The application of climate models to improve our understanding of how the system works is a valuable application of these tools……the term sensitivity study [can be used] to characterize a process study. In a sensitivity study, a subset of the forcings and/or feedback of the climate system may be perturbed to examine its response. The model of the climate system might be incomplete and not include each of the important feedbacks and forcings.
Diagnosis: The application of climate models, in which observed data is assimilated into the model, to produce an observational analysis that is consistent with our best understanding of the climate system as represented by the manner in which the fundamental concepts and parameterizations are represented. Although not yet applied to climate models, this procedure is used for weather reanalyses (see the NCEP/NCAR 40-Year Reanalysis Project).
Forecasting: The application of climate models to predict the future state of the climate system. Forecasts can be made from a single realization, or from an ensemble of forecasts which are produced by slightly perturbing the initial conditions and/or other aspects of the model…..”
I concluded in that post that
“… the IPCC and US National Assessments appropriately should be communicated as process studies in the context that they are sensitivity studies. It is a very convoluted argument to state that a projection is not a prediction. The specification to periods of time in the future (e.g., 2050-2059) and the communication in this format is very misleading to the users of this information. This is a very important distinction which has been missed by impact scientists who study climate impacts using the output from these models and by policymakers.”
See also our related papers; e.g.
Pielke Sr., R.A., 2002: Overlooked issues in the U.S. National Climate and IPCC assessments. Climatic Change, 52, 1-11 http://pielkeclimatesci.wordpress.com/files/2009/10/r-225.pdf
and
Rial, J., R.A. Pielke Sr., M. Beniston, M. Claussen, J. Canadell, P. Cox, H. Held, N. de Noblet-Ducoudre, R. Prinn, J. Reynolds, and J.D. Salas, 2004: Nonlinearities, feedbacks and critical thresholds within the Earth’s climate system. Climatic Change, 65, 11-38. http://pielkeclimatesci.wordpress.com/files/2009/10/r-260.pdf
I look forwward to further comments on this thread, and urge you also to publish your post in BAMS or other similar venue.
Roger, thanks for the links. Seems like we are both onto the fit for purpose issue.
Thanks for posting this topic and allowing discussion. Your initial post covers a number of the issues that I have had regarding the modeling process ussed by various and sundry in justifying their thesis and the predictions that result from that.
One fact has and always will remain as a primary issue for modelers, the input data. If you do not have all the data nad you do not know how the data interacts your model will not be worth much.
The biggest issue with climate modeling is that we do not at present know what all the variables are that effect climate let along know how they interacte with each other. Until that is know comprehensive models of the “climate system” are a major problem. Not knowing theses interactions can create a system, of kludging the model to give the required result for a project. This kludging is not nessisarily a bad thing the problem with it is is you think your model should show “this result” and you are getting “that result” you assume that your model is inadequete for some reason and adjust your inputs in order to get a result that conforms to reality. All the while the model is doing what it is expected to do the real problem is something that you did not include because you were unaware of it.
Modeling is a useful tool , and when it can be used to confirm data observations it is a good thing. Using a model that can not be accurate because you do not know what to put into it is a waste of everybodies time.
It might be helpful to summarise the math/computer modelling process (HR’s point 2).
1. Start from some very complicated physical system (eg the climate).
2. Make a lot of simplifying assumptions – which are the key processes, what can be ignored, how can it be simplified. There is nothing wrong with this, as long as you clearly state all these assumptions up front. (eg heat transfer by conduction is negligible, air flow is 2D).
3. Write down some equations based on your assumptions in 2.
4. Solve these equations on a computer. This involves further approximations and simplifications, which again need to be clearly stated.
5. Get the output from your model and compare it with reality. It probably doesn’t fit very well, so go back to stage 2 and reconsider your assumptions.
As Judith says, there are many reasons for working with such models, besides merely predicting the future. In particular, by conducting ‘numerical experiments’ you can learn a lot, such as the effect of different parameters, or you can discover types of behaviour in the system that you wouldn’t have thought of without the computer.
HR’s point 1 is also very important. It is easy to fall into this circular argument (hypothesis posed, hypothesis built into model, results confirm hypothesis). I have seen many scientists do this (without realising it themselves) and I suspect that it goes on sometimes in climate science. So the cynical answer to Judith’s question “What can we learn from climate models?” would be that we learn about the preconceived ideas of the scientist who designed the model.
Paul, the steps you outline would be key elements of a model verification and validation process
Paul
As an infectious diseaseresearch scientist I know generally know my hypothesis, I know the sort of experiments to perform and I generally know what results will confirm my hypothesis. Unfortunately I’m constrained by the natural laws which mean the bugs often don’t react in the way I wish. It does seem that with these virtual climates that there are no binding laws the scientist is free to set parameters as she sees fit. Critical appraisal by your peers would seem the only control on acceptability and I guess here is where the problem of ‘group think’ may come in.
John Christy just emailed me with a link to his new paper
What Do Observational Datasets Say about Modeled Tropospheric Temperature Trends since 1979?
http://www.mdpi.com/2072-4292/2/9/2148/
I have commented the short answer: I’m not sure, on my blog
http://claesjohnson.blogspot.com/2010/10/on-curry-on-climate-models.html
Claes
From your link to:
“Series on Mathematical and Statistical Approaches to Climate Modeling hosted by the Isaac Newton Institute for Mathematical Sciences”
http://www.newton.ac.uk/programmes/CLP/
We have:
“The non-linearity of the governing physical processes allows energy transfer between different scales, and many aspects of this complex behaviour can be represented by stochastic models. However, the theoretical basis for so doing is far from complete.”
The second sentence would appear to be something that needs attention.
If there is no sound basis for constructing statstical arguments on certain model outputs then perhaps all such arguments (such as confidence intervals) ought to state whether they rely on such a hypothetical basis or not.
At the level that climatic change is commonly argued (the headline global temperature anomaly), the variance in opinion could be characterised based on whether belief in such a basis exists.
The common default IPCC posistion seems to be that the climate (not just the climate models) is chaotic but that arguments based on essentially stochastic variance are valid (the warming from 1970-date is significant but little, if anything, besides volcanic cooling, and ENSO is).
The opposing view, that the anomalies are essentially deterministic, and either:
The climate is chaotic and the anomalies are pattern rich and that the warming is essentially natural and deterministic.
or
The climate is periodic and warming is essentially natural and deterministic.
Not to know whether a stochastic basis for the majority of the variance is valid would imply (at the headline level) that the significance of the warming post 1970 is a matter of opinion. Not a good state of affairs.
Alex
Judith
I will only comment on Stainforth et al. (2007) as this is what you have chosen to illustrate the uncertainty problems.
As this is one of many papers coming from the same perspective (climate models), it suffers of the same shortcomings.
A small point: “pandemonium” is not a term used, the right term used in papers is “spatio-temporal chaos” or just short “chaos” when there can’t be confusion with the classical temporal (Lorenzian) chaos.
An example and an excellent monography by one of the leading experts in spatio-temporal chaos dealing with your issue in a comprehensive even if rather technical way is : http://www.math.missouri.edu/~cli/book2.pdf.
Btw if you could get a guest post or at least an opinion of Y. Charles Li about the relevance of climate models to spatio-temporal chaos, this would be awesome because he belongs to the few scientists who really know what they are talking about in this domain.
The following comments to Stainforth, will use the quoted monography and other Y. Li papers.
In the context of constant boundary conditions, and specifically no changes in atmospheric GHGs and therefore radiative forcing, weather is chaotic and climate may be taken as the distribution of states on some ‘attractor’ of weather.
Weather is chaotic regardless if the boundary conditions are constant or not.
Let us be accurate here: boundary conditions are Cauchy conditions for a system of PDE that describe the weather and climate.
The end of the statement seems to mix temporal chaos and spatio temporal chaos.
In the classical temporal chaos the orbits are in a finite dimensional phase space and each point of the orbit represents indeed a physical state of the system.
An attractor is then just a set of authorised states of the system where all orbits live.
Nothing such in spatio-temporal chaos.
What is the equivalent of the phase space is an infinite dimensional Banach space. The points in this space are functions and not states. For example a “fixed point” in the Banach space (quivalent to an equilibrium in temporal chaos) is a standing wave.
Therefore an “attractor” in the Banach space would be an infinite dimensional set of functions or, with Stainforth’s words “a distribution of distributions of states”.
Once when he problem is correctly formulated, it is clear that this definition of climate doesn’t make much sense in the context of spatio temporal chaos.
The whole complexity of the problem lies precisely in the fact that in spatio temporal chaos we deal with distributions of distributions and not just distributions.
Often a single picture tells more than 1000 words.
Everybody knows the Lorenz system and its “butterfly” attractor in the 3D phase space.
Its equivalent in spatio-temporal chaos is the Kuramoto-Sivashinsky equation.
A picture of a solution in the one dimensional case f(x,t) is here : http://eom.springer.de/common_img/k130070a.gif.
This picture is one point in the Banach space.
So an “attractor” would be an infinity of such pictures.
It appears clearly that the amounts of “chaos” and “regularity” vary with space – there may very well be spatial regions that are periodic and very reasonably behaved while other spatial regions are fully chaotic.
And of course with time chaotic spatial regions may become regular while regular regions become chaotic.
Given our time scale of interest, it can be useful to distinguish two forms of ICU depending on whether or not the model’s future climate distribution would be better constrained by reducing the current level of uncertainty. Macroscopic ICU is found in state variables with relatively ‘large’ slowly mixing scales such that the predicted distribution is affected by our imprecise knowledge of the current state of the system. Microscopic ICU results from imprecise knowledge of ‘small’ rapidly mixing scales.
This distinction is really not useful but on the contrary misleading.
What could make sense and what Y.Li is also discussing is the asymptotic behaviour of solutions. But this is completely different from some artificial dichotomy in “small and fast” and “large and slow”.
It is doubly misleading because the temptation is great to jump immediately to an unjustified hypothesis that the “small and fast” is “random” and “averages out”.
Distribution of the ICE mean change with doubling of atmospheric CO2 concentrations, in 8-year mean Mediterranean basin DJF precipitation, in a grand ensemble …
This chart and thousands of similar charts are another version of the problem mentionned above.
The space averaging region is arbitrary. The reason why the precipitation is averaged over the Mediterranean bassin are cultural, pragmatic or simply because that is where one has data.
There is no reason that the real dynamics of the system (completely unknown) which are given by the distribution of the distributions produce a time invariant and initial conditions invariant probability for an arbitrary space average like this one.
Of course computer runs will deliver such charts by design but I can’t still understand where comes this confidence from, that this tells us something else than how numerical models behave.
Much, much more time and effort must be invested in fundamental understanding of spatio-temporal chaos (along the line of Y.Li&Co work) and much less in developping yet another numerical model.
I don’t think the book by Li is very helpful. It is very technical – even to someone with a math training. Also it deals almost exclusively with the Hamiltonian case, whereas climate models are dissipative (in fact much more disspative, ie damped, than they should be). Having said that, yes, understanding spatiotemporal chaos is important – it’s what we see all around us in the weather and probably long-term climate variations as well.
Tomas thanks for your insightful post. Do you have a “readers digest” explanation of spatio-temporal chaos; the Li paper is too heavy for me and presumably most of the audience here. My interpretation of the challenge is how we should design and interpret experiments using a model of a system of spatio-temporal chaos, more so than in the design of the actual numerical model? I don’t know Li, but I’ll send him an email, who knows, maybe he will respond if he has given any thought to the climate model problem.
Sorry Judith but I do not think that such a thing exists.
Spatio-temporal chaos is probably the most difficult scientific problem that I’d put on a par with the string theory.
As there are more questions than answers, nobody has to my knowledge tried a kind of comprehensive handbook.
My mention of Li has been triggered by a footnote in the Stainforth paper
The development of a well-founded coherent jargon for this case would be of value in many fields of study. because this “jargon” (technique) exists, is well founded and is what Li uses.
While climate science is right in the middle of this field, I can’t understand why the people never thought of asking real experts.
It’s like Mann trying some very creative statistics and never thinking of asking an expert statistician for help and advice.
I’m not at all sure that it has much to contribute.
You are not going to get an analytic solution for climate, not on a planet with randomly shaped land masses at any rate. Furthermore, the degree of internal variability in the system – i.e. the bound for this ‘spatio-temporaral chaos’ – is pretty small.
I think that if you want climate science to acknowledge what you consider to be a major issue, then you really do need to write up an appropriate paper demonstrating the impact of this chaos over and above the known climate signal. There is a limit to how far you will get posting on blogs.
I am not at all sure that you understand the point but that is the problem with people who dismiss what they didn’t study.
Of course that nobody will get an analytic solution with or without randomly shaped land masses, that is the point.
There is no well defined “degree of internal variability” in the spatio-temporaral chaos.
And there certainly is no metric which would separate it in “small” or “large”.
Inform yourself and learn by studying people like Li or York who are experts of the issues , it is not blogs that will help you understanding.
What you really need to understand is that the spatio-temporal chaos is the “climate signal” .
What you keep writing over and over is equivalent to saying “Why should I bother reading about Navier Stokes? It has no relevance to fluid dynamics and I can explain and predict everything with my simple model. Besides prove that turbulence exists.”
Fortunately few scientists adopted this kind of attitude else we’d still be stuck with the understanding of the 18th century.
I am not dismissing what you are saying, but I am questioning how important it is. Relating your ideas to physical reality would help.
Oh, and the climate signal is, as far as I am concerned, the 15 year average of global temperatures.
For instance, it could be argued that ENSO is an example of ‘spatio-temporal chaos’ in the system; but the magnitude IS small (Perhaps 0.2K) compared with the AGW signal, or the effects of a major volcanic eruption. As far as I am aware no comparable effect exists on longer timescales.
You might get more usefulness from ‘spatio-temporal chaos’ when looking at medium term (10d<1000d) modeling on regional scales, which is a much harder problem, although not strictly in the domain of climate modeling.
Just a correction.
I would be flattered if this were my ideas :)
But they are not, I have just studied them from the real experts – Li is an example.
The question about physical reality of dynamical equations is not even asked. This is obvious.
As for what you take for “climaet signal” it is for me very slightly relevant noise. Why would this parameter be relevant to anything? Why not a 17.5 years global average of cloudiness? Or about anything else?
The importance of a climate signal as temperature averaged over a sufficient time period and geographical area is that it removes the whole spatio-chaotic stuff.. and also provides something you can more easily match up to, for instance, ice-core isotope records which arguably do exactly the same thing via physical processes.
It removes chaos?
How?
Dr. Curry,
Thank you for opening this topic up. It is fascinating.
Judith,
Your essay was interesting but you neglected two important criticisms of climate models. The first is found in the book by Duke University professor Orrin Pilkey titled Useless Arithmetic: Why Environmental Scientists Can’t Predict the Future. Pilkey is an environmentalist focused on oceans and coastlines. He basically says nature is too chaotic to be modeled with any certainty.
The second criticism is from the field of scientific forecasting. I cannot understand why climatologists have neglected the insights from this field. It certainly is a young field, even younger than climate science, but there are four peer reviewed journals on the subject now and few, if any, climatologists have ever bothered to study the principles of scientific forecasting as a way to improve climate models. See http://forecastingprinciples.com/index.php?option=com_content&task=view&id=11&Itemid=11
Ron, I don’t disagree with Pilkey’s point about nature being too chaotic to model with any uncertainty. But the point of the climate models is to understand complex system processes and to make projections of possible future states. The chaos and complexity require characterization of uncertainty, which my post was mostly about. Re scientific forecasting using computers, the weather community pretty much invented this field. Re statistical forecasting, the climate system is too complex for this to be of much use. The topics in the journal you link to seem to deal with much simpler systems, where the challenges aren’t as extreme for climate models.
Judith,
Yes, I noticed your post was about uncertainty. Pilkey’s book adds perspective from shorelines which is able to get feedback much more quickly than climate projections. I believe his book needs to be read by more climate scientists.
I understand the point of the climate models is to make projections of possible future states. My point, and the point made by J Scott Armstrong, is that climate scientists have not made any effort to deal with the insights gained from the principles of scientific forecasting. According to Armstrong, the principles do apply to climate projections. What would be the harm of a climatologist studying the principles to see which of them might make the models better?
Ron, I am not sure what you mean by the principles of scientific forecasting (in terms of dynamical forecasting, this was pretty much invented by the numerical weather prediction community). You referred to a journal, which addresses some fairly simple forecasting problems. The weather and climate models are based on physics, not statistics. And the challenge in producing an actual forecast for a chaotic system is use of an ensemble of forecasts, elimination of systematic and random error, and interpreting the distribution of the adjusted ensemble to provide a forecast.
The principles of scientific forecasting is not limited to statistics. I’m afraid a cursory reading has given you a wrong impression. It is used in forecasting earthquakes and earthquake risk, which I don’t believe is based strictly on statistics. And Armstrong has written a critique of climate model projections which I would be interested in reading if I was a climate modeler. Armstrong says climate modelers violate a number of the principles of scientific forecasting. Why not learn what they are? http://forecastingprinciples.com/files/WarmAudit31.pdf
Thanks for this Judith – a much needed review.
1. Whilst climate models currently leave much to be desired (see point 2) I believe they are important for future planning. Currently all weather related infrastructure (dam spillways, urban drainage, wind stress on bridges, etc) is based on the assumption that, statistically, the past will repeat itself. This is patently false and a new paradigm is needed taking account of climate change.
2. I believe modellers should more ‘up front’ about how well their models have simulated the 20th century. IPCC reports only have two 1/8 page figures of precipitation and temperature anomalies. However when you look at actual (i.e. not anomalies) values of how these two variables there are simulated there are large differences from observed values.
http://www.climatedata.info/Temperature/Temperature/simulations.html and click on figure 6 for temperature,
and..
http://www.climatedata.info/Precipitation/Precipitation/global.html and click on figure 1 for precipitation
Ron, I totally agree that insufficient effort has been given to evaluating 20th century climate simulations, especially in the satellite era. This will be the topic of a future post. In terms of future planning at the regional level, i think relying on climate model simulations is a mistake, although I totally agree that that the classical “return time” analysis for extreme weather events based on the assumption of a steady climate is totally inadequate. I will address this issue and how we might think about it on a post (in the next week or two) on future climate scenarios.
Judith
Slightly off topic, feel free to erase.
I have just got a mail from Anastassia Makarieva and Victor Gorshkov.
I have given them the information about your blog and informed that you could be interested by a guest post about their hurricane theory.
Thx, i also received an email from Anastassia, and invited her to do a guest post. I’m sure they have their hands full at the moment after returning from the field.
An excellent summary, Judith. Thanks for taking time to produce this report.
The previous discussions at Climate Audit, to which you linked, is a pretty good one. Especially considering how such discussion can frequently go off on a perpendicular.
Fitness for duty is the bottom line. My impression, and I think it is in agreement with many others who have commented on the issue, is that results from the models provide guidance to implementation of policy that affects the health and safety of the public. However, it sometimes seems that this in itself is an open question.
If our impression is correct, we see a significant dis-connect between the requirements imposed on all other software in all other public-policy-making arenas and those imposed on climate science software, all of it, not just the GCMs. There seems to be no requirements imposed on climate science software, none. In all of science, engineering, and technology, climate science seems to be the singular exception.
We are frequently informed of the V&V / SQA-like activities that take place within the organizations that develop climate-science software. Those of us who have software development experience and expertise have all performed the exact same procedures and processes. There is absolutely nothing new in the approach taken within the climate science software development community. I should limit this statement to those procedures and processes that are correctly carried out. Verification should always precede validation, IMO.
These SOP activities do not begin address the V&V and SQA requirements imposed in all other public-policy-making arenas.
As to complexity and non-linearity, these are always present in both the physical domain and software domain for real-world problems; we live in a non-linear universe and we make software to describe that universe. All the non-complex linear problems have been left for play in the academy sand-box. The non-linearity aspect seems to be invoked by way of the ill-posed nature of the ( extremely ) simple, un-complex, very low order systems of IVP ODEs that exhibit temporal chaotic response. Temporal chaotic response is not necessarily related to the real-world situation at even the spherical-cow level. Spatio-temporal chaotic response is a cow of a different color, you might say.
It would help to move the discussions forward if the complexity and non-linearity issues could be taken as given, moved off the table, and the focus moved directly to spatio-temporal chaotic response. “My problem is more complex than your problem and my code is much bigger that your code”, don’t add to the discussions.
The special pleading with respect to Verification is, well, especially grievous. All software can be Verified, there’s no question about that. And, actually, we have seen initial efforts in this activity getting underway. Just as these efforts have focused on the individual component models of the Earth’s climate systems, isolated from the others, the initial attempts at Validation can proceed in the same way. The usual characterization that there’s no data is wrong; if there is no data there are no models. The models are based on some data from somewhere, otherwise the models are pipe dreams.
The accepted approach to Validation is from the bottom. From the individual models for specific physical phenomena and processes, to limited coupling between a limited number of important components, to more nearly complete coupling that includes all the important models for a given application. In one sense, Validation is a confidence-building exercise. And if the full model cannot possibly be Validated, there should be sufficient information in the Validation of the individual pieces to lend confidence. I think there is universal agreement that solution meta-functionals which map an enormous number of results from the models and methods to a single number, are not valid indications of Verification or Validation. Comparisons of the numerical values of these meta-functionals as calculated by different models, methods, and application procedures does not change this lack of validity.
On the other hand, if the public-policy implications are of sufficient importance, then costs and priority, of necessity, should be allocated to the Validation, and Verification, efforts.
Formal Verification and Validation exercises will provide information that can be used to characterize the degree of certainty, or uncertainty, associated with all aspects of the models, methods, software, and application procedures.
It is unfortunate that the IPCC reports, among many others, have focused solely on the numbers generated by climate science software prior to providing information that demonstrates the degrees of uncertainty associated with both the software, as software code, and the models and methods and application procedures.
Dan thanks for this very lucid analysis.
So, what you are basically saying is that Climate scientists should get a vastly increased software development budget?
I would, however, reject the notion (common on here) that all climate science is dependent on GCMs. You can get useful information out of 1D and 2D models; and you can deduce overall climate sensitivity without reference to computer models at all. So it’s not as if your GCM is unconstrained.
From a scientific perspective, maybe if the ‘skeptics’ stopped carping on and started writing their own GCM – with all of the QA, validation, etc mentioned here – they could show how current GCMs are wrong. It would certainly be a more convincing argument than throwing around software engineering terms. After all, the geosyncline theory was not overturned by people poking holes in it; no scientific theory ever has been.
There is no doubt that quality software, software that has been shown to be fit for production-grade applications, costs more. But, where is the arithmetic that shows that “a vastly increased software development budget” is required.
More importantly, where is the justification that all software used in climate science is exempt from the Independent V&V procedures and processes that are required of all other software that is used to provide guidance to policy-making decisions?
No where have I stated that “all climate science is dependent on GCMs”. Can you point to any examples of anyone anywhere having made such a statement? On the other hand, the importance of the GCMs in climate science is almost always mentioned in papers, books, and the IPCC reports.
It is impossible for individuals to research, develop, construct, apply, and qualify / certify a brand new from scratch alternative GCM. The hardware alone costs hundreds of millions of US$. Such suggestions are commonly found whenever someone is trying to deflect discussions from the subject. AKA Yet Another Strawman, short, thin, single-straw version.
When it comes to answering your first paragraph, I would refer to your fourth paragraph. Perhaps more to the point, if – as people are requesting here – if GCM software is to be produced to a gold-plated standard then it will cost more.
As far as your second paragraph goes, I *seriously* doubt that all software used to provide guidance to policy is subject to independent verification. That would require actual evidence for me to believe.
And as far as you third paragraph goes, this belief does appeart widespread on this site, in your words:
t is unfortunate that the IPCC reports, among many others, have focused solely on the numbers generated by climate science software
could indeed be interpreted this way.
As to your final point, I am serious; this is science and independent replication is pretty critical. All non-trivial models will of course contain assumptions and simplifications, which will in turn mean that ‘skeptics’ can endlessly carp on about these same assumptions; it would be preferable for the ‘skeptics’ to demonstrate that a model founded on their preferred assumptions performed better. Asking people to get off of their backside and do some work is not a strawman.
(And no, you don’t actually need vast amounts of hardware, at least not for a few years)
ndrew,
Are you suggesting that climate science has been under funded these years?
No, I’m suggesting that the logical conclusion of people who say:
‘Climate models have too poor programming standards, insufficient validation and insufficient verification to be used as a guide on policy’
Are implicitly arguing for an increase in funding for GCMs. Decent programmers cost money..
Even if chucking money at the programming would improve its quality, why do so, in preference to funding countless other public enterprises and obligations? Where’s the value to the public, unless you beg the question, and assume the existence of a public hazard for which you have, axiomatically, provided no evidence? Why not spend it on hospitals, or research into hip displasia in overbred spaniels? What’s so exceptional about climate research?
Again you are making the argument that the only evidence for climate change comes from complex computer models.
This is false.
We have a high confidence that the planet is warming due to human influence; GCMs would be looking as much as possible at the knock on effects of this.
As far as funding goes, thats down to politics.
And do not forget that except for models the evidence that any of this is dangerous or will become dangerous is zip.
Dangerous climate change (i.e. what the heck is it) is the topic of my next post
Dr. Curry,
Fantastic. You are covering in your few first few posts more ground than has been covered to date by RC, Romm, etc., combined. And you are doing with civility.
You are definitely underpaid. ;^)
By your reasoning, Andrew, those people who don’t have faith in God should go and make their own God. God MkII. A better God than the one people have been placing faith in all this time, so as to prove that God MkI doesn’t function as gods should.
Let’s not go there, please?
hehe! My point is that Andrew presumes there is predictive value (realised or not) in climate models. This is not necessarily a view shared by those who are sceptical of the potential value of climate models. It would be a lot of time and effort to be able to ultimately say “see, I told you this was a waste of time”.
In the USA that would be the Code of Federal Regulations ( CFR ).
The USA CFR is accessible through an electronic version, e-CFR, here. You can browse the pieces parts and several search methods are also provided.
The various federal organizations that might specify V&V and SQA requirements are found under the Title ‘N’ sections, where N presently runs from 1 to 50. A few examples follow: Energy (NRC) falls under Title 10, Aeronautics and Space (FAA, NASA) under 14, Food and Drugs (FDA) under 21, National Defense (DoD) under 32, Protection of Environment (EPA) under 40.
For example under Boolean Search:, searching Title Number 14 to Retrieve ‘launch’ within Part AND ‘verification’ within Part finds PART 417-Launch Safety. And many others
Searching Title Number 40 to Retrieve ‘software’ within Part AND ‘verification’ within Part finds 194.23 Models and computer codes. And many others.
Searching Title Number 10 to Retrieve ‘software’ within Part AND ‘verification’ within Part finds Appendix B to Part 50—Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants, among others.
Checking the Web sites for specific agencies will very likely lead to more specific info.
Can you name a single exception in any area that affects the health and safety of the public? Any. Single. One. At. All.
I’m British.
The perspective that a GCM can perform an experiment is not scientific. The scientific perspective rests solely on actual observations of nature. The point is that no GCM can show another GCM to be wrong. The same can be said of any model. Only scientific experiments can confirm or falsify models. Models must make predictions that can be tested against reality. Period.
As mentioned in Dan’s analysis, his perspective is one of confidence building. He mentions that in most other scientific/engineering fields, V&V processes have converged. Why should climate science software V&V be any different? Let’s adopt processes we know can build confidence.
I don’t actually have a problem with better processes, improved quality, V&V etc. With of course the remark that it costs money, so people banging on about it must support more money for climate modeling.
Oh, an you can fairly easily do a run against observations by dint of starting your model at 1900, for instance. If you boil the oceans by 1970 you can be pretty sure your model is a crock. I would never say that a GCM run was an ‘Experiment’ anyway.
Andrew, you say: “I don’t actually have a problem with better processes, improved quality, V&V etc. With of course the remark that it costs money, so people banging on about it must support more money for climate modeling.”
But actually, unlike many of the sceptics, the modellers already have funding. Shouldn’t the discussion be about how effectively they are using the resources given to them? Surely climate science modellers should be complying with expected standards as a given?
The argument that to undertake quality control costs more money, therefore we should have more funding doesn’t wash with me. Sorry.
Yes. The first rule of holes is to stop digging.
I think the AGW promotion industry has been able to dig a really really deep hole. Intervention is called for.
Oh, I forgot. As for “throwing around software engineering terms”, the subject of this thread is software. If the subject was PDEs and the numerical solutions of these, we would be “throwing around” mathematics terms like continuous equations, discrete approximations, consistency, stability, convergence, ICs, BCs, well-posed, and a host of others.
Funny how that works.
The fundamental goal of any simulation is to reproduce (and then hopefully predict) the behavior of the physically measurable variables in the system. These ideas go back to the start of quantum mechnical models in the 1920’s. In the case of climate models, the basic variable is the ‘surface temperature’. The claim of global warming is that the 1.7 W.m-2 increase in ‘clear sky’ downward flux from a 100 ppm increase in atmopspheric CO2 concentration has produced an increase in surface temperature of 1 degree C. This is based on the use of the meteorological surface air temperature record [‘hockey stick’] as a proxy for the surface temperature.
The idea of radiative forcing goes back to Manabe and Weatherald in 1967.
The underlying assumption of radiative forcing is that long term averages of dynamic variables can somehow be treated as though they were in equilibrium and that perturbation theory can be applied. The downward flux of 1.7 W.m-2 from CO2 is added to the flux from a surface at an ‘equilibrium surface temperature’ of 288 K. This gives a temperature rise of ~0.3 C and the rest of the 0.7 C temperature rise is claimed from ‘water vapor feedback’. This is to say the least, totally absurd.
The surface energy transfer is dynamic and the surface flux terms vary between +1000 W.m-2 for full summer sun and -100 W.m-2 for low humidity night time cooling. The surface flux is also coupled into the ground, so the thermal conduction down into a cubic meter of soil has to be added to the calculation. The surface temperature during the day, for dry bare soil/summer sun conditions can reach over 50 C. Now do the averaging correctly, with half hour cumulative flux terms and the whole global warming problem goes away. 1.7 W.m-2 for 0.5 hour is ~3kJ.m-2. The full solar flux for 0.5 hour is 1.8 MJ.m-2. The night time cooling is 0.18MJ.m-2/0.5 hr. The heat capacity of the soil is over 1 MJ.m-3. Do the math and there can be no measurable surface temperature rise from 100 ppm [CO2].
Now, remember that this is the real surface temperature. The one under our bare feet when we walk around. The MSAT is the temperature of the air in an enclsosure at eye level. The change in the MSAT is the change in the bulk air temerpature of the local air mass of the weather system as it passes through. This is usually set by the ocean surface temerpatures in the region of origin of the weather system, with a little help from the local urban heat island effects. When the ocean surface temperatures and the urban heat island effects are included, CO2 can have no effect on the MSAT.
I learnt long ago to leave the Navier Stokes equation well alone, but make sure that the output of the fluid dynamics models was firmly anchored to real, measurable variables. There was also a very good indpendent check. If the models were wrong, the rocket engine could blow up ….
There is no problem with the fluid dynamics in the climate models. Make small changes and validate often. They may not be very accurate, but we do not need them to predict global warming. They are research tools, so keep them out of public policy. Weather and climate follow from ocean surface temperatures and sunspots etc.
The problem is that the empirical assumption of CO2 induced warming has been ‘hard wired’ into the models using ‘radiative forcing constants’. This climate astrology not climate science. The IPCC is a political body that has created anthropogenic global warming where none exists. Once radiative forcing is removed and replaced with realistic surface energy transfer algorithms for the air-land and air-ocean interfaces, the climate models should begin to behave much better.
The global warming surface temperature predicted by the IPCC is not a valid climate variable.
References
R. Clark, Energy and Environment 21(4) 170-200 (2010) ‘A null hypothesis for CO2’.
R. Clark, ‘California Climate is caused by the PDO, not by CO2’
http://scienceandpublicpolicy.org/originals/pacific_decadal.html
R. Clark, Gravity rules over the photoncs in the greenhouse effect
http://hidethedecline.eu/pages/posts/greenhouse-effect-vs.-gravity—guest-post-by-roy-clark-201.php
[Note: Figure 6 in this ref gives the half hour flux terms for a real surface]
J. D’Aleo and D. Easterbrook, Multidecadal tendencies in ENSO and global temperatures related to multidecadal oscillations
http://scienceandpublicpolicy.org/reprint/multidecadal_tendencies.html
A. Cheetham, A history of the global warming scare
http://scienceandpublicpolicy.org/reprint/history_of_global_warming_scare.html
[This shows the IPCC liars we are really dealing with]
Lindzen, R.S. & Y-S. Choi, Geophys Res. Letts., On the determination of climate feedbacks from ERBE data, 2009, 36 L16705 1–6
[Goodbye water feedback]
Nicely put Dan.
looking at the number of runs that a team typically does for an IPCC report, it would seem to me that there would be some benefit to taking one model and freezing it and running it repeatedly for a given 21st century scenario. The approach of building and improving models for the next big test really doesnt give one the ability to characterize the performance of what you are changing. is there a solid baseline for say “modelE” performance, say a few hundred runs of the 2oth century under BAU. I mean, I cant even imagine going off and changing any code without having that form baseline.. with all its warts.. you wanna know the warts inside and out, as well as the good parts.
( so cue up the scientists versus engineer fight on this one )
Dear Dr Curry
Problems with a model
First it must be understood that while many people understand modelling and terms like Monte Carlo (in more than one sense) the subject passes the majority of the public by. I suspect that statements like ‘models do not produce evidence’ are largely of no interest to many of the public. That having been said;
Broadly, there are two types of user/builders of models:
• Engineers who build and use models for systems analysis and design.
• Scientists who build and use them to test hypotheses and to, for example, conduct experiments on new scientific and mathematical principles.
There are broadly two approaches to the design of models:
• Engineers tend to build models using well established mathematical and engineering principals. In engineering models there are very few, if any, ‘unknown’ or indescribable parameters. General characteristics of an engineering model may include:
o All mathematical relationships within and between subsystems are known and understood.
o The models are verifiable, do not produce unphysical results and predict test data measured to be within acceptable design limitations under all end user specified system configurations (validation).
o Models meet rigorous engineering design and coding standards.
o Every step in the design, re-design and coding of engineering models is documented to accepted standards.
o For any given set of input conditions the outputs will be absolutely consistent, trustworthy and robust. For this reason, unless knowledge of the model’s ability to represent a system accurately is not perfect, extrapolation is not generally permitted.
o Safety of the end product is often paramount and software coding standards reflect that. Aeroplanes, cars, bridges and skyscrapers spring immediately to mind.
• Some scientific models, say, for experimental evaluation may be designed using mathematical and/or scientific methods that may not yet be well understood; (indeed that may be the very reason for the models’ development). In this case there may be many ‘unknowns’ and perhaps even ‘knowns’ whose importance is unknown and/or that may not even be renderable mathematically. General characteristics:
o Models must, as far as it is understood, accurately represent any embryonic/theoretical mathematical or scientific principle that is under test.
o The end result is more likely to be knowledge than a usable product, so, in very many cases the design and use of scientific models will not be safety critical. Therefore model design and coding standards may be rather more relaxed than would be acceptable in the engineering world. This is not bad; it is pragmatic but does need to be recognised as an issue.
o The ‘knowledge’ obtained from an experiment might be that an hypothesis falls; therefore the next version of the model might be very different as the hypothesis changes. The scientists writing their programs need to be reactive to the changing circumstances of their experiment. They require a flexible approach to programming.
o Strict regulation of modelling standards might not be compatible with the flexibility required in the design and writing of scientific models
o Whether a model represents a system accurately may not be known (again, that might be the experiment)
o Validation may not be possible.
o Extrapolation may be the aim of the model (as for example is the case for climate models). But if it is not known that the model represents the system accurately such extrapolation would, generally, not be acceptable to the engineering fraternity.
Another, and arguably the least qualified, user of model results is the politician.
• Perceptions:
o In the layman’s eyes engineers and their methods are invisible.
o The tag ‘scientist’ equates to romantic infallibility. Therefore, when results of ‘scientific’ experiments are published in the mainstream media, the public whether simply receptive, gullible and/or ignorant tend to believe them. (Whether or not they are correct)
o If politicians can see a way to send a message that they are sympathetic to the public mood and the results of an experiment give them a vehicle to increase taxes without the public squealing they will take it.
o Similarly if entrepreneurs can see a way to make a quick buck they too will exploit such results
o Clever marketing ensures public acceptance of the message. If the message is ‘we’re all doomed’ then the politicians will have the public’s undivided attention and carte blanche to do what is required.
o The public therefore ends up with government policy being based on model results that may be purely experimental and that may not have been subject to the rigours of an engineering model.
Government due diligence
Before a government uses experimental results from a model to drive policy it should, in my opinion have, as a minimum, to confirm that the model has been:
• Verified
• Validated
• Regulated
• Shown to be:
o Relevant
o Robust
o Accurate (within acceptable tolerances)
o Consistent
Finally, bearing the above in mind, would you want to fly on an aeroplane whose design had been based on models that had not been subjected to the rigorous engineering standards required in the aviation industry? Of course not; why then is it acceptable for governments to base policy (that will most certainly be life affecting) on models that may not have been subject to the necessary design and safety rigour?
There is a multi-year consortium/collaboration known as CLIVAR (Climate Variability and Prediction). One of its main goals is particularly relevant to this discussion–
The presentation by Dr Anna Pirani at the CLIVAR SSG16 meeting in 2009 — the “PAGES (Past Global Changes)-CLIVAR Intersection” — provides more detail on the mutual interests of climate modelers and paleoreconstructionists (large PowerPoint file) . It describes the goals:
This strikes me as either constructive or counterproductive… depending.
Constructive, if and only if modelers and paleoclimate reconstructors are acutely aware of the limitation of proxy-based reconstructions, in terms of the actual uncertainties regarding regional and global climate in the centuries prior to the pre-instrumental era.
Counterproductive, if either party believes that the narrow error bars that accompany most reconstructions are reflective of current knowledge of past climate regimes.
Evaluating models by precise but inaccurate “hindcasts” would seem to offer the potential for generating mistaken confidence in the robustness of some models, while unduly dismissing others.
I do not know whether these concerns are sufficiently appreciated by the organizers of this effort, or not. It might be interesting to offer a guest-poster spot to one of the CLIVAR-PAGES scientists, to see how they view the issues being discussed in this thread.
Dusty thanks for your thoughtful post. With regards to your last questions, we need some insights from social scientists.
I have found science is okay with being just general and not being exact.
When I want to know something, I want exactly or why ask the question?
Dr Curry, Your blog is captivating. I do numerical simulations of enhanced oil recovery processes for oil companies where we are confronted with making predictions of oil recovery while often not knowing well the geology of a formation. One of the difficulties we face is convincing managers of the predictability of our simulations. We typically present them the assumptions that go into the models and do parametric studies to estimate a range of oil, water, gas and sand production. Since these managers are often engineers, physicists or geologist they have a “feel” for the validity of these assumptions. In the case of climate modelling you are studying a system which is orders of magnitude more complex, with non-climatologists deciding on policies based on numerical prediction from GCM s. I don’t think they have a “feel” for the validity of the assumptions since they are often not trained in climatology and are not aware of the parametrizations that go into the climate models. My question is how can the policy makers get a better judgment of the numerous assumptions and parametrizations that go into the models to intuitively get a realistic confidence level in the simulations?
Dusty – great post – writes
“The ‘knowledge’ obtained from an experiment might be that an hypothesis falls; therefore the next version of the model might be very different as the hypothesis changes.”
It seems more and more to me as though this is what does NOT happen in climate science – CAGW hypotheses are not ALLOWED to fall, and as a consequence (among several) the stimulus required to create a better model – or experiment – is absent.
Food for thought Dusty
Tom you beat me to it. Although I was going to ask it as a question.
It does seem the presumption of climate sceptics is that everything but the hypothesis is allowed to fall. Is this an accurate statement? I’ve read some pretty dis-heartening papers where I felt that the bar was set sufficiently low to allow a AGW hypothesis to stand. And in a later papers, from the same group, the bar was set much higher to discount alternative explanations. I could go into detail but I don’t think that is what this blog is about.
There seems a subtle difference between using models to test if a hypothesis falls and using a model as supporting evidence for a hypothesis. Are both valid approaches?
As I hope Judith will not mind me saying once more – the null hypothesis is studiously disregarded in CAGW science, and seems not even to be properly understood. A poster here with claims to scientific expertise wrote bizarrely of the consensus “becoming the new null hypothesis”. To the extent that it is understood at all, it is respected only at certain stages deemed to be “milestones”, instead of being “at the experimenter’s side” at every stage in his experiment.
Not only does this blindness to the null hypothesis lead scientists down blind alleys, it deprives them of the intellectual stimulus to – in Rutherford’s words – “think of a better experiment”.
Absolutely wonderful post. I’ve been working my way through the various links. In recent years, I spent a few years on a modeling team so I have some empathy for the awesome challenges that are facing these modelers. Given that climate is a chaotic system and that there are still huge issues re: initial conditions and feedbacks and sufficiency of data, etc, etc, it is obvious that it will take many years before any team is able to successfully model the climate. Having said that, the topic is quite absorbing and fascinating.
Regarding parameters that are set by humans rather than extracted from data, normally those can be considered fudge factors and a sign of lack of understanding of physical reality.
‘Climate is chaotic’?
I have not seen this demonstrated.
Well try that : http://environment.harvard.edu/docs/faculty_pubs/tziperman_chaos.pdf
There are literally hundreds of papers of this kind. Look at Tsonis work too.
Of course you will need to wait for the mathematical demonstration for some decades because the spatio-temporal chaos is not well understood yet.
But looking at the results obtained on the simpler cases like the Kuramoto-Sivashinsky equation, there is no reasonable doubt about the system being chaotic (also hundreds of papers on this issue).
And now for you. Demonstrate that the climate is deterministic but not chaotic.
Oh, I’ve got not problem with the system (I hesitate to call it ‘weather’ or ‘climate’) being chaotic over a period of less than 10 years.
Indeed, that’s why any practical definition of climate has to use a timescale over 10 years.
As far as demonstrating determinism goes, I can simply look at any of the climate reconstructions that cover the past 1000 years and note that until recently, the variations stay within a narrow band:
http://www.ncdc.noaa.gov/paleo/pubs/pcn/fig6-10b.png
If you look at the pre-1900 area, no matter which data set you look at, your range is <1K, and obviously this set is perturbed by events such as volcanic eruptions and solar changes:
http://www.ncdc.noaa.gov/paleo/pubs/ipcc2007/fig614.png
(top graph)
This is an empirical demonstration that if there is a chaotic/random walk element to climate then it cannot be large.
Andrew, do a google search of abrupt climate change holocene and see what pops up (see esp the younger Dryas). Esp see this presentation, the graph 5 slides in. Also, the ocean oscillations (e.g. PDO, NAO) have fairly large multidecadal signals. There is no getting around the fact that the global average temperature increase between 1910-1940 was as large as 1970-2000 (check the CRU temperature anomaly graph); the 1910-1940 warming was apparently unforced and a manifest of chaotic internal variability. More significantly, in spatio-temporal chaos, you can have very large regional manifestations (which average out to some extent when taking a global average). Also with regards to the reconstructions over the past 1000 years, they do a poor job of capturing the magnitude of decadal scale variability.
Thank you for that reply – succinct and accurate. End of that line of resistance
Hi,
Are you seriously suggesting that the Younger Dryas ‘just happened’?
And yes, I am aware that climate can change in a large scale, catastrophic way from natural causes (although good luck characterizing any of this as ‘internally generated’). This is not exactly a reason to feel complacent.
The period from 1910-40 had a lull in volcanic activity, increase in solar irradience and a small buildup in GHGs, which combined to give some warming (although less than the current unless you do some blatant cherry-picking) – you can ask Tamino more if you are interested:
http://tamino.wordpress.com/2010/08/23/antrhopogenic-global-cooling
So to characterize it as ‘unforced’ is not entirely accurate.
As far as variations go.. we have very little evidence that decadal variation in global climate is particularly large; and this is something that you have to actually demonstrate if you want to use it to demonstrate a point.
The large regional variations may or may not happen, but that is not relevant to global climate on the timescales under discussion.
There is increasing evidence that the “irregularities” in the global temperature record prior to 1970 are in a large part unforced, see this discussion at Die Klimazwiebel
Really are you serious ?
Do you mean that showing 2 scrambled charts with unknown assumptions and dubious reconstructions of some global parameter, with uncertainty bands that go to heaven demonstrates anything at all ?
And it’s about time to register that chaotic and random are 2 completely different things.
If you want to bring a proof about something as complex as a non linear dynamic system, then you must do much better than 2 random spagetti graphs.
Tomas, I frequently see this mixing / interchangeability of ‘chaotic’ and ‘random’ in the climate science literature. Just this morning, a few minutes before seeing your comment, I ran across this statement on page 21 in the Muller and von Storch book”
On the other hand, maybe the authors don’t subscribe to the, Chaotic response of a complex, non-linear dynamical systems view.
The subtitle of the book, by the way, is “Building Knowledge”. An excellent summary of the situation, in my opinion.
Dan
The right formulation should be :
Climate models do not forecast. They simulate plausible realizations of a chaotic process. Different initial conditions lead to different trajectories which may have identical statistics.
Even written in this correct way there is still an assumption .
The assumption is that a climate model CAN simulate a plausible realization of a chaotic process.
You are sufficiently familiar with numerical solutions of non linear ODE in chaotic regime to know that this assumption is not a given.
As for the identical statistics , this condition is also known even if apparently Muller is not familar with it – it is called ergodicity.
If the system, and it is not important whether we call it weather or climate because the latter is just some averaging of the former, is ergodic then there will be some invariant PDF in the infinite time limit
For example the Lorenz system of ODE in temporal chaos is ergodic.
Whether the weather system will be temporally and more importantly spatially ergodic is a question that is fully open.
It is the misunderstanding or more probably the ignorance of ergodicity that leads these people to equate chaos with randomness.
Of course in reality chaos/complexity don’t imply ergodicity pretty much as non linearity doesn’t imply chaos.
Ergodicity is a pretty strong constraint and it can’t be just assumed . There are many examples of non ergodic (but chaotic) systems.
Most are in the spatio temporal domain (infinite dimensional phase space) but Hamiltionian conservative systems are an example of a low dimensionnal chaotic and non ergodic system too.
So what Muller writes is just an unproven assumption and the odds are great that this assumption is wrong.
I have never seen it seriously disputed.
The answer to your question may be correct, and your conclusions may be sound. The way you got there in my opinion is wrong! I have a background in mathematics and physics, a PhD in modelling and currently I do not use GCMs, but use SCMs and other simple models.
You have just given a very extreme and unfair representation of modelling per se. A key to modelling is the famous expression “Keep it Simple Stupid”. Always, always, start simple. Slowly over time make it more complex when there is no other way to explain a given phenomena. I think you will find the vast majority of modellers, climate and otherwise, work like this. The modellers you are critiquing above, did not just pull a GCM out of their pocket over night, but took years to build it piece by piece.
As an example of a mirespresentation, while discretization, oscillations and numerical dispersion are definately issues that arise in modelling, these issuse are well known and understood in the vast majority of cases. They are there whenever you solve differential equations numerically, just like there is always bacteria in a hospital. That does not mean I dont go to hospital for a complex operation, though the risk of failure is higher due to the bacteria.
We modellers do not want complex models, they are too complex and hard to work with. I can guarentee you, climate modellers will curse if they have to add more complexity to an already complex model.
You will find, I think, the vast majority of modellers are geeks like me sitting in their rooms trying to understand parts of their models. They are doing a dilegent job trying to overcome some problem that has been identified or understand the physics, if any, behind a particular solution characteristic. They will be well aware of the many pitfalls you mention. They will not be running around better their chest saying how fantastic their model is (maybe a few do, but the vast majority dont).
To imply modllers are ignorant of all the things you mention, is just a plain unfair.
If you put all those caveats in place, then what you say is probably correct ;). I would suspect that the main issue is that modellers are not given the time they need (it is a complex task, after all). There is always a hurry to get results out to get in the next IPCC report or have the next big paper in Nature/Science. The art of checking something, just one more time, is being lost. Models are very useful, if you understand their limitations! Modellers often do (in private atleast), but others probably take their models too far.
Glen,
“Modellers often do (in private atleast), but others probably take their models too far.”
And therein lies the crux of the issue.
Glen,
I would seem in many sceptic blogs that your profession is under attack. I’m glad you ended your post in the way you did because I think for some of us that’s where the distinction lies. I have 100% support for geeks striving to better our understanding of the world, I have greater worries about policy needs driving forward science. You seem to be critical of the IPCC’s involvement in your science, does that go as far as thinking it is subverting (maybe too strong a word) it in some way?
Very instructive ve article; thanks Judith
Global Warming and Cooling Cycle Reporting in the Media:
1)1910s Global Cooling
24-Feb-1895, NY Times: Geologists Think The World May Be Frozen Again
2)1940s Global Warming
15-May-1932, NY Times: Earth Is Steadily Growing Warmer
3) 1970s Global Cooling
21-May-1975, NY Times: A Major Cooling Is Widely Considered Inevitable
4) 2000s Global Warming
3-April-2005, Time Magazine: Special Report On Global Warming, Be Worried, Be Very Worried
http://bit.ly/aVQyxO
Here is the observed data that supports the above cycles:
http://bit.ly/cDRQxM
Judith
Could you consider to add features in your blog for easy printing of your posts?
Thank you.
Judith: I found your discussion interesting, but short on tangible, dramatic evidence. Below are two such tangibles that illustrate the problems with the IPCC’s use of models more clearly for me than your thoughtful detailed presentation. Perhaps I become too used to a world were short “sound-bites” like these dominate the news?
1) Buried in AR4 WGI Box 10.2, there is a statement explaining that the IPCC’s models represent “an ‘ensemble of opportunity’ not designed to sample modelling uncertainties systematically or randomly”. However, when Stainforth et al (Nature (Jan 2005) 433, 403-406) did systematically vary six model parameters describing clouds and precipitation within constraints known from physical experiments, the equilibrium climate sensitivity of the Hadley AOGCM varied from 11 degK for 2X CO2. (www.gfy.ku.dk/~kaas/Bornoecourse/Material/Stainforthetal2005.pdf). The foundations of everything the IPCC has projected based on models is resting on scientific quicksand, a fact that was known when AR4 was being written.
2) The main purpose of climate models is to provide information that will help us avoid or mitigate disasters caused by increasing greenhouse gases. About 5000 years ago, the world experience a well-documented example of the type of climate disaster society wants to know about and probably would pay to avoid – the creation of the Sahara Desert. At that time, the earth was very similar to today: the massive glaciers of the LGM had melted and sea level and greenhouse gas concentrations had reached preindustrial values. The main difference at that time (or “forcing” compared to present) is that our orbit brought the earth was closest to the sun during summer in the Northern Hemisphere. As a consequence, forests grew to the shores of the Arctic Ocean (without harming the polar bears) and the Sahara turned into a desert (probably because the ITCZ began to stay too far south). Unfortunately, none of our climate models currently project how and why the Sahara became a desert under these circumstances.
Frank,
The Sahara desert is a very interesting example. The Sahara has cycled between wet and dry periods many times over the last several hundred thousand years. In the warm, wet periods the Sahara was transformed, becoming much greener and more hospitable to life.
If climate models can’t tell us about these cycles, can they tell us anything useful about AGW?
I hope I can say this in a fairly short write, here.
I am about 25% done reading the Shackley et al paper. I am at section 3.2.1 Response from GCMers.
I am not at all surprised to read this. Not long ago I read about the early days of climate modeling. It was probably one of the simpler ones, not a GCM. In that source they talked about how a Japanese modeler came in and pretty much consciously overrode the deterministic code, because at one point it always went out of control – and the other modelers were very happy that he had “solved” the problem. As I recall, it said the Japanese Mr Fix-It became more or less a legend among modelers. (That is poorly paraphrased – sorry…)
The whole issue of letting the end result dictate seems to me to make make their effort inductive, rather than deductive – just making adjustments until the final output looks good. Bill Gates’ people at MicroSoft are famous for all their patches upon patches on their Windows products. That seems a fair parallel from where I sit, reading Shackley et al.
But the thing I want to add to the discussion is the question of field testing the “first principles” and intermediate outputs. I’d want to have that done on ALL principles involved. If there are reductionists in there – and it sounds like there are – isn’t it best to take the time to make sure of the pieces, and that in doing so the final output will get more and more precise – and thus closer to reality? I would find it VERY hard to believe that they only go by the end result – but much I am reading in Shackley et al seems to hint at that (so far).
I’m glad you included mechanical engineers in your welcome, JC, because that is what I am. I don’t have a degree, but worked my way up. Along the way I also spent 7 years in R&D. I went looking for the Navier-Stokes equations and in reading it saw they dealt with flows and Reynold’s numbers. The R&D work I did included spreadsheet work with heat flows for a pretty darned complex industrial application. One Doctorate-level boss of mine floundered for three years understanding the processes. He was eventually replaced by another. The new PhD’s first order of business was to do a matrix on each of the basic processes, so we could measure and/or observe where the process was taking us, so we would know where to make adjustments to our control settings. Within three months, we were discovering all kinds of new information about the processes – at the intermediate stages. By doing that, we got MUCH more control over the entire process. It was one of the most impressive samples of the scientific method I have ever witnessed. At the same time, it was obvious the earlier PhD should have done this all much earlier – but he really wasn’t very well organized, to put it bluntly.
So, in terms of your asking what we can learn something from the models this approach certainly seems desirable: to simply make SURE of those low-level processes, each and every one. The matrix was a great took, because we actually had a reference to go to to compare our intermediate results – and then to DO something about it.
They have been at it for 25 years or so, but it seems like much of it was like that first PhD I worked with – disorganized, and trying to extrapolate back from final results in order to make process changes. That is TOO HIGH a level, too un-specific. It means too much inductive reasoning, trying to explain what caused the final results. But that should not be happening – not after 25 years. Prdon me for being hard on them, but they need to get their act together, at the LOWER levels of model output – Do those REALLY align with reality? Or are they all assuming?
Like Shackley said – getting “the right result for the wrong reason.” I will tell you, we did that for three years, before we had someone steer us better. And we used the scientific method. What a revelation. And wow, did we have a handle on things after that. Fixes were quick, because we’d learned what to look for – and to recognize what we saw, and what it meant.
So, my input is this question: Can they really be learning much of anything if they aren’t absolutely sure their OUTPUT at lower levels coincides with empirical reality?
If they are already doing this, good. But that focus on getting the final result argues that this may not be the case.
Ref Original Article, quote by Heymann (2010):
…”How did they manage to reach conceptual consensus in spite of persisting scientific gaps, imminent uncertainties and limited means of model validation? Why, to put the question differently, did scientists develop trust in their delicate model constructions?”
Science has never been easy, and I doubt that anyone here would argue that it is now any less complicated by our use of compters today –super or not. We have reached the point where the indivudual is more incapable of achieving breakthroughs independently. That teams and teams of teams are now required; and that as time goes by, groups and groups of groups will be reauired. That a single field of study cannot work independent of other fields on any area of significance.
The super scientist today requires a super team of experts in a specific field (or fields) and experts in various other, related, complementary and supplementary fields capable of the mechanics needed to support the research, experimentation and simulations undertaken. Tomorrow the size of the super team will only grow to become the super group. Of course their budget is also vast and will only become more so.
Who independently assesses and duplicates the findings? Who can? We seem to have arrived at a new plateau and require new rules upon which to maintain the trust and the integrity of science.
My comment about ‘impeccability’, if read carefully, referred only to the data. The algorithms and the code were, even in retrospect (after 30 years more experience in IT or whatever you want to call it), somewhat better than ‘not bad’, given the hardware limitations of the time.
Nevertheless, point taken.
My point is merely the historically arbitrary nature of predictive models with large numbers of degrees of freedom, and the frustration caused when one realises that one is merely ‘tinkering with parameters’. One sometimes wishes that climate modellers would think upon these things.
This is merely a reply to ‘Kan’ October 5, 2010 at 12:32 am
Judith
Great start.
One major problem with IPCC appears to be that they have dismissed natural/solar/cosmic causes as inconsequential climate drivers. Consequently, climate models are force fit to other parameters. Voila -> anthropogenic causes!
Could we begin by asking if Climate Models can reproduce statistically significant historical evidence – back before just the end of the 20th century?
Precipitation/Flow
W.J.R. Alexander analyzed more than 100 years of river flow and precipitation data from the Southern Africa region, including 11 804 cumulative years of data from 183 stations. Development of a multi-year climate prediction model, W.J.R. Alexander, ISSN 0378-4738 = Water SA Vol. 31 No. 2 April 2005 209-218 http://www.wrc.org.za
Alexander showed an
SOLAR ACTIVITY AND CLIMATE CHANGE — A SUMMARY, W.J.R. Alexander and F. Bailey ENERGY & ENVIRONMENT VOLUME 18 No. 6 2007
Alexander shows dry to wet flow reversals in 1933, 1954, 1973, and 1995. The three lowest years before a sunspot minimum had average flow of 52. The three following years average was 300. i.e. a 577% increase. In 2008 Alexander predicted a major drought in South Africa. That began in 2009.
Alexander WJR, Bailey F, Bredenkamp DB, van der Merwe A and Willemse N, (2007). Linkages between solar activity, climate predictability and water resource development. Journal of the South African Institution of Civil Engineering, Vol 49, No 2, June 2007, pages 32-44, paper 659. Alexander compiled a “474-page technical report entitled Climate change and its consequences – an African perspective (Alexander 2006). It includes 51 tables, 33 figures and 218 references.” (available on CD.)
Don Easterbrook has tracked the 60 year Pacific Decadal Oscillation and their correlation with temperature. In 2001, Easterbrook predicted global temperature trends and transitions through to 2100 starting with a transition from warming to cooling ~ 2003. This now appears to be happening.
Multidecadal Tendencies in ENSO and Global Temperatures Related to Multidecadal Oscillations J D’Aleo, D Easterbrook – Energy & Environment, 2010 – Volume 21, Number 5 / September 2010 CategoryResearch-Article Pages437-460 DOI10.1260/0958-305X.21.5.437
Hurst Phenomena
Furthermore, W.J.R. Alexander observes:
So could we begin by asking Climate Models to show:
1) The 95% correlation Alexander finds between precipitation/runoff and the ~22 year magnetic solar cycle?
2) The 60 year Pacific Decadal Oscillation and corresponding global temperature warming/cooling trends?
3) The ~ 0.70 H correlations across 1 to 20 centuries shown in the Hurst phenomenon?
Can we then ask the Climate Models to then predict the these flow/ precipitation variations over the next several solar cycles and Pacific Decadal Oscillations?
When they can reproduce such historical / paleoclimatic evidence, perhaps they may be believable in their estimates of appropriately scaled “anthropogenic” causes!
David, I agree that the PDO (and the AMO and the NAO) have substantial global signals in the 20th century, which is all but dismissed by the IPCC. I also agree that the indirect solar effects are poorly understood and in the “white zone” of the italian flag. Hurst phenomena is new to me, do you have a reference? thx
Just search for Hurst on CA, or search on mandlebrot and hurst
Per Mosher, see:
http://climateaudit.org/?s=hurst
Weather and Climatology: Mandelbrot’s View Reference:
Mandelbrot and Wallis, 1969. Global dependence in geophysical records, Water Resources Research 5, 321-340.
Demetris Koutsoyannis etc.
Thanks to everyone that supplied these references
Judith re Hurst Phenomenon
Alexander et al (2007) above cite Hurst 1951, 1954.
Hurst, H E 1951. Long-term storage capacity of reservoirs. Transactions of the American Society of Civil Engineers, Paper 2447.
Hurst, H E 1952. The Nile. London: Constable.
Hurst, H E 1954. Measurement and utilisation of the water resources of the Nile Basin. Proceedings of the Institution of Civil Engineers, volume 3, part III, pp 1–26, April 1954: discussions pp 26–30, correspondence pp 580–594.
Google Scholar has 819 hits for Hurst Phenomenon
Happy hunting
Judith
See also “Hurst-Kolmogorov “, especially by D. Koutsoyiannis:
Markonis, Y., and D. Koutsoyiannis, Hurst-Kolmogorov dynamics in paleoclimate reconstructions, European Geosciences Union General Assembly 2010, Geophysical Research Abstracts, Vol. 12, Vienna, EGU2010-14816, European Geosciences Union, 2010.
Tyralis, H., and D. Koutsoyiannis, Simultaneous estimation of the parameters of the Hurst-Kolmogorov stochastic process, Stochastic Environmental Research & Risk Assessment, 2010.
Tyralis, H., and D. Koutsoyiannis, Performance evaluation and interdependence of parameter estimators of the Hurst-Kolmogorov stochastic process, European Geosciences Union General Assembly 2010, Geophysical Research Abstracts, Vol. 12, Vienna, EGU2010-10476, European Geosciences Union, 2010.
Dialynas, Y., P. Kossieris, K. Kyriakidis, A. Lykou, Y. Markonis, C. Pappas, S.M. Papalexiou, and D. Koutsoyiannis, Optimal infilling of missing values in hydrometeorological time series, European Geosciences Union General Assembly 2010, Geophysical Research Abstracts, Vol. 12, Vienna, EGU2010-9702, European Geosciences Union, 2010.
Perhaps, as a start, the extensive effort at “infilling” global temperature data could benefit from Hurst-Kolmogorov analyses, and then see if such infilling can actually satisfy evidence of long term Hurst-Kolmogorov properties.
Then apply to climate models to see if they show such evidence and what is needed to achieve such.
Happy processing.
Judith
Here is a good rather simple paper about Hurst phenomenon : http://www.itia.ntua.gr/getfile/612/2/documents/2004EncyclHurstPP_.pdf
Dmetris has published many papers about Hurst and hydrology .
Hurst phenomena is new to me, do you have a reference?
There is a good example in Nicolis and Nicolis in the download chapter 1.4.2.
There are perhaps analogies with what has been learned from ecological models, a field in which many case studies exist and experiments are difficult but not impossible.
1) When processes have been assumed to be computable by subtraction, the result has often proven to be wrong. For example, early plant carbon balance models turned out to be lacking in leaf leachates when it rains, leaf aromatics production, root loses of carbon to fungal symbionts, and root exudates.
2) Idealized models have often turned out to be wrong. Fixing them has meant separating animal age classes, considering animal digestion, 3D ray tracing of sunlight paths through tree canopies, and considering all sorts of heterogeneities.
Surely what Craig has said is highly relevant. So far as I can see none of the models used by the proponents of CAGW have been shown to be capable of solving any of the problems investigated. Which means that there must be a high probability that the answers obtained are just plain wrong.
Since the IPCC is a advocacy group, and not scientific, we can be certain that any errors are on the low side; e.g. the IPCC would never guess a low number for something like climate sensitivity. So there is every indication that the claimed value for climate sensitivity is very likely to be one or two orders of magnitude too high.
Judith
Another check is how well do the climate models include and/or fit the three very large non-CO2 anthropogenic changes?
1) 3rd world deforestation
2) US Clean Air Act capping SO2
3) Saddam’s torching Kuwait
David, the whole issue of aerosols is hugely important, this will be a topic of a future post.
One implication of a chaotic system is that modeling it is incredibly difficult, since any small understanding of the system causes consequences that are many, many orders of magnitude larger than the magnitude of the original misunderstanding (butterfly effect).
It is well understood that mathematical errors in the GCM model will have similar effect to an initial value error or climate system behavior misunderstanding. Beyond these problems, it also seems obvious that any bugs in the GCM software will have the same catastrophic consequences.
In computer science there is the concept of program proving, where you go through the software line by line and prove its correctness (with an input assertion before each line and an output assertion after each line). Obviously this is extremely time consuming but very useful on such critical, extraordinarily sensitive software. Have any of the GCM’s been proven? This topic would be an interesting subtopic of the the bigger topic of GCM software engineering.
Another interesting computer science related topic would be an examination of the GCM software WRT to modularity and maintainability. Since this GCM software has been around for quite a while, it seems likely that the software might not be as well structured as required. Modern software is typically object oriented, which reduces cross coupling between modules (among other things). In the old days of fortran, a lot of scientific software was very spaghetti like, difficult to understand and maintain. Has there been an independent software audit of the various GCM software implementations to determine the quality of their software engineering?
The more object oriented the GCM software is, the easier it is to replace classes with alternative classes to try out different climate science ideas. Obviously there is a tradeoff between performance and maintainability/modifiability.
How large are GCM software implementations in terms of lines of code? What languages are they written in? How old are they? What is the best GCM?
With [abject] apologies to Steve – who writes extremely knowledgeably, but only addresses part of the problem.
Lipstick on a pig? Doesn’t any GCM model depend upon thousands (more?) of assumptions (what is the sign of CO2 -> [water vapour &c.->] cloud feed back, parameters (what [numerically] is the amount thereof, &c., &c., &c.). Degrees of freedom???
Steve, there I agree there are huge problems with modelling – all those you raise, and more. Until the modellers are up front with us about their parameters and their reasoning, i.a., instead of hiding behind the fallacy of Appeal to Authority, their models are, prima facie, irrelevant. (I could use stronger language).
Many apologies to, genuflections and abasement before to those who know so much more (99% of previous, and no doubt future commentors) than do I. Isn’t this the point?
When I do sensitivity analysis to get confidence intervals, I will make 1000 or 10000 runs. Because GCMs take so long to run, few have ever been run more than 20 times for the same scenario–far too little to characterize model uncertainty.
Maybe what we learn from modeling a an incredibly large, complex, chaotic system that is beyond our current capabilities is that it would be better to take a bottom up approach than a top down approach. Maybe there should be more focus on modeling subsets of the environment with extremely fine granularity models and sensor measurements. In the future, when there are more sensors (and types of sensors) deployed and we have a longer record of fine granularity reliable measurements and we have finer granularity modeling software, maybe the problem will be more solvable when we have the computer power to run these finer granularity simulations.
The climate system being chaotic decreases the credibility of GCMs but it increases the importance of the precautionary principle.
The climate system being chaotic decreases the credibility of GCMs but it increases the importance of the precautionary principle.
I would say exactly the contrary for the precaution “principle” (which has nothing of a principle and everything of a circular statement).
Chaotic systems are extremely constrained and pseudo-cyclical.
They never escape from their domain of phase space and can’t “diverge” and go to some strange places.
In a sense, whatever you try to do to them, they always wander only among already visited places by just changing the order and the speed of visit.
I understand what you are saying but was actually going to ask what is it that constrains the earth’s climate at the top end of temperatures?
Having said that, it is true that:
* the earth’s climate is chaotic (small input changes can lead to enormous climate changes)
* we don’t understand the climate process in a detailed way (tremendous uncertainty)
* undergoing a large, rapid change in climate would be traumatic and painful
Given the above, why wouldn’t we be cautious about how we stimulate the climate system?
Steve, are you referring to the Krugmann argument, ostensibly based on Weitzmann’s paper that greater uncertainty increases the importance of acting under the precautionary principle? I will take this argument on in a post (about 3 weeks from now, need to build up to it in my sequencing).
Judith,
Maybe (but not knowingly).This realization occurred to me after reading this post from you. If the climate is chaotic, then a change of 100 ppm in CO2 might be enough to trigger a big change (who knows?). The precautionary principle says don’t take unnecessary risks (wouldn’t be prudent) for tiny gain. So our very inability to accurately model the weather decreases our confidence in predicting the impact of changing CO2. For sure we don’t understand the feedbacks. That argument is more persuasive (to me) than the “science is settled” argument.
My theory is that we should move towards non CO2 emitting energy sources as rapidly as we can without screwing up the economy (i.e. switch over to them when they are commercially viable/competitive but not before).
Agreed, thanks for the clarification.
Steve,
Since we know CO2 has been higher in the past and no great tipping has occurred, I believe that you are misapplying the precautionary principal.
A much better application would be to stop obsessing on CO2, stop wasting money on windmills, and develop power sources that are more reliable and cleaner than coal.
Every time I hear a windmill supporter use the price of oil as an excuse to impose wind power, I know I am listening to someone who is either ill-informed or seeking to deceive. Oil has nearly nothing to do with power production. It is a transport fuel and chemical feedstock. Windmill power has nothing to do with transportation.
Why do windmill promoters confuse this?
Solar has been five years away from usefulness for decades now.
Wind power is so over sold as to be not much different from a big con job.
That leaves only one power source, and oddly enough most in the AGW community avoid even mentioning its name, and the enviro community is mostly dead set against it.
For the questions that the public cares about (long term global temperature forecasts) it seems to me that these models are probably overly complex and ill suited. It’s possible they may be useful for regional climate change predictions on medium time scales, but they haven’t really been promoted that way yet.
For long term temperature forecasts, we really just need an radiative energy balance. If these models are good/useful for determining net feedbacks including cloud formation and water vapor issues in the atmospheric column, let’s test them against satellite observation and use them for that. If our observations aren’t good enough, let’s improve the data gathering. If all the energy is disappearing deep into the oceans – let’s find it.
We need to focus on what questions we want to answer and get a broad consensus that the way we are trying to answer these questions is appropriate and testable, and that the answers we get will be useful.
For long term temperature forecasts, we really just need an radiative energy balance
You surely wanted to say imbalance because there is never radiative balance as temperature variations show.
And you surely don’t mean that you would want to measure this imbalance unless you want to put the whole Earth in a giant detector what would be a quite ambitious project.
Besides even the knowledge of the radiative imbalance doesn’t allow to say anything about ground temperatures anyway.
Why ?
Well because the atmosphere and the oceans are fluids. They move. They accelerate. They dissipate energy. They change of phase and free or capture latent heat. And they do all that in a very complex coupled way.
If you ignore these energy and mass transports then even if you knew the radiation imbalance at every moment (what you don’t and will never do) you would have no clue about the surface temperatures , kinetic and latent energies and their evolutions.
Not even mentionning clouds and albedo variations.
I agree, the climate system is perpetually in imbalance, and there is no sensible averaging period to define a “balance.” My other problem with energy balance climate models is the simplistic way they actually determine the surface temperature, which is quite complicated for land, ocean and ice surfaces.
Steve
It’s not a vendetta (commenting at the same time – similar time zones? Although one should never forget Samuel Johnson’s insight about ‘the finest sight …’).
You’re absolutely right – although my training would lead me to consider ‘reduction in the number of degrees of freedom’ a.k.a. (or at least related to) number of parameters.
On the precautionary principle – yes, its importance may increase (relatively) but its value (absolute – whatever that means) must, surely, decrease? After all, if the models posit no CAGW (whatever – use your favoured term) within their confidence limits (or, simply, limitations), why take precautions? If they are useless, why spend $many trillions (or even $1) on mitigation?
What can we learn from climate models?
For many years into the future we will learn that there is much yet to learn about, (1) the physical domain, (2) modeling of the physical domain at the continuous equation level, (3) the discrete approximations to the continuous equations, (4) the numerical solution domain of the approximations, (5) the coding of these in the software domain, and (6) the application domain.
Each is a separate and distinct domain that requires deep expertise and close analysis. There is an iterative loop around all of these, between groups of two or a few of these, and within each of the domains; it’ll take some time. The critical importance of some of the mechanistic, empirical, heuristic, ad hoc parameterizations will be determined. We will learn that a broad frontal attack on all of this simultaneously within the framework of a single piece of software might not be the best way to approach the problem. Validation of the really important parts will require that data be measured and those experiments will be designed, developed and carried out.
Judith – many thanks again for your work. Although much of it goes scientifically over my head, it does seem as if you have a forum here which is hammering out some kind of quality protocol for GCMs, even if this is largely by applying standards that have long existed in other fields. Steve Mosher’s brief 4-step protocol seemed ideal
Would it be completely impractical for you to host a forum for GCMs whose creators are willing to subject them to appraisal in this way?
In July 2010 Pielke Sr. pointed his readers to an Economist interview with Brian Hoskins, and in particular to his comments about the limitations of models.
http://pielkeclimatesci.wordpress.com/2010/07/25/brian-hoskins-interview-by-the-economist/
Roll on quantum computers!
Judith,
Congratulations, your blog is a breath of fresh air.
I have two daughters who graduated from Georgia Tech. One, a mechanical engineer, once told me that you know you are an engineer if you have ever assumed a horse to be spherical in order to simplify the equations. I’m wondering if that might now be modified to apply to climate modeling.
Donald Rumsfeld once commented on why climate models have to be used with caution. “There are known knowns. These are things we know that we know, models handle them adequately. There are known unknowns. That is to say, there are things that we know we don’t know, models can partially deal with these by making assumptions and running the models over and over with different assumptions. But there are also unknown unknowns. These are things we don’t know we don’t know, it is very difficult for models to deal with them at all. Models may then become merely intellectual exercises.”
If he didn’t say it quite like that I’m sure thats what he meant to say.
Cal, delighted to hear that your daughters went to georgia tech. Re Rumsfeld’s quote, I used it in m Uncertainty Monster post, check it out.
Firstly I don’t believe that weather is chaotic. I think it’s just subjected to to forces that we don’t completely understand and that these act with various strengths and durations. Some examples – What triggers rainfall and what stops it? What makes a hurricane suddenly change course? Why is wind flow sometimes very uneven?
Secondly, Lloyd’s contention that climate models are confirmed by various instances of fit between their output and observational data is rather juvenile. There are multiple ways by which the same result can be produced. The number of sets of 3 numbers between 1 and 99 that together total 100 is 833, and sum of each set produces the same total. The only way that models can be confirmed is by demonstrating that every climate force is completely and accurately described.
Thirdly, I want to draw your attention to table 2.11 page 201) of the WG I contribution to the IPCC’s 2007 report. That table lists only climate forces associated with radiative heat transfer and in the column describing the level of scientific understanding (LoSU) we find that 13 of the 16 forces have an LoSU below “medium”. Until such time as these forces – and probably others dealing with other methods of heat transfer – are better understood then climate models are merely glorified computer games.
Ms Curry:
Thank you for your thoughtful and informative post.
In my opinion, your most cogent point is:
“Particularly for a model of a complex system, the notion of a correct or incorrect model is not well defined, and falsification is not a relevant issue. The relevant issue is how well the model reproduces reality, i.e. whether the model “works” and is fit for its intended purpose.”
However, in the case of climate models it is certain that they do not reproduce reality and are totally unsuitable for the purposes of future prediction (or “projection”) and attribution of the causes of climate change.
All the global climate models and energy balance models are known to provide indications which are based on the assumed degree of forcings resulting from human activity resulting from anthropogenic aerosol cooling input to each model as a ‘fiddle factor’ to obtain agreement between past average global temperature and the model’s indications of average global temperature. This ‘fiddle factor’ is wrongly asserted to be parametrisation.
A decade ago I published a peer-reviewed paper that showed the UK’s Hadley Centre general circulation model (GCM) could not model climate and only obtained agreement between past average global temperature and the model’s indications of average global temperature by forcing the agreement with an input of assumed anthropogenic aerosol cooling.
And my paper demonstrated that the assumption of anthropogenic aerosol effects being responsible for the model’s failure was incorrect.
(ref. Courtney RS ‘An assessment of validation experiments conducted on computer models of global climate using the general circulation model of the UK’s Hadley Centre’ Energy & Environment, Volume 10, Number 5, pp. 491-502, September 1999).
More recently, in 2007, Kiehle published a paper that assessed 9 GCMs and two energy balance models.
(ref. Kiehl JT,Twentieth century climate model response and climate sensitivity. GRL vol.. 34, L22710, doi:10.1029/2007GL031383, 2007).
Kiehl found the same as my paper except that each model he assessed used a different aerosol ‘fix’ from every other model.
He says in his paper:
”One curious aspect of this result is that it is also well known [Houghton et al., 2001] that the same models that agree in simulating the anomaly in surface air temperature differ significantly in their predicted climate sensitivity. The cited range in climate sensitivity from a wide collection of models is usually 1.5 to 4.5 deg C for a doubling of CO2, where most global climate models used for climate change studies vary by at least a factor of two in equilibrium sensitivity.
The question is: if climate models differ by a factor of 2 to 3 in their climate sensitivity, how can they all simulate the global temperature record with a reasonable degree of accuracy.
Kerr [2007] and S. E. Schwartz et al. (Quantifying climate change–too rosy a picture?, available at http://www.nature.com/reports/climatechange ) recently pointed out the importance of understanding the answer to this question. Indeed, Kerr [2007] referred to the present work and the current paper provides the ‘‘widely circulated analysis’’ referred to by Kerr [2007]. This report investigates the most probable explanation for such an agreement. It uses published results from a wide variety of model simulations to understand this apparent paradox between model climate responses for the 20th century, but diverse climate model sensitivity.”
And Kiehl’s paper says:
”These results explain to a large degree why models with such diverse climate sensitivities can all simulate the global anomaly in surface temperature. The magnitude of applied anthropogenic total forcing compensates for the model sensitivity.”
And the “magnitude of applied anthropogenic total forcing” is fixed in each model by the input value of aerosol forcing.
Kiehl’s Figure 2 can be seen at http://img36.imageshack.us/img36/8167/kiehl2007figure2.png
Please note that it is for 9 GCMs and 2 energy balance models, and its title is:
”Figure 2. Total anthropogenic forcing (Wm2) versus aerosol forcing (Wm2) from nine fully coupled climate models and two energy balance models used to simulate the 20th century.”
The graph shows the anthropogenic forcings used by the models show large range of total anthropogenic forcing from 1.22 W/m^2 to 2.02 W/m^2 with each of these values compensated to agree with observations by use of assumed anthropogenic aerosol forcing in the range -0.6 W/m^2 to -1.42 W/m^2. In other words, the total anthropogenic forcings used by the models varies by a factor of almost 2, and this difference is compensated by assuming values of anthropogenic aerosol forcing that varies by a factor of almost 2.4.
Anything can be adjusted to hindcast obervations by permitting that range of assumptions. But there is only one Earth, so at most only one of the models can approximate te climate system which exists in reality.
The underlying problem is that the modellers assume additional energy content in the atmosphere will result in an increase of temperature, but that assumption is very, very unlikely to be true.
Radiation physics tells us that additional greenhouse gases will increase the energy content of the atmosphere. But energy content is not necessarily sensible heat.
An adequate climate physics (n.b. not radiation physics) would tell us how that increased energy content will be distributed among all the climate modes of the Earth. Additional atmospheric greenhouse gases may heat the atmosphere, they may have an undetectable effect on heat content, or they may cause the atmosphere to cool.
The latter could happen, for example, if the extra energy went into a more vigorous hydrological cycle with resulting increase to low cloudiness. Low clouds reflect incoming solar energy (as every sunbather has noticed when a cloud passed in front of the Sun) and have a negative feedback on surface temperature.
Alternatively, there could be an oscillation in cloudiness (in a feedback cycle) between atmospheric energy and hydrology: as the energy content cycles up and down with cloudiness, then the cloudiness cycles up and down with energy with their cycles not quite 180 degrees out of phase (this is analogous to the observed phase relationship of insolation and atmospheric temperature). The net result of such an oscillation process could be no detectable change in sensible heat, but a marginally observable change in cloud dynamics.
However, nobody understands cloud dynamics so the reality of climate response to increased GHGs cannot be known.
So, the climate models are known to be wrong, and it is known why they are wrong: i.e.
1. they each emulate a different climate system and are each differently adjusted by use of ‘fiddle factors’ to get them to match past climate change,
2. and the ‘fiddle factors’ are assumed (n.b. not “estimated”) forcings resulting from human activity,
3. but there is only one climate system of the Earth so at most only one of the models can be right,
4. and there is no reason to suppose any one of them is right,
5. but there is good reason to suppose that they are all wrong because they cannot emulate cloud processes which are not understood.
Hence, use of the models is very, very likely to provide misleading indications of future prediction (or “projection”) of climate change and is not appropriate for attribution of the causes of climate change.
Richard
Richard, your post is very insightful. The whole cloud-aerosol issue is at the heart of the sensitivity uncertainty, I am planning several posts on tis topic. The aerosol forcing fiddle factor in the 20th century attribution analysis introduces a major element of circular reasoning into their argument.
Perhaps you could answer something for me?
Do the modelers find their models to be “better” if they match the climate record (say HadCRUT3) closely?
If so why?
I can see no reason to believe that the record to be a indicator of the underlying froced trend to within a significance of +/- 0.2C or perhaps a greater interval.
I doubt that the wiggle 1940 – 1970 is statistically significant and it seems to be used as a justification for “tuning” the aerosol forcing. (However I suspect the real justification is to match the century scale increase.)
Why on earth do they not just let the models rip with their best aerosol values judged on any other criterion bar matching the warming rate, and see what comes out.
If they then get the warming “wrong” then one or more my still have actually got the science right, providing “wrong” is not spectacularly wrong (say worse than a 10,000 year event). If they are all fudged to meet the same idealised objective then they are probably all wrong as the actual record is in a sense an unlikely version of the 20th century, (that is it is just one of many possible outcomes).
It is a bit like I just rolled a full house at poker dice and the models all tended to run things close to full houses. I would deduce that they were biased as I know that my roll was a rare event.
Unfortunately we are denied such certainty but still I would not look for the best climate model based on the best fit to the record, it must have a plausible fit but nothing more than that. What it must have is a good basis for its parameterisations and fudge factors other than the record itself.
On a slightly different topic, why are there not more control (picntrl) runs archived? (Climate Explorer)
I think that they hugely important as they answer the question: “What do the models do when you don’t tell them what to do?”
That is things like:
How wiggly are they?
Do they get the phase of the season, or diurnal cycle right?
Do they “drift”?
(Again that is not a condemnation unless we think that we are blessed by being spot on the equlilibrium. Why should model runs that drift a little 0.2C/Century be excluded from some analyses?)
What do they tell us about climatic sensitivity?
(The sensitivity is an inherent property of the model and exists even when unforced, but it would show up very slowly in that runs may have to go on for a millenium or more and then it might be detectable in spectral analysis. or similar techniques. The compensating advantage is that variables such as the relative efficiencies of variying different trace gases is irrelevant as they wouldn’t vary.)
In my view matching the wiggles of the 20th Century is more than likely a guarrantee that you have come up with the wrong model.
I remember when one of the early models thought that the Sahara should not be a desert. Actually that is not so laughable, as it was recently a lot wetter and may be again.
However if a model tells you anything other than that the Namib is always going to be a desert then it is probably very wrong as it has been more or less since the dinosaurs roamed (based on the fossil evidence, mostly fossil dunes).
A model has not got to get things spot on to get the science right, and if the current state of the MOC is a fluke then being out by a few degrees might be an asset not a defect.
Alex
Alexander, these are very good questions and get to the issue of fit for purpose. Most of the model “tuning” is done for the current climate where we have good data, say the last 30 years. Smaller variations like diurnal cycle and response to volcanic eruptions are used evaluate model performance also. You would be surprised at how poorly the regional seasonal cycles (esp of precipitation) are simulated. Most of the models use some sort of tuned aerosol forcing particularly prior to 1940. The modeling groups select different aerosol and solar forcing data sets, presumably choosing the ones that give the best fit. Not only does this kind of fitting introduce circularity into the attribution arguments, but this kind of fitting is actually detrimental to trying to understand climate sensitivity.
Judith,
“You would be surprised at how poorly the regional seasonal cycles (esp of precipitation) are simulated. ”
Actually I wouldn’t, I am a seasonal cycle nerd, I know diddly-squat about precipitation but I have an gridded analysis of the seasonal temperature cycle based on the HadCRUT3 climatology that I have tried to spot check against the Hadxxx models.
“Not only does this kind of fitting introduce circularity into the attribution arguments, but this kind of fitting is actually detrimental to trying to understand climate sensitivity.”
Well that is what I think too. Given that temperature record could be held out and the tuning done to meet other criteria and the question “Do the models predict the general form of the temperature record without being forced to?” Could be asked, but even then failure to match the record could just be a happenstance.
As I understand it, AGW theory never was an empirical temperature record based hypothesis but was, is, and would always be a atmospheric science based theory, and would still be non-invalidated were we to drop into another ice-age.
To my mind, slavishly tuning to the 20th Century temperature record might demonstrate a failure of confidence in the underlying science.
I am very much of the “publish and be damned” school of thought, if the models give values that are infeasibly far to low or far to high with the best science, look for better science for the parametrisations but look anware else except to the record that you wish to project and hence end the circularity.
To make this clear, for me the question is not, “How hot will it get?”, but how uncertain are we about the consequences of AGW theory and what must, or even could, we do to minimise that uncertainty. To my mind gagging the models is precisely the wrong approach. Training them to make a match seems like a political decision with, as you suggest, must have a detrimental effect on projective skill.
I still fear that I have not got across just how extreme my views are on this point. We have paid the earth to build these models but then we tell them what the correct answer is and bet the earth on this strange admixture of their skill hampered by our prejudice.
Alex
A naive question from this geologist (if the computer doesn’t work, I hit it with a hammer):
Is the complexity of the GCMs helpful or a hinderance?
To my normal understanding, a model is a simplification of the system overall, and designed to improve our understanding of a few key elements of the system. I appreciate that we are discussing modelling a very complex system that contains chaotic elements both in the real world system and in the mathematical model equations, but given that there is still a necessity to calculate the results of model ensembles rather than gaining much information from an individual run (or in some cases even an individual model), wouldn’t equally useful predictions / projections be obtained by running a much less complex model many more times? (the complex GCMs are perhaps necessary in improving understandings of mechanisms in operation in the real world)
As another point, is there a tendency amongst those both within the modelling community and particularly those taking their outputs and interpretting them to implicitly assume that ‘complex’ necessarily means ‘more correct’ where in fact all it is adding is further layers of uncertainty?
Of course, also as a geologist I wonder why there isn’t more investment in collection of real world data and observations, rather than an over-reliance on computer models, simulations and attempts to calculate or ‘correct’ data fundamentally collected for a different (if related) purpose?
Minor comment: Judy, you write “Elisabeth Lloyd nails it in this statement”. But the quote is fromWendy Parker (2009). Parker uses the notion of fitness for purpose in criticising Lloyd.
You are correct, thanks for spotting that! I will fix the text.
Judith Curry says : “What can we learn from climate models? Short answer: I’m not sure.”
Ian Blanchard asks a “naive” question: “Is the complexity of the GCMs helpful or a hinderance?” and wonders whether “‘complex’ necessarily means ‘more correct’ where in fact all it is adding is further layers of uncertainty?”
As a humble layman, I would ask any climate modelers out there, this simple question (in all seriousness): “Have the GCM’s produced any unique insights into the workings of the climate that: 1) have been empirically verified and 2) that could not have been deduced from the basic physics of greenhouse gases and a simple model (i.e. calculations set forth on a couple sheets of paper)
Paul, the pros and cons of simple vs complex models are discussed in several places, see this article by Petersen. See the wikipedia for a description of a range of climate models from very simple to highly complex.
Complex models are definitely needed to simulate the feedbacks and regional variations, and the interplay between natural and forced variations. But simple models can also provide important insights.
Pauld,
I strongly second that as far as point 1) and should like to know the answer. Point 2) is very climatic change centric, the models should inform us about the climate system irrespective of AGW which is an oddity in very long history of climate, but I take your point.
Alex
Judith,
Yet another great thread.
My modeling experience is limited, but my experience with complex and not fully understood systems is more extensive. It seems to me that the apparent overconfidence in model certainty and model predictions is a bit of a “pay no attention to the man behind the curtain” meme. The kludges/parameterizations that you talk about can (and I strongly suspect do) morph unintentionally into sophisticated curve fitting of the model to historical data. CGCM’s are always said to be reasonable representations of physical reality, not curve fit exercises, of course. But it is clear that models which diagnose high climate sensitivity also consistently presume that much (or even most!) radiative forcing due to well mixed greenhouse gases has been canceled by man-made atmospheric aerosols, while those that diagnose lower sensitivity presume lower aerosol off-sets of forcing.
The correlation is far too strong to be a coincidence, and this makes me honestly question how much credibility one can give to the predictions of the models, or even to their utility to provide useful information about climate processes, short or long term. There really has been only one historical evolution of man-made aerosols, even though this is unknown. The kludge of different assumed aerosol effects fairly well screams that most models must have important errors in how they represent the climate system.
Stephen Schwartz has described (in talks) atmospheric aerosols as the “pipe wrench” of climate science, which I presume means that you can use aerosols to apply about as much force as needed to make a model fit. I find this a fair critique.
I think that so long as there remains large uncertainty in the applied net forcing (mostly due to aerosol uncertainty) there will never be any good way to make a fair judgment of model performance, and no way to rationally discard a model which is a poor representation of reality… nor to rationally believe a model’s projections. I believe that public funding needs to be focused on generating improved climate data, and most importantly, focused on those areas (like aerosols) where uncertainty remains large. Progress in validating climate models seems to me to depend entirely on having much more certain forcing data. I find the focus on “improving” the models, in the face of vast uncertainty in the data, badly misplaced.
So, “What can be learn from climate models?” I think not very much right now.
For me, the clouds-aerosols issue is both perplexing and disconcerting.
Perplexing because it has been known to be very important since the very first days of investigations into the effects of the composition of the Earth’s atmosphere on radiative energy transport and yet to this day remains basically un-quantifiable. At that time, the effects of clouds were investigated, and found to be of first-order importance. I don’t know if other matter suspended in the atmosphere was considered back then. There’s lots of other matter in addition to the various phases of water in the atmosphere. Some of this matter interacts with the radiative energy transport aspects in inherently complex manner. While there are first-priniciple based models for some highly idealized situations, in general the interactions cannot be described from first principles. Additionally, the computational burden for the solution of the equations are such that first-priniciple models are not generally used in GCMs.
Disconcerting because these critically important aspects of radiative energy transport, with their large range of uncertainty and unknown-ness are used to tune the results of GCM applications when hindcasts are made. So, we have a situation in which a critically important aspect which has not yet been sufficiently investigated to narrow its uncertainties is none-the-less employed in attempts to ‘get the right answers’, you might say. If I recall correctly, the radiative energy transport parameterizations were changed only after initial hindcast calculations did not agree with data. I also understand that the actual state of the atmosphere during the period of the hindcast cannot be determined. Let me know if I’m wrong on these two points.
My experience has been that whenever a critically important aspect of a problem has been identified, efforts to narrow the ranges of uncertainty are moved to the top place in priority. After about 150 years it seems that little improvement has been attained. The characterizations of the physical problem in the modeling domain remain at the parameterization level. And, even with the fuzziness it is taken that it’s ok to tweak the parameterizations so as to tune the models. Under these conditions, getting the right answer for the wrong reasons seems highly probable.
This is not minimize the difficulties associated with the problem. Its an exceedingly difficult problem and part of an inherently complex physical situation.
Some of the aspects of the suspended matter are also important relative to modeling the initiation of precipitation. Some of the suspended matter act as condensation nuclei onto which water vapor condenses and the resulting droplets grow to form rain. It would be interesting to know how these two different aspects are handled in the models. Are the physical characteristics of the matter that are common to energy transport and condensation nuclei treated the same in the models. Or, are separate characterizations used; one for radiative energy transport and another for condensation nuclei?
Corrections for all incorrectos will be appreciated.
Perhaps an honest summary of knowledge gained through modeling of climate would be this:
“We have not succeeded in answering all our problems—indeed we sometimes feel we have not completely answered any of them. The answers we have found have only served to raise a whole set of new questions. In some ways we feel that we are as confused as ever, but we think we are confused on a higher level and about more important things. So this report does not purport to give final answers, or to claim that we now “know how to do it”. We see more need for revision than ever. But we are doing better than we did. And this is a progress report, rendered with humility because of the unsolved problems we see now which we could not see before.”
http://quoteinvestigator.com/2010/07/11/confused/
good statement
I am pretty sure that this statement originated with the Rand Coporation.
Dan Huges,
It looks like we have once again proved that great minds think alike… and often at the same time!
Steve Fitzspatrick says: “I think that so long as there remains large uncertainty in the applied net forcing (mostly due to aerosol uncertainty) there will never be any good way to make a fair judgment of model performance, and no way to rationally discard a model which is a poor representation of reality… nor to rationally believe a model’s projections. ”
Well stated. This fundemental point strikes me as an absolutely critical issue for policy makers to understand in evaluating the usefulness of climate models’ projections/simulations.
We need to teach climate models a lot more before they can teach us.
A question implicit in this thread is “how complex do GCMs need to be if they are to be skilful”? I think there is a case for looking through the other end of the telescope.
If we accept that the only truly authentic model we have of the climate is the climate itself, and if we also accept that constraints on computing power and imperfections in our understanding of its drivers will for the foreseeable future – probably forever – prevent us simulating it in its entirety, isn’t it reasonable to ask “Can a model as complex as the climate survive the REMOVAL of ANY complexity and remain skilful?” If not, then attempts (or at least publicly-funded attempts) to construct GCMs for predictive purposes (of any kind) should be discontinued. If it can, then how much complexity can safely be removed, is the complexity that remains susceptible of skilful modelling?
TomFP:
You make a good point concerning model complexity.
Climate modellers assume additional complexity increases the abilities of their models. And that assumption is false.
The draft of IPCC AR4 contained an assertion that increased model complexity had provided “Major improvements in the results”, but the draft did not present any comparison of model output with empirical data to substantiate the assertion.
My review comment of the assertion said:
Page 2-28 Chapter 2 Section 2.4.4 Line 5
Replace “Major progress over the results” with “Substantial developments of the models used to provide the results” because the statement is incorrect. Additional complexity of a model does not necessarily mean “Major improvements in the results”: it may result in the opposite. The draft Report provides no evidence that the additional model complexity has made any “progress” in the results they provide.
But my comment was ignored.
All my many review comments on the AR4 drafts that concerned climate modes were ignored. And the following is a selection of some of them
Richard
Page 1-2 Chapter 1 Executive Summary Lines 45 to 48
Replace all of from “The remarkable success ..” to “… fortunately impossible on the actual Earth.” with “Despite the limitations of these models, their use permits investigation of potential climate behaviours many of which are fortunately impossible on the actual Earth.”
This replacement is needed because the statements in Lines 45 to 48 in the draft grossly overstate the abilities of existing models and ignore the failures of existing models to emulate real climate in terms of, for example, spatial distributions of precipitation and temperature. Indeed, these statements conflict with Page 1-2 Lines 52 to 55 of the draft: why bother to “increase the trustworthiness and reliability of model predictions” if the models are sufficiently good that scientists can use them to “learn” how a modified Earth climate would behave?
Page 1-2 Chapter 1 Executive Summary Lines 50 and 51
Replace the sentence “The rapid development of … understanding of the climate system” with “The rapid development of increasingly sophisticated climate models permits testing of our understandings of climate and climate behaviours.” because the models are – and can only be – formulations of existing understandings. They can be used
1. to test those understandings against empirical data,
2. to explore the limitations of those understandings against empirical data,
3. and to assess the possible behaviours of the climate system according to those understandings
but that is very different from “contributing significantly to our understanding of the climate system”. The models are not the real world. They are merely simplified descriptions of part of the real world, and the climate system is the part of the real world they attempt to emulate.
Page 1-19 Chapter 1 Section 1.5.7 Line 56
For accuracy, between “wisdom” and “at least” insert “among climate modellers” because most scientists do not agree that anything is indicated by the average of two model results from models that are not validated for prediction of those results: average ignorance is not “wisdom”.
Page 1-20 Chapter 1 Section 1.5.7 Line 18
Replace “the scientific community” with “the climate modelling community” because the climate modellers and those who work with them are a very small part of the scientific community. (It may be useful to consider what Freud would have thought of this misuse of the phrase “the scientific community”).
Page 1-21 Chapter 1 Section 1.5.7 Line 6
For accuracy and completeness, after “… years to come.” it is very important to add, “However, it should always be kept in mind that the models provide results which are only valid to the degree that those results can be validated against empirical data from observation of the real climate system so, for example, all their predictions have great uncertainty for climate effects from elevated atmospheric greenhouse gas concentrations above those measured in reality.”
Page 1-23 Chapter 1 Section 1.5.9 Line 18
For accuracy and completeness, after “…Gates et al., 1996)” it is very important to add, “because this need for flux corrections indicated the models contained significant error(s) and/or omitted significant climate processes.”
Page 1-23 Chapter 1 Section 1.5.10 Line 55
For accuracy, after “… years to centuries” add “according to the understandings of the climate system built into the model” because the model is – and can only be – merely a representation of those understandings and not of the real climate system.
Page 1-24 Chapter 1 Section 1.5.10 Lines 9 and 10
For accuracy, replace the words “could not be explained by natural internal variability” with the words “could not be explained by their model within its limits of natural internal variability” because the study was of the model’s indication of climate behaviour and was not of the behaviour of the real climate. (The entire Section 1.5.10 seems to show that its authors have extreme confusion concerning the difference between model emulation and empirical observation of reality: this error is one example of the confusion).
Page 1-24 Chapter 1 Section 1.5.10 Lines 27 and 28
Delete the sentence, “The evidence from this body of work strengthens the scientific case for a discernible influence on global climate” because it is simply not true. The sentence is an assertion that derives from the extreme confusion concerning the difference between model emulation and empirical observation of reality that is repeatedly displayed in this chapter. The models’ results are not reality: they are merely the outcome of understandings of reality that are built into the models. Studies of the models’ results show the behaviour of the models, and studies of the real climate system show the behaviour of the real climate system. Differences between findings of these studies inform about the models and not the climate system, because the climate system is reality and the results of the model emulations are merely virtual realities. Also, the failure of a virtual reality to match reality without inclusion of an effect in the virtual reality indicates nothing concerning the existence of any particular postulated effect. So, the difference between a model’s results and observed reality informs about the model, and this difference is not “evidence” for the existence or otherwise of any postulated effect – for example, anthropogenic global warming – in the real climate system. (If the authors of the chapter cannot grasp this simple point then they should consider the following. Computer models based on fundamental physical laws can very accurately emulate the behaviours of battling spaceships, but this cannot provide any “evidence” for the existence of alien monsters in real space.)
Page 2-3 Chapter 2 Line 17
Replace the phrase, “give high confidence” with “agree with the suggestion” because the statement in the draft title is simply not true. The sentence is an assertion that derives from the extreme confusion concerning the difference between model emulation and empirical observation of reality that is repeatedly displayed in this chapter. The models’ results are not reality: they are merely the outcome of understandings of reality that are built into the models. Studies of the models’ results show the behaviour of the models, and studies of the real climate system show the behaviour of the real climate system. Differences between findings of these studies inform about the models and not the climate system, because the climate system is reality and the results of the model emulations are merely virtual realities. So, model predictions do not – and cannot – give confidence about anything that occurs in the real climate system. The model predictions can only indicate the confidence that can be applied to the understandings built into the models because agreement of those predictions with empirical data obtained from the real climate system can provide confidence that those understandings are correct.
Page 2-5 Chapter 2 Lines 10 to 12
Delete everything on lines 10 to 12 because they are complete and utter nonsense. If “the scientific understanding is low” then a “best estimate” has little if any value. And no estimate provided by computer models of the climate system has any value of any kind when “the scientific understanding is low” because the models are merely formulations of understandings of the climate system.
Page 2-47 Chapter 2 Section 2.6.3 Line 46
Delete the phrase, “and a physical model” because it is a falsehood.
Evidence says what it says, and construction of a physical model is irrelevant to that in any real science.
The authors of this draft Report seem to have an extreme prejudice in favour of models (some parts of the Report seem to assert that climate obeys what the models say; e.g. Page 2-47 Chapter 2 Section 2.6.3 Lines 33 and 34), and this phrase that needs deletion is an example of the prejudice. Evidence is the result of empirical observation of reality. Hypotheses are ideas based on the evidence. Theories are hypotheses that have repeatedly been tested by comparison with evidence and have withstood all the tests. Models are representations of the hypotheses and theories. Outputs of the models can be used as evidence only when the output data is demonstrated to accurately represent reality. If a model output disagrees with the available evidence then this indicates fault in the model, and this indication remains true until the evidence is shown to be wrong.
This draft Report repeatedly demonstrates that its authors do not understand these matters. So, I provide the following analogy to help them. If they can comprehend the analogy then they may achieve graduate standard in their science practice.
A scientist discovers a new species.
1. He/she names it (e.g. he/she calls it a gazelle) and describes it (e.g. a gazelle has a leg in each corner).
2. He/she observes that gazelles leap. (n.b. the muscles, ligaments etc. that enable gazelles to leap are not known, do not need to be discovered, and do not need to be modelled to observe that gazelles leap. The observation is evidence.)
3. Gazelles are observed to always leap when a predator is near. (This observation is also evidence.)
4. From (3) it can be deduced that gazelles leap in response to the presence of a predator.
5. n.b. The gazelle’s internal body structure and central nervous system do not need to be studied, known or modelled for the conclusion in (4) that “gazelles leap when a predator is near” to be valid. Indeed, study of a gazelle’s internal body structure and central nervous system may never reveal that, and such a model may take decades to construct following achievement of the conclusion from the evidence.
I tried to choose a word which summarises your post, Richard, but I’m stuck between “withering” and “punishing”.
Suffice it to say that I agree with every syllable in your post and reviewer’s comments. One does not need to be a modeller to see the impeccable rationality of your words.
That your review comments were ignored is damning.
Thanks, Richard. A couple of years ago I started asking questions about the science on which CAGW is founded – your post answers most of them.
Now if you can just explain why an ill-educated schmuck like me can see this stuff, and half the western world can’t…..
I really do feel the inverse question has to be addressed:
What can climate models learn from us?
If we engage in a process of training highly complex models to exihibt certain prefered characterisitcs then we must consider the behaviour of a bigger system, the naked model and its educators, and how that system is worked on by external agencies.
I believe that there is at least anecdotal support for the concept that when you attempt to teach complex systems by entraining their behaviour you can never be really sure what you have taught them. One may think that their behaviour has improved but it may have become pathological. Test such a system with the scenarios that it has been conditioned to and it will perform according to its conditioning. Give it some new scenarios and its behaviour may be perverse.
There are anecdotes of such behaviour with regard to training neural networks, also when training dogs, and children.
In some cases, what we have taught the system is how to be devious, how to game its instructor; certainly true of dogs and children in my view.
Given the extended climate model (computer, programmers, theorists, operators, supervisors, tweakers etc.) one could consider how such a system is likley to behave once subject to perturbation by powerful external agencies. Are there tendencies towards deviation? Will the system start to game its outside agencies? Will it become pathological and unfit for more general purpose? Well maybe. At any event if the outside agencies impose agenda with political or merely expedient imperatives then the possibility that these will give rise to deliberate or unintended gaming, is at least a plausible scenerio.
Perhaps the simplest significant example is a tendency to minimise variance through over-fitting. This is well understood, at least in principle. For example, Multivariate Regression, exhibits a form of gaming in that it seeks to construct the best fit vector by optimising the weightings of the component vectors no matter how bizarre the weightings.
In more complex systems gaming may not be quite so obvious, for instance how would one know whether some subtle adjustment to a Cloud Parameterisation Scheme is in fact a gaming response to correct for a weakness in the Surface Exchange Scheme. If it is then one has “balloon squeezing”; the overall competence of the system has not been improved but merely redirected to meet certain external directives.
I think that is enough to get my point across.
Alex
Someone wrote re: more funding for software development:
“But actually, unlike many of the sceptics, the modellers already have funding. Shouldn’t the discussion be about how effectively they are using the resources given to them? Surely climate science modellers should be complying with expected standards as a given?”
..which tells me that someone does not have experience with how science gets funded (in the USA at least), and believes that grant authors have absolute control over what parts of their proposals get funded…and hence how resources get allocated.
Here again is a place where input from actual grant-writing climate scientists would be illuminating. ..scolding from commercial software developers, somewhat less so.
Actually, in the U.S. (and I suspect elsewhere), the large climate modelling efforts are funded by block funding in fairly large amounts: NCAR (NSF) and GFDL (NOAA). So this doesn’t depend on the vagaries of the peer reviewed process of applying for individual grants
It’s the responsibility of those doing the work, and doing the grant asking, to ensure that sufficient funding is available to cover all important activities.
Been there, done that.
Numerical climate models should be used to learn what is poorly understood in the conceptual climate model and identify key data gaps. This should drive data collection, conceptual model revision and update the numerical models.
Repeat, repeat, repeat
RealClimate’s Rasmus Benestad reviews
A Vast Machine: Computer Models, Climate Data, and The Politics of Global Warming , by Paul N Edwards
http://physicsworld.com/cws/article/indepth/43625
Sample quotes:
“I particularly liked the way that Edwards deliberately breaks down old barriers by blurring the difference between data and models. One common misconception about climate science is that conclusions drawn from “data” are more accurate than predictions based on “models”.”
Uh, Rasmus, I think you may be part of the problem here…. and I very much doubt R. Feynman would have agreed with you.
“Although some climate scientists have been criticized for being unwilling to share data with their critics (as exposed in the – misnamed – “Climategate” controversy), in fact both the meteorology and climate-research communities have long traditions of openness and data sharing.”
I believe Steve McIntyre (and others) would strongly disagree with this “open data-sharing” assertion.
Rasmus’s review brings to mind journalist Clive Crook’s comment,
“The climate-science establishment … seems entirely incapable of understanding, let alone repairing, the harm it has done to its own cause.”
http://www.theatlantic.com/politics/archive/2010/07/climategate-and-the-big-green-lie/59709
Best, Pete Tillman
[Note: I mistakenly posted this at the “Doubt” thread, & just discovered that Google couldn’t find my mispost. Nor could the internal search feature.]
May I extend and put into words, a notion that Dr Roy Clark has previously alluded to but has not since been entertained in model uncertainty discussion.
That is:
WE HAVE NOT TO DATE BEEN PRESENTED WITH ACTUAL SIMULATION COMPARISONS BETWEEN COMPETING CLIMATE DRIVER HYPOTHESES AND COMBINATIONS OF SUCH.
Where are the ensembles that are pitted against AGW-centric simulations?
The PCMDI project that supposedly makes model inter-comparisons is a massive group-think exercise and somewhat incestuous.
The IPCC’s assertion that: well, we took out CO2 forcing and ran 15 simulations on 5 different models using natural forcing only (Lean solar) with OUR RF methodology and the simulations failed to mimic 90’s warming, JUST DOES NOT STAND UP TO SCRUTINY.
Both the IPCC’s ACO2 forced AND the naturally forced simulations, failed to mimic the 1930’s warming AND the ACO2 forced simulations are now diverging from the observed condition (points of inflexion across several metrics in the mid 2000’s).
Where are the models that mimic observable natural phenomena?
e.g.
CO2 fails to account for the 1930′s warming but sunspot cycle length correlates with temperature over the entire warming period:
http://www.appinsys.com/GlobalWarming/GW_Part6_SolarEvidence_files/image013.jpg
CO2 fails to correlate with Arctic-wide Surface Air Temperature anomalies:
http://www.appinsys.com/GlobalWarming/GW_Part6_SolarEvidence_files/image024.gif
But solar irradiance does:
http://www.appinsys.com/GlobalWarming/GW_Part6_SolarEvidence_files/image023.gif
And then there’s the reduction in early 90’s planet albedo.
It is specious to be addressing model uncertainty, when the rest of the competition has not been evaluated, and does it even exist?
What is the state-of-play in natural forcings modeling work?
Is there an umbrella group like WCRP, SPARC, PCMDI or CMIP5? Any direction or link in this regard would help me considerably. Where do I look for extensive analysis (to the same degree as AGW-centric) of natural forcings simulations that do actually attempt to mimic the above examples?
Richard, the issue of model forcings and the 20th century attribution by climate models will be addressed in a future post (two weeks)
Again, fascinating! The last time I came across ‘ontic uncertainties’ was whilst reading Heidegger!
http://journals.ametsoc.org/doi/abs/10.1175/2010JCLI3814.1
This paper in press seems relevant to this discussion.
HR, thanks for catching this, I can use this paper in an upcoming post
Given that we lack verifiability of the models ability to measure impact of CO2 change (since we can’t replay Earth), AND the phenoma are relatively complex and coled (much more complex than designing a structure and doing mech e calcualations for instance), we end up needing to have some “feel” for how effecitve the models are likely to be.
I remember asking Ed Z about this and a little bit challenging the use of dynamic models. For instance, if we wanted to understand boiling water, we would use a Molllier chart, not some kinematic model of the molecules running around. We would have some abstraction (stat mech or classical). Ed (who is a condensed matter “jock”) said , well Poly there are no abstractions of climate system the way there are for a pressure vessel of water. But then you have Annan saying, “just because the models don’t predict weather after a few weeks, that doesn’t mean they can’t tell climate. Further, Ed thinks transport is important, thus dynmaical modesl are the way to go.
But I end up wondering, if they are really useful. In parrticular, I wonder if the “fluid dynamics loverz” are the cause of us using these hyper complicated models. IOW, it is a bit of a stylistic preference. I end up wondering why having time-dependant features (that we KNOW) don’t reflect reality and having area resoltuion wich we pretty well know allso doesn’t match reality…..why is the model better for having those things that don’t really even “work”.
I can understand more simple scalar modeling arraments. For instance, we can try modeling how temp drives more H20 into the air and then what effect that has. Or how a temp change affects the albdeo by melting some icecap. IOW, estimating the positive feedbacks.
Of course, simpler EBMs won’t detect strange transport changes or the like. But wdo we really have any feel tha gCMs will?
Even for transport questions, I wonder if we are better off with simpler examinations (for instance looking at how a different delta T drives a different flow of water). And not sure that a GCM is the way to really best do that. Maybe something where you drill down more to the problem at hand (of course you end up needing to have boundaries and aslo maybpe figure out a way to deal with seasonas and the like, but still maybe a tradeoff workth making).
Sorry the criti is unfocused. But that’s all I can do.
IMO, a hierarchy of different model types and complexity is needed to sort all these issues out and understand them. Trying to make sense of the complex global climate models is the topic of my next post (hopefully up later tonite).
A mathematical model of boiling water is an exceedingly difficult problem. It is an exceedingly difficult problem for the same reasons that climate modeling is equally difficult. Both involve many coupled physical phenomena and processes each of which is inherently complex. That is, the total problem is not simply complex because several interacting phenomena and processes are involved. Instead, each individual phenomenon and process is inherently complex and their interactions, in many cases, equally inherently complex.
Just as climate modeling involves multi-physics, multi-fluids, and multi-scale phenomena and processes so does boiling water. Equally important, for some applications, complete descriptions of all the various scales are required of the model equations and they must be resolved in the solutions of the model equations.
While almost all useful models do not yet involve description at the molecular level, there are models of boiling water based on the Boltzmann equation. Modeling and tracking the individual regions of liquid and vapor, and the associated interfaces, is SOP for some, limited applications. Not a trivial problem.
The Mollier diagram is a projection of the equilibrium thermodynamic equation of state of the phases of water onto the enthalpy-entropy ( h-s ) plane. Such projections can be made for any material and can extend far beyond the usual solid, liquid,vapor phases. Mollier and water and h-s are linked to the projection due to historical reasons. Mollier because he was the first, water because it was then and remains to this day the most prevalent working fluid, and h-s because the states of a working fluid as it flows through engineering equipment, or under-goes processes in the natural environment, are easily displayed on this plane. The h-s plane is especially useful for isentropic processes, highly idealized processes in other words.
As it represents only a useful projection of the equation of state, and the particular projection is of limited general usefulness, the Mollier chart, or any other projection, says nothing, absolutely nothing, about the physical phenomena and processes that the fluid experiences. Just as in the case of climate modeling, many situations of interest associated with boiling water require that the local-instantneous physical phenomena and processes occurring within the fluid and between the fluid and any associated equipment and between the regions occupied by liquid and vapor phases and at the liquid-vapor interface be modeled and resolved.
These phenomena and processes are not represented by the Mollier chart for water just as phenomena and processes in climate modeling are not represented by the Mollier chart, or any other projection, for any of the fluids of interest in climate.
Another thing is was wondering is how adding complexity affects the impact of CO2. IOW is there some simple ranking we could make of different omplexity models and then see how adding more features (I’m hung up on the dynamic thing, but I’m sure there is more)…how that effects the CO2 doubling impact. IOW, does more complexity which hopefully is closer to reality, does it tend to drive you in the direction of a lower or a higher CO2 impact?
I just think that would be interesting to know. But then I really worry that there are so many signifcant tuning parameters remaining in even the most complex modesl that it is not a simple isuee of adding complextity. Even the biggest modes complex GCMs still have a lot of “play in them, particularly wrt aerosol tuning. So could we even try what I want to know?
I think if I were working in the area of modelling, I’d be very worried about sheer bugs. I noticed that some of the software released from the CRU, was written in Fortran – which is notorious for the kind of errors that are possible. In particular, unless Fortran code is compiled in a special mode that prevents array access errors (such as referring to the eleventh element of a 10-element vector) programs can execute totally invalid code.
Many years ago, I was surprised to discover that scientists are not always aware of the problems this can produce. I was shown a fairly large monte carlo calculation that was crucial to interpreting an accelerator experiment. Apparently this program would abort from time to time (presumably due to an array access error), but this was handled by discarding the aborted run, and the next run – with different random numbers seeded from the date/time – would normally run OK!
Even with a modern object oriented language, there are many, many subtle traps that produce totally invalid results. For example, if you have many objects, all interacting, it can be hard to predict in what order the calculations will be done – so for example a calculation may get done with an obsolete value of X because X is yet to be re-calculated.
I don’t think program proving is much use for serious sized programs, I used to work on compilers, and we had a suite over over 2000 test programs that we would run through the compiler to check it out, and even so some errors got through.
Compilers and operating systems are complex, but at least there are good ways to check what is going on, whereas with complex numerical models, this is much less obvious – mistakes can just get averaged out!
Bugs of the kind I am discussing are obviously in addition to issues of chaos, numerical instability, etc. I’m not sure I believe any result based on large computer models!
Judith
(The question and comment below may have been addressed already in some of the 200 comments following your very informative post).
Are you aware of any studies by mathematicians addressing boundaries of complexity in climate models?
My understanding (as a non-mathematician) is that applying methods of Computational Complexity Theory in the design of highly complex systems, limits in complexity can be predicted beyond which problems are no longer computable.
http://en.wikipedia.org/wiki/Computational_complexity_theory
How accurate can a climate model become assuming availability of unlimited resources and computational capacities to cope with the highest computable complexity?
Federico, really interesting question that I have no answer for, and thanks for introducing me to computational complexity theory, this seems really relevant.
Hi Judith
Please delete my unfinished previous post.
Enjoyed your post and just pondering upon the question you have set.
One approach to answering the question is looking at the details of what models are good at doing in general and specifically in the context of climate change and the questions they are addressing.
Another possible approach, and not the only one, could be to explore the relationship of this question to other questions. For example: What are the questions we needs answers for within the climate change arena? What are the tools we can use to answer them? Where do climate models fit in with respect to these other tools? How will this relationship change over time as our knowledge matures?
Dr. Curry, excellent evenhanded post and responses – I finally had time to read through.
Couldn’t agree more.
==============