by Judith Curry
My paper “Reasoning about climate uncertainty” has now been published online at Climatic Change; it looks like mine is the first to make it online of the papers in the special issue entitled Framing and Communicating Uncertainty and Confidence Judgments by the IPCC.
Also of relevance, there is a new working paper from the LSE Grantham Research Institute entitled “Scientific uncertainty: a user’s guide” (h/t Bishop Hill).
Reasoning about climate uncertainty
Judith Curry
Received: 1 April 2011 / Accepted: 14 June 2011 # The Author(s) 2011.
Climatic Change DOI 10.1007/s10584-011-0180-z
This article is published with open access at Springerlink.com
Abstract. This paper argues that the IPCC has oversimplified the issue of uncertainty in its Assessment Reports, which can lead to misleading overconfidence. A concerted effort by the IPCC is needed to identify better ways of framing the climate change problem, explore and characterize uncertainty, reason about uncertainty in the context of evidence-based logical hierarchies, and eliminate bias from the consensus building process itself.
You may recall the previous thread here where I posted an earlier draft of the paper for comments. I GREATLY appreciate the comments that I received on this thread, and in my acknowledgements I made the following statement:
Acknowledgements. I would like to acknowledge the contributions of the Denizens of my blog Climate Etc. (judithcurry.com) for their insightful comments and discussions on the numerous uncertainty threads.
Its always interesting to go back and read your own paper after not thinking about it for a few months. I’m still pretty happy with it, but upon rereading section 2, it seems to have gotten a bit muddled in all the editing.
Once the special issue is published, we will have a thread discussing the other papers.
Scientific uncertainty: a user’s guide
Seamus Bradley
Abstract. There are different kinds of uncertainty. I outline some of the various ways that uncertainty enters science, focusing on uncertainty in climate science and weather prediction. I then show how we cope with some of these sources of error through sophisticated modelling techniques. I show how we maintain confidence in the face of error.
Contents
1 Two motivating quotes . . . . . . . . . . . . . . . . . . . . . . . 5
1.1 Laplace’s demon: ideal scientist . . . . . . . . . . . . . . 5
1.2 On Exactitude in Science . . . . . . . . . . . . . . . . . . 6
1.3 Characterisations of uncertainty . . . . . . . . . . . . . . 7
1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Data gathering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1 Truncated data: imprecision . . . . . . . . . . . . . . . . 9
2.2 Noisy data: inaccuracy . . . . . . . . . . . . . . . . . . . 9
2.3 Deeper errors . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Meaningfulness of derived quantities . . . . . . . . . . . 11
3 Model building . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1 A toy example . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Curve fitting . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Structure error . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Missing physics . . . . . . . . . . . . . . . . . . . . . . . 15
3.5 Overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.6 Discretisation . . . . . . . . . . . . . . . . . . . . . . . . 17
3.7 Model resolution . . . . . . . . . . . . . . . . . . . . . . 18
3.8 Implementation . . . . . . . . . . . . . . . . . . . . . . . 18
4 Coping with uncertainty . . . . . . . . . . . . . . . . . . . . . . 19
4.1 Make better measurements! . . . . . . . . . . . . . . . . 19
4.2 Derivation from theory . . . . . . . . . . . . . . . . . . . 20
4.3 Interval predictions . . . . . . . . . . . . . . . . . . . . . 22
4.4 Ensemble forecasting . . . . . . . . . . . . . . . . . . . . 23
4.5 Training and evaluation . . . . . . . . . . . . . . . . . . 25
4.6 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.7 Past success . . . . . . . . . . . . . . . . . . . . . . . . . 29
5 Realism and the “True” model of the world . . . . . . . . . . . 29
6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
IMO the chapters on Model building and Coping with uncertainty are the most interesting. There are aspects of the coping with uncertainty chapter that I don’t agree with, particularly the section on robustness. A main theme of the paper is “ways we have of maintaining confidence in our predictions despite these errors.” I don’t think the paper makes a very convincing case for this.
New paper by Jonassen and Pielke Jr
Just after posting this, I spotted this Pielke Jr’s blog
Jonassen, R. and R. Pielke, Jr., 2011. Improving conveyance of uncertainties in the findings of the IPCC, Climatic Change, 9 August, 0165-0009:1-9, http://dx.doi.org/10.1007/s10584-011-0185-7.
Abstract. Authors of the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) received guidance on reporting understanding, certainty and/or confidence in findings using a common language, to better communicate with decision makers. However, a review of the IPCC conducted by the InterAcademy Council (2010) found that “the guidance was not consistently followed in AR4, leading to unnecessary errors . . . the guidance was often applied to statements that are so vague they cannot be falsified. In these cases the impression was often left, quite incorrectly, that a substantive finding was being presented.” Our comprehensive and quantitative analysis of findings and associated uncertainty in the AR4 supports the IAC findings and suggests opportunities for improvement in future assessments.
Some excerpts:
If we confine our attention to those findings that refer to the future, one can ask how many IPCC findings can be expected to become verified ultimately as being accurate? For example, if we consider findings that refer to future events with likelihood in the ‘likely’ class (i.e., >66% likelihood) then if these judgments are well calibrated then it would be appropriate to conclude that as many as a third can be expected to not occur. More generally, of the 360 findings reported in the full text of WG1 across all likelihood categories and presented with associated measures of likelihood (i.e., those summarized in Table 2 below), then based on the judgments of likelihood associated with each statement we should logically expect that about 100 of these findings (~28%) will at some point be overturned.
Although the IPCC has made enormous contributions and set an important example for global assessment of a vexing problem of immense ramifications, there remain clear opportunities for improvement in documenting findings and specifying uncertainties. We recommend more care in the definition and determination of uncertainty, more clarity in identifying and presenting findings and a more systematic approach in the entire process, especially from assessment to assessment. We also suggest an independent, dedicated group to monitor the process, evaluate findings as they are presented and track their fate. This would include tracking the relationship of findings and attendant uncertainties that pass up the hierarchy of documents within AR5. Strict rules for expressing uncertainty in findings that are derived from (possibly multiple) other findings are needed (see, e.g., the second example in the Supplementary Material).
It is not the purpose of this note to discuss other, related scientific assessments of climate change knowledge; but, we do note that our preliminary analysis of the U.S. Global Change Research Program Synthesis and Assessment Products suggests a far less systematic application of the guidance supplied to authors of those documents and far less consistent application of the defined terms. We believe that the concerns we have expressed here, and the resulting recommendations, apply more broadly than the IPCC process.
You can find the full text here. The full dataset of IPCC findings is online here.