by Judith Curry
I am starting to see some encouraging signs that people (including the IPCC) are paying more attention to the uncertainty issue as it relates to climate change. Nature has an editorial on this issue that summarizes the situation as:
IPCC members last week considered the best way to quantify uncertainty. They are not alone in needing to do so — the media must also take a firm line when it comes to scientific reporting.
The public seems to want and expect an acknowledgement of uncertainty. If there is anything to be gleaned from the silly Scientific American survey, it is that uncertainty scores big:
1. Should climate scientists discuss scientific uncertainty in mainstream forums?
- No, that would play into the hands of the fossil fuel lobby: 3.0%
- Yes, it would help engage the citizenry: 90.3%
- Maybe, but only in serious venues: 6.7%
Well, this survey has attracted a self-selecting sample. But for reference, 34% of the respondents said we should do something about climate change (the implication being that a majority of these respondents also voted for uncertainty). Only half of these respondents (17.5%) thought the IPCC was an effective group of government representatives, scientists, and other experts (with 81.8% thinking it was a corrupt organization prone to groupthink with a political agenda). My interpretation of this is that besides being an essential element of science, acknowledging uncertainty increases the public credibility of the science and the scientists. Overconfidence comes across as selling snake oil.
I would like to think that my arguments are helping to elevate this issue into importance, but you can never be sure how many people are actually reading the blog (I’ve no idea how to relate individuals to number of blog hits) and who is actually reading the blog other than the people that post comments. Well, at least one person on the “Hill” has been reading the blog, a congressional staffer who contacted me about possibly testifying in a possible hearing (more on that if/when it materializes).
From my perspective, the most interesting (and potentially important) development on the uncertainty front is the preparation of a special issue in the journal Climatic Change (founding editor Steve Schneider) entitled Framing and Communicating Uncertainty and Confidence Judgments by the IPCC. I have been invited to submit an article, and I have accepted.
So I am submitting an article to a mainstream journal criticizing the IPCC (in my invitation, it was noted that they expect me to submit a critical paper). I view this as strong evidence that I am not an apostate, by working within the system to try to change the system. But perhaps my assertion should be checked by Michael Tobis with an Italian flag analysis, supervised by Keith Kloor.
This thread is an opportunity to extract and summarize the highlights of previous uncertainty threads, and make some suggestions for what I might include in the paper (and bring newcomers to Climate Etc. up to speed on what has been our major topic of discussion).
IPCC’s foundation for characterizing and communicating uncertainties and confidence levels is described by Moss and Schneider (2000). The “Guidance Paper” by Moss and Schneider recommended steps for assessing uncertainty in the IPCC Assessment Reports and a common vocabulary to express quantitative levels of confidence based on the amount of evidence (number of sources of information) and the degree of agreement (consensus) among experts.
The actual implementation of this guidance in the AR3 and AR4 WG1 Reports focused more on communicating uncertainty rather than on characterizing it (e.g. Peterson, 2006), adopting “judgmental estimates of confidence” whereby a single term (e.g. “very likely”) characterizes the overall confidence. Since physical scientists generally prefer to consider uncertainty in objective terms, why did the WG1 authors choose the subjective perspective, or judgmental estimates of confidence, and focus mainly on communicating uncertainty rather than evaluating it? Peterson (2006) suggests that lack of time given other competing priorities and the simplicity of using a scale in a table rather than a more thorough examination of uncertainties enabled the assessors to continue with only a minimal change to their usual working methods. Defenders of the IPCC uncertainty characterization argue that subjective consensus expressed using simple terms is more easily understood by policy makers.
The recommendations on characterizing and communicating uncertainties made by the IAC that recently reviewed the IPCC were:
“All Working Groups should use the qualitative level-of-understanding scale in their Summary for Policymakers and Technical Summary, as suggested in IPCC’s uncertainty guidance for the Fourth Assessment Report. This scale may be supplemented by a quantitative probability scale, if appropriate.”
“Chapter Lead Authors should provide a traceable account of how they arrived at their ratings for level of scientific understanding and likelihood that an outcome will occur.”
“Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event (e.g., based on measurement, expert judgment, and/or model runs).”
“The confidence scale should not be used to assign subjective probabilities to ill-defined outcomes.”
“The likelihood scale should be stated in terms of probabilities (numbers) in addition to words to improve understanding of uncertainty.”
“Where practical, formal expert elicitation procedures should be used to obtain subjective probabilities for key results.”
At the recent IPCC meeting in Busan, the IPCC prepared a response to the IAC recommendations, the relevant statement regarding the treatment of uncertainty and confidence levels is:
The Panel decided to improve the IPCC guidance on evaluation of evidence and treatment of uncertainty. It is implementing the six recommendations in the IAC Review as part of a broader package of updates to procedures and guidance notes. The Panel noted with appreciation the Draft Guidance Note for Lead Authors of the Fifth Assessment Report on Consistent Treatment of Uncertainties (Appendix 4) and requested the Co-Chairs of Workings Group I, II and III to present the final document to the Panel at its next Session. The final document should provide more detail on traceable accounts, the evolution of the guidance since AR4 and explain how each of the six recommendations in the IAC review is addressed. The Panel urges the Co-Chairs to take any necessary steps to ensure that the guidance note is implemented in the development of its work.
In Appendix 4 to this document, further clarity is provided on their proposed methods for treating uncertainty in the AR5.
Well, these are steps in the right direction, but barely scratch the surface of issues that I and others have raised in Climate Etc.’s series of uncertainty threads:
- The uncertainty monster
- No consensus on consensus
- What can we learn from climate models?
- The culture of building confidence in climate models
- Do IPCC’s scenarios fail to comply with the precautionary principle?
- Overconfidence in IPCC’s detection and attribution. Parts I, II, III
Special issue in Climatic Change
The editors have invited papers on the topics of assessing the treatment of uncertainty in previous IPCC reports, the IAC critique, the new guidelines established for the AR5 reports, the social psychology of communicating and understanding uncertainty, how information on uncertainty is used in the policy process, perspectives from critics of the IPCC (i.e. me), other approaches for communicating uncertainty, perspectives from the user community, and the defense/security community perspective on exploring uncertainty.
Well, it still seems like the emphasis remains on communicating uncertainty rather than actually understanding, characterizing and reasoning about it. The perspective from the defense/security community should be really interesting; what a novel idea for the IPCC that uncertainty should be exploring uncertainty.
My article has a word limit of 3000 words. I’ve already written well over 10,000 words at Climate Etc. on the subject, and I’m just getting started. I would appreciate your thoughts on ideas about what should be included in my paper, including arguments that you thought were effective (or not) in the previous uncertainty threads. Your help in identifying the most important comments in the previous threads (made by yourself or by someone else) would be most appreciated.
Moderation note: please keep this issue focused on the broader issues of uncertainty and the IPCC. Discussion about WHAT we are uncertain about (e.g. the temperature record, climate models, whatever) should be conducted at the Disagreement thread or a previous Open Thread.