by Judith Curry
Two previous threads (here and here) have presented sections of my draft paper on Climate Science and the Uncertainty Monster. Here is an additional section on Uncertainty and the IPCC.
Climate Science and the Uncertainty Monster
This paper is a very long one, so I elected not to try to discuss the paper all in one post but to present sections. Further, most of the material is not new, I have developed these arguments on previous uncertainty threads. Once the paper is in press, I will post the entire thing for discussion.
1. Introduction (draws mainly from my original post on the uncertainty monster)
Sidebar: Uncertainty lexicon (with material from the original uncertainty monster post)
2. Uncertainty of Climate Models (draws heavily from the post what can we learn from climate models?)
3. Uncertainty and the IPCC (see below)
4. Uncertainty in the Attribution of 20th Century Climate Change
5. Taming the Uncertainty Monster
3. Uncertainty and the IPCC
“You are so convinced that you believe only what you believe that you believe, that you remain utterly blind to what you really believe without believing you believe it.” Orson Scott Card, Shadow of the Hegemon
How to reason about uncertainties in the complex climate system and its computer simulations is not simple or obvious. Scientific debates involve controversies over the value and importance of particular classes of evidence as well as disagreement about the appropriate logical framework for linking and assessing the evidence. The IPCC faces a daunting challenge with regards to characterizing and reasoning about uncertainty, by assessing the quality of evidence, linking the evidence into arguments, identifying areas of ignorance, characterizing uncertainty and assessing confidence levels.
3.1 Characterizing uncertainty
“A long time ago a bunch of people reached a general consensus as to what’s real and what’s not and most of us have been going along with it ever since.” Charles de Lint
Over the course of four Assessment Reports, the IPCC has given increasing attention to reporting uncertainties (e.g. Swart et al. 2009). The “Guidance Paper” by Moss and Schneider (2000) recommended steps for assessing uncertainty in the IPCC Assessment Reports and a common vocabulary to express quantitative levels of confidence based on the amount of evidence (number of sources of information) and the degree of agreement (consensus) among experts.
The IPCC guidance for characterizing uncertainty for the AR4 describes three approaches for indicating confidence in a particular result and/or that the likelihood that a particular conclusion is correct:
1. A qualitative level-of-understanding scale describes the level of scientific understanding in terms of the amount of evidence available and the degree of agreement among experts. There can be limited, medium, or much evidence, and agreement can be low, medium, or high.
2. A quantitative confidence scale estimates the level of confidence for a scientific finding, and ranges from ‘very high confidence’ (9 in 10 chance) to ‘very low confidence’ (less than 1 in 10 chance).
3. A quantitative likelihood scale represents ‘a probabilistic assessment of some well-defined outcome having occurred or occurring in the future.’ The scale ranges from ‘virtually certain’ (greater than 99% probability) to ‘exceptionally unlikely’ (less than 1% probability).
Oppenheimer et al. (2007), Webster (2009), Petersen (2006), and Kandlikar et al. (2005) argue that future IPCC efforts need to be more thorough about describing sources and types of uncertainty, making the uncertainty analysis as transparent as possible. The InterAcademy Council (IAC) reviewed the IPCC’s performance on characterizing uncertainty. In response to concerns raised in the review, the IAC made the following recommendations regarding the IPCC’s treatment of uncertainty:
- “Each Working Group should use the qualitative level-of-understanding scale in its Summary for Policymakers and Technical Summary, as suggested in IPCC’s uncertainty guidance for the Fourth Assessment.” This is a key element of uncertainty monster detection.
- “Chapter Lead Authors should provide a traceable account of how they arrived at their ratings for level of scientific understanding and likelihood that an outcome will occur.” Failure to provide a traceable account is a symptom of uncertainty monster hiding.
- “Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event (e.g., based on measurement, expert judgment, and/or model runs).” Using quantitative probabilities when there is insufficient evidence is uncertainty monster simplification.
The recommendations made by the IAC concerning the IPCC’s characterization of uncertainty are steps in the right direction in terms of dealing with the uncertainty monster. Curry (2011a) argued that a concerted effort by the IPCC is needed to identify better ways of framing the climate change problem, explore and characterize uncertainty, reason about uncertainty in the context of evidence-based logical hierarchies, and eliminate bias from the consensus building process itself.
3.2 Reasoning about uncertainty
“It is not so much that people hate uncertainty, but rather that they hate losing.” Amos Tversky
Many of the key conclusions from the IPCC AR4 WGI Report are quantitative assessments of likelihood. The IPCC characterization of likelihood is based upon a consensus building process that is an exercise in collective judgment in areas of uncertain knowledge. The general reasoning underlying the IPCC’s arguments for anthropogenic climate change combines a compilation of evidence with subjective Bayesian reasoning. This process is described by Oreskes (2007) as presenting a ‘consilience of evidence’ argument, which consists of independent lines of evidence that are explained by the same theoretical account. Oreskes draws an analogy for this approach with what happens in a legal case.
Given the complexity of the climate problem, expert judgments about uncertainty and confidence levels are made by the IPCC on issues that are dominated by unquantifiable uncertainties. Curry (2011a) argues that because of the complexity of the issues, individual experts use different mental models for evaluating the interconnected evidence. Biases can abound when reasoning and making judgments about such a complex problem. Bias can occur by excessive reliance on a particular piece of evidence, the presence of cognitive biases in heuristics, failure to account for indeterminacy and ignorance, and locial fallacies and errors including circular reasoning. The IAC states that “Studies suggest that informal elicitation measures, especially those designed to reach consensus, lead to different assessments of probabilities than formal measures. Informal procedures often result in probability distributions that place less weight in the tails of the distribution than formal elicitation methods, possibly understating the uncertainty associated with a given outcome.”
Oreskes (2007) draws an analogy for the consilience of evidence approach with what happens in a legal case. Continuing with the legal analogy, Johnston (2010) characterized the IPCC’s arguments as a legal brief, designed to persuade, in contrast to a legal memo that is intended to objectively assess both sides. Along the lines of a legal memo, Curry (2011a) argues that the consilience of evidence argument is not convincing unless it includes parallel evidence-based analyses for competing hypotheses, and hence a critical element in uncertainty monster detection. Any evidence-based argument that is more inclined to admit one type of evidence or argument rather than another tends to be biased. Parallel evidence-based analysis of competing hypotheses provides a framework whereby scientists with a plurality of viewpoints (including skeptics) participate in an assessment. In a Bayesian analysis with multiple lines of evidence, it is conceivable that there are multiple lines of evidence that produce a high confidence level for each of two opposing arguments, which is referred to as the ambiguity of competing certainties. If uncertainty and ignorance are acknowledged adequately, then the competing certainties disappear. Disagreement then becomes the basis for focusing research in a certain area, and so moves the science forward.
JC note: Curry (2011a) is part of a special issue on framing and communicating uncertainty for the IPCC, in Climatic Change. The electronic version of this issue should be published later this summer, with print version in Sept. This should be a very interesting issue.
No disagreement here.
In my paper in the same issue, I make the point that the IPCC has put a lot of effort into the internal consistency of the language to describe uncertainties. That is very useful for people who want to read the entire 3,000 page report.
Unfortunately, the IPCC has put less effort into external consistency, that is, reasonable estimates of the uncertainty about this or that.
AR3 had medium confidence that the economic impacts of initial warming would be beneficial. That was based on a single study.
AR4 had low confidence that the economic impacts of initial warming would be beneficial. Four additional studies had confirmed the first study referred to in AR3. No new study had reached the opposite conclusion.
AR3 had medium confidence that the economic impacts of initial warming would be beneficial. That was based on a single study.
AR4 had low confidence that the economic impacts of initial warming would be beneficial. Four additional studies had confirmed the first study referred to in AR3. No new study had reached the opposite conclusion.
Interesting…was any reason for the change in confidence level?
The IAC’s recommendation for a “traceable account of how they arrived at their ratings” is one that resonates with me. It is not only vital to present and deal with plausible contrary viewpoints, but also to provide some transparency into the process of how you came to your conclusions.
Doh!
That should read “was any reason given for the change in confidence level?”
When I asked them the first time, face-to-face, I was met with stunned silence: They had not even noticed the increase in evidence and the decrease in confidence.
The second time, they offered waffle.
Written by magic and sustained in secrecy.
=============
I should have noted that I was a lead author on this chapter for AR3, but not AR4.
I argued that a statement based on a single study could have no more than low confidence. I was adamant, thought, that it should be in the summary because it was the only substantive change in knowledge since AR2 (when all impacts were thought negative). The rule applied was that everyone’s favorite statement would get medium confidence (because anyone’s favorite was someone else’ pet hate — equal treatment was used to avoid retaliation).
How sad. Your reasoning on the appropriate confidence level for AR3 makes perfect sense and “showing your work” would have been better than applying the simplistic equal treatment rule. In light of the “stunned silence”, I’m guessing no one had compared the relevant section between the AR3 and AR4.
Gene, reflect, this was a critical point for policy.
==========
Understood. But even without delving into motive, it’s a pretty sad statement on the amateurish nature of the undertaking that they were caught completely unaware of the conflict between the two reports.
Oh my, what an insight into the “traceability” of these confidence levels
Political turtles, all the way down.
Certainty is the only weapon we have against uncertainty and certainty comes from what we can physically see or measure. We can measure global temperature within an acceptable degree of error and we can do the same with atmospheric CO2.
For the past nine years since 2002 HadleyCRU, GISS, and NCDC monthly temperature anomaly datasets show over 0.02°C of cooling to present day so we are certain that the world is currently cooling.
For the past nine years since 2002 the Mauna Loa Observatory CO2 data shows an average annual increase in atmospheric CO2 concentration of 2.095ppmv/year, so we are certain that atmospheric CO2 is currently increasing.
Since we are certain that the Earth is currently cooling and we are certain that CO2 is currently increasing; we are therefore certain that at least for the past nine years increased atmospheric CO2 has not caused global cooling.
For the previous nine years the global temperature increased by over 0.1°C and the atmospheric CO2 concentration grew at an average rate of 1.731ppmv/year. Over these nine years from 1993 to 2001 we can state the possibility that CO2 caused the increase in global temperature but we can only do so with a great degree of uncertainty because no causal relationship can be established from any direct physical evidence.
If we compare the current nine years to the previous nine years we can state with certainty that for half the time there is no possible causal relationship between CO2 and global warming and for the other half of the time we can only state with a high degree of uncertainty that there is such a relationship.
For the past fourteen years since the Kyoto Protocol on climate change came into existence there are five years of uncertainty about a causal relationship between CO2 and global warming and nine years of absolute certainty of no such relationship.
For the past ten years since the 2001 IPCC TAR there is only one year of uncertainty about a causal relationship between CO2 and global warming and nine years of absolute certainty of no such relationship.
For the past four years since the 2007 IPCC 4AR there are no years of uncertainty about a causal relationship between CO2 and global warming and nine years of absolute certainty of no such relationship.
With the current solar cycle 24 closely matching the onset of the Dalton Minimum that brought an extension of the Little Ice Age, and the current prognosis that solar cycle 25 may herald solar dormancy equivalent to the Maunder minimum that brought on the Little Ice age; the only uncertainty is the question of what possesses scientists to still promote the uncertain concept of AGW and support the Kyoto Accord dictates that have ruined the economy and killed off hundreds of thousands of the world’s poor by taking away their food and using it to make biofuels.
Dear Norm:
You are correct, and we can be certain in our climate projections for the cause of uncertainty is only temporary and caused by cycles whose effect vanish within seven years at most. These cycles include ENSO and volcanic events. However, in the long run, the trend is certainly a warming one that can be projected reasonably. Please see Earth’s Magic on my website http://www.global-heat.net.
Dear Nabil,
I started reading your writeup, and quickly ran into nonsense. I did not go much further, since you already made untrue statements. There were several, but this is not the place to edit your book, so I will just limit my comments to a couple of points. I do agree the addition of CO2 (and removal of equal amounts of O2) do increase the lapse rate the small amount you stated, although I got a change in M of 0.004%, not 0.005% for 100ppm. It was shortly after that you stated this changed the chemical properties of the atmosphere? What is that all about. You then clearly stated this would cause a reduction in condensation and an increase in water vapor, which then condenses and the latent heat of condensation condensation warms the Earth. This is nonsense,
and logically gibberish. You then say this extra water raining down, along with melting ice causes rising sea level. If most of the melting ice is sea ice, this does not cause the sea level to rise. Also, the evaporating water is from the sea, so rain just returns it. The water content in the entire atmosphere is small, so readjustment of that content would make little difference. Also, the major claimed feedback needed to increase the small direct CO2 effect is increase in total water vapor content in the atmosphere. In fact data shows little increase and not any decrease, so you
and the positive feedback theory are wrong. You go down hill from there so I won’t bother to expand further.
Since we are certain that the Earth is currently cooling and we are certain that CO2 is currently increasing; we are therefore certain that at least for the past nine years increased atmospheric CO2 has not caused global cooling.
ERROR should read:
Since we are certain that the Earth is currently cooling and we are certain that CO2 is currently increasing; we are therefore certain that at least for the past nine years increased atmospheric CO2 has not caused global warming.
Perhaps you should refer in some way on the reaction of IPCC to the IAC recommendations. The issue has been discussed at two IPCC meetings, first in Oct 2010 and further in May 2011. Some material based on these meetings is now available. This list of decisions taken includec the
The agreed reference material for AR5 contains one short page (page 16) on handling of uncertainties refers in point 11 to the guidance note for lead authors..
It’s not clear, how much the recent emphasis finally helps, but the issue has certainly come up within the IPCC.
jcurry 6/17/11, Uncertainty and the IPCC
Assigning probability–like, or confidence level-like. numbers to subjective beliefs or evaluations does not create objectivity. This is fakery, not science, and it brings discredit on the IPCC. It’s politics. It is a sheep in wolf’s clothing.
Justice and science both hold scales, but only justice is blindfolded.
Science doesn’t guess about the accumulation and application of evidence. It accumulates facts statistically. Where courts worry about the subjective bias of cumulative evidence, science accepts all facts while measuring diminishing returns. While courts have subjective rules about admissible and prejudicial evidence, science tests for completeness, and acquires new facts targeted to stress predicted regions of uncertainty.
Every step in science must be objective. No exceptions.
My message got garbled. I hope this is clearer.
Perhaps you should refer in some way on the reaction of IPCC to the IAC recommendations. The issue has been discussed at two IPCC meetings, first in Oct 2010 and further in May 2011. Some material based on these meetings is now available. This list of decisions taken includes the point 11, which refers to the guidance note for lead authors..
The agreed reference material for AR5 contains one short page (page 16) on handling of uncertainties.
It’s not clear, how much the recent emphasis finally helps, but the issue has certainly come up within the IPCC.
Judy:
You could do an entire post on Richard Tol’s comments above. What an indictment of the IPCC process!
Somewhere I recall an essay by McKitrick that related the following. The TAR had very low confidence level for solar forcing. A draft of the AR4 had medium confidence. Somebody pointed out this discrepancy, and they changed it to low confidence. This “expert judgement” of confidence does not exactly inspire confidence. The IAC call for traceability in these judgements is essential, lets see what happens in AR5.
Dear Dr. Judy:
Please correct me if am wrong. IPCC is not a scientific organization and relies on our work to draw conclusions. If IPCC has done a poor job, then we are to be blamed as well.
Making uncertainty, confidence, or statistics as a new approach to address climate science will definitely fail. There is no substitute to mathematics backed by observations. These are the absolute truth and we have to develop climate mathematics.
ENSO and volcanic events are short term cycles whose effect vanish within seven to ten years at the most. They do vary surface temperature up and down and cause uncertainty of importance for meteorological considerations; however, their effect in the long run for climate considerations cancels out and have no impact on the climate. The present climate trend is CERTAINLY warming and can be projected reasonably.
Dear Mr. Swedan,
May I suggest taking a longer perspective?
1) “Climate” is generally held to be “average” weather over 30 years.
2) The sun has come out of a ~30-year grand maximum (and may be entering a minimum).
3) Our “best” data is over a period of about 40 years.
4) Although ENSO may be a “short term cycle”, the cumulative PDO index appears to vary cyclically (+ to – to +) over ~60 years.
5) Persistent effects of the PDO state appear to teleconnect to other ocean basins.
6) The state of the ocean basins affect the observed temperatures of land and sea.
7) These observations affect what we perceive to be the “climate”.
Judy: re: “whereby scientists with a plurality of viewpoints (including skeptics)”
To properly evaluate “uncertainty”, we need to have the FULL range of viewpoints included, not just “a plurality”. This must include “outliers”.
The “outlier” may end up being he truth.
e.g. Galileo vs Aristotelians.
Bacteria causing ulcers.
Plate tectonics etc.
Recommend stating:
“scientists from all viewpoints (from orthodox to skeptics to “outliers”).”
Outliers are far more likely to be right than In-liars.
In my opinion, there are two basic problems with the IPCC / UNFCCC / SBSTA case:
1) “Oreskes (2007) draws an analogy for the consilience of evidence approach with what happens in a legal case. Continuing with the legal analogy, Johnston (2010) characterized the IPCC’s arguments as a legal brief, designed to persuade, in contrast to a legal memo that is intended to objectively assess both sides. ”
Were this a legal case, the prosecution would not be allowed to be prosecutor, jury and judge. Neither would the prosecutor be allowed to pick a jury without challenge.
2) “Curry (2011a) argues that the consilience of evidence argument is not convincing unless it includes parallel evidence-based analyses for competing hypotheses, and hence a critical element in uncertainty monster detection.”
True, and the evidence presented would have to be shown to be accurate, unaltered, independently verified and open to challenge or alternate interpretations.