by Judith Curry
I just came across a paper that I view to be remarkably important, entitled “Influence diagrams for representing uncertainty in climate-related propositions,” by Hall, Twyman and Kay. This paper integrates a number of themes that we have been discussing here: reasoning about uncertainty, consilience of evidence, attribution of extreme events, floods, and even the Italian flag(!). And it does the best job I’ve seen of assessing uncertainty and confidence in a climate-related proposition.
INFLUENCE DIAGRAMS FOR REPRESENTING UNCERTAINTY IN CLIMATE-RELATED PROPOSITIONS
JIM HALL, CRAIG TWYMAN and ALISON KAY
Climatic Change (2005) 69: 343–365
Abstract. In order to respond to policy questions about the potential impacts of climate change it is usually necessary to assemble large quantities of evidence from a variety of sources. Influence diagrams provide a formal mechanism for structuring this evidence and representing its relationship with the climate-related question of interest. When populated with probabilistic measures of belief an influence diagram provides a graphical representation of uncertainty, which can help to synthesize complex and contentious arguments into a relatively simple, yet evidence-based, graphical output. Following unusually damaging floods in October–November 2000 the UK government commissioned research with a view to establishing the extent to which the floods were a manifestation of hydrological climate change. By way of example application, influence diagrams have been used to represent the evidential reasoning and uncertainties in responding to this question. Three alterna- tive approaches to the mathematization of uncertainty in influence diagrams are demonstrated and compared. In situations of information scarcity and imprecise expert judgements, methods based on interval probabilities have proved to be attractive. Interval probabilities can, it is argued, represent ambiguity and ignorance in a more satisfactory manner than the conventional Bayesian alterna- tive. The analysis provides a quantified commentary on the uncertainties in the conclusion that the events of October–November 2000 were extreme, but cannot in themselves be attributed to climate change.
I’ve posted the paper on the web [hall twyman kay influence diagrams].
JC summary points and background info
The specific problem that is addressed in the paper was a study commissioned by the UK Government that sought to establish the extent to which very severe floods in the UK in October–November 2000 were attributable to climate change.
An overview of influence diagrams is provided by the Wikipedia. A Bayesian Network is a type of influence diagram. Influence diagrams are hierarchical and can be used as the basis for a logical hypothesis hierarchy.
The annalysis method and application to the specific flood problem included the following elements:
• Formulation of a source proposition and two child propositions
• Assembly of 24 items of evidence that were each connected, through a series of propositions, to one of the two child propositions
• Population of the influence diagram with quantified estimates of:
- a probabilistic estimate of belief in each sink proposition;
- conditional probability estimates describing the strength of relationship between propositions joined by a link; and
- for Interval Probability Theory only, pair-wise dependency estimates at each node in the diagram.
• Expert elicitation to construct a belief measure for each proposition, to determine point and interval probabilities
• Calculate probabilities for each of the propositions, using three different methods for propagating uncertainty in the influence diagram:
- Bayesian belief networks (point probability)
- Support Logic Programming (interval probabilities)
- Interval Probability (interval probabilities; foundation for the Italian flag analysis; for a practical guide to applications of interval probability, see the Tesla document)
The results are displayed in Table III using point probabilities for the Bayesian analysis and Italian flag interval probabilities for the SLP and IPT analyses, whereby the number on the left is belief for, and the number on the right is belief against the proposition.
A summary of the conclusions from the analysis:
The source proposition receives a support of 0.58 from the Bayesian belief network, [0.21, 0.86] from SLP and [0.37, 0.72] from IPT. The probability measures for proposition B1 indicate that the 2000 flooding and rainfall were highly unusual in the historical context. The existence of hydrological climate change is much less certain, with nearly as much evidence against Proposition B2 as for it. Support for the proposition that ‘Upward trends are present in historical flooding and rainfall data’ (shorthand: ‘statistical trends’) is 0.49 from the Bayesian belief network, [0.10, 0.68] from SLP and [0.23, 0.65] from IPT. The three results demonstrate more belief against the proposition than in favour of it, and SLP and IPT indicate major uncertainty.
Punchline: the Bayesian belief network analysis supports the hypothesis, whereas the interval probability analyses do not support the hypothesis and indicate major uncertainty.
Conclusions
The conclusion section of the paper is reproduced here in full:
The use of influence diagrams for evidential reasoning applied to a prob- lematic climate-related proposition has been demonstrated. The influence dia- grams have provided a structured commentary on the conclusion that the events of October–November 2000 were extreme, but cannot in themselves be at- tributed to climate change. Three alternative inference mechanisms have been tested on the same influence diagram structure. Support Logic Programming and Interval Probability Theory both deal with interval bounds on an unknown probability measure and are attractive in being able to represent in a straightfor- ward way legitimate imprecision in our ability to estimate probabilities. This is particularly useful in situations where evidence is scarce or ambiguous. Inter- val Probability Theory has the added attraction of being able to represent de- pendency relationships between evidence that are not implied by the network structure.
Influence diagrams can help to synthesize complex and contentious argu- ments into a relatively simple, yet evidence-based, graphical output. The graphical structure can formalise expert reasoning, facilitating dialogue between experts, policy makers and other decision stakeholders. In the case of the October–November 2000 floods in the UK, influence diagrams have demonstrated sources of uncertainty and conflict in the available evidence. The analysis has demonstrated how the reluctance of scientists to commit themselves to conclusions about the floods was due to insufficient evidence in the pivotal arguments that related available data via expert reasoning to the hypothesis that the events were a manifestation of climate change. Thus the process of constructing, populating and testing an influence diagram has generated valuable insights.
JC comments: This is how it should be done. This paper has made obselete my draft Waving the Italian Flag: Part II. This paper is featuring prominently in my draft paper “Reasoning about Uncertainty” that will be submitted to the special issue of Climatic Change on framing and communicating uncertainty for the IPCC (a draft of which will hopefully be posted by the end of the week. Perusal of the citations of the Hall et al. paper on Google Scholar shows only 6 citations; hopefully this post will bring further attention to this important paper.
Note to James Annan: this is why I am not impressed with your Bayesian analyses of sensitivity.
Note to Fred Moolten: this illustrates how complex the consilience of evidence argument is, and simple methods for reasoning about multiple lines of evidence can be very misleading.
Moderation note: this is a technical thread that will be moderated for relevance.