by Judith Curry
Is “best available evidence” a new, improved “reframing” of the so-called “consensus” (that is not really holding up too well, these days)? Is it simply a way of sweeping aside the validity of any acknowledgement/discussion of the uncertainties? Or is it something completely different?! – Hilary Ostrov
Dan Kahan has two interesting posts at Cultural Cognition:
- What “climate skeptics” have in common with ‘believers'” a stubborn attraction to evidence-free, just-so stories about the formation of public risk perceptions
- So what is the “best available scientific evidence” anyways?
In addition to the main posts, some of the comments are quite interesting. Excerpts from Kahan’s first post:
When I address the sources of persistent public conflict over climate change, though, it seems pretty clear to me that those with a practical interest in using the best evidence on science communication are themselves predominantly focused on dispelling what they see as a failure on the part of the public to credit valid evidence on the extent, sources, and deleterious consequences of anthropogenic global warming.
I certainly have no problem with that! On the contrary, I’m eager to help them, both because I believe their efforts will promote more enlightened policymaking on climate change and because I believe their self-conscious use of evidence-based methods of science communication will itself enlarge knowledge on how to promote constructive public engagement with decision-relevant science generally.
A comment from Hilary Ostrov:
Dan, in your initial post you mention “best available evidence” no less than six times. And you may also have reiterated the phrase in some of your comments.
Perhaps you have identified your criteria for determining what constitutes “best available evidence” elsewhere; but for the benefit of those of us who might have missed it, perhaps you would be kind enough to articulate your criteria and/or source(s) for us.
It is a rather nebulous phrase; however, I suppose it works as a very confident, if not all encompassing, modifier. But as far as I can see, your post doesn’t tell us specifically what “evidence” you are referring to (whether “best available” or not!)
Is “best available evidence” a new, improved “reframing” of the so-called “consensus” (that is not really holding up too well, these days)? Is it simply a way of sweeping aside the validity of any acknowledgement/discussion of the uncertainties? Or is it something completely different?!
This comment from Hilary Ostrov motivated the second post, excerpts from Kahan’s 2nd post:
But I certainly don’t have a set of criteria for identifying the “best available scientific evidence.” Rather I have an ability, one that is generally reliable but far from perfect, for recognizing it.
JC comment: sounds a bit like pornography.
I think that is all anyone has—all anyone possibly could have that could be of use to him or her in trying to be guided by what science knows.
For sure, I can identify a bunch of things that are part of what I’m seeing when I perceive what I believe is the best available scientific evidence. These include, first and foremost, the origination of the scientific understanding in question in the methods of empirical observation and inference that are the signature of science’s way of knowing. Basic techniques for recognizing the best available scientific evidence. But those things I’m noticing (and there are obviously many more than that) don’t add up to some sort of test or algorithm.
Moreover, even the things I’m seeing are usually being glimpsed only 2nd hand. That is, I’m “taking it on someone’s word” that all of the methods used are the proper and valid ones, and have actually been carried out and carried out properly and so on.
Nullius in verba–the Royal Society motto that translates to “take no one’s word for it”– can’t literally meant what it says: even Nobel Prize winners would never be able to make a contribution to their fields — their lives are too short, and their brains too small–if they insisted on “figuring out everything for themselves” before adding to what’s known within their areas of specialty.
What the motto is best understood as meaning is don’t take the word of anyone except those whose claim to knowledge is based on science’s way of knowing–by disciplined observation and inference– as opposed to some other, nonempirical way grounded in the authority of a particular person’s or institution’s privileged insight.
Amen! But even identifying those people whose knowledge reflects science’s empirical way of knowing requires (and always has) a reliably trained sense of recognition!
So no definition or logical algorithm for identification — yet I and you and everyone else all manage pretty well in recognizing the best available scientific evidence in all sorts of domains in which we must make decisions, individual and collective (and even in domains in which we might even be able to contribute to what is known through science).
I understand disputes like climate change to be a consequence of conditions that disable this remarkable recognition faculty.
Chief among those is the entanglement of facts that admit of scientific investigation in antagonistic cultural meanings.
This entanglement generates persistent division, in part b/c people typically exercise their science-knowledge recognition faculty within cultural affinity groups, whose members they understand and trustwell enough to be able to figure out who really knows what about what (and who is really just full of shit). If those groups end up pesistently divided about what the best available science is, their members will be persistently divided about that too.
Even more important, the entanglement of facts with culturally antagonistic meanings generates division b/c people will often have a more powerful psychic stake in forming and persisting in beliefs that fit their group identities than in “getting the right answer” from science’s point of view, or in aligning themselves correctly w/ what the ‘best scientific evidence is.”
After all, I can’t hurt myself or anyone else by making a mistake about what the best evidence is on climate change; I don’t matter enough as consumer, voter, “big mouth” etc. to have an impact, no matter what “mistake” I make in acting on a mistaken view of what is going on.
But if I take the wrong position on the issue relative the one that predominates in my group, and I might well cost myself the trust and respect of many on whose support I depend, emotionally, materially and otherwise.
JC comment: Joshua, pay attention. This pretty much defines tribalism in context of the climate debate. This is the strategy of the ‘consensus police’, whereby scientists who deviate in a minor way are made extremely uncomfortable. Note, this does not happen on the skeptical ‘side’
But first, back to the questions that motivated the last post.
To answer them, I hope I’ve now shown you, you won’t have to agree with me about what the “best available scientific evidence” is on climate change.
Indeed, the science of science communication doesn’t presuppose anything about the content of the best decision-relevant scientific evidence. It assumes only two things: (1) that there is such a thing; and (2) that the question of how to enable its reliable apprehension by people who stand to benefit from it admits of and demands scientific inquiry.
JC comment: Light bulb time. Take note of the presupposition of ‘the best decision-relevant scientific evidence’. I will return to this later.
But here goes:
Climate skeptics (or the ones who are acting in good faith, and I fully believe that includes the vast majority of ordinary people — 50% of them pretty much — in our society who say they don’t believe in AGW or accept that it poses significant risks to human wellbeing) believe that their position on climate change is based on the best available scientific evidence — just as I believe mine is!
About the science communication problem–by which I mean precisely the influences that are preventing us, as free reasoning people, from converging on the best available scientific evidence on climate change and a small number of other consequential issues (nuclear power, the HPV vaccine, the lethality of cats to birds, etc)? Converging in the way that we normally do on so many other consequential issues–so many many many more that no one could ever count them!?
I hope that because I would like to think that once we get this sad matter behind us, and resume the patterns of trust and reciprocal cooperation that normally characterize the nonpathological state in which we are able to recognize the best available scientific evidence, there will be some better science of science communication evidence for us all to share with each other on how to to negotiate the profound and historic challenge we face in communicating what’s known to science within a liberal democratic society.
A statement from Kahan in the comments:
But I believe what I believe, and expect others to respect my right to use and rely on my own reason as I see fit.
Well it is difficult to know where to start with this, Kahan’s posts raise many fundamental issues that provide insights into the conundrum of communication of climate science.
First, what constitutes ‘evidence’ in climate science? Scientific evidence is generally regarded to consist of observations and experimental results. In complex natural systems, the epistemic status of observations is not straightforward, with homogenization adjustments, model assimilation of observations, retrieval algorithms to interpret voltages measured by satellites, etc. Plenty of room for debate and uncertainty regarding the establishment of climate data records, and relatively few data sets have actually met the relevant maturity matrix requirements.
Evidence in climate science seems to be more broadly defined to include any information that is presented in support of an assertion about the climate system. In climate science arguments, such evidence commonly includes results from climate model simulations and arguments presented in refereed journal publications. The epistemic status of this more indirect evidence in scientific arguments is something that should be debated more than it has been.
For the sake of argument, lets assume that all of the major climate data sets (surface temperature, ocean heat content, paleo reconstructions) and climate model simulations have high epistemic value as evidence. Even if we agree on this evidence, why might scientists disagree about topics such as attribution?
Arguments about causality in complex systems are not straightforward. Scientific debates involve controversies over the value and importance of particular classes of evidence as well as disagreement about the appropriate logical framework for linking and assessing the evidence.
Different scientists may rationally draw different conclusions and make different assertions based upon the existing observational evidence: the wide range of recent estimates of climate sensitivity is a case in point. Kahan assumes that there is some subset of these different assertions or arguments that constitutes ‘best available evidence’ and that the inferences drawn from observations are staightforward. There is nothing straightforward about making inferences about the complex climate system; see my paper Reasoning about climate uncertainty. We are then generally left with expert judgment in terms of determining ‘best available evidence’, the outcome of which depends on which experts you select and the method by which their expertise is elicited (see my paper No consensus on consensus). So there is no objective ‘best available evidence’ when uncertainties are large and unknown unknowns loom.
If there is no ‘best available evidence’, or no consensus, how can we proceed with making decisions? My no consensus paper addresses this issue (I won’t repeat that text here).
A useful example is the decision of the Bush 43 administration to go to war with Iraq, ostensibly based on the best available evidence that the Iraqis possessed weapons of mass destruction. The best available information turned out to be incorrect, but there were other reasons that the Bush administration wanted to go to war with Iraq.
Two summary points:
1. For a complex problem with large uncertainties, there is no objective way to to determine ‘best available evidence’; expert judgment methods subjectively depend on which experts you select.
2. If uncertainties are large, ‘best available evidence’ may lead to bad decisions. There are numerous approaches to decision making under deep uncertainty that do not require a consensus or ‘best available evidence.’
Lets return to Kahan’s main issue, that of communicating science. I had an epiphany on this subject during the panel discussion among myself, Gavin Schmidt and Richard Betts on the subject of advocacy by scientists. [link] Gavin Schmidt made the point that scientists should speak out on their opinions about topics related to their area of expertise. I have no problem with this, if your objective is to respond to someone’s request for your opinion, or you want your opinion for some reason to be a matter of public record or for people to consider your opinion and comment on it.
It is a different ballgame, however, if your objective is to change the minds of the people in your audience, to support your view on the science and/or policy. This seems to be the objective of ‘climate communication’, which seems to me to be a euphemism for policy advocacy whereby the ‘best available scientific evidence’ leads to self-evident policy prescription. So there is a not so subtle distinction in a scientist stating his/her opinion about policy simply for the purpose of clarifying what the scientist thinks, versus trying to change someone’s mind about the science with the objective to spurring them them to taking action on the self-evident policy.
So why does ‘climate communication’ seem to be failing? Here is my take. Dan Kahan seems to think that rational people are able to identify the best available scientific evidence, i.e. they know it when they see it (sort of like pornography). I say that identifying best available scientific evidence is difficult even for scientists if the uncertainties are large. What rational people are able to to do is to identify BS (see BS detectors). Overconfidence, failure to present evidence that does not support your thesis, dismissal of skeptics and skeptical arguments, appeal to consensus, advocacy, etc. all can act to trigger someone’s BS detector. Its about trust; without trust, expertise does not equal credibility.
p.s. read the comments on Kahan’s threads, all of the comments provide cogent and provocative inputs (something for the Climate Etc. denizens to aspire to).