by Judith Curry
The issuing of the BEST press release prior to peer review of the papers raises some interesting and provocative issues.
From the BEST FAQ: Why didn’t Berkeley Earth wait for peer review?
Such traditional and open peer review has many advantages. It usually results in better papers in the archival journals, because the papers are widely examined prior to publication. It does have a disadvantage, however, that journalists can also pick up preprints and report on them before the traditional peer-review process is finished.
Perhaps because of the media picking up on talks and preprints, a few journals made a new rule: they will not publish anything that is distributed as a preprint or that is discussed openly in a meeting or colloquium. This policy has resulted in more attention to several journals, but the restrictive approach had a detrimental effect on the traditional peer review system. Some fields of science, for example String Theory, objected so strongly that they refuse to publish in these journals, and they put all their papers online immediately.
The best alternative would be to have the media hold back and not report preprint material. Unfortunately they refuse to do that. The situation is made more difficult by the fact that many of the media misreport the content of the preprints. For that reason Berkeley Earth has tried to answer the questions given to us by the media, in hopes that our work will be more accurately reported. The two page summary of findings is also meant to help ensure that the media reports accurately reflect the content of our papers.
WUWT: The Best Whopper Ever
Anthony Watts refers to the BEST FAQ as the Best Whopper Ever. An excerpt:
I call absolute total BS on that. Why?
Because BEST contacted media in advance of the release of their papers and provided preprints. The October 20th release by BEST was planned and coordinated with media, such as the Economist, Guardian, NYT, New Scientist, and Nature, all of which contacted me before the release on October 20th. This FAQ on peer review was added sometime after that date, I don’t know when, but the FAQ headline obviously refers to past tense.
Mike Hulme’s essay
Mike Hulme has written a very interesting essay on this, some excerpts:
So what does this do to the conventional journal peer-review process? Those asked to review these manuscripts for JGR will now conduct their personal reviews in the full knowledge of the parallel public review which is on-going. And unless they shut-off all their communication platforms for the duration they will hear and see what others’ judgements on the manuscripts are. Whether for better or worse it’s difficult to see how this will not change the (conventional) peer-review process.
This is rather similar to the situation with juries in court cases. Jurors are sealed-away from extraneous media-based interpretations, speculations and judgements while they decide on the verdict of their case. Scientists do not have the same obligations nor possibilities.
The BEST team didn’t have to play it this way, but they did. What interests me then is how this illustrates the changing nature of peer-review in ‘hot science’ (see also my article in Science as Culture on Kilimanjaro’s glaciers, Guy Callendar and Al Gore). What new name do we give this parallel form of peer review: it is certainly not conventional and it seems to me something more interesting than Ravetz’s extended peer review? And whatever it is, does it make for more (or less) legitimate public knowledge?
One interesting parallel I can think of is with Maarten Haijer’s recent idea of ‘authoritative governance’: governance which establishes its legitimacy and authority on the basis of how well it performs in pressurised and real-time media spaces.
What we are witnessing with BEST is scientific knowledge which is being judged on its ‘performative’ successes in an open society as much as it is being judged on conventional scientific norms of thoroughness, clarity and logic. Knowledge is here being made ‘in the open’ for all to see.
Where does conventional journal peer review for ‘hot science’ go after this?
JC’s take on this in Mixing Politics and Science paper
I raised this issue in my 2006 paper Mixing Politics and Science in Addressing the Hypothesis that Global Warming is Causing a Global Increase in Hurricane Intensity:
Some of the most relevant scientific debate on this topic is not being undertaken at meetings sponsored by the relevant professional societies and government agencies, but rather in the media and via blogs, and only slowly in the professional scientific journals. After reading The World is Flat: A Brief History of the Twenty-First Century (Friedman 2005), we were prompted to reflect on how broadly the new technologies are influencing the scientific process on topics of high relevance. As the media debate proceeded, and certainly in the process of researching the material for this paper, we made extensive use of online media articles, blogs, Wikipedia.com, and other Web sites. As pointed out by Friedman, the challenge is how to think about the new technologies and the associated changes that have irreversibly changed the intellectual commons and manage it to maximum effect. The new scientific process will eventually sort itself out among the new technologies, the need for the scientific review process, and the need for information by the public and policymakers. However, during this sorting-out period (which may end up being a period of continual evolution as new technologies emerge), the use of science to inform policy, particularly on issues of high relevance, will almost certainly become confused with the decentralization of scientific authority previously vested in scientists that have published on the subject in refereed journals. While this decentralization provides a better guarantee that the best possible information and analysis is out there somewhere, it becomes increasingly difficult to identify the best information and analysis in this new environment, providing more fodder for the politicization of science.