by Judith Curry
The UK House of Commons Select Committee on Science and Technology has launched an inquiry into peer review.
The committee invites evidence on the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication.
The Committee welcomes submissions on all aspect of the process and among the issues it is likely to examine are the following:
- the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public;
- measures to strengthen peer review;
- the value and use of peer reviewed science on advancing and testing scientific knowledge;
- the value and use of peer reviewed science in informing public debate;
- the extent to which peer review varies between scientific disciplines and between countries across the world;
- the processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases;
- the impact of IT and greater use of online resources on the peer review process; and
- possible alternatives to peer review.
Here are some of the submissions that caught my eye:
Professor Michael Kelly (a member of the Oxburgh Committee)
Problems in peer review are the symptom, not the cause, of deeper problems in the modern scientific enterprise. It is these deeper problems that should be debated and solved, so that peer view or a timely alternative can do its job again. Since popular discourse sets 2050 as the date by which the world must be transformed, should science and technology be made more directly the handmaiden of such a transformation?
Richard Horton
More importantly, intensified post as well as pre publication review would put uncertainty – its extent and boundaries – at the centre of the peer review and publication process. This new emphasis on uncertainty would limit the rhetorical power of the scientific paper (50), and offer an opportunity to make continuous but constructive public criticism of research a new norm of science.
David Taylor
Where a consensus has developed in the scientific community both reviewers and editors will be very unlikely to accept any new material which seeks to challenge that viewpoint. Thus the peer review process is an excellent method for guarding the consensus view against attack. This is a positive benefit where the consensus view is actually correct e.g. as seems to be the case with HIV as the causal agent of AIDS. However scientific endeavours are littered with examples of where a strongly held consensus has eventually been overturned, often after a considerable struggle e.g. the continental drift theory of Wegener and the Helicobacter pyloritheory of ulcer formation which eventually led to a Nobel Prize for Marshall and Warren. In such cases the gate keeping activities of the Peer Review system can be seen to have had a very serious negative impact on scientific advance.
Annabelle Mark
The issue for both academic communities and the public, who have increasing access to publications because of the internet, is where do I look for work I can trust, and where do I look for work that is innovative – in some instances the answers I suggest may be mutually exclusive, so a re-evaluation of this singular approach to what matters is long overdue.
Nikolaus Kriegeskorte
Presents a vision for open publication and evaluation
Lawrence Souder
Addresses issues related to civility (and references the climategate emails)
Mark Bretscher
The main failure thus lies with the journals’ scientific editors. They are too busy to act as more than a postal centre – there are far too many papers chas ing around the journals , each being handled several times rather than just once . E ditors are mostly “PIs”, senior scientists long removed from bench work who manage large groups from office desks. It matters little to them if an extra year or more is needed by a student or postdoc to fulfil the whim of a referee. T hat’s the way the system works. Another concern is that referees may use their un bridled influence to obstruct competitors. I also sense that the freedom of editors to disregard points raised in referees’ reports may be curtailed by a journal’s publishers “in the interests of fairness”.
RI Tricker
Peer reviewers can act as barriers not gateways. The review process can put a straight-jacket on new thinking, alternative insights, and new paradigms when reviewers are hostile to theoretical insights that do not fit their own theoretical position or preconceptions. A good example can be found in the study of corporate governance. The agency theoretical methodology of financial economics has dominated research published in the field over the past 20 years. Whilst this has provided statistically neat and replicable results, the methodology tends to use published data, and researchers have no need to meet real company directors. Consequently, developments in the thinking and practice of the subject have been reactions to corporate collapses which led to corporate governance codes based on conventional wisdom, not serious research. If a subject becomes dominated by a particular theoretical paradigm, it becomes impossible for new thinking and alternative paradigms to be accepted.
Donald Gillies
READ THE WHOLE THING. Arguably the closest thing to what Kuhn would have said.
This is as far as I got, #22 in the list (after reading Donald Gillies, I didn’t think anything could top that). Let me know if you spot anything else of interest.