by Judith Curry
The UK House of Commons Select Committee on Science and Technology has launched an inquiry into peer review.
The committee invites evidence on the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication.
The Committee welcomes submissions on all aspect of the process and among the issues it is likely to examine are the following:
- the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public;
- measures to strengthen peer review;
- the value and use of peer reviewed science on advancing and testing scientific knowledge;
- the value and use of peer reviewed science in informing public debate;
- the extent to which peer review varies between scientific disciplines and between countries across the world;
- the processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases;
- the impact of IT and greater use of online resources on the peer review process; and
- possible alternatives to peer review.
Here are some of the submissions that caught my eye:
Professor Michael Kelly (a member of the Oxburgh Committee)
Problems in peer review are the symptom, not the cause, of deeper problems in the modern scientific enterprise. It is these deeper problems that should be debated and solved, so that peer view or a timely alternative can do its job again. Since popular discourse sets 2050 as the date by which the world must be transformed, should science and technology be made more directly the handmaiden of such a transformation?
Richard Horton
More importantly, intensified post as well as pre publication review would put uncertainty – its extent and boundaries – at the centre of the peer review and publication process. This new emphasis on uncertainty would limit the rhetorical power of the scientific paper (50), and offer an opportunity to make continuous but constructive public criticism of research a new norm of science.
David Taylor
Where a consensus has developed in the scientific community both reviewers and editors will be very unlikely to accept any new material which seeks to challenge that viewpoint. Thus the peer review process is an excellent method for guarding the consensus view against attack. This is a positive benefit where the consensus view is actually correct e.g. as seems to be the case with HIV as the causal agent of AIDS. However scientific endeavours are littered with examples of where a strongly held consensus has eventually been overturned, often after a considerable struggle e.g. the continental drift theory of Wegener and the Helicobacter pyloritheory of ulcer formation which eventually led to a Nobel Prize for Marshall and Warren. In such cases the gate keeping activities of the Peer Review system can be seen to have had a very serious negative impact on scientific advance.
Annabelle Mark
The issue for both academic communities and the public, who have increasing access to publications because of the internet, is where do I look for work I can trust, and where do I look for work that is innovative – in some instances the answers I suggest may be mutually exclusive, so a re-evaluation of this singular approach to what matters is long overdue.
Nikolaus Kriegeskorte
Presents a vision for open publication and evaluation
Lawrence Souder
Addresses issues related to civility (and references the climategate emails)
Mark Bretscher
The main failure thus lies with the journals’ scientific editors. They are too busy to act as more than a postal centre – there are far too many papers chas ing around the journals , each being handled several times rather than just once . E ditors are mostly “PIs”, senior scientists long removed from bench work who manage large groups from office desks. It matters little to them if an extra year or more is needed by a student or postdoc to fulfil the whim of a referee. T hat’s the way the system works. Another concern is that referees may use their un bridled influence to obstruct competitors. I also sense that the freedom of editors to disregard points raised in referees’ reports may be curtailed by a journal’s publishers “in the interests of fairness”.
RI Tricker
Peer reviewers can act as barriers not gateways. The review process can put a straight-jacket on new thinking, alternative insights, and new paradigms when reviewers are hostile to theoretical insights that do not fit their own theoretical position or preconceptions. A good example can be found in the study of corporate governance. The agency theoretical methodology of financial economics has dominated research published in the field over the past 20 years. Whilst this has provided statistically neat and replicable results, the methodology tends to use published data, and researchers have no need to meet real company directors. Consequently, developments in the thinking and practice of the subject have been reactions to corporate collapses which led to corporate governance codes based on conventional wisdom, not serious research. If a subject becomes dominated by a particular theoretical paradigm, it becomes impossible for new thinking and alternative paradigms to be accepted.
Donald Gillies
READ THE WHOLE THING. Arguably the closest thing to what Kuhn would have said.
This is as far as I got, #22 in the list (after reading Donald Gillies, I didn’t think anything could top that). Let me know if you spot anything else of interest.
Great news!
Thanks,
Oliver
The Deep Roots of the Problem
Dr. J. Marvin Herndon identifies below the deep historical roots of problem:
1. NSF instituted the anonymous peer review system for evaluating grant proposal reviews soon after NSF was formed in 1951.
2. Beginning with this deepest known root of the problem in the USA, the “President of the United States could sign an Executive Order forbidding the U.S. Government from engaging in anonymous peer review and forbidding the U. S. Government from doing business with organizations that do.”
3. Given the involvement of a former Democratic Vice President in the climate scandal, it might be more realistic for the Subcommittee on Commerce, Justice and Science of the US House Appropriations Committee to restrict government funding to any agency of the U.S. Government that engages in anonymous peer review and forbidding the U. S. Government from doing business with organizations that do.”
4. In the UK, the House of Commons Select Committee on Science and Technology might accomplish the same goal by restricting funding.
Oliver, I have to think that the formation of the NSF in 1951 had to have something to do with the Manhattan Project and the perceived wonderfulness of government-funded research. IMHO, this confluence was the beginning of “Big Science.”
Identifying peer-review as the culprit vs the bottomless pockets of the U.S. government is not a conclusion I would arrive at. I see the money as being the distorter and anonymous peer-review as one of possibly several selected mechanisms to ensure that the politically connected have first dibs on the flow of the money.
I don’t think they gave two thoughts to it impeding scientific progress. I’d bet they actually thought the flow of government money would increase scientific progress. And a prima facie case could be made that it did exactly that. The acceleration of the growth of “knowledge” since the late 1940s has been noted many times, with people assigning several causes, including alien technology from Roswell that was reversed engineered. But the strongest case for that increase would reasonably be assigned to the influx of government money, IMHO.
Therefore, there has been a lot of upside, whether the involvement of the NSF was a necessary part of it or not. (I think it was.) Some government agency had to be the funneler of the money; that was the intent of the NSF, as I understand it (though I do not know for a fact). So, the positives almost certainly outweigh the negatives.
However, once set in place, this institutionalization within the government of the granting of research money had to have some downside. It is impossible for them (then or now) to come up with a perfect system. And that downside had to come to the fore sooner or later.
In the exposure of anonymous peer-review as a weak link, we are seeing this dirty laundry being done in public for perhaps the first time. It is being seen as “science’s little secret.”
This outing of it is perhaps overdue, really. It will be interesting to see how and how long this will all play out.
I agree, Steve. Nobody intended to corrupt government science when NSF was first established and anonymous peer reviews were initiated.
However ten years later, in his farewell address to the nation, former President Eisenhower warned of the danger that a federally-funded “scientific-technological elite” might one day take control.
http://mcadams.posc.mu.edu/ike.htm
or youtube.com/watch?v=GOLld5PR4ts
My experience suggests that government research agencies have actively encouraged investigators to hide, ignore, and/or manipulate unexpected experimental observations over the past four or five decades about:
a.) The Sun’s origin,
b.) The Sun’s composition,
c.) The Sun’s source of energy, and
d.) The Sun’s influence on Earth’s changing climate.
The climate scandal exposed these practices in (d).
Thus, the climate scandal is only the visible tip of an unclean government iceberg that grew, out of sight, almost since the time former President Eisenhower warned about this danger in 1961.
A manuscript in press gives a brief summary of the experimental findings concerning items (a.-c.)
http://arxiv.org/pdf/1102.1499v1
i think your talking rubbish Manuel
I detect contamination by PNS in some of the quotes above … social controls on science eviscerate its value as a source of reliable “reality checks”.
The following two links document the corruption of American science, mainly as a consequence of “peer review”:
http://nuclearplanet.com/Anerican%20Science%20Decline.html
http://nuclearplanet.com/Peer_Review_Folly.html
Please read and reason. Thank you.
J. Marvin Herndon
Thanks, Marvin, you hit the nail on its head!
And thanks, Professor Curry, for your role in getting this issue before the UK House of Commons Select Committee on Science and Technology
In my opinion, the problem is not peer-review, but anonymous peer review.
Oliver K. Manuel
Marvin, as someone who studies US science policy I think your claims about corruption and that “American science has continued to decline toward third-world status” are ridiculous. Have you published any serious analyses supporting these outrageous claims, in the science policy literature? Your diatribe webpage http://nuclearplanet.com/Anerican%20Science%20Decline.html has no links to anything.
So
When does the US get its own panel of this sort?
Come to think of it, the mechanisms of both House of Commons and also of Congressional Committee Hearings as forms of review, also long overdue for exactly such questions too.
I’d have liked to see the phrase ‘adverse reward systems’ used more.
Paragraphs 5 and 6 from Professor Kelly’s submission seem particularly troubling:
5. The resulting paper in various forms has so far been turned down ten times by seven top international journals devoted to solid state physics, nanotechnology or microelectronics based both in the US and in the UK. These are all journals that publish the positive results including many of the 102 citations described above. I can table correspondence from the editors (who are not peers), but I do not include them here, as such material is considered confidential: see http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5635418&tag=1 The paper has not been rejected as wrong, trivial or derivative, but for a number of reasons which all boil down to ‘we don’t want to be associated with such a profoundly negative paper, and the type of debate it might spur’…
6. I am still trying to get my paper published in the peer review literature, before going to the press on the issue of the integrity of peer review. Until it is published, the paper will always suffer from the unanswerable put-down that it has not been peer-reviewed, a common critique used of much work in other fields such as critical climate change science and impacts…
I’ve seen some of the arguments for why peer review should involve confidentiality and anonymity (Phillip Campbell’s submission deals with this in detail), but the potential for abuse seems very great.
I’ve give an opinion on it if I could find a copy. My guess is he’s using statistical fluctuations of material constituents to come up with the limitation. He could also be using an “edge roughness / surface roughness” approach, but I can’t really tell from what he says. It all cascades from being able to design useful circuits, you have to have certaindevice properties within a particular range for the devices. To get the properties within the range, you have to have electrical / chemical / mechanical / topological properties within a certain range. To get this, you have to have manufacturing processes which can actually achieve the goals.
John Nielsen-Gammon has an interesting article about the rigors of peer review on his blog.
Predictably, nothing about the moderation policy at RealClimate. Another whitewash.
1. Peer review is only as useful as the attempt to falsify the hypothesis is. If it is not a indepth and broad attempt to break the Hypothesis, then one should expect the hypothesis to be torn apart once Published.
2. Communicate that Peer Review is not the final step in the Scientific Method, it is only a proof reading step to give the authors of the item to be published reassurance that the skeptical community will not tear their methods or findings apart. They will attempt to.
3. See #1, a Peer Review is only as valuable as the attempt to falsify the work was. If it was rigorous, then the value is likely to be high, if it was a rubber stamp, the value is likely to be low.
4. See #1 and #3. Once public debate is starts, Peer Review is rendered useless until the Skeptics have had a chance to respond.
5. Peer Review varies upon the Reviewer. I would add that where Scientific Authorities are in play, Skeptics should be far more rigorous in their attempt to falisfy the Hypothesis. ‘Authorities’ have been known to accept Credentials instead of repeatable methods.
6. Peer Review is not a Skeptical Review, requisite skills and knowledge of reviewer is up to the Authors looking to publish. Expect that a Skeptical Review will be both rigorous and hostile, choose your Peers wisely.
7. IT certainly allows authors looking to publish works to share info with their colleagues prior to publishing.
8. There are only two possibilities with Peer Review, have it done prior to Publishing, or do not. One should choose a colleague that they trust to keep their data and methods to themselves (until Publishing, of course) and trust that they are capable and willing to attempt to break the hypothesis. If they are competent and cannot break the Hypothesis, it is likely safe to publish without the risk of embarrassment.
Without question, one should expect a very rigorous and hostile (depending upon several factors such as novelty, alarmism, social and economic costs, etc) by the most ardent of skeptics.
As a veteran of much experience as both reviewer and reviewee, I’m inclined to paraphrase Winston Churchill’s dictum on democracy by proclaiming that peer review is the worst possible system, except for all the others. I state that as a scientist whose most important contribution (in my own view) was rejected many times before it was eventually published, while my lesser and more pedestrian efforts skated through the process with little resistance.
The process is encumbered by cliquishness, resistance to innovation, favoritism, competitiveness, and poor judgment, but most of all, in my view, by the difficulty journals encounter in cajoling expert reviewers to volunteer the time and effort needed to do an adequate job. (Similar problems apply to the review of research grants).
Other than recognizing the problems and encouraging potential reviewers to avoid them, I’m not sure what approaches to improving the process are actually feasible. However, there are some encouraging phenomena emerging. One of these involves simply the existence of alternative forms of the process, including experiments in extended interactions between authors, reviewers, and editors, and also including open access journals – whether these latter are a step forward or backward is not yet clear in my estimation. More broadly, the Internet itself acts to some extent as a means of pressuring the peer review process to avoid its worst sins, and of course, the blogosphere has played an important role in this type of oversight.
Despite all the emphasis on the deficiencies of peer review, however, I think some perspective is in order. To the best of my knowledge, no important advance in modern science has been stifled by peer review, but at worst, only delayed. The cited examples are the most egregious (e.g., plate tectonics, and I would add Prusiner’s prion theory of neurodegenerative diseases), but most delays have been more abbreviated, and have been the price paid for the benefit of keeping much false, misleading, fraudulent, and in some cases harmful material out of the literature. Certainly, the process has not been stringent enough to exclude all such errors, although most have been less serious than Andrew Wakefield’s reprehensible attempt to link vaccines to autism.
There is another test, however, of any hypothesis that the world is being denied valuable scientific importance by peer review obstructionism. That is the existence today of alternatives to prestigious journals in the form of other journals designed to be hospitable to contrarian viewpoints, and more saliently, the availability of Internet sources that publicize material that has not or cannot traverse the peer review gantlet. If valuable material has been blocked by peer review, we might expect to find it in these alternative locations. One such location is the non-peer reviewed arxiv, and blogs on scientific themes are another. How do these fare?
In my experience, when it comes to scientific innovation, the answer is dismally. Both sources are dominated by poorly informed and often nonsensical content, sometimes emanating from individuals who fancy themselves unappreciated geniuses the world has rejected to its detriment. Genuine scientific advances have been at best rare, and significant advances non-existent to my knowledge, although obviously I may have overlooked examples.
That is not to say that no content of value has emerged among the worthless items. That content, however, has consisted mostly of perceptive criticisms of peer-reviewed material rather than a description of positive advances in their own right. At this point, then, what I see is an uneasy but perhaps useful tension between the value of peer review as a screening process and the value of the critics in pointing out its flaws, including flaws in the papers that navigate the process and are published.
When it comes to peer review, I am not a panglossian who believes we live in the best of all possible worlds. At the same time, I endorse peer review as a concept until something better comes along. There are only 24 hours in a day, and I appreciate the fact that when I read a scientific article, someone has gone to the trouble of reducing the possibility that I am wasting my time.
“If valuable material has been blocked by peer review, we might expect to find it in these alternative locations. One such location is the non-peer reviewed arxiv, and blogs on scientific themes are another.”
I’d look in the patent office -this always lags, but individuals and small groups working on valuable innovations are most likely to patent, if they have something, and publish if they need press (or more money).
Non-peer reviewed arXiv BLACKLISTS educated, experiences, and capable scientists see
http://nuclearplanet.com/Science%20Suppression%20by%20the%20NSF.html
Thank you, thank you, Marvin, for posting that information for all to see.
It is these anonymous, out-of-public-view decisions that corrupted science and eventually produced the climate scandal.
With kind regards,
Oliver K. Manuel
To the best of my knowledge, no important advance in modern science has been stifled by peer review ..”
I am reminded of the question the visitor to the site of an Indian ruin in Arizona asked the guide. “How many undiscovered Indian ruins are there?”
The problem with biases towards status quo inherent in the peer review process are real, but to argue that somehow we would have been better off as a society absent the peer review process seems like a a bit of a stretch.
For all the important scientific discoveries that were rejected by peer review processes and later validated I would guess that there are many multiples of: (1) important scientific discoveries that were validated by peer review processes and, (2) invalid scientific theories that were correctly rejected by peer review processes. These are factors which tip the balance in the direction of a flawed by ultimately beneficial peer review process.
Certainly, it is at least theoretically plausible that there are “unknown unknowns” out there – some number of correct scientific theories that were rejected by peer review processes and that will never see the light of day and have opportunity for wider validation as a result; but who can say with any certainty that such “unknown unknowns” could tip the balance in the opposite direction?
To the best of my knowledge, no important advance in modern science has been stifled by peer review, but at worst, only delayed.
============================
This is sheer pap. How would you falsify this statement? It is like AGW thesis. Here’s my pap contribution “Peer review is only as good as the peers reviewing it” or “Peer review does not kill papers, reviewers do”
There is another test, however, of any hypothesis that the world is being denied valuable scientific importance by peer review obstructionism. That is the existence today of alternatives to prestigious journals in the form of other journals designed to be hospitable to contrarian viewpoints, and more saliently, the availability of Internet sources that publicize material that has not or cannot traverse the peer review gantlet. If valuable material has been blocked by peer review, we might expect to find it in these alternative locations. One such location is the non-peer reviewed arxiv, and blogs on scientific themes are another. How do these fare?
Fred,
As a last resort, papers could be submitted to one of these alternative locations, but wouldn’t that pretty much foreclose its ever making it’s way into the peer reviewed literature? Professor Kelly’s submission that I touched on above outlines the dilemma.
Gene,
If a scientific result is accepted as valuable, it can be brought to the peer reviewed literature in many ways. Perhaps it cannot be published exactly as it has already been published, but valuable results can be used in further work and partially republished in that connection, they can be described in review articles, and they can be used as references in peer reviewed research articles. All these ways will give the result the value it reserves. The author may get somewhat less credit in later job applications, but the result itself survives.
Fred M: Following up on your Churchillian analogy that peer review, like democracy, is the worst system in the world except for all the others, I would say that there is democracy and there is a democracy under a political machine. They are not the same.
My problem isn’t with peer review — it strikes me as a reasonable system — but Climategate revealed that top climate scientists were rigging the peer review system behind the scenes, and the response of climate scientists and scientists in general was a big shrug.
Likewise the publication of the Hockey Stick, which became the poster graph for the IPCC and the whole climate change movement, and the Stick’s attendent problems and corruption. Another big shrug from scientists.
Skeptics complained loudly enough, so scientists grumpily agreed to investigate, then did a classic whitewash. So now scientists happily slam down further complaints with “exonerated on all counts.”
The problem isn’t peer review IMO but the mafia version of peer review that climate science now provides and that scientists seem happy enough to live with and defend except for the rare brave one like Dr. Curry, who is naturally treated like a traitor.
Hux,
“Climategate revealed that top climate scientists were rigging the peer review system behind the scenes, and the response of climate scientists and scientists in general was a big shrug.”
I am not sure that this problem with peer review is confined to climate science although climategate shone a bright light on the matter. It has always been a problem in science hence the ‘big shrug’ by others in the field. The ‘consensus’ reviews new material and then tends to defend its own position when new ideas pose a challenge to established thought. Not trying to justify it, just pointing out the reality of the situation.
RobB: Quite right. I enjoy reading layman’s physics books and clearly it’s a problem in physics. See Lee Smolin’s “The Trouble with Physics” or Joao Maguiejo’s “Faster than the Speed of Light”. In both cases views outside the consensus are frozen out of circulation.
It just seems worse in climate science because it goes beyond defending the consensus by ignoring competing views to actively conspiring behind the scenes plus the corruption of the Hockey Stick upheld by peer review.
I think in recent decades science has entered a new, more corrupt phase with big money, big teams and academia slanted way over to the Left.
@ Fred: “To the best of my knowledge, no important advance in modern science has been stifled by peer review, but at worst, only delayed.”
Fred, other commenters have drawn attention to this apparently naive and self-defeating statement. What is it supposed to mean? How can anybody know what you claim? If you let it stand your own credibility seems to be at risk.
Judith –
Gillies’ words apply not only to scientific peer review but also to engineering practice in many cases. I’ve watched many a brilliant engineering solution deep-sixed because it was “different” or because it was simply not understood – and so not funded. And I’ve watched a lot of garbage solutions end up creating major problems later – usually after the perpetrator ensured that they no longer had to live with the results (usually by leaving the operation) . Many times bad solutions were adopted, not because they were worth having, but because of who did the design. And more than once a bad design embodied more than one of the above.
I’m happy that peer review is coming under scrutiny. It needs that. Now if we could only get a similar review of engineering methods – but I won’t hold my breath.
And the strange thing is that I still feel that engineering practices are better than the peer review process. Probably partly because I’ve also seen some “better even if different” engineering solutions adopted as well. :-)
Thank you.
“Gillies’ words apply not only to scientific peer review but also to engineering practice in many cases. I’ve watched many a brilliant engineering solution deep-sixed because it was “different”
“Many times bad solutions were adopted, not because they were worth having, but because of who did the design. ”
You must have worked some of the places that I have! ;)
Could be – I was a NASA contractor for 40+ years, mostly at GSFC – Nimbus, Landsat, UARS, HST, TDRSS, +++++.
Well, at least it isn’t the House of Lords enquiring. I suppose such committees can enquire into anything they like, but publishing papers is mostly a private enterprise activity, and its success is determined by the readership. It is the editors who seek peer review to improve the quality of their product. That’s a business decision.
If the HoC decides thay can improve the system, HM Government could start up journals to implement it. Or maybe HM could lean on the RS.
I’d endorse Fred M’s point – the big practical issue with peer review is getting highly qualified people to invest enough time to do it properly. Alternative theorists need to explain how they would solve that one,
“publishing papers is mostly a private enterprise activity, and its success is determined by the readership. It is the editors who seek peer review to improve the quality of their product. That’s a business decision.”
Agreed. I’d be very wary of a govt. remedy to a problem that is caused by the govt. themselves. Peer-review isn’t the problem, the problem is the validity governments and individuals lends it.
Anyone that has been in the climate discussion for any length of time has seen peer-reviewed works touted as gospel only to see it a short time later invalidated or at least challenged to the point of less certitude.
I used to have a load of wonderful ideas about how to fix peer review, but then I took notice of something Nick said while we were chatting in Lisbon.
People do seem to forget that science publishing is a business. That insight might lead to a different tack on the issues.
I think we have way too many journals, way too many papers. using journals as a method of documenting the progress of science is just weird.
Nick, not sure about this, but if the reviewers are doing it for the good of science, i.e., not getting paid, then maybe whatever university or company that has hired them could make it mandatory that they spend 5-10% of their paid time on review.
Lack of credit for time spent reviewing is a common complaint, and correcting that would have some of the same effect. The anonymity of reviewing is a complication here.
Very good point.
Nick – It may be unrealistic to expect a company or even a university to pay for those they hire to do work for someone else on company time. On the other hand, one system that we use would accomplish some of the same goals. It’s called “presubmission review”, and although targeted mainly at research grant submissions, it also applies to manuscripts. The idea is that the grant proposal or the manuscript be reviewed in-house by relevant scientists (from inside or outside) to ensure that the material is optimized from both the scientific and presentation point of view. We occasionally recommend that a proposal or paper be deferred for further data, or even abandoned, but we almost always have suggestions for improving the product. The process is voluntary, but reviewers do get credit, which is justified by the University in that it enhances the University’s record and reputation. Those who undergo review consistently achieve higher grant funding percentages than those who don’t, for whatever reason, and all appear to appreciate the experience. We have no data on the success rate in terms of journal acceptance, but it’s reasonable to think that the process has some benefit.
My research organisation reviews all papers internally before submission. It compounds, though, the problem of draining the time of those well able to do the review, and I think overall the result is patchy. The two review layers can clash. However, the most frequent outcome is that the internal reviewers that could be found are not specialists (they are on the author list) and the review is rather on issues of style.
It also doesn’t help with the complaints voiced here about conformity etc.
I’ll just note for the record Nick that your comments on the business aspects of this have me thinking differently. not about the goals (better science) but about paths to that. thrwoing out ideas about what journals should do is easy.
If it can’t detect fraud and it doesn’t detect blatant statistical or software screwups, the present peer review system can’t be worth much. If it also stifles innovation and advancement, as Gilles contends, abandon it.
Or at least instruct the public that it means very, very little. That way we could look forward to not having idiots recite “peer review”, “peer review”, “peer review” any more. That would surely be an advancement for mankind.
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m28.htm (peer review horror story)
And a grandstand for climate change:
”
14 Another area of global concern is the Amazon. 2010 has seen another “100-year probability” drought, the second in five years, with some major rivers almost dry. A multiyear repeat would leave very large areas vulnerable to a catastrophic forest fire. This would increase global CO2 level instantaneously as well as removing a portion of the world’s ability to absorb carbon from the atmosphere. (This is of course on top of, and separate from, deforestation.) Again it cannot be proved that these droughts are a direct result of global warming just as it cannot be proved that any of the following are:
— Central Russian heatwave last year.
— floods, typhoon and fires in Australia.
— UK record cold temperatures and snow (100 year low December 2010)
— East USA record snowfalls down to Kentucky ! (never !)
— record high temperatures in Greenland.
— reduction in Indian monsoon on decadal timescale
15 There are however scientific papers linking each of these, and the Amazon droughts, to the effects of global warming. (3) No one paper can be proof, but at a 10% level, each of these might be caused by global warming. If all are taken together, then it is more likely than not, that global warming is the cause and that these symptoms will get very much worse as overall world warming continues from the present 0.7°C towards 2° and beyond.
Surely, again, it is on this basis that public policy decisions should be made and not on the basis that we can’t prove that these events are caused by global warming.
”
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m30.htm
Harold- your idea that countries around the world should/will adopt energy policies eliminating CO2 based on some report “fearing” a potential fire in South America…..is thankfully unlikely.
Not my idea, it’s from the testimony I reference right below the long excerpt.
Don’t make stuff up. Cite the purported scientific papers that support the nonsense you have listed.
Ken and Rob,
Those are quotes from the submission he linked to at the bottom of his comment:
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m30.htm
The one he referred to as a “grandstand”.
A great many advances in science have been held up by peer review. Why? Because the conclusion of the sudy contradicted something another scientists in a position of power had proposed. Until that scientists retired or died, the new discovery could not receive a fair hearing.
Peer review should not concern itself with the conclusions of the study. Rather, it should only be about the methods. Was the study conducted in a scientific fashion, with appropriate controls so that some faith might be placed in the result.
IF the study follows the scientific method, if the math used to analyse the data is sound, if the appropriate controls were used to eliminate bias, then it should not matter one iota to the reviewer what the study concludes, so long as the conclusion follows from the findings.
Any study that uses the word “could” or “might” should be rejected out of hand.
Ferd-
“Peer review should not concern itself with the conclusions of the study.”
No, conclusions are actually the important part. The rest of the work ( data gathering, analysis, methods used, etc) has to support the conclusions. I don’t view that the conclusions , however, have to be justified / reconciled with previous theories / papers. etc. If the paper doesn’t adequately support its conclusions, it has to be written before publication.
Consider the conclusion of a study that says “bacteria cause ulcers”. Now, why would that study be rejected? Because a lot of people didn’t like the conclusion. They stood to lose if it was accepted. That is the reason they would oppose the study, regardless of how well the study was conducted. This type of problems is at the heart of peer review problems. They aren’t reviewing the science in how the study was conducted, it is the conclusion they are really evaluating, then working backwards to try and find holes in the methdology if they don’t like the conclusion.
So, you end up with studies that say “X causes global warming”. You can prove just about anything with numbers, so long as you can control the data going into the sample. The reviews are quickly approved, not because of “X”, but because of “causes global warming”. The question the review should have asked was “who controlled the data?”, but instead they worked backwards from the conclusion. They liked the conclusion, so they likely the report.
“They aren’t reviewing the science in how the study was conducted, it is the conclusion they are really evaluating,”
Yes, how it fits into their view of things and existing work. How a conclusion fits into the larger body of work is the part I don’t think is necessary, but it’s usually done.
Ferd, the conclusion must be scrutinized so it can be determined whether or not it follows from a sound methodology.
Well yes in a trivial sense. Peer review can look at data, method and hypothesis and then, having confirmed their validity ask whether the conclusion logically follows if data, method and hypothesis have been, or are assumed to have been correctly done.
I’m not sure of your point.
Ferd said that the conclusions should not matter one iota to a reviewer.
It seems obvious to me that in addition to validating methodology, a reviewer needs to examine conclusions and evaluate whether they logically follow from the methodology and data.
Obviously, a reviewer should not use the conclusions in and of themselves to validate the data or the methodology. Can it happen? I have little doubt that to some degree, not that infrequently, a reviewer’s alignment with a paper’s conclusions can bias their assessment of a study’s methodology and data interpretation, but in my experience a blanket assertion that such biases are characteristic of the entire peer review process are unrealistic.
Joshua, Ferd is correct in what he says – the conclusions are examoned first, and if they don’t fit the current understanding, then it’s much more likely the paper is blocked. From a reviewer perspective, examining the conclusions to see that they are reasonable in light of what is generally accepted helps to identify areas for added scrutiny. The time that it takes to actually go through the paper and understand it in detail can be more time than the reviewer wishes to spend, it really depends. How a reviewer handles it depends on the reviewer.
Harold.
Typically when I read an analytical article, I read the abstract, determine what the thesis is, and then and jump to the conclusions. If I agree with the conclusions I might not read the rest of the article in much detail – particularly if a quick reading of the body of the paper confirms that I understand and/or am familiar with the methodology involved. If I have no opinion about the conclusions, or if I disagree with the conclusions, I begin looking at the methodology in more detail, how the data were interpreted, and then how the authors argue that the data support their conclusions.
In the end, I give greater scrutiny to articles where I have no prior opinion on or disagree with the conclusions, trying to find holes in the author’s analysis or to understand better why my view of the matter at hand was lacking.
If a peer-reviewer follows a similar process, it is likely that they would miss poor methodology or poorly interpreted data if they agree with an article’s conclusions. They would give greater scrutiny to articles with conclusions that they didn’t agree with prior to their reading. Assuming that peer reviewers do follow such a process (at least in some measure), it seems likely that a outcomes of peer review would reflect the biases of the reviewers. Entirely plausible. A reviewer should not differentiate in their level of scrutiny based on their agreement or lack thereof with the conclusions. Ideally, reviewers should look with great scrutiny at the methodology and data interpretation even in studies where they disagreed with the conclusions. Not doubt, we live in an imperfect world.
However, Ferd is saying that the reviewer should disregard the conclusions. As I see it, a huge component of a reviewer’s objective should be to ascertain the validity of an article’s conclusions (along with the validity of the methodology and data interpretation that supports those conclusions).
When you’re saying Ferd’s right, does that mean that you agree that if a reviewer reads an article, finds the methodology and data interpretation to be valid, they shouldn’t bother to assess whether the author’s conclusions are supported by the data? Really?
I would think that if reviewers routinely follow the process that Ferd recommends, the outcomes of peer review would be significantly less valuable than the outcomes of a process which, at least to some degree, in aggregate, reflects the biases of the reviewers.
“When you’re saying Ferd’s right, does that mean that you agree that if a reviewer reads an article, finds the methodology and data interpretation to be valid, they shouldn’t bother to assess whether the author’s conclusions are supported by the data? Really?”
No, this is what I said:
“Joshua, Ferd is correct in what he says – the conclusions are examoned first, and if they don’t fit the current understanding, then it’s much more likely the paper is blocked.”
Gillies, quoting Black: “The anonymous peer review process is the enemy of scientific creativity. … Peer reviewers go for orthodoxy.”
Our local shining examples are Leif S. and Fred M. Fred sez, above: “To the best of my knowledge, no important advance in modern science has been stifled by peer review, but at worst, only delayed.”
Um, how would you know if it were successfully ‘stifled’, Fred? By definition, it’s then dead until some future generation revives it. There may well be dozens of such cases, and you’d be the last to know or suspect.
As for alternatives and fixes, I don’t have access to Gillies’ book, but he references: “I also believe that it would be easy to eliminate most of the present use of peer review. However, I cannot, for reasons of space, give here the details of how this might be done. They are to be found in my 2008 book [Gillies, D. (2008) How Should Research be Organised? College Publications], Part 3, pp. 63-130.”
So your lack of imagination is not universal.
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m33.htm
Proposes replacing current peer review process with an on-line pre-publication discussion forum open to many.
”
1.24 Peer review cannot usually detect intentional fraud or artefactual results caused by experimental methodology that has not been described by the authors.”
We’ve seen this…
”
1.25 Peer review (and open literature) is now almost the exclusive territory of the academic research sector which the public cannot access (cost) which results in the public being more dependent on un-reviewed material.
”
The obvious solution is to make it free to the public, and have subscribers pay. I would have thought that a ridiculous business model, but internet business has made me a believer that business models which couldn’t possibly work are probably worth billions…
Left out the location:
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m38.htm
Scathing:
“1.2 These principles continue to be widely ignored, both by researchers and by the peer reviewers who judge proposals for new research and the reports of new research submitted for publication (5, 6, 7). As a result, people have suffered and died unnecessarily (8), and resources for health services and research have been wasted (1). Furthermore, current regulation of biomedical and health research has done very little to reduce these serious problems. Indeed, because of the delays it has caused in addressing uncertainties about the effects of treatments, research regulation has resulted in avoidable suffering and deaths of millions of patients, most of whom have not been participants in research (9).”
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m47.htm
read up to #50. It seems a lot of people want paid reviewers, or at least compensated in some way. Publishers think the review process is great, the rest is a mix.
“As a result, people have suffered and died unnecessarily…”
Is there a count of how many people’s suffering has been alleviated, or lives saved, by discoveries that were validated by peer review processes? How about a count of the suffering or lost lives that might have resulted from malignant theories that might have been implemented absent peer review processes.
Peer review should be subjected to scrutiny. I would suggest that polemics will not serve to create beneficial refinements.
DDT (outright ban) versus effects of Malaria.
I’m not sure what your point is.
Brevity is sometimes counter-productive. Sorry, your point is well taken.
After “Silent Spring”, the politicians got together and banned DDT, which was effective against malarial mosquitoes. Malaria got worse in Africa. (I don’t have a reference to how many died or were crippled.) Nets were not fully effective.
The solution was to spray the interior of residences with DDT.
I would suggest that you read a bit more about the DDT debate from a variety of sources.
Malaria was on the increase in a number of locations where DDT was being used prior to bans which were directed at the overuse of DDT for agricultural purposes, which had a deleterious environmental impact as well as created the development of DDT resistant mosquitoes. As to comparative effectiveness, in a technical sense it depends on which areas you’re talking about and the types of mosquitoes, types of housing, attitudes of the public towards the smell, staining, and other problematic aspects of DDT usage. Proper DDT usage requires well-established governmental controls in countries with significant resources. The statements about “bans” of DDT have also been widely over-stated.
In the very least, if you’re going to make assertions about increased prevalence of DDT attributable to banning of its usage for agricultural purposes, you need to at least attempt to account for the environmental damages that would resulted from continued improper usage of DDT (which continued even after its diminished usage) as well as likely increased prevalence of malaria due to increased mosquito resistance to a primary control mechanism.
Good points. Thanks for the leads.
Joshua: Done. Glad to accommodate. :-)
Edwards, J. Gordon, and Steven Milloy. 1999. 100 things you should know about DDT. Scientific Blog. JunkScience.com. http://junksciencearchive.com/ddtfaq.html
Covers the following list of issues:
I. Historical Background. Discovered by accident, DDT became one of the greatest public health tools of the 20th century. Overuse harmed its efficacy — and made it politically unpopular.
II. Advocacy against DDT. DDT was demagogued out of use.
III. EPA hearings. DDT was banned by an EPA administrator who ignored the decision of his own administrative law judge.
IV. Human exposure. Actual human exposures have always been far lower than the “acceptable” level.
V. Cancer. DDT was alleged to be a liver carcinogen in Silent Spring and a breast carcinogen in Our Stolen Future.
VI. Egg-shell thinning. DDT was alleged to have thinned bird egg shells. To the extent egg shell thinning occurred, many other substances and conditions could have been responsible.
VII. Bald eagles. DDT was blamed for the decline in the bald eagle population.
VIII. Peregrine falcons. DDT was blamed for the decline in the peregrine falcon population.
IX. Brown pelicans. DDT was blamed for the decline in the brown pelican population.
X. Bird populations increase during DDT years. Widespread declines in bird populations during the DDT years is a myth.
XI. Erroneous detection. Gas chromatography was universally used for pesticide analysis in the mid-1960’s.
But it often failed to differentiate between DDT residues and other chemicals.
Note: The information presented here has been largely drawn from materials compiled by J. Gordon Edwards, professor of entomology at San Jose State University. Dr. Edwards testified at the 1971-1972 EPA hearings on DDT. Some research and all editing/formatting was done by Steven J. Milloy, publisher of junkscience.com.
Joshua
Good point about the running total; I do think I cautioned about its accuracy.
I think if you go to the Edwards paper, you will find the sources cited by Edward’s summary are [square bracketed]. No, I have not read them all. I have read parallel items. One could race down a never-ending rabbit trail forever reading every possible published item.
Dixie – I really don’t think that you can take Edwards’ info at face value. His dismissal of the issue of mosquito resistance seems like advocacy more than science. Take this, for example (from Wikipedia):
More from the same source. To be sure, the clock (currently 104,609,969 avoidable deaths since the ban took place) is just one of those running totals based upon the passage of time. As for the source, Milloy is admittedly no fan of eco-regulation.
Milloy, Steven. n.d. The Malaria Clock — A Green Legacy Of Death. Scientific Blog. JunkScience.com. http://junksciencearchive.com/malaria_clock.html
Dixie,
I suggested that you read a variety of sources. Gordon and Milloy are clearly outliers. That doesn’t make them wrong, necessarily, but if you aren’t reading a variety of sources then you won’t see that they are outliers. And they are clearly banging a political drum. Take, for example, their dismissal of resistance as an issue because mosquitoes avoid DDT; yet, well before DDT was banned for agricultural usage, countries were shifting away from using it because of resistance problems. If your source fails to account for such factors, and then goes ahead and creates a “calculator” without accounting for such factors, it needs to be taken with a grain of salt.
Judith,
Lawrence Souder link doesn’t work.
Full list of submissions here:
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/contents.htm
Peer review is the quality control system used to separate science findings/ideas/theories “wheat from the chaff”. The UK HOC inquiry shows there appears to be problems with peer review recognized by the scientists who use it. (Those of us interested in or are actively participating in the field of climate science recognize the same problems with peer review)
So we have a quality control system and the system is recognized (by its users) as not being perfect.
What mechanisms are in place to review/change peer review policy?
When I finished graduate school I left academia for commercial/industrial endeavors. Standardizing methods/processes/systems is common course of action. Standardizing organizations exist, such as ASTM or ISO, to review and change standard practices, methods, specifications etc.. on a periodic and frequent basis, to keep them current. Experts in the various fields of technology, which use such standards, belong to these organizations to commonly maintain them.
Take Aerospace Manufacturing as a further example. A world wide quality control auditing organization, PRI/Nadcap, exists to bring ALL the prime manufacturers together to standardize processing and testing auditing for the their supply chain vendors. Any business who wants to supply special processes, such as heat treating, chemical processing, NDT…etc, to the aerospace community must acquire and maintain Nadcap accreditation. Everyone who participates is audited to the same checklist (set of rules and policies). Task groups, comprising of aerospace manufacturing prime QA representatives and the suppliers for them, work together to review/change/update the audit checklist. The basis for this is to ensure all the components manufactured and assembled together to make aircraft are done properly, are traceable, are to specification and are using the best accepted practices to ensure the number one issue… SAFETY.
Is there such an organized effort available within the scientific community to ensure scientific findings generated by researchers and published in journals is done so in a standardized, universal and periodically reviewed process? If not, why not?
Quality control is needed in 2 areas: reviewers who are nasty and obviously biased should not be used again, but since there are so many journals, editors don’t find out about a reviewer until they use them; and editors who do not over-rule or ignore nasty/biased reviews should not continue editing, but I have never heard of this happening.
I hope somebody asks the question: For what problem is peer review the solution?, and what shall it cost a knave to violate the intent and trust that peer review promises?
One cost rises to the top: loss of tenure and position
Here’s another: banishment from the community of scientists. Let them pound nails thereafter.
They’re coming in a flood: Retraction of all papers to which they were principle contributors.
And no cookie!
A small player in the academic world, I have experienced the highs and lows of peer review, and struggled with doing some reviews. I say that part of the solution is for people and groups to actively do Open Science. For the last year I have been doing just this with a modest project that is gradually maturing — see: http://www.oz-energy-analysis.org/
I commend this approach to others, and hold it up as an example (the project content being another matter). For a place on the OzEA site to comment on Open Science, please see Discussions / Open Science. Would like to make more extensive comments here, but have a three day bush walk to leave for.
Here is my suggestion
Do away with the peer review process.
Publish papers on the Internet, which are accessible to anyone for a small fee.
Let others publish on the Internet to refute or confirm the papers.
Let still others publish literature review on the Internet on the subject.
This does not allow the “redefining of the peer review process” by the powerful.
The problem with pier review is that eye flaps and tunnel vision develope
with existing knowledge. It is frustrating to see many scientists on the cutting edge, not having a chance to have their papers noted. One only needs to look at the last 100 years for such.
Dare I say look beyond?
The problem we have is that we do need a filter of some sort, we need science people with modern understanding of current day to day issues and that the process needs to be updated.
My suggestions:
1- All reviewers names and positions should be published with the paper they reviewed.
2- Make it an ‘offense’ to review a paper that directly critiquies one of your own (not the same as denying a right to reply- the author can be consulted, but not allowed to review).
3- Have a 2 stage review process:
Stage 1- the current system (though modified as above)
Stage 2- when the work is replicated indipendantly (which i always thought was a pre-requisit for the paper to be ‘truly’ peer-reviewed…). A paper is classed to be stage 1 reviewed if it passes the former and FULLY reviewed if it passes the latter.
4- Use relevant expertise in the review- if the paper relies heavily on statistics- a statastician must be employed. If it relies heavily on atmospheric physics, then an atmospheric physicist must be employed, etcetera.
5- no paper will be considered without fully archived data, model info, calculations and other critical information available at point of review. This supplementary information should be available for purchase alongside the paper and open to requests from future reviewers.
These 5 easy steps would solve it pretty much over night.
Wait a second. How do these examples indicate any problem with “peer review”? Wegener did publish his ideas, right? So did Prusiner.
Argh. Ignore the above message (somehow it got teleported to the wrong thread).
Bob Carter and Lewis (ryan et al) may be off interest ref climategate gatekeeping and Steig reviewer A
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m28.htm
http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m15.htm
When you type a query into GOOGLE, you get back a pretty good selection of the most useful responses. GOOGLE grew out of its ability to do this. The system works (as I understand it) by examining the whole graph of references between web pages.
I wonder if something vaguely similar could be set up for papers published on the internet. Each paper would contain references to other papers, together perhaps with some “weak references” – references that the author did not wish to be considered as endorsements of the other work. Maybe links with negative weight would be permitted too, “I have read *this* paper, and this is what is wrong with it….”
Searching for a scientific paper would then be an automatic process based on graph theory. Nobody would be censored, but it would be clear to the casual reader when a paper was not mainstream or otherwise not worth reading.
Just because something is not mainstream does not make it ‘not worth reading’.
This method is effectively just another ‘pal-review’ where the most citations gaurantee prominence, rather than worth of content.
No – because the ‘pals’ get picked automatically. Say Judith publishes a paper that critics like, it will rise up the rankings, just because others refer to it in a positive way. GOOGLE have solved essentially the same issue – we all rely on it every time we use the internet!
One such location is the non-peer reviewed arxiv, and blogs on scientific themes are another. How do these fare?
In my experience, when it comes to scientific innovation, the answer is dismally. Both sources are dominated by poorly informed and often nonsensical content, sometimes emanating from individuals who fancy themselves unappreciated geniuses the world has rejected to its detriment. Genuine scientific advances have been at best rare, and significant advances non-existent to my knowledge, although obviously I may have overlooked examples.
.
Just top of mind I’d mention this : http://arxiv.org/abs/math/0303109
It is true , this paper is not merely significant . It is revolutionary and solves one of the most difficult problems (Poincaré conjecture) which was worth not only a Fields medal (Nobel in mathematics) but also 1 M$ from the Clay institute.
I could mention many Arxiv papers with top authors for instance in QFT which are major too. Etc.
I consider that Arxiv is at least as good as the classical “peer reviewed” process.
Even if there are some dubious papers, in balance it is good.
Actually I’d say that peer review is best for pedestrian papers, those that are easy to read and extremely consensual.
As soon as a the preprint is technically difficult and, God forbid, challenging the installed nomenclature (you know all this small circle of established “Godfathers” where the reviewers are always chosen) the peer review may become an unpassable barrier.
It is well known that the amount of open mind and the ability to change one’s opinion decreases, sometimes dramatically so, with the age and the amount of what’s at stake.
As the peer review is dominated (more so in physics than in mathematics) by old people with everything at stake from the carrier point of view, it is an understatement to say that a challenging paper will not have an easy life to pass through.
Of course not all challenging papers are correct and not all challenging correct papers are blocked for a long time.
But this is clearly an issue which has been partly solved by places like Arxiv.
Although I was referring primarily to scientific findings that were rejected by peer review, and I have no familiarity with the mathematical literature, are you suggesting that the particular paper you cite was in arXiv because it could not get past peer-review into a journal or other standard means of describing advances in mathematical theory? In other words, was the author compelled to use arXiv because of universal rejection elsewhere? My own familiarity is with papers in areas of science of interest to me, including climate science and biomedical science, where peer-reviewed publication is the accepted practice for authors who want their work to be taken seriously. If there are exceptions in other scientific disciplines, I have to admit I was unaware of them.
Fred,
in physics, Arxiv is sometimes used as a ‘preprint’ of sorts. Meaning a group knows that the bulk of work they’ve recently done will get published, but they want to get it to the community faster or in what they feel is a more compelling/understandable form. They may even feel that some work that was not part of a published paper deserved recognition in light of what they have published.
But it should be noted that I have read several papers in the optical/chemical physics journals that have CITED Arxiv papers. In science, I don’t think there is a higher form of flattery.
Maxwell – Those are good points. I was focused on addressing the question as to whether arXiv or other non-peer reviewed sources provide evidence that major advances are denied publication because they are rejected by all the relevant peer-reviewed journals. I haven’t seen such evidence, but it may exist in scientific fields I’m unfamiliar with.
Fred,
I interpreted your comment as making two points. First, that peer review hasn’t completely held back the advance of controversial science indefinitely in known case, with which I agree. Second,
‘My own familiarity is with papers in areas of science of interest to me, including climate science and biomedical science, where peer-reviewed publication is the accepted practice for authors who want their work to be taken seriously.’
to which I was pointing out, in physics at least, readers of arXiv take the submissions quite seriously. Perusing the chemical physics list, most of the submissions have been also submitted to main stream journals.
It would be interesting if someone did a study on whether the work on arXiv is accepted to the desired journal, at what rate and how much of it gets accepted at any journal with what rate. We could put it on the arXiv.
So I agree, peer review seems to work at advancing science, albeit slowly in the case of some discoveries. But I think that in physical science at least, open-access journals like arXiv seem to play an important role in communication between serious researchers. Maybe there is a larger role for it to play in other fields as well.
Yes, my focus is on your first point. I don’t believe that with today’s plethora of journals, peer review delays any important advance very long at all, and the problem is more the papers that should have been delayed (or discarded) but aren’t.
Maxwell,
Can you tell, whether people in those fields follow regularly, what is published in arXiv, or are they informed through other routes about the publications. In this latter case arXiv would serve rather as a depository for the reports than a publication channel in the same sense as journals.
Years ago (in 1970’s), when I was working in theoretical elementary particle physics, the preprints of well known institutes were the real publication channel. Most important results were distributed as preprints and more or less outdated, when they came out in journals. The role of journals was more one of a permanent storage place several months or a year after everybody had already read the paper.
Pekka,
I was starting to notice a little of the opposite effect with respect to the arXiv. There were papers that had been published in well-read journals months ago, that are just now getting on the arXiv. It was almost that one or more of the authors felt that they might get a larger audience by posting it on the arXiv than in Physical Review B.
Is the arXiv becoming the repository?
I think something, as a researcher, to keep an eye on.
In my field (experimental quantum computing) most results are published both in journals and on arXiv. The standard stuff is often put on arXiv first and then submitted immediately afterwards: for journals in the Physical Review family you can even submit a manuscript just by giving them the arXiv reference number, and this is how I handle most of my work.
The top level journals like Nature and Science are a bit more precious about this sort of thing, and so authors will frequently wait until the paper is accepted or even printed before copying it to arXiv, but even so almost everything turns up eventually: journals which tried to enforce an absolute ban on arXiv preprints found that they didn’t get man manuscripts submitted.
In terms of learning about new papers, arXiv is unquestionably my dominant source: I read the article list in quant-ph first thing every morning.
Theoretical quantum computing has gone a stage further: many key results are only found in arXiv preprints and have never been formally published. That may perhaps reflect the fact that theorists are much more comfortable about their own ability to assess how good other researchers are, and don’t feel a need for the imprimatur of impact factors and H-indices.
Perelman is a real exception. He has declined tenured jobs at top universities already in the 1990’s, and he has declined the Field medal that was awarded to him in 2006. He has also left his job in St. Petersburg and told that he is now working with something else than the mathematical theories that he has developed.
It’s unlikely that would have had difficulties in getting his work published elsewhere. He was also so well known in 2002 that his paper in arXiv got rapidly as much attention as a paper in mathematics can get.
Remember that peer review is a workflow procedure in the publishing industry. Nothing more.
Science thrived before it came along and science will thrive after it’s gone.
Gillies says: “This shows why Type 1 errors are much more serious than Type 2 errors, and why funding bodies should make sure that some funding at least is given to every research school and approach rather than concentrating on the hopeless task of trying to foresee which approach will in the long run prove successful.”
I would say right diagnosis, wrong prescription, at least in part. What he is describing is what conservatives mean when they criticize centralization of decision making authority outside the private sector as “picking winners and losers.” Peer review as described above is simply another example of letting a select “elite” decide what is and is not valuable to pursue.
I would suggest that the error in the prescription is in the recommendation that “funding bodies should make sure that some funding at least is given to every research school….” Funding bodies, whether governmental or quasi-governmental, simply will not operate this way. It is antithetical to human nature. The incentive for any bureaucrat in such a position is to avoid funding the “wrong” alternatives. “Wrong” in this context usually means that which is contrary to the bureaucrat’s own opinion. Bureaucratic committees are no better than bureaucratic individuals in this regard.
The best way to obtain the type of open ended research proposed by Gillies is to remove the government, or other centralized groups (including peer reviewers), from the process. Trial and error on numerous fronts will almost always have a higher probability of success than a centralized process where a select few decide who gets funded, in accordance with their existing opinions. This decentralized system already exists in the private sector, think pharmaceuticals, computers, telecommunications. There is no reason to believe it will be any less successful in areas of energy development, adaptation etc.
“Peer review” could still be valuable, in a decentralized system, if it were a process of verifying research results, ie. detecting errors in proposed papers by attempting to verify methods and replicate those results. Maybe we could call it…peer auditing instead. Not as impressive sounding as peer review, but perhaps more useful given what we have seen lately.
I see peer review as a tool for speeding up the progress of science, not a method for giving a final quality judgment of individual scientific papers. It is needed, because any scientific journal can publish only a limited number of papers and because each individual scientist can read only a limited number of papers written by other scientists. We need a filter that helps in selecting the papers that fall within the limits of publishing and reading. A good filter passes with high probability the papers that should be published and also makes them better before they are published. A good filter also blocks a majority of papers that are finally worthless and returns for further processing those, which have valuable content but also severe deficiencies.
The present peer review system does not perform ideally, but it has the essential additional advantage that it is itself not regulated but maintained by the science community. The informality of rules is in my mind a very essential feature of the scientific process. The alternative approaches to peer review are continuously in competition and free to develop, when needs change.
The scientists criticize the peer review system, when good papers have difficulties in getting published and when they see a paper published that is in their view not worth it. The non-scientists have recently criticized the peer review system, when it does not perform tasks that it’s not even supposed to perform. Many of they appear to think that peer review should be a quality control of the conclusions presented in the paper, but making peer review to do comprehensive quality control is impossible and trying it would be detrimental for the progress of science. Another group of non-scientists would like any text claimed by somebody to be science to have the same routes for publications. This is not realistic, as I discuss above.
I disagree, i see the biggest weakness of the peer review system to be the lack of quality control. The cAGW theory would not be in the mess it is in now were it for proper QC at the peer review point.
Academia likes to think it can play by different rules to the rest of the (scientific) world, blissfully unaware that it opens itself up to massive (legitimate) criticism by doing so.
Academia and science in general would benefit greatly if QC procedures were in place at peer review.
I think that you understood that my view is that quality control may be very important, but that the peer review cannot satisfy that need, and trying that too strongly would be detrimental for science.
I missed that distinction- apologies.
Though i’m sure you could quite easily include QC in peer review. All industry scientists are trained to adhere to and operate under strict QC. To say that adding this extra ‘string’ to the academic’s bow would be inordinatley difficult is, odd.
If anything it will aid their work and make their own lives easier (especially wrt FOI’s and data requests). No, i think it is easy to implement and wholly beneficial to science.
Science is about creating new knowledge, very often using new methodologies. These activities are prone to error and insisting on some QC procedures would seriously slow down the progress.
There are separate tasks within the scientific work, where QC is appropriate and should be applied, but that is not the main task of peer review. The reviewers must look at the paper, insist that it is well enough written and tells enough about the work done, but the overall judgment is very much influenced by the novelty of the work, and often the parts that are most valuable from this point of view are those, where formal QC is impossible, or at least impossible for the reviewers.
The reviewers are not auditors, their task is to give an overall evaluation of the paper and to insist on improvements to the serious weaknesses of the paper.
Yet part of the QC process revolves, significantly so, around the soundness and quality of the data to which you are basing all your conclusions on.
I am by no means suggesting that a full audit be performed for each paper, but is it too much to ask for the reviewers to check the equipment used was calibrated? Or that the raw data has been treated properly (and is included)? Or that the methodologies followed were controlled enough to eliminate mitigating factors?
QC DOES add time to research and experimentation (i do it all the time)- but it ensures quality work and that any conclusions are sound and as such i don’t see it as an optional extra.
If i look at a paper and see that the data quality is poor- i generally stop right there as all conclusions therafter are suspect- it’s what happens in industry, so again i ask- why should academia be any different?
To put it another way, you will probably lengthen the review process, but you’ll save much more time in not having to counter obviously shoddy/flawed papers via FURTHER peer review.
i.e.. reworks
Pekka,
You say:
“The reviewers are not auditors, their task is to give an overall evaluation of the paper and to insist on improvements to the serious weaknesses of the paper.”
Not necessarily so. Although the reviewer may not go through a systematic number of checklist questions asking to verify answers, like an audit would, the reviewer still drills into the content of the research and where it is found to be weak or not well explained or lacks good scientific evidence “insists on improvements” or corrections to findings which is what auditing is all about.
John,
I think we do not disagree significantly, we just emphasize differently the words. My comments do also include actions that are in some sense auditing, but I wanted to emphasize that a full audit might involve far too much effort, and that some important parts of the publication may be even speculative, which must of course be stated clearly, if that is the case.
The real QC comes from the scientific process through additional related research. That may take time, and nobody is expected to tell, how far it has proceeded. It is just known by the community, and that’s the beauty of the scientific process. Scientific knowledge gets confirmed, but usually not finally proven.
I agree with you Pekka.
I am less concerned about flawed papers getting through the system, than I am about the failure to publish truly innovative papers or papers that go against the prevailing ideas.
well then we have two issues.
One of basic quality control and one of deliberate supression of opposing viewpoints.
This is exactly the reason that I consider excessive emphasis on quality control to be against the idea of science and detrimental for the scientific progress.
As we need some filter for practical reasons, we have a dilemma that cannot be resolved in an ideal manner. The internet has helped on this point. First of all anything can now be brought publicly available, but perhaps even more importantly, the net provides also an effective way for spreading knowledge on an interesting paper as soon as some people have realized its value.
I agree with you.
I should not have focused solely on that one aspect of what a standardized system could achieve. A standardized system could address your concern as well as part of the overall process. There should be a mechanism that addresses this aspect of publishing scientific ideas. I suspect there is such a mechanism, but does it follow some sort of universally accepted standard method? Specifically, on what basis would an editor be able to reject a claim, how would the editor be able to back up the rejection? Who reviews that mechanism to see if it working properly. It appears the individual journals perform this mechanism “in house” without any outside review.
I am suggesting an oversight mechanism of the review process. Maybe this is not desirable to some or many, but it may be worth looking at. Nobody likes to have there problems open for public display, but in my world it is a normal course of business.
pekka, you’ve clearly not worked in industry if you think excessive quality control is in ANY way detrimental to science.
It helps science 100%. The only thing it takes is time in the initial implementation, but i promise you, it saves FAR more time down the line.
In my view there IS no option- quality control is central to good science.
If i can look at your paper and find errors/quality issues in the data then all your conclusions become suspect.
Labmonkey,
I have worked with industry 20 years, but the question here is about science, not about industrial research.
Hmm. i think you’re splitting hairs. Would you declassify scientific research simply because it occurs in an industry laboratory?
I work in industry, but (currently) work almost exclusively on research projects.
I think you’re talking about the percieved ‘science’ rather than the actual scientific process. I.e. the presented science in journals etc etc rather than the practical aspect- which is where i’m coming from.
I still maintain though, that good quality science is integral to the whole process. I do not accept that academic research should be allowed a more ‘lax’ approach simply because it is in it’s own interests or because it’s difficult.
The place, where the research is done is not the essential issue, the essential issue is in the purpose and goal of the work. Industrial research is typically searching for a solution to a practical problem and it must be brought to the level of reliability that it can be used to that purpose.
When peer review is an essential step in publishing scientific research, the purpose of the publication is usually to bring the results to the knowledge of other scientists, who will not accept it without normal level of skepticism and without knowledge that errors do occur in science. The published work becomes a part of the scientific knowledge and of the incremental scientific process that leads gradually to better and better knowledge. The value of the paper is determined by, what it contributes to this process, and there the novelty of the results is usually more valuable than a comprehensive QC. These two issues are often in contradiction, and this must not slow down the creation of new knowledge.
Pekka,
Understood there are differences between the way industry and acedemic scientists work and to what ends. But acedemics have observed there is a problem with their peer review process (topic of this thread offered to us by JC). What is acedemia going to do about it is the question?
We scientists in industry are offering a doable solution, that should not impede the creativity of the science, but will make it more accountable. Your reaction to our solution, incedently, is completely normal. I myself took your position years ago when I was initially introduced to such a system. I have since been “enightened” to its benefits.
Curryja, 3/18/11 10:24 am,
When you wrote,
I am less concerned about flawed papers getting through the system, than I am about the failure to publish truly innovative papers or papers that go against the prevailing ideas.
you seem to have taken a view suitable and applicable to any science in the public domain. Science qua science. Your own topics show the problem has boiled over into technical challenges to the AGW model, and that pro-AGW forces have laid their challenged science before the public for funds and regulation, all without debate.
The public is currently buying out potassium iodide because the scientifically illiterate media is marketing panic. Japanese workers “frantically” trying to prevent a “meltdown”. Fallout drifting over US territories. Another Chernobyl. Ever widening safety zones: did I say 20 miles? It’s 50 miles; no, no, it’s “Evacuate Japan!” Three Mile Island failure upgraded from a 4 to a 5, then to a 6, vs. the ultimate, Chernobyl at 7. The government is lying! This is a 6 plus. Where are the helicopters? Where are the troops?
The broadcasters regularly reveal their ignorance of the difference between extreme background levels of radiation, and the fatal flux from nuclear bombs. It’s all Radiation! They are ignorant that potassium iodide is far from certain to lessen the risk of radioactive iodine poisoning, and let their audience conclude KI is a cure for the coming radiation epidemic.
No climate threat would exist but for the IPCC. The public readership of peer-reviewed scientific or pseudo-scientific papers is approximately zero. It’s far less than its appetite for junk science. The public is being asked to accept severe regulations over energy usage, private and commercial, with its reduction in their standard of living, and generously to fund insatiable and skewed, peer-approvable projects, all based on IPCC’s panic mongering. As the President said recently, “This is not the time to panic.” I.e., panic later.
This reminds me of my son who tells of an actual fire in his grammar school when he was in the third grade. The teacher who had been in the hall, burst through the door, and to the kiddies yelled, “Run!”
What the higher authority, i.e., the public, not science, needs is an appreciation of how the IPCC story came to be, drawing as it did from the peer-reviewed scientific literature. The public and their representatives need to know that what has been and is being published in those journals is tainted as acceptable dogma. They need to know not about competing models or alternative data sets to the IPCC Reports, which they won’t read either, but about the fatal errors in IPCC’s AGW model. The problem should not be reduced to a legalistic adversarial contest between experts or political parties, but to the basic offenses to science literacy being foisted upon the public. It should be the substantive domination of science, with all its limitations, over its wayward stepchild, peer-reviewed climatology with its manufactured certainty, consensus, and sometimes even physics.
John,
I do not think that there is a problem, rather there are all kind of problems and deficiencies in the process. Perhaps the largest obstacle in improving the process is in the fact that good peer review needs resources, and it needs resources of the best quality. Those are the same resources that are producing the original science and it doesn’t make sense that the review would take a very large share of the best resources.
Consequently the review goes on basically as before unless somebody can develop better alternatives for that. Certainly it is possible to provide reviewers with better guidance and the editors can improve their part in the process. The reviewers are, however, essentially volunteers that cannot be commanded. If the journals start to set too strict requirements for the reviewers, they are soon without reviewers. Making reviewing a paid activity is also problematic due to the costs, and it might lead to other problems as well.
The present peer review system has developed over years to what it is. There are certainly many good reasons for its present form, but it’s hardly a unique optimum, and it may well be that analyzing its working will lead to significant improvements. It’s certainly worthwhile to bring these issues to open discussion every now and then. The Internet and other recent developments may also mean that something that was the best choice in the past is not any more as optimal.
Pekka, you make some interesting points; specifically the dissemination of the gained scientific knowledge (especially in industry).
My point is that it wouldn’t slow things down in the academic field to have QC. Think of it as the best of both worlds- non profit based science (mostly) with very accurate data and quality. win win!
Pekka,
I have inadvertantaly left out a critical aspect to my argument that, after reading through the thread, needs to be explained.
When I speak of oversight and review of the peer review process, I mean oversight by a third, unbiased and neutral party.
Let me return to my example upthread of aerospace manufactering. The manufacturing primes who make the aircraft want to be sure the supply chain is making their components properly. In the past, they would perform the auditing themselves to ensure their needs were addressed. There were many problems associated with this approach, especially when one supplier might perform services for multiple prime manufacturers.
A third party auditor was determined to be more objective. The hitch was to get all the prime manufactuerers to “buy” into the system and for them to all agree on what the baseling of the audit should be. It has taken years and years of volunteer effort by all interested parties to make this happen. Only the third party auditing entity, PRI, charges a fee for the actual audit. The primes and the suppliers have to self fund themselves to do all the rest. The auditors are typically experienced, retired, experts from the industry.
I understand that this is a far cry from the way acedemics operate, I have been there before, and is like comparing apples with oranges (but they are both fruit). The idea of a third party, neutral, unbiased entity available to oversee the peer review process that takes place between journal, reviewers and submitters should be examined as a way to provide a more “fair and balanced” result. I don’t know how this can be achieved, only that it is one possible way to approach the problems with peer review at hand.
Thankyou for a very good discussion, I hope you now understand what I am suggesting.
Pekka,
Please don’t misunderstand what I mean by “quality control” as I discuss upthread. I was not saying peer review is used as QC for scientific conclusions. What I meant was, peer review is there to weed out those papers with no scientific merit vs those that do. Is this not the case? As a simple example, I probably couldn’t get a paper published in a respectable journal on perpetual motion. The reasons are obvious.
What I am asking above is: is there a standardized framework upon which the peer review process is built that is periodically reviewed and updated by the users of the process? I think the asnwer is no, I don’t know since I am no longer involved with acedemic research. Such frameworks are common in other fields of discipline. Why could one not be adopted in publishing of acedemic research?
Judith,
The problem I have is that current laws and theories were put in place and are too generalized.
This then created a problem of being accurate as they were traditionally passed down and taught and expanded upon.
A famous E=MC2 is incorrect.
Why? What energy is being represented?
There are many different types from rotational energy, stored energy, compressed energy, forward momentum energy, electro-magnetic energy, gravity, solar energy, passive energy, centrifugal force energy, kinetic energy, infused energy, etc.
So, unless we understand individually how these interact, the generalized theory makes no sense.
This then brings us to the peer reviewer that protects the traditional science as fact and will disregard anything that may correct of interfere was has been taught to them.
Judith,
Why should I “bust a nut” to contribute to the world of science?
Overall, I understand the how and why we were created and the mechanical process that has taken us to this point in time.
But it greatly conflicts with the current theories even though I can back up my research with far more physical evidence, mechanical evidence and time lines.
There seems to be a serious misconception here about the nature of scientific publishing. It is a multi-billion dollar industry, and very competitive. The competition is to publish the most important works. If important works are missed it is not deliberate, it is a blunder.
I agree that that this competition exists at a high level (e.g. administrators in publishing houses, professional societies), but at the level of individual editors and reviewers, more parochial interests take over.
I have served as a reviewer of both papers and grants. Basically the only question I was asked was how important is this stuff? On the grant side NSF has a nice ranking form for asking this question. As I recall there are four or five levels of importance.
This is not to say that science is not driven by fads, fashions and fallacies. Of course it is, because we are talking about human communities deciding what is important at the time. I have used a disease model to track fads, because ideas are contagious. See my “Population modeling of the emergence and development of scientific fields.” http://www.osti.gov/innovation/research/diffusion/PopModeling.pdf
Nanotechnology, informatics and genomics are classic examples of scientific fads. But the fault lies not in peer review, but in human nature, which we are not going to legislate away. More specifically, the problem with climate science is politicization, not peer review.
I would agree. And I would add that it is a fundamental problem of human nature, not only a problem with those who that think that GW might be A.
What I find interesting is when people (on either side of the debate) say that they aren’t politically motivated, or who say that they are only interested in the validity of the science, attribute those problematic human attributes only to folks one one side of the debate, or who think that they are more characteristic of folks only on one side of the debate.
Dr. Curry,
Thank you for this post. I have not had a chance to read any of the papers yet, only the excerpts you quoted. I was very interested in the quote of Richard Horton regarding pre- and post-publication review. In the pharmaceutical world, drugs are continually tested and adverse reactions are reported after marketing approval has been granted. Sometimes drugs have to be pulled off the market. Sometimes papers have to be pulled. I cannot explain why MBH98 has not been pulled. It is a travesty.
MBH98 has not been pulled because if you take the data they used, and apply the methods they used, you will get their results. Scientists have great freedom to play with data and methods. Doing science is not like selling drugs. It is how MBH98 was sold in the policy marketplace that is the problem, but that is not science.
David, the methods used in MBH98 were invalid resulting in an Artificial Hockey Stick when the data is trendless red noise. The paper is not valid and should have been pulled years ago. It was found by McIntyre and confirmed by von Storch and Zorita. Everyone knows this.
I think what we discovered is that principal component analysis does not work for noisy data. That is no reason to pull a paper. Papers are pulled for fraud, not failure. Nor do the authors even agree that it failed. I do not support censorship based on disagreement.
David,
Papers are pulled when they are shown to be wrong, not just fraud. MBH98 has been shown to be wrong. Whether the authors agree or not is not important.
Ron, your dogmatism is showing. MBH98 has been heavily criticized, in many cases on political grounds. I followed this closely and in many cases the criticisms were questionable, while in some cases I concurred.
In such a controversial and political context, what mechanism do you propose for pulling papers, when the authors do not accept the criticism as valid? Do you want a Court of Scientific Truth? Do you even realize what you are asking for, which is basically political censorship of science.
The fact that you do not like the paper is no reason to pull it. You even offer a consensus argument — “Everyone knows this.” On the contrary this is probably the most widely discussed paper in climate science history, and many people still support it. It is correspondingly important.
David Wojick,
MBH98/99 was comprehensively discredited by McIntyre and McKitrick, as later conformed by the Wegman committee and the NAS panel, both under oath before a US congressional committee. One can read Andrew Montford’s book for a detailed chronology of what happened.
The Wegman committee concluded:
[Wegman was Chair of the National Academy of Science’s Committee on Applied and Theoretical Statistics.]
To the specific question by Committee chairman Barton:
Climate scientist, Dr. Gerald North replied:
Statistics professor, Peter Bloomfield added:
So much for the record of the various gentlemen under oath.
It seems quite clear to me that MBH98/99 was scientifically “trashed”. Why a few people discredit themselves by trying to resuscitate it is beyond me. It should R.I.P.
But more on-topic here is the scathing account the Wegman committee gave of the “in-bred” peer review process, which allowed this bit of bad science to slip through, in the first place.
But even more important is Judith Curry’s comment concerning the censorship aspect of peer review:
You asked:
Of course not. But that’s exactly what a consensus-controlled peer reviewed censorship process would be, so let’s hope that the UK committee will expose this if it exists in climate science today.
Max
David,
One can be dogmatic when in the right. I am quite dogmatic that 2+2=4 as well. The fact MBH98 is invalid is just as demonstrable as elementary math. It has been demonstrated and confirmed. To defend the indefensible only makes you look absurd.
I would just add that MBH 1998 made at least one false claim about the method used, namely standard principal components. What they in fact used only became clear some years later – so-called short-centred principal components which seems to be a previously unknown method (or possibly jsut a mistake). Moreover other parts of their methodology were not mentioned in the article – such as how they dealt with the problem that many of the proxies claimed to be used did not extend as far back in time as the supposed reconstruction (what they actually did was splice together different reconstructions each using different proxy combinations). I#d love to see the original peer review comments to see what tehy said about methods.
Ron suggests that published papers later found to have wrongly survived peer-review , whether faulty or fraudulent (perhaps only pal-reviewed), should be pulled. This might include those that failed to publicise all data and code.
David opposes this, on the grounds that this would be a form of “political” censorship of science.
===
Publishing inherently involves gatekeeping, which is a a form of censorship. And if it is wrong to censor by pulling a paper, it is equally wrong to censor by not publishing it in the first place. Which means all papers must be published – which I’m sure we all agree is senseless. But by the same token, not pulling is just as senseless. (And in both cases the blogosphere is still there as a fallback).
Furthermore, if peer-reviewing has been captured and tainted by politics, why would pulling or not publishing a paper be any more or any less politically motivated than publishing it? Is David worried about political censorhip of science, but not political boosting of science?
IMO, a published article should not be pulled, even if it turns out to be wrong. (Reviewer comments and author’s responses should also be published.) Transparency and responsibility are good things, I’m told.
A published article, if disproved, serves as a marker for future investigators. Where an article is total botch, it may warn others not to waste their time. If, on the other hand, it is flawed in one respect, it may inspire others to correct the defect and advance understanding.
Dixie,
I agree that transparency is a good thing, but if you have ever invested time in a paper (perhaps even designing systems and protocols around its conclusions) and then find out later it has been discredited by a later paper, you would understand my frustration.
Perhaps a definition of terms would be helpful. When papers are voluntarily withdrawn by authors, I don’t think the actual hard copies are retrieved from libraries across the country. It is more that an announcement by the author that he has new information that invalidates his former conclusions and life goes on. Actually, pretty rare in the history of science. When a journal pulls a paper, usually for fraud, it is similar. Indexes of literature no longer contain references to these papers.
I am proposing is that a new journal, Post-publication Review, would publish criticisms of papers that would result in the same action – that the paper with the invalid conclusions simply does not show up in indexes or online searches (unless perhaps a science historian searches for the exact title). This would have the result that researchers would not cite the errant paper for support and the wrong conclusions would eventually wither and die.
I thought that the problem was not that principal component analysis didn’t work for certain types of data, but that it had been improperly applied, i.e., that the procedutre had been bungled, which led to hockey sticks from even random data. That would be a case of failure to follow valid techniques for analyzing the data, and thus put the results obtained into the useless column.
That is not my understanding, but it has been a long time. But only the handle of the hockey stick comes from PCA. The blade is the instrumental record. If you get the handle from random data then that indeed shows that it is a result of noise. PCA on random data will not give you the blade.
Actually David Wojick, you won’t. One of the most obvious examples of this is MBH claimed its conclusions were robust to the removal of tree ring data. This isn’t true.
Also, MBH didn’t publish validation results they calculated which their paper failed (while publishing ones which were good). This alone should be enough to warrant retracting the paper.
I think you folks are pushing a new, overly stringent and politically motivated standard for retraction.
You folks? I’m curious just who you are grouping me with.
Regardless, all I’ve said is a paper which deliberately deceives readers by publishing positive results while hiding negative results should be retracted. I say this because readers have no way of judging the paper due to the deception. This is in no way politically motivated.
…pushing a new, overly stringent and politically motivated standard for retraction.
Clever. Charges of political motivation, are dismissed by saying anyone who detects political motivation must themselves be politically motivated.
The net result is an incentive to just let politically motivated junkscience go uncriticised, the underlying political motivation kept off the agenda. Which is is exactly what the consensus wants. More than just clever actually.
There is a certain degree of absurdity in his response, but I don’t think we should assume it was intentional. It seems far more likely he just didn’t think much before he responded.
Then again, he made several more comments after I pointed out the absurdity of his accusation, but he didn’t respond to me again. There are any number of possible explanations, but a certain amount of annoyance is appropriate.
I recommend that if anyone hasn’t done so, they read the story of John Harrison and the chronometer. The book is called “Longitude”. Harrison was stonewalled by a consensus of scientists who wanted the credit for inventing a device that could measure longitude.
What John Harrison illustrated to me is just how vast is the aptitude of some people to both imagine and to take into account a high number of factors impinging on their field of study, in his case, chronometers. We have people like that working in climate now, and none of them know for sure the effect of Carbon Dioxide on climate.
===================
By Newton, no less!
Re UK SciTech peer review inquiry,
Science thrives in industry with peer-review and publication noticeably absent. In the fields of physical science and engineering, industry at one time employed about six times as many PhDs as academe, and industrial scientists could complete projects in half to a third of the time of their counterparts in universities and government labs. This was an observation from a government edict lasting several decades that industry was to carve out their Independent Research and Development projects as funded tasks to universities for execution by professors and their graduate students. Those IR&D projects came from military, other government, and commercial sectors.
Industry relies on the scientific method for what academe ostensibly expects from academic peer-review, but industry relies on secrecy, the reverse of publication and open peer-review. Industry relies not on public peer-review, but on most private superior review. Often industry will eschew patents, relying instead on keeping trade secrets.
If one did not already recognize à priori what the scientific method was, the contrast between industrial and academic research drove home the difference. Peer-review and publication are part of the method as viewed by academics, not by industry. In short, the scientific method is about models of the Real World, whether natural or manmade, which have predictive power.
Rewards in industry come à posteriori from what can be sold competitively at a profit, not from anyone’s list of publications or his rank. They come from products that work, satisfying well the needs of customers. Academia and government laboratories enjoy freedom from that burden, but have instead a duty to their employers, ultimately the public. That duty is ensconced in ethics.
On a personal note, it is the failure of that duty in post-IPCC climatology that first stimulated my interest in the field.
Nor is the Popper concept of falsification any part of the method, although the process of validation of predictions can be rationalized as falsification just to satisfy Popperists. Psychologist Popper misunderstood science, thinking that it was about universal generalizations instead of simple hypotheses. Popper believed definitions do not matter, when they are an essential core of models. He seemed not to recognize the stochastic nature of scientific observations, that every observation has an error, and that every prediction must have a prescribed accuracy. Not all crows are black, Dr. Popper; most crows are black, sir — so far.
At some risk of redundancy for Climate Etc., the following missive from Richard Horton, Editor, the Lancet, needs to published again on this thread.
The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability – not the validity – of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed [ jiggered, not repaired], often insulting, usually ignorant, occasionally foolish, and frequently wrong.
http://en.wikipedia.org/wiki/Peer_review, citing from http://www.mja.com.au/public/issues/172_04_210200/horton/horton.html .
Such peer-review journals practice advocacy science, not science.
Horton published this in 2000, but the failure of peer-review dates from the introduction of commercial interests in the modern scientific journals. Einstein reportedly skipped peer-review for five papers he published circa 1905, presumably because his work broke with the conventional wisdom of his peers. Four years before Revelle & Suess (1957) conjectured that sea water might fractionate CO2 appeared in presumably peer-reviewed Tellus to kick off AGW, Watson & Crick (1953) skipped peer-review in order safely to publish their findings on DNA. Excruciating delay and inordinate risk, and neither scientific quality nor validity, come from peer-review.
Jeff,
Your points are mostly valid and well-stated (although I like Popper’s ideas). The problem, of course, is that industry cannot figure out how to sell predictions about climate. Without a profit motive, industry will not invest in climate science. The only exception I can think of is insurance industry where predictions of coming disasters can raise premiums and increase profits – to the detriment of consumers everywhere if the predictions are not valid. But again, the profit motive tends to skew predictions to more catastrophic scenarios just like the current drive to increase taxes and limit freedoms.
For science to work properly, there must be an even playing field in the open market place of ideas, data and interpretation. We don’t have that today.
Out of curiosity, where did Watson and Crick publish if they skipped peer review? I’ve never heard that story before.
Ron Cram, 3/18/11 12:02 pm,
1. mostly valid? Oh, dear.
2. You are correct about industry and the profit motive. That’s why industry doesn’t invest in green energy and technologies, except by government pump priming.
However, a market could exist for a commercial climate model with predictive power. Gambling on commodity futures is a huge business. If the climate model were near term enough, it might be useful in investment strategies.
In fact, how would we know whether such a model exists? Would it be published? Or, would it be held as a trade secret? In fact we know such a model does not exist. The best model, outside peer-reviewed circles, is that Earth’s climate is governed on all time scales by the interaction of the Sun with the extant state of Earth’s variables. The climate problem in our life times is to predict the behavior of the Sun during Earth’s current warm state, and that is a long way off.
The profit motive does skew predictions, but only when the model is offered for sale. That is seen for software products that predict just about anything, from markets to lotteries. The models with predictive power are kept secret. They might be copyrighted, sometimes patented, and hidden in some other product. If I discover how to predict damaging earthquakes with some usable lead time, I’m not likely to publish it. I’m going to put a black box on the market to do the job.
The libertarian view of a free market is laissez faire, which leads to monopolistic practices. In my model, a free market exists if and only if an auction exists for the product or the function.
Publication of Watson and Crick’s 1951 paper on the structure of DNA in Nature. This paper was not sent out for peer review. John Maddox stated that “the Watson and Crick paper was not peer-reviewed by Nature… the paper could not have been refereed: its correctness is self-evident. No referee working in the field (Linus Pauling?) could have kept his mouth shut once he saw the structure” (Nature 426:119 (2003)). The editors accepted the paper upon receipt of a “Publish” covering letter from influential physicist William Lawrence Bragg.
http://en.wikipedia.org/wiki/Talk%3APeer_review/Archive_1
Wikipedia relegated this observation to its archive. For what appears to be the only criticism and off-point to peer-review at that, read on at the link.
wow…looks like Connolley misses the point…as usual
Popper was an epistemologist; not a psychologist. If you had read any of his books, you would no that.
Ken Lydell, 3/18/11 1:08 pm,
According to Wikipedia, Karl Popper earned his doctorate in psychology. He was also a secondary school teacher. I wouldn’t dispute that his doctorate was equivalent to a PhD, giving him some credentials in philosophy. But then, we’re all philosophers when we opine about ideas. http://en.wikipedia.org/wiki/Karl_Popper
He wrote about epistemology, and some not-too-critical circles regard him as an epistemologist. What might the requirements be for that honor?
Linus Pauling wrote about the health benefits of Vitamin C. Carl Sagan wrote about the nuclear winter certain to come from the Kuwaiti oil fires. Fleishman and Pons wrote about cold fusion. Schön published peer-reviewed articles about nanotechnology. Sokal published a peer-reviewed paper on Transformative Hermeneutics of Quantum Gravity. Mann, Bradley & Hughes wrote about Hemisphere Temperatures During the Past Millennium. Revelle & Suess wrote about the ocean’s buffer against CO2 absorption. Someone’s going to have his name attached to Chaos Theory, the “theory” that some things can’t be modeled, and it might not be altogether flattering. And Popper wrote about science. Some honors are albatrosses.
What counts are models with predictive power — exclusively. That is the antithesis of chaos. Once those predictions are validated, a theory or law results, and sometimes the modeler gets hits name attached. Sometimes the models fail spectacularly, and a name gets attached.
Science is stingy with its honors. Some are good, others not so good.
The problem isn’t the peer-review process. The problem is the weight and validity it is lent. Currently, the impression is if a paper went through the peer-review process, it is seen as gospel. If a paper does not, then no validity is lent to it. The process is a tool and nothing more. If I have a novel thought or idea, it isn’t validated because someone generally agrees with me or couldn’t find a reason why it would be invalidated, nor is it invalidated because I didn’t find someone to agree. It is simply silly to view peer-review in such a manner.
The fact that governmental agencies also lean on this concept is even more troubling.
Worse, that a government wishes to “improve” a private practice shocks me. Governments shouldn’t be so dependent upon this practice or any other process of such. The U.S. employs many scientists. I assume, so to, does the U.K. The governments should use tools available, but should have their on processes independent of peer-reviewed papers in scientific journals.
I am glad someone wrote this. I agree completely. To me peer review is merely a screening system to see whether the science is good enough for the specific publication. Some require very good science; some are prepared to publish things that are little more than guesses.
But we need all types of publications if science is to prgress. Thank heaven for the internet!!!
‘The problem isn’t the peer-review process. The problem is the weight and validity it is lent. Currently, the impression is if a paper went through the peer-review process, it is seen as gospel.’
This is patently false.
In fact, there was a fairly amazing paper in Physical Review Letters (the single most prestigious physics journal mind you) recently on superradiant light scattering from collective quantum mechanical systems. It seemed like an interesting path for future research…until the guy who co-discovered real life quantum collective systems with atoms (a Nobel laureate no less) claimed the work wrong in a comment to the journal published just today.
Apparently he didn’t find it ‘gospel’.
I’m sure a debate will rage for the next couple years on the topic.
I, as a scientist, have been told by no small number of mentors and colleagues that one is mistaken if one does not assume a published result is wrong.
The ‘problem’ with peer review, inherently, is that it’s not perfect.
With respect to climate science, the ‘problem’ is that the peer review process has become part of some scientists’ argument from authority. But within in the scientific community at large, no such ‘problem’ exists. It’s the political aspects of the climate debate that create issues.
I couldn’t agree more – and the political influence and “tribalism” exists on both sides of the debate. I find it bizarre when denialists/skeptics or warmists/”believers” characterize only the other side as being significantly influenced by politics.
Josh,
if I remember correctly, it was not ‘denialists/skeptics’ that asked for Congressional hearings on global warming 20+ years ago. It was not ‘denialists/skeptics’ who created a UN organization to promote a particular policy path with respect to climate change mitigation.
It’s true that there exists ‘tribalism’ on both sides of the climate policy debate.
But it’s my opinion at least that much of this comes from scientists who feel obligated to ‘inform’ the public as what the answer to this policy question is. From there, ‘denialists/skeptics’ pushed back to the extent that opinions on climate change follow party lines in the US almost exactly.
But the ‘denialists/skeptics’ certainly didn’t push first.
Two points:
First, I remain unconvinced by analysis that locates the origin of the “tribalism” at particular point when scientists were advocating for policies in light of theories of AGW. From what I’ve seen, that’s an misleading simplification. Theories were offered, and along a related timeline there was advocacy for policies in response, but coincident to that were attacks from “skeptics/denialists” on the theories, the scientists offering those theories, the policy recommendations related to those theories, and in fact, the notion that very notion that government should play a role in attempting to develop regulations that would affect industry for the purpose of limiting climate change. And the political tribalism related to climate change, (largely based on political opposition to government regulation), was directly linked to political tribalism on a larger scale – relating to ideological (financial/political) opposition to the idea that government should have any sort of regulatory role in our economic system.
The tribalism was bi-directional from early stages of the debate, and trying to locate the origin as a single point seems to me to ignore the inextricable relationship of the debate to the larger political context.
The second point I’d make is that even if it were true that you could validly locate the origin so specifically with the “warmists/believers” at one particular point in time, to do so suggests, at least, a view of tribalism that is discordant with the view that tribalism is an inherent part of human nature. I tend to be dubious about charges of tribalism whenever it isn’t accompanied by at least a good faith effort to control for the effects of tribalism in one’s own argument. Yes, you do say that tribalism exists on both sides, but I’m left wondering about the concluding point that is essentially, “Well, they did it first.” That seems to me to be utlimately dismissive of the impact of tribalism among “denialists/skeptics.”
Josh,
In the context of this thread, my point is that a small group of advocate-scientists began to ‘inform’ the public on the possible dangers of global warming/climate change 20+ years ago. Part of that ‘informing’ was utilizing the peer-review process in scientific literature to dismiss dissenting views on the appropriate political response to the perceived threat.
I do agree that there were and still are vested interests who viewed and still view action against global warming/climate change as ‘problematic’ to other political and economic goals. Some energy companies/lobbies might fall into this category. When the topic of global warming/climate change was brought to the fore by ‘warmists’, I agree there was likely a immediate response from these interests as ‘denialists’.
But the tribe of ‘denialists’ is hardly just composed of oil tycoons and energy hawks. As far as the public is concerned, I personally think that it was the dismissive attitudes and abuse of the peer review label that created a response from the average Joe who feels entitled to an opinion. And it took quite a bit of time for many people’s belief in the claims of papers covered by the media to erode. That process of ‘informing’ the average Joe and lambasting dissent with constant calls for citations has very much cut away at the legitimacy that climate scientists enjoyed before such behavior.
‘That seems to me to be utlimately dismissive of the impact of tribalism among “denialists/skeptics.”’
And what ‘impact’ would that be?
10 years ago, many more people believed the apocalyptic forecasts. Now, more and more people think such claims are an exaggeration of reality. That didn’t happen overnight.
Were all those people reading Roy Spencer’s blog?
I don’t think so.
With one side using op-eds, well publicized press conferences and large-scale music festivals, I find it hard to believe that someone can make the argument that there has been any kind of symmetry in the ‘tribalism’ surrounding the political response to global warming/climate change. I mean, Al Gore’s first Live Earth was broadcast on several tv networks across the world. How many also covered the last Heartland conference on global warming/climate change?
I would say that the counterpart to that was a concerted effort to campaign against theories that GW might be A – which comprised attacking the science, the scientists, the notion that government should play a role in mitigating the impact of CO2 on the climate, and more than that, the very notion that government should play a regulatory role in our economy. For example: “in 1990 Singer set up the Science & Environmental Policy Project.” Although there are disagreements about some of the information in the following Newsweek article, I think it makes a credible argument that there really was an effort to “inform” the public from the other side of the debate that extends back quite far: http://www.newsweek.com/2007/08/13/the-truth-about-denial.html.
Singer, and folks like Frederick Seitz had a clear and easily identifiable political orientation and their “scientific expertise” was used to establish their credibility just as “peer review” might be used . Tribalism has been well in evidence on the “denier” side of the debate for a long time. The Bush (the elder) administration formally politicized anti-AGW policy development.
Beliefs about “elitists scientists” and “leftist/statist social engineers,” and “tax and spend liberals” who want a “nanny state,” where a “socialist” government interferes in the free market in a way that damages the economy didn’t just pop out of the country’s political womb in the early 1990s when scientists began talking peer reviewed articles about global warming. And, it is incontrovertible fact that folks who held beliefs along those lines were well-represented in the immediate response from “skeptics/denialists.” These were no less “dismissive attitudes” than those attributed to climate scientists. Now, digging in closer to the heart of this thread, it does seem likely that the process of “peer review” was utilized, at least to some degree, by “warmists/believers” as part of the arsenal in their tribalism – but to some degree, at least, such use of peer review was in direct response to politically motivated attacks against “elitist scientists.” The counterpart to the “peer review defense,” if you will, was used not only in the climate debate, but against scientific expertise emanating from “ivory towers” more generally – basically on political principle and regardless of the particular issue being debated. Thus, I think that you’re jumping in mid-stream when you talk about the use of “peer review” by “warmists/believers,” because it was part of a larger dynamic that predated the moment that you identify as the origin.
And I believe that the overly-broad and categorical attacks on scientists who think that GW might be A, and the process of peer-review, are reflective of tribalism. Yes, the basic criticisms seem valid to me, but the polemical rhetoric is evidence of tribalism.
I know I’m repeating myself, but my point is that that immediate response was part of the “positive feedback loop” of tribalism in the climate change debate. I think it is not really possible to locate the origins of that tribalism on one side or the other (and question whether there would be a point in doing so). Further, I contend that trying to do so: (1) isn’t consistent with the “human nature” aspect of tribalism and, (2) leaves one in the position of effectively downplaying the “tribalism” on one side or the other.
See comments above.
10 years ago there was significant skepticism about theories of AGW. It is more now than it was then, but the increased skepticism is not simply a matter of people raising reasonable questions about peer review, it is not simply a matter of well-reasoned critiques of theories of AGW, and it is not simply a matter of an understandable response to “tribalism” among climate scientist. The increased skepticism can also, obviously, be linked to “tribalism” among political, ideological, and economic stakeholders. Fox News, Rush Limbaugh, Glenn Beck, have built very tall speaking platforms.
See my comment above. If you want to narrow the debate to increased skepticism among what Dr. Curry called #2s or even #3s, – who might read Spencer’s blog – then I think that you need to (1) document an increase in skepticism among that group (I would guess it has increased but think it remains poorly quantified), and 2, articulate a reasonable counter-argument to the assertion that that increased amount of skepticism among #2s and #3s is not politically influenced. If this blog is an example, I see many self-identified “skeptical” #2s and #3s, many of whom say that they were radicalized at some fairly recent point (possibly because of “climategate”) with a clear political orientation that seems to have deep roots that extend out beyond the debate about the science of climate change.
With one side using op-eds, well publicized press conferences and large-scale music festivals, I find it hard to believe that someone can make the argument that there has been any kind of symmetry in the ‘tribalism’ surrounding the political response to global warming/climate change. I mean, Al Gore’s first Live Earth was broadcast on several tv networks across the world. How many also covered the last Heartland conference on global warming/climate change?
Both Bush administrations used the force of government to influence the debate. They had more than op-eds at their disposal.
And Al Gore has been systematically ridiculed, attacked, and used as a rallying point for the denialist/skeptical tribe (as has the Heartland Inistitute by the “warmist/believer” tribe – with great effect – to the point where, for large segments of the “skeptical/denialist” public, the scientific validity of AGW theory can be evaluated on the basis of what Gore does or doesn’t say (or how big his electric bill is). Of course, “warmists/believers” share responsibility for that state of affairs, but it has been a deliberate tactic on the part of the “denialist/skeptic” side.
Josh,
I didn’t read your whole comment due its length. It seemed like you were repeating several points that don’t really support your disbelief in my suspicions. The whole inclusion of the Science and Environmental Policy Project proves my point.
The first Congressional hearings on the matter of climate change and the governmental response to it took places two to three years BEFORE the establishment of what you rightly identify as a highly partisan organization also aiming to ‘inform’ the public. But it was a highly partisan RESPONSE to the politically involved action of some scientists and politicians.
The identification of Fox News, Rush and their ilk as using climate change as an wedge issue did not start until just a few years ago. Before Al Gore brought the issue to the forefront of American politics, I’d be surprised if Fox News and Rush even mentioned climate change. Because Gore and his supporters tried to market their message to liberals who already had a track record of supporting environmental issues, it was pretty easy to get American conservatives to take the opposite stance.
I agree that the conservative media played a role in that, but the push by scientists, activists, the UN and less conservative politicians didn’t help the cause either. And I think that it is very, very shortsighted to think that the dip we’ve seen in the public’s belief/disbelief in global warming, the role of people in global warming and our ability to effectively address global warming is because of Rush Limbaugh. Even among the less scientifically literate. When the public is told things like there being no sea ice in the Arctic by 2015 or the major currents in the Atlantic being shut down based on model predictions, eyebrows are going to get raised.
So I think we agree on more than you think. Tribalism with respect to this issue works both ways. Conservatives have used it as a wedge issue pretty effectively. But it certainly doesn’t help when Michael Mann is taking out a substantial op-ed in the Washington Post to tell voters if they care about climate change, they can’t vote Republican. I have yet to see a scientist make the opposite argument.
PS Once you start making an argument (‘tribalism’ is totally symmetric wrt climate change), please don’t ask someone else to cite statistical evidence refuting such claims. If your opinion is good enough for your argument, someone else’ opinion is good enough for their own argument.
…if they care about climate change, they can’t vote Republican
If they care about science fraud, they can’t vote Democrat.
Punksta,
you’ve won the award for the most unsubstantiated claim of the day. Well done.
Anymore gems?
Maxwell,
I don’t think I ever made that argument. I’m saying arguing about “symmetry” is of dubious value, given that the science can’t be divorced from the politics.
I think that you are focusing in on one event and reactions to that event in a way that ignores the political context that predated that event. I’ll offer you the following paragraphs, but I suggest that you go to the link and read more of what you’ll find there. It details a process by which the original calls for scientists were for more studies (ironically, calling for the integration and coordination of scientists from a variety of disciplines) and that they were met with a web of political responses which shaped the future dynamic between the science and the politics – including a general response from the Reagan/Bush administrations of trying to control the message, hostility towards funding research, etc.
BTW, to the extent that you think that the question of symmetry is important, I think you’ll find this link interesting.
Abstract. Debates on global climate change (GCC) have been heavily influenced by such factors as scientific evidence, media coverage, public concerns, partisan interest, and so forth. Focusing on the linkages among the congressional committees, hearings, and invited witnesses (and their sectors), this study investigates the relational conditions under which congressional committees have mobilized climate expertise to discuss climate change issues for the past decades in U.S. Congress. Our findings show that agenda setting and witness selection by the committees significantly differed across the party lines: more environmental scientists were invited to define GCC as a threat in Democratic Congresses, whereas industrial scientists, to search for solutions in Republican Congresses. Except for a few proactive committees, committee jurisdiction was limitedly exercised. Our findings presents strong evidence along which climate policy debates have been framed based on a biased input of climate expertise.
http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1041&context=pnconfs_2010&sei-redir=1#search=“first+congressional+hearings+on+climate+change”
Sorry – here’s the link for the paragraphs I excerpted:
http://tinyurl.com/4qdb4e4
Maxwell,
Do tell us what Al Gore’s party’s take on on the matter of science fraud is then. All in favour as long as it supports more taxes and bureaucrats and green subsidies and climate science grants?
And what of Mann’s comment?
For an example of a close community among scientists, see:
Wegman, Edward J., David W. Scott, and Yasmin H. Said. 2006. Ad Hoc Committee Report On The “Hockey Stick” Global Climate Reconstruction. July 11. http://republicans.energycommerce.house.gov/108/home/07142006_Wegman_Report.pdf
Page 38 et seq: 5. Social Network Analysis Of Authorships In Temperature Reconstructions. In particular, see figures 5.2 and 5.3 (“The classic social network view of the Mann co-authors. Each block or subcluster is represented along an arc”).
Political influence is apparently inherent in Post Normal Science, in that PNS advocates “democracy”. What has happened is not democracy.
Even in “democracies”, participants must be qualified: citizens, of age, not felons. (If only “well-informed” were a qualification as our founders hoped.)
There’s the rub. Who decides “qualified”?
Sorry Maxwell, I wasn’t clear. I should have given caveats with the statement.
My context was in terms of the climate discussion and the parties within the discussion. I would refine the statement, but I’d be no better at it than your last paragraph.
Maxwell, you wrote:
“I, as a scientist, have been told by no small number of mentors and colleagues that one is mistaken if one does not assume a published result is wrong.”
You had some excellent mentors!
It is very unfortunate, to say the least, that so many cAGW adherents seem to regard the label ‘peer reviewed’ as quality sign which says ‘these results are now sacrosanct’.
How do you criticize the sacrosanct?
Why would you dare to even think of auditing it?
This attitude, together with the loathsome attitude of ‘not peer reviewed? We don’t need to even debate with you’ is what has brought us to where we are now.
When will people understand that critique and scepticism is the life blood of science?
Viv,
I think that’s an excellent point. There are some involved in this debate that feel the science motivates a specific policy response. To these participants, the label ‘peer review’ means something that it is not.
I am frequently catching myself telling others not to base real life decisions on the recently published results. That goes for quantum computing as much as climate change.
What’s even worse is those that are willing to take media interpretations of specifics results from specific journal articles at face value. I have yet to read an article on climate change at NPR that does not either overstate the certainty of the results or totally mangle how someone should interpret the paper’s conclusions in the context of the body of the literature on the topic. I think that’s where public skepticism is greatly lacking and the idea of ‘selection bias’ really comes into play.
And both sides of the debate are guilty of this. Punksta’s comment above is a great example of this ‘logic’ working in the other direction.
But, to me at least, the very fact that these advocates (on both sides of the issue) do so little work to substantiate their beliefs is proof enough to me that their position, whatever it happens to be, is not worth my time. I can only hope that over time, people who are really interested in this topic both scientifically and politically are able to find ways not to confirm what they already believe, but rather search for the most complete version of the truth. I don’t think it’s probable, but it’s worth hope.
There are definitely legitimate peer review issues. James Annan has an interesting post on his reviewer 4 encounter. Perhaps you can get him to comment?
Before I go to work, who would you select to represent you for an all-star climate disruption debate? I would pick James Annan and our host, even though I am skeptical.
One of the problems I see is the tendency for the initial success of a scientist to lead to the procurment of huge grants which concentrate research $ into few hands. These big shots have lots of $, techs, lab space, students, and post-docs. They then create a whole cadre of supporters of their views and have lots of influence. But the research tends to dig a deeper channel in the same topics and methods that initially were successful, rather than being innovative. Such people get very favorable reviews because they are so well-known. It ends up being a very conservative system.
Its not quite the way you outline, Craig, but admittedly the Mathew Principal is a relaity in academia, but as much in economics and sociology as science, as far as I’ve seen. The vast majority of climate science, as far as I can tell, does not involve huge grants, but scratching around for pots of money (for example the Tyndall Centre has hardly any money as a unit). Another factor is the need to apply for grants as a large network, which can sometimes stifle creativity as you get everyone on board. The UK government is trying to make sure that UK based researchers have an incentive to apply their research and to network outside the academy by making 20% of the research assessment based on demonstrating use by non academics.
http://mitigatingapathy.blogspot.com/
Here is my experience from my 126 peer-reviewed papers.
Papers introducing a new method or non-controversial topic: pretty easy time (but not always), with many (20+) accepted with no revisions.
Controversial topic (climate change and anything on endangered species or conservation): completely hysterical, mean, angry, comments by reviewers (including insults), including clearly false statements, criticisms that don’t make sense, clear attempts to block the paper because it could be used by bad people (ie, it got the wrong answer), and 1-line rejections. When emotions get involved, objectivity and politeness go out the window.
Another improvement in peer-review would be for journals to adopt a policy that papers showing major errors in previous work (math errors, stat errors, inability to replicate, computer code errors, etc) MUST be accepted–whereas now the journals resist publishing such work.
Agree that a “mea culpa” policy of correcting errors in past reports published by a journal should be a must.
It would have done “Science” a lot of good to have published an errata to the “Mann hockey stick” after it was comprehensively discredited by M+M, as later confirmed by the Wegman committee and rubber-stamped by the NAS panel under oath.
Making an error is human and normal.
But, as Watergate showed years ago, it is the failure to acknowledge errors and the attempt to cover them up that gets people into trouble.
Max
It was “Nature” (not “Science”) that published the hockeystick.
Max
… and the hockey stick wasn’t due to ‘human error’ – to the contrary.
How this got through the peer review at a time when the debate was far less heated or controversial, is just beyond me.
How scientists at that time could not see that one is ‘not allowed to do this in science’ (cf Prof, R.Muller’ video) is simply unimaginable.
Craig,
Even more than that, once the error is shown and the conclusions of the paper are known to be invalid – the paper must be withdrawn. Nature has refused to withdraw MBH98 and it is inexcusable. The paper is trash and everyone knows it.
I am an editor, and a controversial one a that (maligned by RealClimate and Roger Pielke, erroneous comments). Peer review can teach new authors how to communicate better, improve readability. It will not produce truth, only time will tell …. A distinction should be made between ‘pure’ science papers and those based on theories or beliefs or ideologies, i.e. social science papers. For good reasons I can’t explain here but which relate to my own research (into to politics of science, sciecne policy and ‘global warming’/IPCC: ESRC funded) I decided to treat climate science papers more like social science papers, that is starting with the acceptance that these papers ome from different ‘paradigns’ and that you can’t find peer reviewers for one paradigm from people who start with different assumptions or even beliefs. So climate sceptical papers were not peer reviewed by IPCC supporters, or rejection could be predicted. I believe that in very new complex sciences like climate, there is not yet a ‘correct’ scientific foundations and that many voices must therefore be allowed to speak, including ‘way-out’ ones offending the likes of Gavin Schmidt of realclimate.
Peer review is time consuming and usually not rewarded, hence it tends to become a ‘club activity ‘ giving mutual support for peopel with likeminded research interests. For the editor it is sometimes heart breaking and often involves teaching new researchers, especially ‘foreigners’ the ropes and demand good English. Few editors have the time to do this.
Anomymity of the peer reviewer is no salvation and would not apply to the editor anywya, who has to know who made what judgement. But it could be a matter of mutual choice, with the reviewer wanting his name to be known to the author. If requested, I have always encouraged this. For rejections it is less likely to work. A one line rejection is an insult, but sometimes this is all an editor receives…. And in the end, the main problem with the climate issue is its politicisation, with many institutions and interests needing or preferrring a specific answer, and wishing to fund research that will deliever the wished for policy support. This issue of funding policy-relevant science, or rather funding research (not rally ‘science’) to support the desired policy options (i.e. harmful, man-made climate change) is a more fundamental issue that needs an inquiry. It has reduced much academic research to the status of consultancy, and encouraged peer review by mutuallt supportive ‘epistemic communities’.
Most of all, having been peer reviewed should never become a mark of quality (as is now often asserted). Peer review is helpful and it may and usually does improve a papers, but it will not guarantee ‘truth’.
Sonja, thanks for stopping by. apologies, this landed in spam initially.
Your comment that peer review “will not guarantee ‘truth’” is very pertinent, although there may well be a common misconception that this should be the case.
Even so, one should also not expect peer review to act as an arbitrary gatekeeper mechanism, whereby those reports and data that conflict with the consensus paradigm are simply censored out.
And I sincerely hope this investigation will flush out if, how and why this has happened in the past in climate science.
Max
As recent exchanges of comments on this very blog have reminded me, review is invaluable in enhancing quality of communication.
I think we all share this experience, if we are fortunate enough to engage in depth on our ideas with others of strong persuasion (especially on the other side) and perception.
To echo Sonja Boehmer-Christiansen’s conclusion, though I feel how I say it is improved by the process, I doubt anyone thinks my second, or even tenth iteration of an idea, however refined, expanded, better supported, better spelled, more grammatical, more correct mathematically or otherwise improved by the review of knowledgeable and diligent review by other commenters has a mark of quality or is guaranteed truth.
If there are parallels between blogs and peer-review.
Though forfend that any panel of any government stick its nose into the Internet.
”
Most of all, having been peer reviewed should never become a mark of quality (as is now often asserted).
”
I definately agree with this. Peer review is not intended to evaluate research quality. I did notice that a number of the written testimonies think it important to educate the public that peer review is almost a gold seal of quality (not their words).
Judith, thanks for posting this.
Donald Gillies piece is excellent. I did read the whole thing. Although, initially, I presumed that “read the whole thing” was his comment about reviwers. I know some researchers – busy people with the best intentions – that mostly just look at the pictures and read the conclusions.
Dr. Curry,
The inquiry into peer review launched by the UK House of Commons Select Committee on Science and Technology could be a very good thing.
It should not be so watered down that climate science (where the overwhelming problem lies today) is only brushed over in passing.
Even more importantly, it must get to the root of the current problem in climate science, i.e. the militant defense of the consensus paradigm against any scientific challenges through the use of the peer review process as an effective censoring mechanism.
And finally, it should not be allowed to become hijacked by the consensus group in order to simply end up being a whitewash of the status quo. (We have seen too many of these recently, and we do not need another.)
David Taylor’s comment of the pros and cons of peer review as a defender of the consensus position is to the point. Unfortunately, in climate science today, it is quite likely that we are dealing with another “Wegener situation”, where the supporters of the consensus paradigm are working with the scientific editors of journals to block publication of conflicting reports or evidence, which might challenge this paradigm.
Along this same line, RI Tricker adds:
If a subject becomes dominated by a particular theoretical paradigm, it becomes impossible for new thinking and alternative paradigms to be accepted.
Let’s hope the UK inquiry really does expose the problems with the peer review system as it exists in climate science today and that constructive suggestions can be initiated and implemented to improve what appears to many to be a corrupt process.
As one of the climate scientists, who has expressed concern about the dogmatic approach being used by the consensus team, I hope you will contribute your comments and thoughts to this inquiry.
Max
A distinction that has not always been made in the comments is between peer review of submitted papers and peer review of research grant proposals. The latter is constrained by limited funding resources, and so funding agencies are forced to balance the need to support unexciting, incremental scientific advances with the wish to support innovative high-risk/high reward projects. This is not a problem with peer review per se, but with the difficulty of making the optimal judgment. Many funding agencies have set aside special funding resources for the innovative projects, to be judged by different criteria from those used for the incremental ones.
Regarding peer review of papers, I don’t see any evidence that it has prevented the publication of major scientific advances, although delays have occurred. These days, it is almost always possible to find some journal somewhere that will publish a paper, no matter how bad it is. In addition, non-peer reviewed web sources are available, but again, I’m unaware of any significant scientific advance that was compelled to utilize those sources because it was universally rejected by the peer-reviewed journals. If such examples exist, I would be interested in learning about them. The problem, as I see it, lies more toward the inability of journals to filter out papers that did not deserve to be published in the form in which they emerged – some should have been rejected outright, while others deserved to have been substantially revised. This is not to say that peer review could easily have been foregone. Any reviewer or editor familiar with what does NOT get published can attest to the value of peer review as a screening process, however imperfect it may be.
The issue of costs has been raised. Publishers are businesses, and for an adequate profit, they must charge subscribers to help them meet publication expenses. Many also impose page charges on authors. One possible step, however, toward making subscription costs more affordable lies in the establishment of Internet-only journals that have no costs for printing or other expenses related to print journals. I believe a few such journals already exist, but the high impact journals that garner the most attention and respect remain committed to a print plus online format. If this changes, the availability of published articles to interested individuals will probably increase.
To elaborate a bit on the foregoing, I don’t believe the public is deprived of learning about advances because of peer review. However, from the perspective of scientists, inequities in peer review can be frustrating because they often believe, sometimes correctly, that the inequities can prevent a good paper they have written from appearing in a high impact journal where it will receive the greatest attention and respect, and instead be relegated to a lesser journal, after what they perceive as unfair criticism from the first journal(s) they submitted the paper to. Those frustrations are understandable, but ultimately the public will still get to see the paper.
Steve Goddard in Are Scientists Always Smart? highlights the strong opposition to Alfred Wegener and his 1912 theory of continental drift. It took 50 years of severe scientific ridicule before Wegener’s theory was accepted in the light of plate tectonics.
One of the significant comments comes further down the list of submissions, in the one from the Institute of Physics:
“The public should be encouraged to recognise that a peer reviewed result is the ‘gold standard’ in research and will produce the most reliable information in the long term…”
It is sad to see that the IoP seems to have such a wrong-headed view of peer review. The ‘gold standard’ of research is surely results that are reproducible – and that for this to be possible, full access to data and methods/code are required, especially for non-experimental science that inevitably depends on historical data of all kinds and on computer modelling of various types.
One innovation worth discussing here is the rapid growth of peer reviewed, author-pays, open access e-journals, of which there are now several thousand. These were originally derided as “bottom feeders” in the industry, but the major publishers have recently begun introducing versions. Several have announced that scientific importance is not a criterion for publication, merely methodological soundness. They can do this because page space is relatively unlimited, unlike with expensive subscription print journals. It is basically a peer reviewed vanity press. Seems like a reasonable solution to me, unless one believes that every crank should be published for free just in case they are actually right.
Will this review be broad enough — as its terms do seem extremely broad — to catch data hoarding, suppression and cherry-picking often associated with publish-or-perish positions, where original research is at such a premium that some are tempted to keep and hide their figures until they’ve wrung out the last drop of every possible variation of originality?
I personally favor a system where raw data is immediately stored in the cloud, accessible to all, from so early in the process as possible, well-indexed and so easily found as can be managed.
Wouldn’t it be interesting to know what then was the most, and the least, accessed raw data?
Some might find this story interesting
http://climateaudit.org/2011/01/04/crowleys-belated-apology/
According to the current peer reviewed record, Steve McIntyre is a said to be a liar, even though his accuser has apologized for making those false claims about him. The offending journal could apparently care less.
We live in a different society than we did when peer review took hold. Scientists used to battle it out, and eventually the best theory won. Now, with the establishment of peer review, it too easily lapses into gate keeping, and challenges can’t even get published. I say do away with peer review all together. Let publishing follow an open source model, and then you will have the scientific discourse we all deserve.
The inquiry is a good thing, and I would welcome something like this in the U.S., although it would be better if it originated with scientific organizations rather than politicians.
Peer review is useful. As a peer reviewer, I just recommended rejection of a manuscript because it reported biological effects of a chemical on cells in culture at a chemical concentration that would be lethal to human beings or animals. Peer review in most fields of research keeps obviously irrelevant papers from being published, as well as preventing authors from claiming more than the data supports, it identifies lack of proper controls, and prevents publication of results of routine assays that are way out of line with dozens of other published values (suggesting a problem with the technique or the calculations). This is the good side.
The bad side is that human beings are involved and some of them exhibit typical human vices. Peer review can be used to keep people out of the club, block publication of results contrary to those of the reviewer, be used as part of a personal vendetta, or become a source for new (stolen) ideas that the reviewer can then use while stalling the review process. I have suspicions about some of these things occurring with a few papers I have submitted, but these things are very difficult to prove.
The only solution I can imagine is to hold reviewers as well as authors accountable. Anonymous review has some good features. It promotes candid assessments, particularly of well known scientists, who most other scientists would hesitate to criticize if the reviewer’s identity was revealed to the author. However, it also means that reviewers can say almost anything, correct or not, with impunity. Editors probably catch some of this and address it, but I suspect most gets past them. So, perhaps a random audit of reviews in which other reviewers will critique the reviews for rejected manuscripts and manuscripts for which major revision was required and report to the Editor? Maybe it would be even better to engage in an anonymous chat session online after the review has been received but before the Editor’s decision. The authors could be given the reviewers comments a day or two before the chat to allow them time to formulate responses. Then during the chat, the authors would have an opportunity to interact with the reviewers and the editor and address one of the major causes of rejections, misunderstandings on the part of reviewers. This would retain anonymity but would make it much more difficult for reviewers to delay the process and to make continued incorrect assertions that could be easily addressed by the author.
I think this would also enhance the probability of publishing contrarian papers. It would be more difficult for reviewers to appeal to authority if the author could address their criticisms in real time and point out imperfections in current dogma. Of course, this would depend on an objective Editor to judge the outcome. Considering the climategate discussions on peer review, it seems that editors of climate science journals who are too open to contrarians are quickly ostracized (or worse), and I don’t know how that could be fixed.
Stephen – In a perfect world, what you suggest would be easy to implement. In our world, it is more problematic. Having been a reviewer, a reviewee, but also a journal Associate Editor asked to solicit reviews for papers, I can attest to the difficulty of coaxing good people to engage in the review process. The best are very busy scientists in their own right, and to review a paper, they must volunteer their time, free of charge, and in most cases only out of a sense of responsibility to the scientific enterprise, with no reward for themselves. It is surprising how many are willing to do this, but even so, the supply is limited. If reviewing becomes even more arduous, some who volunteer now will have no trouble declining on the grounds that they are too busy, and we will end up worse off than we are now.
I agree with the notion that anonymous reviews are not always conducted ideally, and certainly a journal that can attract named reviewers should be welcome to do so, but I don’t see it as a general solution. Rather, editors must be responsible for monitoring exchanges among authors, and the several different reviewers of a paper so as to arrive at a fair judgment. This is more common today than in the past, when in many cases, authors had no opportunity to challenge negative reviews. Unfortunately, even this takes time, and so it will never operate as well as it should. Fortunately, it doesn’t do too badly today. I like your suggestion of online chat sessions involving anonymous reviewers, but even this is likely to impose additional burden on reviewers (and editors), and so I don’t know how practical it would be.
As a peer reviewer, I just recommended rejection of a manuscript because it reported biological effects of a chemical on cells in culture at a chemical concentration that would be lethal to human beings or animals.
That’s a valid argument for PR. But, if that would have been published, there would have been several subsequent papers all pointing out that error. In today’s world, in the information age, that vital rebuttal can happen much faster than ever before. One negative thing peer review seems to do, is to make a single paper on a topic seem to have more weight, because, well, scientists reviewed it. It weakens the concept that an experiment or process needs to be replicated repeatedly to be shown to have validity.
I don’t know if this is a valid analogy or not, but I’ll throw it out there anyway.
One history professor in college brought up an interesting theory about why the north did so poorly during the first few years of the Civil War. The Generals of both the North and South were all West Point trained, and promoted because they had been singled out as the best, according to West Point standards of the day. In a nut shell, they all thought alike. Therefore they fought each other using the same tactics, thus the quagmire. Though he did well, Ulissis S Grant was not a stand-out per-se, he didn’t fit what was considered the mold of a military leader. He wasn’t that interested in “the science” of past wars. He didn’t care much about Napoleon and his tactics. He was not originally thought of as “General” material by the powers that be. But during the Civil War, he eventually rose to replace McClellan because, unlike him, Grant was not in the West Point mold. Where McClellan was cautious, as his West Point instructions dictated, Grant was bold. He showed an originality in his actions and planning, something that the hand picked West Point Generals were not familiar with.
In a nutshell, Grant thought out of the box. It seems peer review, in climate science anyway, too often makes sure everyone stays inside the box, with no room for maneuvering and going in different directions. In the end, peer review stifles scientific creativity.
Hope this made sense.
Uh Oh, I think the spam filter is acting up again.
Having had to review papers submitted by authors for whom English is not their first language, there is still a need for quality control, requiring good technical English, fixing errors, ensuring graphs are legible, literature cited etc.
However, the major problems with today’s “peer review” is the establishment of orthodoxy or even a minority politically correct view, rather than supporting the full range of scientific inquiry, including “kicking the tires”, exploring contradictory evidence and probing uncertainties. Prof. Frank Tipler addressed peer review problems:
Refereed Journals: Do They Insure Quality or Enforce Orthodoxy?
ISCID Archive – June 30, 2003
The tendency to enforce orthodoxy is a problem. If the methods in a paper are sound but the conclusions at variance, some journal should still publish the paper as a discussion paper. Invited comments by both the orthodoxy and others would give the paper a chance to succeed in an open marketplace of ideas. The more eyes that see it, the more likely someone will spot errors if errors are to be found.
A Sokal proved anything can get set in print. Hansen tells us he has the normal global temp. LOL yeah OK . Peer review… right up there with the washroom walls… for a good time call Jim.
We could eliminate much of the controversy by exempting from peer review all those papers that support the premise of the publishing journal. Since peers are carefully chosen to agree with such premises, there is little point in going through the motions.
Or publish every paper complete with all source documentation, formulas, etc., and let everyone do their own peer review.
I still think people are missing Nick Stokes point about the business of publishing. Everyone is trying to design rules that spit out the “best science”
everything we know says that this is highly unlikely to happen if top down rules are imposed upon the business of publishing. think about revolutionizing the business of science publishing.
Indeed.
And the tools for such a revolution are at hand – and are being used in some cases. The internet changes everything. The internet enables a much freer exchange of ideas. Look at the arXiv.org e-print service. Look at the open “review” process that takes place in the blogosphere.
Publish and be damned! And if you publish and don’t make your data, your methods, your code available for all, expect rocks to be thrown. Openness is the key.
UK readers may be interested in a new BBC radio 4 series: “science betrayed” which is relevant to the subject of this thread. The first of the series may be heard here:
http://www.bbc.co.uk/iplayer/episode/b00zf4ns/Science_Betrayed_Episode_1/
It will be interesting to see if they eventually get round to climategate, etc.
Michael Larkin
“It will be interesting to see if they eventually get round to climategate, etc.”
Don’t hold your breath.
Max
I appreciate the theme of this piece and I appreciate the problems with peer review, but I concur with several commenters that we need some sort of review. Otherwise, humans being what they are, lots of total garbage – analyses full of mathematical errors, sloppy data, etc, would get published.
We need a two-tiered review system. Tier one review would be done by hired reviewers. Their job would be to check data, references, graphics, calculations and verify standard, previously published methods. Rejection at this level would have nothing to do with conclusions or relevance, and could normally be overcome with modest effort.
Tier two would be scientific review. This would be similar to the traditional peer review – except that all reviews would be signed by reviewers and peers would provide guidance to the author, but have no responsibility for recommendation on publication. It would be essentially a courtesy review, and the author would be free to accept or reject comments.
This system would, hopefully, free second tier reviewers from the responsibility of cross-checking data, thus giving them more time to focus on the science. It would allow them to comment constructively with less interference from political factors (because the review does not affect the future publication).
The only problem it doesn’t solve is the potential stealing of ideas. However, this might be solved by limiting the time for review to one month – more than enough time to get the job done for a single paper. It would also speed up publication, since technical details would be handled by a dedicated staff.
JL,
What happen when a whole new area of science is opened up that their are no qualified peer-reviewers?
Currently, we know science has generated mistakes, unless this reviewer understands the mistakes in science, any opposing science will be rejected.
There is no rejection except on the grounds that there is a technical error – an incorrect graphic, data that don’t match a graphic, mathematical errors, etc.
You get what you pay for, and peer-review is an amateur pursuit (I am shocked to learn from various commenters above). As with amateur sport and amateur opera, this more or less makes it fit only for family and friends to consume.
Surely it’s time for it to go professional?
It also seems that peer-review is just a very low-grade audit. Why not a proper job? Gettting paid should make this possible.
And TOTAL openness. No anonymity, and both the papers and every comment that every reviewer makes, made immediately available on the internet. You don’t have to read all of this internal dialog, you can choose to read only the completed review later (as now happens). But this blow-by-blow account will still be there if later it seems all is not kosher; and this radical openness will help keep things kosher in the first place.
Punksta, I think you’re confusing amateur with non-profit. Peer review isn’t even a low grade audit. In an audit, you would actually analyze the data (assuming it’s a data based paper), starting from the raw data. This could take weeks or months (or conceivably years). The peer review I’m familiar with typically takes a day or less. You can view it as a general (rarely detailed) quality check on the process used by the researchers. Mathematics apparently already does essentially audit for their peer reviews.
I was very much in favor of open peer review, the same as you. After reading some of the comments fron areas other than the ones I worked in, I’ll concede there are areas where it seems impractical.
I think you’re confusing amateur with non-profit
Well, I think the problem is exactly the opposite – failure to see the equivalence.
And can we agree though that peer-review is a spectacularly low-grade audit ? At best.
And where is the real problem with radical openness of review?
As long as you can filter papers according to what stage of the review they made, you can have everything you presently have, plus – unlike now – reviewers and journals will be much more held to account and kept on their toes by the vigilance of millions.
1a) For printed journals some way of screening papers to be published is a necessity, as printing everything would be too costly. It is not possible to operate an economically viable publishing business without screening of papers.
2a) Readers of scientific papers need some help in selecting, what to read. Reading everything would take more than 100% of the time available.
3a) Many manuscripts have significant deficiencies and are greatly improved based on the comments of the reviewers.
All these reasons for peer review have been developed during the time of printed press. Electronic publishing has changed much:
1b) The cost of having everything written publicly available is not a problem.
2b) Interactive media may offer new alternatives for screening what to read.
3b) Weaknesses in papers can be corrected easily and at low cost at any moment, not only before printing.
Perhaps it is really time to think again the nature and roles of scientific papers and peer review. This is going on in the alternative channels as stated already in this thread, but perhaps the process could be accelerated.
One important obstacle is in the valuation of scientific work of individuals in career development. That is a very important point for every young scientist and there the publications in highest ranking journals have much more weight than other papers (we can forget the exceptional great papers as the authors of those papers are not usually the ones, who worry about their future employment). Something should be done to this obstacle in a way that the young scientists trust, when they decide, how to publish.
Pekka,
What is currently correct or incorrect in published science?
Observed science has not taken into account of the many “hidden processes” going on nor does it account for different locations and different heights from the sea level, pressure, gravity, etc.
Example: At the equator to the poles generates circular movement of currents and air circulation. These never cross the equator and rotate opposite of each other due to the largest diameter in motion.
Next, everything is inter-related to each other from DNA to planetary motion with other planets. One planet in our solar system “shaved off” the sun at the formation of the solar system and all others followed and interacted with the suns actions.
If you specialize in just one area, you miss all the interactions from many areas. Currently science is studying in a stop time phase and has missed billion of years of interaction to get to this point.
I wonder if a hybrid system where publishing is divorced from detailed review might be an answer (caveat: I have no background whatsoever with scientific publishing – consider this fodder for discussion rather than an actual fully thought out proposal). Institutions (and/or groups of institutions) could sponsor publishing sites similar to arXiv to handle the basic editorial functions, archiving, etc. while Journals could evolve to identifying and reviewing what’s of interest to their readership.
I’d see this addressing items 1a-3a above, as well as lowering the potential for gate-keeping abuses that have been alleged. The journals would be rid of the expense of providing data archiving, but would retain their cachet (“my paper was reviewed by X”). At the same time, the journals would no longer have a tight grip on publication – I could see this kind of a system actually injecting some competition between them to be the first to find and evaluate new and innovative papers (since no one journal would “own” the paper”. Lastly, I’ve read complaints from authors about the space limitations inherent in the print publications. With electronic publishing, that’s no longer an issue.
Gene,
There is a hybrid system of sorts for most fields. IEEE has online peer review plus the normal peer review, which if I remember correctly is an open review. The current process suffers most in the high impact journals in my opinion. Nearly every field has an association with a journal. Publish there and it could get boosted up to a prestigious journal on its own merit.
Judith,
Current science is more interested in the scholastic process and mathematics than following science.
Much of science is in a stop don’t move the planet and observe phase. Meanwhile, many processed are going on that is not observed for a conclusion that is follow in individual areas.
Physics followed that everything moves forever and outside forces stop motion. Not even looking into stored energy, compression or interaction.
Old theories will not die even though new technology shows them to be incorrect. Hard mathematic equations fail when brought back into a different time line when the parameters have changed.
Yet, climate science is stuck just looking at short term temperature trends.
There is a vast wonderful world of science that is being missed by short sighted scientists.
Dr. Curry,
As I mentioned in a comment above, I like the idea of post-publication review. Just like the pharma industry has post-marketing studies and adverse reporting mechanisms which sometimes cause drugs to be pulled from the market, science needs a way to winnow out the bad science. I hate it when I invest a great deal of time understanding a paper and making plans around the conclusions and then later learn the paper had been discredited.
One of the problems with climate science is that when papers are discredited, the journals do not withdraw them. Nature, for example, has failed to withdraw a number of papers with invalid conclusions – MBH98, Steig09, etc. It is a shame a high impact journal like Nature is too embarrassed or too political to pull papers everyone knows are wrong.
I propose a new journal to be called Post-publication Review which will name papers which should be withdrawn by the journals. The Post-publication Review will also rate journals (after a period of time) based on the number of articles published which should have been pulled but have not been. I can imagine researchers saying to one another “Oh, don’t publish in Nature. They have a negative Post-publication number.”
What do you think?
I think this idea of a Post-publication Review is an excellent one.
Perhaps though its domain could widened, to include identifying papers that lack supporting data or code. With a record of these unreliable papers, we could work out which journals have a habit of publishing them, and which policy proposals depend on them. It would be interesting to see which if any IPCC conclusions depend on discredited and data-deficient papers.
A sort of ongoing Peer-Review Review.
Yes, it’s one of the more practicable ideas here. It addresses the problem of editors who are slow to accept that their papers have been discredited by bloggers. And anyone could start up this Post review. Why not here and now?
Maybe Energy and Environment could take it on?
Energy and Environment huh? You mean the one that had the audacity to publish some of the skeptic papers? Yeah, that’s where it should start! All those discredited papers from skeptics. ;)
as if no peer-reviewed papers on “climate science” have ever been shown to be junk….grow up, boy! grow a pair!
I think what Nick actually meant, was that Energy & Environment could run a Post-Publication Review of papers in all journals, not just of papers it itself had published.
There is no rational basis for regulating scientific publishing as though it were drug use.
I like this idea very much!
It might even prevent, ultimately, scientists from tacking onto their conclusions the politically correct statement de jour, even if this contradicts their findings … we all have seen such papers.
I like this proposal as well.
It could apply for all papers that have been published, but would be particularly pertinent for those that were financed by taxpayer funding, in that it would give the ones who ended up paying the bill an insight into what they got for their money.
The audits of these papers would need to be completely open and transparent, particularly if they resulted in the extreme case of a paper being pulled.
And, with time. it would expose journals, which have been too quick to publish papers that later turned out have invalid conclusions just because they happened to agree with the editorial policy of the journal involved. These would then be more cautious before publishing a paper just because it conveyed a desired message.
So it would act as a self-cleansing mechanism.
So much for the “bad papers”.
It would still not address Judith’s concern that many good papers get blocked by gatekeepers who censor out those papers with which they disagree.
Max
Max, the number of peer reviewed papers published very year is on the order of one million. Who do you propose should audit them, and how will the auditors be paid? Do you envision new government agencies around the world doing this, as the FDA regulates drugs? Will only selected papers be audited? Who selects them, and how?
For that matter, why not audit the papers before they are published, instead of peer reviewing and publishing them first?
David,
There are always going to be bad papers that make it through peer review. It costs more than $500 million to get a drug through the FDA approval process (obviously much, much tougher than pal review or even hostile peer review) and still unsafe drugs get approved on occasion.
The question is how should those papers be dealt with once they are known to be unreliable? Will the authors withdraw them voluntarily? Not always. Will the journal pull them? Maybe, but not often enough. So, what is to be done? I am suggested a mechanism that will allow researchers to know a paper’s conclusions are invalid so they do not build on bad science. It is a suggestion. Do you have a better idea?
Ron,
I am asking what the mechanism is and how it will be staffed and funded? We can’t pay $500 million per paper.
The mechanism is a new journal called Post-publication Review. Perhaps it has two or more levels of papers. You probably already know all of this but I review it anyway.
Science papers come in different levels of evidence – at least in medical literature. At the bottom is a case study. It reports on one patient getting a new treatment. One level up is a case series. A series of similar patients getting the same investigational treatment. (The first two may be published in a journal but are considered anecdotal but possibly interesting.) A randomized controlled trial is a higher level and considered real science. A literature review is considered better evidence because it reports on the conclusions of a number of randomized controlled trials. When doctors get a number of literature reviews all reporting the same thing, they will write a guideline.
Perhaps Post-publication Review publishes papers on two or three levels. On one level, it may be a simple comment about a paper. Or perhaps it revisits a series of comments and replies on a controversial paper asking if all of the questions raised have been adequately answered. Perhaps another level is a full paper reporting on failed attempts to replicate a particular study or other errors or problems with the paper. The highest level of paper would be like a literature review. It would revisit the literature generated on a controversial paper, the goal being to either confirm the paper (with corrigenda) or to reject the paper (the equivalent of the community rejecting the paper).
Anyway, that’s the idea. I appreciate people discussing it, pointing out the good and the bad.
How many papers do you plan to review each year? If you can get subscriptions more power to you. On the other hand, if a paper is already controversial why would anyone pay to read criticism that is freely available?
David
Let’s restrict our discussion first to papers on climate science, where the suspected problem lies. Are there a “million” of these per year?
An audit of a paper will obviously take less effort and cost than the paper itself, so we are talking about increasing the total cost by less than doubling.
Most of this is funded by taxpayer money.
If an auditing process results in a few less papers in the first place, but with papers that have been independently audited for quality, maybe the taxpayer will end up getting more for his money than with no independent auditing process in place.
I’d rather have half a million good papers than a million questionable ones.
Wouldn’t you?
Max
Max, I estimate that there are between 1,000 and 10,000 papers in climate science a year, but the topic is very vague, which is your first problem. I do not believe half of them are questionable, in fact I doubt that any of them are worth auditing. But it is your witch hunt so you have to explain what problem you think you are solving. That is the first step in developing a regulatory program.
You are not auditing the paper, you are auditing the work, which might actually be more laborious than the original work. IRS audits sure are. But you still have not told me who will do the auditing and how it will be funded? Research grants are in fact subject to financial audit. Are you thinking of adding an audit of the research itself, by the funding agency, like NSF? How do you select projects for audit and what percentage do you plan to audit? all of them?
I do not support your proposal.
David,
We are “getting into the weeds” on “how” to implement an auditing process and “who” should finance this all, etc.
I am going from the starting premise that there is a perceived problem with the peer review process today in climate science, which results in some bad stuff getting through a “pal review” and some potentially good stuff being censored out by biased gatekeepers.
I am also going from the perception that “insiders” in climate science have to some extent hijacked the peer review process to ensure that papers that do not conform to the “consensus” are systematically rejected.
This is what the committee is charged with finding out, so let’s see what it comes up with.
The first part (bad stuff getting through) could be addressed by an auditing process, as was suggested here.
The second part (good stuff being blocked) would require that some gatekeepers may need to be replaced, but that is a separate discussion.
Max
The second part (good stuff being blocked) would require that some gatekeepers may need to be replaced, but that is a separate discussion.
To detect this, we surely need complete openness of peer-review. No anonymity, all comments and emails being publicly available. Any off-the-record or “please delete such-and-such emails” behaviour would need to be dealt with by lifetime banishment.
And such exposure it may reveal it is necessary to replace more than just some gatekeepeers. It depends on how much of the consensus is crooked.
David
Not to try to solve the problem here (we first need to find out if there really is a problem, but I’d go with the old adage that perception is reality).
Privately funded research is owned by whoever paid for it.
Publicly funded research (i.e. most climate related research) is owned by the taxpayers who funded it, i.e. the public. It should, therefore, be open to the public, including the peer review process.
The UK committee is going to investigate “the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication”.
Let’s see what they come up with.
Max
Often when bad papers do get through, many other better papers reference them and try to show where they went wrong. This is why it makes no sense to withdraw published papers or pretend they didn’t exist, otherwise the later papers’ references would also lose their meaning. It is like on a blog such as this where a bad comment is removed by the moderator, but all those denouncing it are allowed to remain. Should those be removed too, even if they add good points besides?
Conclusion
Form the above discussion of practical problems and merits of peer review, it appears that the single most effective, legislative solution to the problems revealed by the climate scandal would be:
Legislation from the Subcommittee on Commerce, Justice and Science (of the US House Appropriations Committee) to deny government funds from any US research agency that uses anonymous peer reviews and from doing business with any publishers or other organizations that do.
That legal restriction would still allow publishers and federal research agencies to use peer review, but only by reviewers who were willing to identify themselves and thus be accountable for the reviews they write.
With kind regards,
Oliver K. Manuel
Former NASA Principal
Investigator for Apollo
“deny government funds from any US research agency that uses anonymous peer reviews”
Fantastic idea. Anonymous peer reviews are no different than secret courts. Would you trust the results of any inquiry held behind closed doors by people unknown that did not need to give an accounting of how they arrived at the results?
I would add to this that the need to publish the data and mehods with the paper, sufficient to recreate the work. Otherwise the paper itself is anonymous.
The FIRST step of the peer review should be to ensure that the data and methods match the results. If the peer review can’t/didn’t recreate the results, how exactly is it a review?
Anonymous peer reviews are more honest and less influenced by the power or stature of the scientist being reviewed. Would you want that to be gone?
You are talking about something like replication of results, which would be very laborious, expensive, and typically worthless because most work is not worth replicating. Peer review is a simple, quick procedure, taking a few hours at most, and done voluntarily. You apparently do not understand the difference.
‘most work is not worth replicating’
Ummm….if its not worth replicating, why is it worth publishing ..or reading…or citing?
Seems to me that the way academics construct their ‘scorecards’ for career purposes leaves an awful lot to be desired. An awful lot of fourth-rate stuff gets published..and the fourth-rate authors manage to keep getting public funding rather than P45s.
Perhaps that will be the S&T committees next assignment. It all sounds like a crock of s..t to me.
“Often when bad papers do get through, many other better papers reference them and try to show where they went wrong. This is why it makes no sense to withdraw published papers or pretend they didn’t exist, otherwise the later papers’ references would also lose their meaning. ”
I don’t agree. When a paper is withdrawn, the papers that refute the earlier paper are in no way invalidated.
Only those papers that use the earlier study as a foundation to build upon are at risk, as so they should be. If you paper builds upon a wrong idea, your paper is likely wrong as well.
Those later papers make less sense when you can’t even get hold of the article they reference. You wouldn’t be able to evaluate if their criticisms are valid or not for yourself.
Why destroy the old copies? Simple label them “Withdrawn”. This allows them to be studied so that we can still learn from our mistakes.
The problem comes in allowing a mistake to stand. This allows the mistake to be used in a follow-on work, to create another false result.
For example, if we allow 1 = 2 to stand as a result, then we know that since 1 + 1 = 2, then by substitution 2 + 2 = 2, therefore 2 = 0, etc., and we can proceed to turn the world of mathematics upside down.
The harm comes in using a mistake as a foundation for a further result.
Withdrawing a work simply means that the paper cannot be cited as a positive reference for future work. It doesn’t mean you need to destroy all copies, which is next to impossible.
‘Scuse me an all for butting in, not being an academic and stuff, but was it an Act of Parliament that sad that all academics get judged by their citation index? Or is it just that all academics like to do so because its simple to do the sums?
And was it Congress who said that all papers must be peer-reviewed and that doing so gives some form of ‘quality assurance’, depsite lots and lots and lots of evidence that it doesn’t do very much of anything at all other than show that whoever actually wrote the paper can spell their own name correctly.
Because viewed from the outside it seems that the whole kit and caboodle is arranged just for the convenience of the academic community, and like all trade institutions, academia is very resistant to doing anything any way other than suits them.
But you can’t have it both ways. Either citation indices and peer-review actually do something useful, and so can be sensibly shown to the 99.8% of the rest of the world outside academia as trustworthy and reliable. And robust enough to withstand external challenges and review. Or it can’t.
From what I read, very few can find much good to say about either. Can your collective Great Brains not come up with something better? They are both just organisational arrangements…not Laws of the Universe, however much some of you may think they are.
A huge amount of academic work is funded through taxation. Surely it is right that the taxpayers shou;d have confidence in the quality of the work that is done..not just in the conveniences of those doing it.
It is hard to fault the peer-review process in a niche area of research, climate science, an outgrowth of meteorology blended with geology and a host of related specialty interests. Climate science would have evolved, recognizing its deficiencies, particularly in the areas of mathematics and statistics had it not lapsed into advocacy. Maybe climate science was “born” to justify an advocacy position and not the other way around. I certainly don’t know. In any case, the academic modes operandi necessitated specialty journals, hence, the peer-review process. When advocacy begot the IPCC, the whole justification paradigm took over and the first hints that all was not legit, ie, M&M did the escalating dog fight became public consumption. Now the public, in the framework of knowledgable citizen scientists, the blogosphere, and with great trepidation a few climate scientists themselves, is the morass evident. The peer-review process has in the past worked begrudgingly towards more knowledgable editors, a larger cast of reviewers, a broader array of experts, in this case the urgent need for statisticians, and ultimately, better science. Advocacy has been climate sciences’ most self-destructive tool. Until advocacy is put onto the back burner, at least by the illuminaries, climate science, grant proposals, funding will suffer the political whims of Congress. In the mean time, lets get back to developing the science of weather prediction for more than the current two week window. Accurate regional weather data/predictions may eventually morph into better regional climate predictions and then we all “win.”
I certainly welcome the calls for openness and transparency. I’m not so keen on government regulation of science publishing and Im not so keen on impractical ideas of enforcing rules on the publishing business.
here is a thought. If you wanted to start an open “journal” that competed with Nature, what would your plan be? would you invest in such a start up? would you pay to read that science? pay to interact with the authors? pay to read the reviews? Is there a way to make the science publication free and still create a business?
When one looks at complaints on references, the universal gripe is “pay wall.” A very large social network company, Facebook, has yet to build a financial base. I don’t believe there are money solutions to information as long as the world wide web is free, books in the library, or cafe advice is free. Some very smart people have been stumped by this issue. One possibility that has just popped into my mind, government regulate Google’s personal information treasure trove, dispense to all interested parties for a fee, and then companies can purchase segments of personal information for “targeted” advertising. Just possibly Larry Page would object to such seizure of propriety data collection and its regulation, but, all’s fair in love and war.
The success of an internet business rarely results from billing customer’s directly. Facebook, Google give their service away for free and in doing so make money. Think of it this way.
I want to publish a scientific paper. Why? Because if the idea takes off, I could be famous in my field and do very well as a result. So, there will be no lack of authors, so long as the site promises readers.
Now, as sites like WUWT and JC have shown, they can attract readers so long as there is worthwhile information to read. The exchange between JC and GS in near real time sparked a huge interest in JC’s site, to the point where it could barely keep up with volume.
With volume comes value. Millions of people are walking into your store because you have some very nice things on display. Many of them will buy, if you can provide them an opportunity. However, unless you provide a mechanism, there is no opportunity.
A million people enter your store. Not all will buy. Some will. Some won’t. The more opportunity you offer to buy, the simpler you make the mechanism to purchase, the more that will buy, the more the site will prosper.
The key is to act like Google and Facebook. Don’t force your customers to buy to enter the store. Simply give them interesting things to look at and an opportunity to buy should they wish.
Steven,
Invest in a startup? No. But if I was CEO of a science publishing company and had the experience and those economies of scale, I would be very interested in the business opportunity of Post-publication Review as I have described it. I think the interest would be high, especially if the journal was able to attract papers from Steve McIntyre.
Joshua would have us believe that identifying and outing the obvious tribalism and vested interest in alarmism, makes those who point it out just as tribalised. So it was pretty much always win-win for alarmism then.
Either A, alarmists would succeed in pulling the wool over everyone’e eyes and so advancing their tribal/political agenda under the cover of science.
Or B, if rumbled, they would complain that those who outed their tribalism are themselves tribal.
Option A now now rocking badly, the likes of Joshua retreat to B, in an attempt to get discussion way from the endemic, unrepentant dishonesty underpinning alarmism.
The counter-tribalism charge is also a tactic to get discussion away from the facts that
– politics (the state) has a vested in interest in alarmism (more taxes and regulators, controls)
– the state is sponsor of alarmism, in the form of its funding of climate science (almost all of which is alarmist)
– this biased funding is three or more orders of magnitude greater than everyone’s else put together.
The fair point has been made that having peer-review go professional, will entail raising more money. So where will it come from?
Why not have the journals charge authors? So when you get a grant, that is one more thing you factor in. If that means some money for the authors being diverted to reviewers, so be it. There will be some optimal split for good science, and the current 100% – 0% split is perhaps not it.
Anonymous peer-review has advantages, someone argued, since it means one could attack the current paradigm withour fear of harming your career. It could help unsettle a blinkered consensus.
It could. But just as likely, it could help entrcnch one. As with use of a nom de plume :-), it can be both liberating and corrupting.
So I still say scrap it. Publish every comment and email any reviewer makes. Only then can we see what journals are really up to, and what if any value (as a rating service) they provide.
For those interested, Donald Gillies’ has a paper online at University College London, entitled:
Known sceptic, Michael Crichton favoured the use of double-blind peer review. An interesting discussion of that, instigated by the Nature magazine is here:
http://blogs.nature.com/peer-to-peer/2008/02/working_doubleblind.html
I wonder if Judith has suffered the gender bias identified?
I tend to think the bigger problem for is in the sensationalising of mundane papers in order to shift copy or maintain a high profile: Otherwise known as the tabloidisation of science or science by press release.
Counter-consensus papers do get published though and yet they are still ignored by the mainstream. That’s just the way of human nauture. There is no guarantee that doctors would start washing their hands just because a paper was published telling them it might save lives. Indeed even despite our knowlege of germs, the rise in the MRSA virus has been blamed on hospital staff not being clean enough…..And then by way of counterpoint, there is the recent research (which I disctinctly remember James Le Fanu concluding 20 years earlier in a magazine article) that says we are getting more allergies because we are not exposing ourselves to enough dirt. Nature’s slippery: Just when you think you have a bit of it sussed, it reveals another complexity you missed.
I was just about to comment with that same Nature link, alighting on it independently. It seems to be hijacked by a notional gender bias, which I suspect is a red herring. The primary problem always being pre-conceived bias for or against the authors prior tribal convictions, not gender.
Below is a link to an Patrick Michael’s analysis of Arctic conditions and their contribution to climate change. His commentary most certainly should be part of the debate on this thread.
http://www.cato-at-liberty.org/the-current-wisdom-overplaying-the-human-contribution-to-recent-weather-extremes/
I can hardly wait for the discussion. I hope it remains civil and relevant.
Steven Mosher
You write that you are “not so keen on government regulation of science publishing” and “not so keen on impractical ideas of enforcing rules on the publishing business”.
I would agree that added government regulation of private industry, private research or the publishing business in general is not a good thing (many would argue that we are already way over-regulated today).
But here we are talking primarily about taxpayer-funded climate research, and that is a different story.
If the taxpayer has the feeling that he is not getting “his money’s worth” in climate research today, because of a “peer review” process that is either just sloppy or has been hijacked by AGW activists who are trying to establish the “proof” for dangerous AGW, rather than finding the “truth” about what makes our climate work, then it is time to make sure the process is not broken.
Will this committee be successful in doing this? My guess it that there is a 50-50 chance that the investigation will either get so watered down into non-climate related issues or steered in a direction to benefit the “consensus” view, that it will end up being essentially a white wash.
But who knows?
I’d say, let’s assume that it will accomplish something positive.
What then comes out as recommendations to improve the process by making it more transparent, less “insider-controlled”, more open to dissenting views or independent audits, etc. is a second discussion.
But, Steven, the key point is the taxpayer is funding this work, so the taxpayer has a full right to insist that he is getting his money’s worth, rather than just being fed a “party line” by an insider group of activists, as is perceived by many to be the case today.
Max
For government funded research of course one can and should impose quality control on the reports of the funded study.
that is a question for the program manager.
But the idea that the government or anybody else should dictate to Science or Nature what to do sounds wrong headed to me. What sounds right headed is to compete with Science and Nature.
Steven Mosher
The Government “dictating to Science or Nature what to do” is a side issue here.
The main issue is that the government has the right and the duty to spend taxpayer funds wisely and judiciously. If it is spending billions of taxpayer dollars for climate research, then it has the right and obligation to make sure it is not simply spending money to allow AGW doomsday advocates to promote their message but for serious, unbiased and totally transparent climate science. And this includes the peer review process, both from the standpoint of “pal review” and “censorship by gatekeepers”. If those “pals” or “censors” happen to be part of the editorial staff of Nature or Science, then it is the government’s job to tell these individuals “what to do ” (and “what not to do”).
This is nothing more, Steven, than applying “the golden rule” to protect the taxpayer, who is funding the whole exercise in the first place.
Max
Steve Mosher; governments dictating to Science or Nature what to do
Government already dictates what climate science itself does, since it funds it; it chooses who and what gets funded, and who and what doesn’t. Exactly like tobacco companies dictated the science they funded. And in both cases, the funded science finds in favour of its funders’ vested interests.
And since peer-reviewers are volunteers on government checks during the day, government already controls peer-review to some extent, giving it the same bias it gives the science it funds.
So why is that influence ok, but trying (or appearing to try) to influence it in the direction of balance (ie sincere science), not ok?
What sounds right headed is to compete with Science and Nature.
How are they funded?
Some posters, like Steven Mosher, are arguing for a more “laissez faire” approach to what basically boils down to a “quality control” issue related to the peer review and publication of taxpayer funded climate research work.
IMO, this argument leads to a slippery slope.
If the taxpaying public continues to lose confidence in the objectivity of the”climate science” being published today, it will only be a matter of time until it stops the funding of climate science through pressure on its elected officials (this has already happened to a slight extent with the US Congress decision to cut IPCC funding).
It would behoove the greater climate science community to embrace any investigation of the peer review process in climate science enthusiastically and cooperatively, in the hopes that it can again regain the confidence it has lost as a result of the actions of a corrupt IPCC process and a handful of “consensus insiders”.
And, if some “consensus insiders” attempt to derail the investigation or turn it into a whitewash exercise, or later to stonewall against any corrective actions, this would be the worst case for public confidence in climate science and, as a result, for the future of government-funded climate science in general.
Judith Curry and many others have seen this danger.
Eroding confidence of the public that it is “getting its money’s worth” (or even that it is being “bamboozled”) can lead to public wrath, which can very quickly lead to an abrupt halt in public taxpayer funding.
That is what is at stake here, even if many have not quite realized it yet.
Max
The UNFCCC (UN Framework Convention on Climate Change) should be defunded, including its offspring, the SBSTA. The IPCC is the publicity branch; the UNFCCC is the perpetrator. If funded through our dues to the UN itself, offset the dues paid to the UN by 22% of the UN funding to UNFCCC, et al. (The UNFCCC could be considered distinct from the IPCC itself, having been set up by a UN Conference in 1992. This would evade the intent of Rep. Luetkemeyer’s proposal .)
The objective of the UNFCCC is found in “Article 2: OBJECTIVE” of the 1992 United Nations Framework Convention On Climate Change. United Nations. http://unfccc.int/resource/docs/convkp/conveng.pdf
The “UNFCCC Objective” governed the IPCC Scenarios, famed for the catastrophes they “predicted” that did not happen. See the Foreword of:
Hulme, Mike, Tom Wigley, Elaine Barrow, Sarah Raper, Abel Centella, Steve Smith, and Aston Chipanshi. 2000. Using a Climate Scenario for Vulnerability & Adaptation Assessments. Climatic Research Unit, May.
http://www.aiaccproject.org/resources/ele_lib_docs/magicc_scengen_workbook.pdf
Lots of really interesting comments.
In my view it has been shown beyond doubt that peer review in climate science is broken. So, something must be done.
Yes, Richard, “peer-review in climate science is broken.”
That is only a small part of the problem. Sowell’s book, “The Vision of the Annointed” describes the approach that government-funded ‘teams’ used to provide consensus misinformation on –
a.) The Sun’s origin: Galileo-gate [1,2]
b.) The Sun’s composition: Apollo-gate [3]
c.) The Sun’s source of energy: DOE-gate [4]
d.) The Sun’s influence on Earth’s climate: Climate-gate
The US public paid $$$ billions for deception, e.g. over $1,000,000,000 each for Apollo and Galieo Missions alone.
“Team” members are pawns. The root of the problem is in Washington, DC where science bureaucrats “annointed” teams with research fund in return for endorsements of
a.) Standard Solar Model
b.) The Hydrogen-filled Sun
c.) Oscillating solar neutrinos
d.) CO2-induced global warming
With kind regards,
Oliver K. Manuel
Former NASA Principal
Investigator for Apollo
1. “Abundances of hydrogen and helium isotopes in Jupiter”
http://www.omatumr.com/abstracts2005/Nolte_and_Lietz.pdf
2. “Galileo probe confirms ‘strange’ xenon in Jupiter”
http://www.omatumr.com/Data/1998Data.htm
3. “Solar abundance of the elements”
http://www.omatumr.com/archive/SolarAbundances.pdf
4. “Neutron Repulsion”
http://arxiv.org/pdf/1102.1499v1
Open review methodology, copied from Pielke Sr. blog
http://pielkeclimatesci.wordpress.com/
Guest blog post: Dr James Ward 11th March, 2011 [his bio is at people.unisa.edu.au/james.ward]
Point 2. The review process. The journal is open access, and open review. The paper was subject to an initial editorial review, which led to minor revisions, but is now open for public interactive discussion until May 3rd. That is, people read the paper online, and make scientific comment, and we (the authors) will have to reply as quickly as we can. The comments and replies are published rapidly, and remain published after the discussion closes (at which point the paper is finally revised for publication in the main journal). Even after the final paper is published, the initial discussion paper, and all comments and author replies, remain online. This means that virtually the entire peer review process is kept in the public realm, in contrast to the usual system of an anonymous and secretive review. It means that if someone wants to challenge our position, they have to be prepared to have their critique published, and possibly rebutted or disproven in public (it also means that we, the authors, could be publicly humiliated if we have made any glaring errors in our analysis that are picked up in the review). Under an anonymous review, neither the authors nor the reviewers have to worry about such things. However, several highly publicised examples have emerged in the blog-o-sphere which – if true – suggest that the traditional (anonymous) peer review process has allowed bias into the system, and controversial ideas are sometimes being blocked from publication without due cause. At least this should be minimised in the open review system.
Let me suggest that a solution is partly in place now, and may be implemented with a few improvements already mentioned here.
Passage of time is a factor. The journal peer review cycle is not adequate. It may take a year or more for a response to be prepared.
Journal articles can be Googled, yielding both the article and current references and citations in arXiv.
Comments from a wider public may identify alternatives, exceptions and extensions to the original paper.