by Judith Curry
But the truth is that peer review as practiced in the 21st century biomedical research poisons science. It is conservative, cumbersome, capricious and intrusive. It slows down the communication of new ideas and discoveries, while failing to accomplish most of what it purports to do. And, worst of all, the mythical veneer of peer review has created the perception that a handful of journals stand as gatekeepers of success in science, ceding undue power to them, and thereby stifling innovation in scientific communication.
Peer review is f***ed up
So begins a post “Peer review is f***ed up” on the blog it is NOT junk, by evolutionary biologist Michael Eisen. Some excerpts:
There are too many things that are wrong with this process, but I want to focus on two here:
1) The process takes a really long time. In my experience, the first round of reviews rarely takes less than a month, and often take a lot longer, with papers sitting on reviewers’ desks the primary rate-limiting step. But even more time consuming is what happens after the initial round of review, when papers have to be rewritten, often with new data collected and analyses done. For typical papers from my lab it takes 6 to 9 months from initial submission to publication.
The scientific enterprise is all about building on the results of others – but this can’t be done if the results of others are languishing in the hands of reviewers, or suffering through multiple rounds of peer review. There can be little doubt that this delay slows down scientific discovery and the introduction to the public of new ways to diagnose and treat disease [this is something Pat Brown and I have talked about trying to quantify, but I don’t have anything yet].
2) The system is not very good at what it purports to do. The values that people primarily ascribe to peer review are maintaining the integrity of the scientific literature by preventing the publication of flawed science; filtering of the mass of papers into to identify those one should read; and providing a system for evaluating the contribution of individual scientists for hiring, funding and promotion. But it doesn’t actually do any of these things effectively.
The kind of flawed science that people are most worried about are deceptive or fraudulent papers, especially those dealing with clinical topics. And while I am sure that some egregious papers are prevented from being published by peer review, the reality is that with 10,000 or so journals out there, most papers that are not obviously flawed will ultimately get published if the authors are sufficiently persistent. The peer reviewed literature is filled with all manner of crappy papers – especially in more clinical fields. And even the supposedly more rigorous standards of the elite journals fail to prevent flawed papers from being published (witness the recent Arsenic paper published by Science). So, while it might be a nice idea to imagine peer review as some kind of defender of scientific integrity – it isn’t.
And even if you believed that peer review could do this – several aspects of the current system make it more difficult. First, the focus on the importance of a paper in the publishing decision often deemphasizes technical issues. And, more importantly, the current system relies on three reviewers judging the technical merits of a paper under a fairly strict time constraint – conditions that are not ideally suited to recognize anything but the most obvious flaws. In my experience the most important technical flaws are uncovered after papers are published. And yet, because we have a system that places so much emphasis on where a paper is published, we have no effective way to annotate previously published papers that turn out to be wrong: once a Nature paper, always a Nature paper.
The Scientist has a post “I hate your paper,” which identifies additional problems with peer review. Some excerpts:
Solution: Eliminate anonymous peer review ( Biology Direct, BMJ, BMC); run open peer review alongside traditional review (Atmospheric Chemistry and Physics); judge a paper based only on scientific soundness, not impact or scope (PLoS ONE)
One of the most hotly debated aspects of peer review is the anonymity of the reviewers. On the one hand, concealing the identity of the reviewers gives them the freedom to voice dissenting opinions about the work they are reviewing, but anonymity also “gives the reviewer latitude to say all sorts of nasty things,” says Kaplan. It also allows for the infiltration of inevitable personal biases—against the scientific ideas presented or even the authors themselves—into a judgment that should be based entirely on scientific merit.
An alternative way to limit the influence of personal biases in peer review is to limit the power of the reviewers to reject a manuscript. “There are certain questions that are best asked before publication, and [then there are] questions that are best asked after publication,” says Binfield. At PLoS ONE, for example, the review process is void of any “subjective questions about impact or scope,” he says. “We’re literally using the peer review process to determine if the work is scientifically sound.” So, as long as the paper is judged to be “rigorous and properly reported,” Binfield says, the journal will accept it, regardless of its potential impact on the field, giving the journal a striking acceptance rate of about 70 percent.
Problem #2: Peer review is too slow, affecting public health, grants, and credit for ideas
Solution: Shorten publication time to a few days (PLoS Currents Influenza); bypass subsequent reviews (Journal of Biology); publish first drafts (European Geosciences Union journals)
A handful of other journals have taken a different tactic altogether to tackle the problem of publication time lags—keep the traditional peer review process but first publish a preliminary version of a submitted paper. Atmospheric Chemistry and Physics, launched by the European Geosciences Union (EGU) in 2001, along with the 10 or so sister journals that have subsequently been launched by the EGU, employs a “two-stage” process of publication and peer review, concurrent with an interactive public discussion. After a quick prescreening by one of the journal’s expert editors, a submitted manuscript is immediately published on the journal’s website as a “discussion paper,” and is available for anyone to see and comment on for 8 weeks. At the same time, the manuscript is passed on to referees who are familiar with the subject, and their comments (for which they can claim authorship or remain anonymous) are also posted alongside the discussion paper, public comments, and authors’ replies. The manuscript can then be accepted for publication, at which point a revised paper is published in the main, open-access journal.
Open post publication peer review
From an article “Open post-publication peer review” posted on the blog The future of science:
Open: Any scientist can instantly publish a peer review on any published paper. The scientist will submit the review to a public repository. The repository will link each paper to all its reviews, such that that readers are automatically presented with the evaluative meta-information. Peer review is open in both directions: (1) Any scientist can freely submit a review on any paper. (2) Anyone can freely access any review.
Post-publication: Reviews are submitted after publication, because the paper needs to be publicly accessible in order for any scientist to be able to review it. For example, a highly controversial paper appearing in Science may motivate a number of supportive and critical post-publication reviews. The overall evaluation from these public reviews will affect the attention given to the paper by potential readers. The actual text of the reviews may help readers understand and judge the details of the paper.
Peer review: In an open peer-review system, writing a review is the equivalent of getting up to comment on a talk presented at a conference. Because these reviews do not decide about publication, they are less affected by politics. Because they are communications to the community, their power depends on how compelling their arguments are to the community. This is in contrast to secret peer review, where uncompelling arguments can prevent publication because editors largely rely on reviewers’ judgments.
Signed or anonymous: The open peer reviews can be signed or anonymous. In analyzing the review information to rank papers, signed reviews can be given greater weight if there is evidence that they are more reliable.
From another post on the same blog:
Free instant publishing: Once open post-publication peer review provides the critical evaluation function, papers themselves will no longer strictly need journals in order to become part of the scientific literature. They can be published like the reviews: as digitally signed documents that are instantly publicly available. Post-publication review will provide evaluative information for any sufficiently important publication. With post-publication review in place, there is no strong argument for pre-publication review. Publication on the internet can, thus, be instant and reviews will follow as part of the integrated post-publication process of reception and evaluation.
So, does post publication peer review work? The biological journal PLoS has been experimenting, and so far, there hasn’t been much of an impact.
JC comments: During the past few weeks, we have seen two interesting examples of peer review: the pre-publication extended peer review of the BEST papers, and the post-publication extended peer review of the Ludecke et al. papers. The extended peer review in the blogosphere was far more substantial than the papers were likely to receive in the normal peer review process. In both instances, the extended peer review of these papers conducted in the blogosphere were not part of the formal peer review process. Scientists who do not check the blogs might be completely unaware that this extended peer review has occurred.
I am a big fan of preprint servers such as ArXiv, and also the online discussion journals such as Atmospheric Chemistry and Physics. ACPD actually posts the reviews along with the paper, and allows people to comment during the peer review process. Extending this to include blog discussions on the paper would be great.
The prestige journals such as Nature, Science, and PNAS do not allow any pre-publication of the paper and serve a gate-keeping role in determining “significance.” Personally I am not a fan of this approach, but it seems to have worked in terms of generating high impact factors for these journals.
Your thoughts and ideas?
We are at the point now where is nothing can be accomplished until we get government out of the classroom.
“We are at the point now where is nothing can be accomplished until we get government out of the classroom.”
Because, news is not objective, and peer reviewed isn’t science?
Something everything should know, but they don’t?
Because have public baby siting instead of public education?
But we don’t need to the correct news, nor the best peer review, nor even
get rid public education, but we need more competition.
And as it happens, this is all occurring. And main driver has been the internet. So that’s good news [in world that is spinning out control:)].
As for our peer review, it’s basically avenue to get a job- it provides status, promotes your work. Like a form baboon hierarchy. Which is fine.
A question is what is wanted something which also about status- perhaps a much higher status.
I am not a scientist but seems Nature etc are rather insignificant, it seems there tons of scientific association which allow papers to be published- they have conferences all the time. Not political conferences- like Durban, but conferences which involve scientists relating to scientists. These are intended to be a service to scientists. Whereas Nature, etc is more of public relation type operation- or a non gadget oriented Popular Science- science theory rather than technology magazine.
It seems if one wants status, one needs a board with bunch scientists with a diversity of all the disciplines [not limited to a small board- could grow to hundreds]. I would imagine the most important aspect of such a board is to have this board activity waste the least amount of their time.
So one need to lowest “cost” in terms of the administration part- meaning having a system and/or have good administrators.
So I would say what needed is talented hard working executives. It’s more of management issue rather than a science issue.
Dr. Marvin Herndon has written several good articles on the evils of anonymous peer reviewers, a practice started by the National Science Foundation (NSF) in the early 1950s.
http://nuclearplanet.com/Science_Suppression.html
http://nuclearplanet.com/Anerican%20Science%20Decline.html
With kind regards,
Oliver K. Manuel
Peer review only has a problem because its purpose has been distorted.
There is an assumption that since a paper is Peer Reviewed, its CONCLUSIONS are correct. This is a fallacy.
The ONLY thing that Peer Review is capable of checking is that the METHODOLOGY is correct.
Peer Review does not attempt to confirm the conclusions of the study, thus it has nothing valid to say about the conclusions.
This distortion has come about because of the large number of scientific papers that have published conclusions that CANNOT be tested. These papers attempt to imply that Peer Review is a SUBSTITUTE for TESTING.
There is no substitute for testing. The solution to Peer Review is quite simple. The scientific community need only apply the standards that already exist.
Peer Review shows that the methodology in the study was (likely) correct. However the conclusions of the study are not in any way confirmed by peer review.
Confirmation requires independent testing. Until that is done, nothing meaningful can be said about the conclusion. They may be right or they may be wrong . No one knows. Folks that do comment one way or the other are simply guessing.
“There is an assumption that since a paper is Peer Reviewed, its CONCLUSIONS are correct. This is a fallacy. The ONLY thing that Peer Review is capable of checking is that the METHODOLOGY is correct.”
Right on! Once I had the misfortune of working in a field in which conclusions were assumed to be correct because the paper was peer reviewed though the methodology was incorrect. Fatal methodological errors persist in this field, twenty five years later.
Wagathon,
Can you give me some examples of non-governmental science of which you approve?
I’m a big fan of intermittent windshield wipers –they save lives and provide utility that people voluntarily pay for — unlike the filing cabinets full of worthless AGW junk science toy models
Wagathon,
Right. I’ll take that as a no!
Well said.
Also have a look at hypothes.is project which aims to bring peer-review to the Internet (blogosphere) utilising tong-term crowd-sourced reputation models to asses credibility of the information.
Very interesting project: “An open-source, community-moderated, distributed platform for sentence-level annotation of the Web.”
Check out their Kickstarter which ends in few hours http://kck.st/nbvi0G
thanks. looks cool.
Corruption of Peer Review
Contrast the conspiracy within “The Team” to explicitly corrupt peer review – with Paucharia,head of IPCC, acquiescing, as exposed by the ClimateGate II emails:
Tribalistic corruption of peer review – the Chris de Freitas incident
Climategate 2 – Salinger puts the boot into De Freitas
Climategate 2 and Corruption of Peer Review
Climategate 2 and Corruption of Peer Review – Part II
“In both instances, the extended peer review of these papers conducted in the blogosphere were not part of the formal peer review process. Scientists who do not check the blogs might be completely unaware that this extended peer review has occurred.”
I could create a blog post with 10 invented flaws in the BEST papers and 10 invented strengths in the Ludecke papers. Since “extended peer review” is a self selected self publishing process how is the validity and quality of that review to be determined?
A crowd-sourced long-term reputation system? ;)
Check out http://hypothes.is for possible solution to the problem you highlighted. It IS being worked on and it’d work for all content, not only scientific papers …
“A crowd-sourced long-term reputation system? “
Once you construct a system of rules the system will be subverted and gamed. See any internet ranking system ever devised for examples.
“Check out http://hypothes.is for possible solution to the problem you highlighted. It IS being worked on and it’d work for all content, not only scientific papers …”
This looks a prime example of a system just waiting for the aforementioned subversion.
Do you think the future of science should be thinkprogress and WUWT directing people to vote up their favourite papers? Talkorigins and creation wiki voting down the credibility of each other’s positions?
Until software is able to independently judge the quality of work it will simply act as a tool to publish the criticisms of others. The wider the net you cast for those criticisms the lower their average quality will be.
Computers can allow hundreds of millions of people to share their views on the latest blockbuster movie. It can’t tell you which of those views are any good, which of them are relevant or which of them will appeal to you.
Nothing is perfect, I could make similar condescending comments about current form of peer-review: you want for a few chosen-ones decide what is acceptable and what not in secret behind closed doors with no way to check?
Well, that doesn’t even sound too removed from reality, does it? ;)
The point is that the proposed system may end up being much superior to classical peer-review, not perfect … nothing is perfect.
And it’s not about “voting”, did you miss the part about long-term reputation? There is currently NO SYSTEM that implements anything remotely like that … all are dumb voting machines, which I completely agree with you are useless when deciding veracity of somehting … but this is NOT THAT. I recommend to look deeper …
“Nothing is perfect”
Well yes, that’s my point. The current system has flaws sure but before switching to another flawed system we need to be sure it’s less flawed. not more.
“you want for a few chosen-ones decide what is acceptable and what not in secret behind closed doors with no way to check?”
I want the people with the expertise to do so yes. Wrapping your arguments in elitist language does little to advance them.
“The point is that the proposed system may end up being much superior to classical peer-review,”
That’s a hope, not a point. Arguments need to be made for how to realise this hope.
“And it’s not about “voting”, did you miss the part about long-term reputation?”
Reputation will still have to happen in one of two ways:
1. At the whim of “a chosen few”.
2. As determined by an algorithm that can be analysed and gamed.
Have you ever participated in an open moderated community? They still need chosen editors just to function and even then gaming regularly overwhelms votes that comply with the guidelines. Everything becomes a grab for popularity.
Anything can be told to be “analysed and gamed”, that’s not a valid point unless applied to a specific process.
For example the prime example is peer-review in climate science that has been gamed by few gatekeepers conspiring to prevent publication of papers they don’t like, boycotting papers if they do publish and keeping any such papers from the IPCC reports – that is a definition of a gamed system.
And would climategame never happened we would never be able to prove this was going on and you would make the usual comments about some unbelieveable “conspiracy” that is only in our heads … thank god for climategate so we have specific communications to prove every point I mentioned.
“Anything can be told to be “analysed and gamed”, that’s not a valid point unless applied to a specific process.”
Systems operated by human judgement are difficult to game. A system like the current peer review process is nigh impossible to game, instead people wanting to publish whatever establish their own crappy journals for it and suffer the consequences of being a crappy journal.
“For example the prime example is peer-review in climate science that has been gamed by few gatekeepers conspiring to prevent publication of papers they don’t like, boycotting papers if they do publish and keeping any such papers from the IPCC reports – that is a definition of a gamed system.”
This charge is not in evidence. If your argument for switching from the current peer review system hinges on the reality of it then you’re going to have to do a lot better than blog posts and papers retracted for rampant plagiarism.
“thank god for climategate so we have specific communications to prove every point I mentioned.”
I didn’t realise we were playing the skeptic bingo game, you forgot to mention realclimate moderation for a perfect score.
Dude, where have you been, in Mongolia?
The 2011 play-at-home version also requires mention of Skeptical Science moderation policies, an assertion that “the warming has stopped,” a whine about BEST not being peer-reviewed, and at least two assertions that all climate scientists say that “the science is settled” at minimum semi-annually.
I think that we should just send all articles that might potentially be published to Wagathon and let him decide.
It’s the only way that we can overcome the “bias form personal motives” of traditional peer review.
There is no “bias from personal motives” in blog discussions of scientific analysis – as is proven by the blog discussions of BEST and Ludecke et al. Particularly once Wagathon gets involved.
“Every man takes the limits of his own field of vision for the limits of the world.” ~Arthur Schopenhauer
“I am the wisest man alive, for I know one thing, and that is that I know nothing.” – Socrates
“I am the wisest man alive, for I know one thing, and that is that I know nothing.” – Socrates
I am not the wisest man alive , for I know one thing and that is that when a peer eviewed paper predicts catastrophic global warming from increased CO2 emissions and for the past nine years the world has been cooling with record levels of CO2 emissions; Peer review is f***ed up!!!!
How do you determine the validity and quality of anonymous pal reviews, that you never have an opportunity to see? Extended peer review is open for all to see. Weigh the arguments against the counter-arguments and make a judgment. Or take a poll and go with the crowd.
Or to put it another way, the system needs to be better than Amazon.com, where there have been documented cases of people writing reviews of books that hadn’t been published yet. Having things “open” on the internet is just begging for organized and unorganized mischief.
“How do you determine the validity and quality of anonymous pal reviews, that you never have an opportunity to see?”
I don’t, the editor does.
“Extended peer review is open for all to see.”
Open for all to see but with only a few qualified to judge.
“Make it open” sounds nice and everything, I mean who isn’t against things being open? The argument is really not between “open” versus “not open” but rather between “selection” and “no selection”. In the no selection scenario all voices are equal, regardless of quality and accuracy. In the selection scenario someone has to do the selecting.
Pointing out flaws in the current selection process is all well and good but before anyone can agree no selection is better the problems of not selecting need to be addressed: Namely how to extract quality arguments from non-quality ones.
The editors comply with the pal reviewers. If they don’t they get into trouble.
Those of you defending the pal-peer review system against blog review are making the assumption that someone has suggested that it be replaced by blog review. Try to understand the topic, before you shoot your mouths off.
“The editors comply with the pal reviewers. If they don’t they get into trouble.”
Ah the old conspiracy theory defence. I never knew climate scientists were so uniquely powerful.
“Those of you defending the pal-peer review system”
I’m asking people that support an open system to address the problems with such a system, something you’re clearly unable to do.
“are making the assumption that someone has suggested that it be replaced by blog review.”
You’re posting on a blog which explicitly references blog review. Blogs are simply a manifestation of an open publishing system. Maybe the “other people” understand the topic just fine and you don’t.
Climate science peer-review has been re-defined to protect the dogma. Catch up.
Do you think that Judith’s post suggests that blog review, should replace peer/pal review? It seems that is your fear. What are you getting so upset about? The consensus Big Climate pal review system is safe, for the moment.
But I will have to give you credit for not being as vicious as temptyranus.
This is what one of the editors you seem to have such a high regard for says about peer-review:
“When it comes to journals and publications, I’m highly skeptical that [the peer review] process adds much value at all,” adds Richard Smith, former editor of the British Medical Journal, who has written extensively about peer review. “In fact, it detracts value because it wastes a lot of time of a lot of people,” he says. “There’s lots of evidence of the downside of peer review, and very limited evidence of the upside.”
Here is some background on the “arsenic” paper.
http://www.slate.com/articles/health_and_science/science/2011/05/the_discovery_of_arsenicbased_twitter.html
Judith,
I suspect that this post as at least partly influenced by a recent failure, by you and your husband, to get a paper past the peer review process.
Furthermore, whatever its deficiencies might have been, I’m sure you can point to examples of pro-consensus papers which were no better, and yet somehow managed to get through. I’m sure it is harder for non -consensus papers to be accepted.
But its often been pointed out the consensus has sometimes been wrong and those who have challenged it have been unfairly treated. Yes they have. That’s why they often get a Nobel prize by way of an apology when they are finally shown to be correct.
So by all means challenge the consensus if you are sure you are correct. Fame and fortune await if you are shown to be correct. On the other hand, if you aren’t, well … er maybe we shouldn’t go there!
PS I’m not an expert in climate science, but if you’ll allow me to give you a little tip: if you do decide to resubmit your “uncertainty monster” paper, you might want to think about putting in some data, graphs, equations, mathematical models etc. I’ve noticed that most papers tend to include these.
tt – what is your evidence that the uncertainty monster paper has been rejected or do you have inside information? Or are you just practising disinformation?
RobB,
http://judithcurry.com/2011/10/27/defending-the-uncertainty-monster-paper/#more-5482
I take all this to mean it’s had the thumbs down? Or at least in its initial form it has.
But, if I’ve misunderstood, then I’m open to being corrected.
“…you might want to think about putting in some data, graphs, equations, mathematical models etc.”
Good call. The more data, graphs, equations and models, the more important the paper. Far better than a paper with a sound argument, at any rate.
It’s all very well to muse on the extensive blog reviews of a few papers. But they are few indeed; the productivity is very low. The IPCC 2007 report referred to about 18500 papers. Who is going to review those?
Open review ideas are fine too. But will the reviewers come to the party? This is the resource limit in any scheme. I noted, for example, the recent open review paper of Anna Makarieva, reviewed by Judith. For a long time no other reviewer could be found, despite the publicity the paper received. And the process just seems to have fizzled out.
Traditional peer review has many faults. Many alternatives are theorized about. But where will the reviewers come from?
The IPCC 2007 report referred to about 18500 papers.
Name 5 papers in the 21st century where uncertainty is less then in the previous reviews.Rank them in order of importance.
There was a very simple solution suggested to this. If you want to have your paper reviewed you need to review 3 other papers – problem solved :)
The reviews are on voluntary basis even now, nothing would change … there is certainly not going to be less reviewers than currently if journals just transition to open form of peer-review.
“There was a very simple solution suggested to this. If you want to have your paper reviewed you need to review 3 other papers – problem solved”
Have you seen what happens when people are forced to satisfy some blocking criteria in order to get to what they really want? They half ass it.
yeah that would be so much different than what is it now /sarc
especially when no one other than the editor checks it, that just beggs to be half assed, doesn’t it?
anyway, if anyone “half ass[es] it” in an open review their reputation will suffer as a result, I do not think anyone would risk that if they hope to have any career in science …
“anyway, if anyone “half ass[es] it” in an open review their reputation will suffer as a result”
Unless some people like the answer and vote it up purely on that basis.
I already said the system doesn’t rely on pure votes, why are you so intent on mirepresenting it and knocking down a strawman … why would you be threatened by science being done in the open?
If it doesn’t work, it will just wither away … but it seems your fear is that it actually will work. Hm …
“The IPCC 2007 report referred to about 18500 papers. ”
Imagine how many they would need to review if they actually READ the papers that contradicted the bogus CO2 theory …
>The IPCC 2007 report referred to about 18500 papers. Who is going to review those?<
Well, about 1/3rd of them by Greenpeace, of course, Nick
http://reviewipcc.interacademycouncil.net/report/Climate%20Change%20Assessments,%20Review%20of%20the%20Processes%20&%20Procedures%20of%20the%20IPCC.pdf
http://reviewipcc.interacademycouncil.net/Comments.pdf
You haven't been paying attention, have you ?
“1/3rd of them by Greenpeace”
One third? I searched both linked documents for corroboration of your assertion, Ian. It’s not there. No percentage of any kind is given. There are Greenpeace references in WGII and III, but few, if any in WG1, “The Physical Science Basis.”
Peer review primarily serves the interests of the journals, not the authors or the readers. The process is far too cursory to pick up all but egregious errors or fundamental misunderstandings; beyond this basic sanity check most journals are really only interested in the assessment of “importance”, which it itself driven as much by current fads and fashions as by anything objective.
Journals are businesses. They compete. They have to find a system which authors will submit papers, reviewers will provide their time, and readers will pay money. All three are necessary. The current system is what has evolved to balance these.
Nick Stokes sez:
“The current system is what has evolved to balance these.”
Nick, then it’s now showing the evolutionary “punctuated equilibrium”.
I often wonder, when people say that ‘the process is far too cursory to pick up all but egregious errors or fundamental misunderstandings’, whether they are talking about the reviews they write, an assessment of their peer’s abilities to write reviews, or the reviews of their papers that they have had to answer. None reflects well on the person saying it…
Jonathan Jones | November 12, 2011 at 5:08 pm
——————–
Jonathan Jones,
Thank you for your perspective. I basically agree with the thrust of your comment.
My view is that the journal process per se, with its current range of ‘peer review’ methods, is neither necessary nor sufficient for the scientific processes to occur. They are an historical artifact only. They appear to be a convenience today but not essential for the scientific process to persist.
Science would still advance if they self-dissolved into the mists of time.
The scientific community is capable of talking to itself and to all interested minds in the world. N’est ce pas?
John
My view is that the journal process per se, with its current range of ‘peer review’ methods, is neither necessary nor sufficient for the scientific processes to occur. They are an historical artifact only. They appear to be a convenience today but not essential for the scientific process to persist.
And yet new journals are created by active scientists all the time. My favorite, The Annals of Applied Statistics was created just 5 years ago. When the McShane and Wyner paper was accepted for publication in AOAS, it was posted on line and bloggers wrote reviews. Then the reviews were revised for publication and published with the original, and with a rejoinder, and supporting online material was made available online. This was a good combination of approaches, imo, but the printed journal is still necessary.
There are so many adverse rewards in the system of scientific research as practiced that the minor complaints about the inconvenience of peer review seems like registering an issue with the temperature of the bathwater on the Titanic.
Researchers continue to hoard data toward publication or for other less admirable reasons (such as keeping it out of the hands of rivals, keeping contrary evidence from seeing the light of day, sloppiness, sloth, vanity, to extort or trade future favors or prestige, etc.) as there is such pressure to publish new research — meaning new data, at least ‘new’ to the world.
This plunderous attitude toward raw observations is an immediate detrimental result of inverted values, and in my opinion far more damaging than the prolonged torture of waiting for peer review.
Worse, we frequently research the wrong thing or fail to research the right thing — fail to examine areas of study we can absolutely say for certain are valuable or crucial — for frivolous reasons. This is as true in atmospheric sciences, where we have known for over a century at least that we ought be measuring gas and aerosol levels carefully to develop some profile of our air (and yet failed even while weather stations track — badly — temperature), as it is in medicine, where it’s well known that the popular cancers and diseases get research funding, while many true scourges of substantially greater rate of mortality and economic cost or treatability go unfunded.
Internet prestige ought obliterate the very precept of journal prestige.
Online distribution of ideas, with a full open data from cradle to grave philosophy, ought run data hoarders out of town on a rail.
Topical international panels with some rational foundation for shining a light on the lapses and lacks in research, while at the same time promoting the most important and interesting results, ought become the general rule in science while science bodies with lackadaisical what-interests-our-membership or catering-to-corporate-agenda approaches ought slink away in shame.
My thought… the major thing that needs to change is timeliness.
The major thing that needs to be kept is the notion that the reviews are not just scientists, but scientists who are actively working and deeply familiar with the subject matter of the paper. Peers in the sense of working in the same field.
The thing about the reviewers being very narrow experts is both a feature and a bug at the same time. Depending on the nature of the paper, it may need to be looked at by the narrowest experts inside the field, experts outside of the field, or often both.
Usually both!
“Open post publication peer review” would have prevented AGW from ever have becomiing an issue.
Hansen et al was published in the Journal of Geophysical Research Volume 93 on August 20, 1988, but Hansen made his presentation to Congress on June 23, 1988 before this was published and before the greater scientific communioty had access to it and could point out its many shortcomings.
This paper was first recieved by the Journal of Geophysical Research on January 25, 1988; revised May 6, 1988 and accepted May 6, 1988.
Obviously accepting the revised version and accepting it on the same day indicates that there was no proper peer review of the revision so the many shortcomings of the paper were never exposed before Hansen used this paper as the basis for his June 23 1988 presentation to legislators who had no scientific knowledge and accepted what Hansen stated as fact leading to the unfounded panic that created the climate change issue.
Climate model parameters and output are in units of energy flux in W/m^2 but temperature is based on energy which would be in units of energy such as kWh and since no such units are in the climate models it is an impossibility for climate models to make any predictions about temperature without establishing a relationship between energy flux and global temperature. This is not done anywhere in the paper so any proper peer review would have rejected this paper until such a relationship was demonstrated and verified.
The paper also relates CO2 emissions increases to increases in atmospheric CO2 concentration without any valid justification for doing so and more importantly establishes a CO2 forcing parameter that has no documentation on either how it was derived or what its value or range of values was. It was not until much later that it was revealed that this forcing parameter was based on the observed 0.6°C of warming being attributed to the 100ppmv increase in CO2 concentration from the preindustrial level of 280ppmv to recent value of 380ppmv as this warming took place.
This relationship to the best that I can tell was
5.35ln(380/280) = 1.634 W/m^2
Since this change in flux is stated to have caused 0.6°C of temperature change the the derived climate sensitivity for this model should have been
0.6°C/1.634W/m^2 = 0.367°C for each Watt per square metre.
This paper states that the models are based on a climate sensitivity of 4.2°C for a doubling of CO2.
Based on 5.35ln(2) = 3.71 and the 0.367°C/W/m^2 conversion to temperature this only comes to only 1.366°C which 32.4% of the value used by Hansen for the climate models in this paper.
This leads to another problem with the paper. This paper was written a full decade before Mann removed the Little Ice Age and the presence of the Little Ice Age requires that at least 0.5°C of the 0.6°C attributed to the 100ppmv increase in CO2 should have been attributed to the natural warming recovery from the Little Ice Age.
This would reduce the factor relating forcing from CO2 to temperature to one sixth of the 0.367°C/W/m^2 derived value which is just 0.061°C/W/m^2.
If this properly justified value is used a doubling of CO2 producing 3.71W/m^2 of forcing will only produce 0.227°C of warming and not the 4.2°C used by the model for the projection of temperature increase from a doubling of CO2.
CO2 is currently increasing at 2.037ppmv/year so it will be 191.5 years before CO2 doubles from the current 390ppmv concentration and produces 0.227°C of warming.
If “Hansen et al 1988” had either been properly peer reviewed or if the global scientific community had been allowed to do a post publication peer review; predicted global warming of 0.227°C or even the six times higher value of 1.366°C for a doubling of CO2 would not have led to global warming becoming the issue that it has now become.
Deniers/skeptics don’t like Hansen et al 1988 because it’s three scenarios forecast global warming for 1988-2020 and.. Guess What. .. it has been warming. Rather than give Hansen credit, they complain none of the three scenarios has turned out to be exactly right. Are there any denier/skeptic global temperature forecasts made back in 1988 that we can compare with Hansen’s work to see who did best at forecasting?
What is wrong is actually right – because there has been some warming ?
You do see things in an interesting way.
On second thought , I may understand your comment.
Consider the following analogy.
If yesterday the forecast was 2 inches of rain today, and it actually rained only 1/4 inch today, you probably would call that a wrong forecast or a bad forecast. But if yesterday you were considering an outdoor activity for today, such as a picnic or house painting, the wrong forecast would have proved useful, and useful is good.
Let’s mold your analogy to better represent the situation:.
During a rainstorm the weatherman predicts 2 inches more but it turns out to be 1/2 an inch.
“During a rainstorm the weatherman predicts 2 inches more but it turns out to be 1/2 an inch.”
Corporate, that’s as accurate as the forecast in the first analogy, but not as useful. However, it is better than a “no more rain today” forecast.
Got any skeptic / denier forecasts of global temperature that were made back in 1988 when Hansen made his forecasts.
“Got any skeptic / denier forecasts of global temperature that were made back in 1988 when Hansen made his forecasts.”
You might find some farmers almanacs predictions of that time period- but they are probably regarding US temperatures if anything.
Here is Hansen prediction as compared to measured temperatures and later prediction by others:
http://www.appinsys.com/globalwarming/GW_TemperatureProjections.htm
If appears Hansen missed by about .2 C or more per decade. Or by 2019 Hansen predicted average temperatures to about 1 C warmer than current temperatures. Anyone think we could get .5 C or higher increase
in less than decade?
I think human emission of CO2 will be double what Hansen predicted by 2019.
I would guess that by 2019 we could be as much as ..2 to .3 C warmer compared to present:
http://www.drroyspencer.com/latest-global-temperatures/
And therefore still sort of flat or slight warming- perhaps still not have a single year higher than 1998, but having higher year to year trend of warming
Though if solar activity continues to be on low side, it’s possible by 2019 it might be .1 to .2 cooler.
I think if by 2019 temperature were 1 C warmer, I would have to agree that CO2 was major cause of this increase. I think if instead it rose 1/2 this amount- it was .5 C- I would have agree that a significant amount warming was due to CO2. But as I expect if it rise as much as .3 C I will continue to assume CO2 may add some warming but it’s rather insignificant. And if continues at more or less current trend I see as evidence that CO2 from human emission is leaving no measurable “fingerprints” upon global temperature.
There were no sceptic/deniers in 1988, in fact the word denier when atttached to a belief was only used in the context of the then fashionable idea among racists that they could make the Holocaust go away. It was subsequently applied to sceptics of the CAGW theory by green fanatics to associate those who challenged the theory of CAGW with Holocaust deniers in an attempt to direct the public’s gaze away from their challenges to the flimsiness of the science by categorising them as evil people. Moreover Hansen was looking at world that had been warming since around 1800. It should come as no surprise to anyone that it continued to warm. In fact the only surprise is that it actually stopped warming around ten years ago.
A good analogy is hard to build from your beginning.
You did not say that it was already raining, and that if it does rain 2 more inches according to prediction, cities will be ruined, and everyone is screwed.
The Blackboard has a good discussion of Hansen’s forecasts.
http://rankexploits.com/musings/2008/ordinary-eyeball-how-did-hansens-predictions-do/
Corporate, I don’t understand your comment.
M.caey
Deniers/skeptics don’t like Hansen et al 1988 because it’s three scenarios
Yeah. And guess what? It has been warming since well before the Industrial Revolution, as we have been recovering from a colder period called the Little Ice Age (did you somehow miss out on that one?).
Hansen’s 1988 forecast was grossly exaggerated, quite simply because he used a climate sensitivity that was too high by a factor of more than 2:1.
In fact, the actual warming was the same as he had forecast for a case where CO2 emissions were frozen, but in actual fact CO2 emissions increased at a greater rate than his fastest case.
One should, however “give Hansen credit”. He created the CAGW panic (and with it the multi-billion dollar CAGW business of today) out of bogus data, which no one checked at the time, but which turned out to be grossly exaggerated.
Max
No we complain because based on this single projection of global temperature hundreds of millions of the world’s poor face starvation from the 85 billion litres of ethanol produced that uses 6.5% of the world’s grain supply as feedstock.
If there were no consequences to this overstatement of projected global temperature I would not be making these comments or be involved in this discussion because I really don’t care whether Hansen’s predictions are right or wrong; I only care about people and all the damage done to the global economy and the global population that stemmed from this paper with its ” f***ed up peer review!
Peer review works exactly the same way, benefits and costs, in commercial publishing (unless the author is employed by the publisher), the social sciences and, I suspect, the humanities. Realistically, it will continue to do so as long as publication depends on human nature.
One of the issues with current peer review is the level of complexity require to impress contemporaries. At one time simplicity seemed to be valued when dealing with the complex. Now complexity seems to be the norm for dealing with the simple.
Instead of a comparison of time series methods to verify that there is not enough information contained in the time series, using a PCA analysis on both trend and de-trended data spliced in short and long segments areally weighted following appropriate smoothing to eliminate pesky signals, it much more appealing.
Linear no threshold modeling of practically imperceptible probabilities as opposed Bayesian methods that might show not statistical significance are in vogue, because there has to be some impact due to something whether understood or not.
Where the lack of sufficient data might preclude more standard approaches, properly selected fabrication of data can really catch the eye of peers and truly wow the public.
I see absolutely nothing wrong with the current peer review process. With out it, Oluster, with its minor side effect of anal leakage, would never have benefited the weight conscious for the approximately two weeks it was on the market.
How To Save Science
A fair and efficient peer review process would seem to:
Have those selected to review and comment on publications randomly selected from a “qualified pool” predetermined as potential reviewers
Maintain confidentially of both authors of articles and reviewers through out the review process
Peer review will die a natural death due to economic and political forces that no one controls. The more information that becomes available, the better, as long as no one is stifled in their ability to criticize it.
The centralization of power over the flow of information by journals and editors is being rejected by the market. Peer review gave us the hockey stick, but it also gave us Steve McIntyre, Anthony Watts, Judith Curry et al. as a response. So the death of peer review may end up being ruled a suicide.
Has McIntyre’s attempt to make a parallel between Mann and child rapist Jerry Sandusky been peer reviewed?
I haven’t seen his comments on the issue. But if he is comparing the thoroughness (or more properly the lack thereof) of the investigations of two Penn State luminaries by Penn State, I would have to say I agree there are similarities.
Guilt by association?
M.carey, you’re certainly telling a big untruth
You will not be able to provide any evidence for that lie.
True, true Mann stabbed all of us in the back not the backside.
Corporate, for McIntyre’s shameful attempt to rub some of the sex scandal off on Mann, see
http://climateaudit.org/2011/11/10/penn-state-president-fired/#more-14925
M. carey | November 13, 2011 at 1:41 am |
M. carey, thank you for the referral. I also encourage people to read the article. The issue Steve McIntyre discusses is not sex. It is the reprehensible actions of the fired Penn State President Graham Spanier in the Mann case as well. As Steve closes by saying,
It’s not just the President. The University Counsel, Wendell Courtney, is also implicated in the current scandal. He was the lawyer who advised the Penn State Inquiry Committee that “exonerated” Michael Mann.
In any case, M. carey, I appreciate your bringing up the opportunity to get the truth out to as many people as possible. ClimateAudit is a great resource, thanks for mentioning it.
w.
Willis, I will break down the McIntyre quote into its three component sentences, comment on each, and pose some questions
1. McIntyre said: “Spanier was fired not because of any personal role in the Sandusky football scandal, but because of negligence on his part in ensuring that the allegations were properly investigated.”
Yes, Spanier was fired by the Penn State Board of Trustees, but did the Board state the reason was “negligence on his part in ensuring that the allegations were properly investigated.”
2. McIntyre said: “This was not the only case in which Spanier failed to ensure proper investigation of misconduct allegations.”
This could be interpreted to mean the Board of Trustees, which fired Spanier, had found him negligent in ensuring other allegations were properly investigations. Is that true?
3. McIntyre said: As noted above, Spanier had falsely reported to the Penn State trustees and the public that the Penn State Inquiry Committee had properly interviewed critics and had examined the Climategate documents and issues “from all sides”.
This could be interpreted as the Board of Trustees’ finding. Is that true?
______
Yes, M.carey ,
There is nothing there which resembles what you claim. In fact, you then apparently need to ( below) begin asking leading questions – instead of providing any evidence for your claim.
Corporate, I believe this is what McIntyre is suggesting in his comments at Climate Audit. Because Spanier may have been negligent in assuring pedophile allegations against Sandusky were properly investigated, he may have been negligent in assuring academic misconduct allegations against Mann were properly investigated by the Committee that exonerated Mann, and therefore Mann may be guilty of academic misconduct.
McIntyre sure likes to speculate about Mann. The irony is McIntyre doesn’t like what he perceives to be speculation about temperature proxies by Mann and other climate scientists.
BTW, those questions in my previous post were not leading. A leading question would be “You are defending McIntyre’s attempt to use Spanier’s firing over the Sandusky scandal to discredit the unrelated exoneration of Mann, aren’t you?
M.
just wait. it get’s better.
Two big and related problems with peer-review are that people are short of time and that reviewers are sloppy adn often poorly qualified. I have received peer-reviews, which i) were obviously written by graduate students or perhaps even undergrads, ii) contained mistaken assertions and egregious errors of statistical reasoning, and iii) were riddled with spelling and grammatical errors! I suspect that harried professors, confronted by too many demands on their time, farm out papers to their students to cut their teeth on.
Although I am sure that some students are excellent reviewers, peer-review can only work if the reviewers are suitably qualified to assess the material. Having correctly qualified reviewers can not guarantee their objectivity or lack of bias, but I suspect that no review system could do so.
I have received peer-reviews, which i) were obviously written by graduate students or perhaps even undergrads, ii) contained mistaken assertions and egregious errors of statistical reasoning, and iii) were riddled with spelling and grammatical errors!
I have to agree there. Farming out papers to clueless undergrads is a recipe for shoddy refereeing. Same for clueless full professors.
Like the Loch Ness monster, the subject of peer review surfaces periodically in this blog and elsewhere. Peer review isn’t going to go away (I hope), but it will undoubtedly evolve, if for no other reason than the power of the Internet to add its own weight to the judgments exercised by journals and reviewers. It’s a process limited by the imperfections of human fallibility and bias, and so for me, the relevant questions relate to how specifically to improve it rather than how to turn humans into paragons of omniscience. At this point, I remain unconvinced of the value of any proposed alternatives to modify the basic process, although some of the supplementary forms of post-publication review mentioned above seem to have merit. That’s a tentative judgment, and I hope to remain open-minded.
My thoughts have some personal basis, because I’ve had decades of experience with peer review. Like Joni Mitchell, I’ve seen it from both sides now, or more accurately all sides now. I’ve been a reviewer who has often spent hours on a difficult and sometimes undeserving manuscript. This has convinced me that even if I had been paid at one thousand times the going rate, it would still have been less than I deserved. (Translation – reviewers work for free out of a sense of obligation. It’s surprising and gratifying that so many take the time to do a good job, even if others do not. One certainty is that any change that makes reviewing more onerous will be a step backward by discouraging good people from participating in the process. This includes making too many new demands on them, or making their efforts seem superfluous).
I have been a reviewee. Some of my papers have been published in Science, Nature, and respected specialty journals, which confirmed for me that the peer review process is an efficient mechanism for identifying and communicating important scientific content. Other papers of mine have been rejected by these same journals, demonstrating that peer review is a worthless exercise run by incompetents whose work could be better done by a panel of trained monkeys. Perhaps more important, though, most papers I’ve ever written have been improved by the process, and I’m grudgingly grateful even when I’ve thought the reviewers were too stupid to recognize the obvious value of what I submitted. Subjectivity in science comes with the territory.
I have been a journal editorial board member asked to solicit reviewers from among a group of busy scientists who have their own work to worry about. This experience convinced me that I should have gone into dentistry, because pulling teeth would be easy by comparison – but again, I’ve been grateful to those who have volunteered their time. Also in my capacity as both a board member and a reviewer, I have seen not only what gets into print in a given journal, but what doesn’t. That comparison is essential, in my view, for understanding the relevance of peer review to scientific advance. In particular, even the most deserving manuscripts usually end up improved, sometimes strikingly so, as a result of the process. Anyone who wants to know what a world without peer review might look like should spend time looking at pre-review manuscripts. A glimpse of some of this comes from reading blogosphere articles on climate that never get published. Wait a minute – didn’t even the Skydragons make it into print?
Most of all I’ve been a reader grateful that I have enough time to sleep, eat, and do other nice things because there are only 24 hours in a day and reading everything would probably take more than 240 of those. Peer review doesn’t stop articles from being published (which is probably a good thing),but it does accomplish two important tasks. First, it identifies articles that have passed the standards of high impact journals that can afford to be discriminating, and that acts as a filtering mechanism when there isn’t time to read everything. Second, it guarantees that individuals with expertise have had a chance to suggest improvements. Let’s take it as self-evident that neither of these processes works to perfection, but by the same token, no reader is forced to forego the opportunity to read low impact journals or to second guess the conclusions in papers that passed peer review in high impact journals.
In my view, concerns about the very real imperfections of peer review are exaggerated. Science doesn’t advance because journals always make the right decisions, but because genuinely important work can be replicated and spurious advances can’t. My experience with journals and their reviewers is that most try to do a creditable job most of the time. The process has improved over the years – for example, most journals no longer use an adverse review as automatic grounds for rejection but encourage a dialog among authors, reviewers, and editors. It’s also my expectation that current high impact journals will be reluctant to abandon a process that has gained them their reputation. It is in their self interest to publish work that is both significant and replicable, and ultimately that self-interest should prevent them from steering too far off course. If it doesn’t, their lustre will diminish and will shine from other journals that do better. My guess is that this Darwinian process will play a large role in determining the future shape of peer review.
FM;
As I understand it, the Hypothes.is project is an attempt to accelerate the “Darwinian” processes you speak of. Some aspects of how they intend to accomplish what they claim are their goals are obscure to me, but weighting of referee comments by the community and some kind of internal linking and ranking logic that filters trivia and nonsense seems to be implied.
We’ll see. What there won’t be is discretionary gate-keeping.
One further: “Science doesn’t advance because journals always make the right decisions, but because genuinely important work can be replicated and spurious advances can’t.” Here is where the publication pressure to scoop “new stuff” and attract paid readers etc. does most damage. Replication, especially failures to replicate, gets little or no journal column inches, and hence little funding. As a result, bogus positives ride the waves for years, before a reality check finally kicks in.
I think that’s true for relatively inconsequential reports, but any paper that claims to be a significant advance will almost invariably inspire attempts to replicate the results. Most spurious claims of significant advances have been refuted promptly, and in almost all other cases, doubt has been cast upon them promptly even if complete refutation had to wait a while.
It depends on how revolutionary the claim is, but cold fusion, for example, was pounced upon pretty quickly and mercilessly, even though it still lives on in the semi-underground world of junk science and technology because it can’t be disproven.
Climate science is different for a long list of reasons, including the fact that often the question to be answered isn’t even clear.
Peer review is false advertising. It is better described as:
PEER REVIEW = FINDINGS UNTESTED, UNVERIFIED
The hockey stick made very significant claims. The elimination of the MWP and LIA. The paper received huge acceptance without ever being replicated, even though it directly contradicted multiple previous findings.
This points to a huge problem in science. The substitution of Peer Review in place of testing (replication), largely as a result of academia and the corrupting influence surrounding publication and tenure.
The correct scientific process was to withhold judgement on the hockey stick until such time as it had been independently replicated. Especially in light of the contradictions with previous findings. Instead the IPCC and Gore made the hockey stick their poster child.
“My guess is that this Darwinian process will play a large role in determining the future shape of peer review.”
Maybe we can start a Darwin Awards Honorable Mention list :)
Fred, I agree that the peer review process often works well. However I do think the current process tends to weed out valuable papers that contradict a large body of previously published work. It also can result in the capture of the process by a group of “elite” scientists as has happened in climate science and I think that is very bad. It can happen in other fields that the influence of money can skew the literature. I think you may tend to downplay some of the more significant failures of the present system particularly in medicine where a lot of people have been harmed and a lot of money wasted on worthless procedures and treatments. I made a more detailed post below.
If I had to point to one thing that is worst, it would be the prejudice against negative results (results that contradict the current fashionable view) in a lot of fields. You see this in climate science, for example, there is a prejudice against results that are negative with regard to the doctrine of high climate sensitivity. It is a problem in medicine, in fluid dynamics, in lots of other fields. A confluence of monetary interest, professional interest in preserving the importance of one’s own contributions, and just plain lack of honesty in evaluating ideas you don’t like can really delay progress.
David – I can agree in part. There is typically some resistance to results that appear to contradict established perspectives. On the other hand, it seems to me that just about everything gets published somewhere if it isn’t obviously delusional, and even some of the latter manages to makes its way into obscure journals – the rest appears in the blogosphere, where it will often find legions of endorsers who share the delusions. If you look hard for a negative report, you can always find one.
What this does, however, is put the onus on readers to decide how to weigh reports depending on the status of the journal that publishes them. It’s a difficult challenge, because outside of our own specialized areas, we rarely have the expertise to identify subtle flaws in what seems to be a convincing case for a conclusion. Like it or not, we are forced to use the journal’s reputation as one element in determining the confidence we can place in the conclusion. If our time is limited, that often means we will not even read the material in low impact journals.
There is no perfect solution. I think we must combine our own powers of judgment about a paper with some degree of judgment about the quality of the journal we are reading, and add to that all the other sources of information we can tap into, including individuals we respect in the blogosphere. If we do that, and understand that judgments should never be final, we are probably likely to be well informed reasonably often and misinformed relatively seldom. On the other hand, I see it as presumptuous for any of us to think that ignoring everything except our own personal assessment of a paper is a more reliable avenue to an accurate understanding.
Fred, I just finished an investigation of McShane and Wyner on the paieoclimate issue. It does seem to me that they got the better of the argument vs. Schmidt, Mann, et al. This whole 12 year controversy does I think illustrate a very big problem with peer review in climate science and that is the capture of the process by an “elite” group with an agenda. It took a statistical journal to really air the issues. It seems to me like this is VERY serious and a big abuse of the literature. You I think Fred simply ignore issues like this with platitudes about how things always work out in the end. Well, that is often true. But we should hold people to a higher standard and your attitude really justifies the lower standard.
I agree with much of Fred’s comment, because I have had comparable experiences. I would add that the peer review system is not perfect, relies on good will, defends the status quo, and can delay the publication of novel papers. I don’t have an alternative to propose other than, in controversial areas, like AGW and its presumed consequences, that critical discussion via the Internet is useful, and that papers subjected to it will be better for it.
An interesting (and very wry) post Fred.
You’ve clearly got more experience of peer review than I, but i’d say i largely agree with what you’re saying. I don’t think it’s the peer review process per say, that needs to change. More the methods of applying it:
The process needs standardising.
ALL journals should have the same standards, it’s unacceptable that they don’t.
Data/methodology should be available on submission and published along side (probably online). This includes RAW data.
Reviewers should sign off the review publicly (i.e. on the paper have a reviewed by, section).
finally, all review comments should be archived with the data.
This keeps the beneficial aspects of peer review, but helps remove some of the less beneficial (pal review, data hiding ect) in a very easy to implement way.
job done.
There can be no “objective” peer-review in a politically intensive topic like climate science. There is just too much to loose by climate scientists; talk about seeing one’s life flash before one’s eyes. This baby is going down. If Penn State made a series of conscious mistakes to preserve the University’s reputation re its football scandal, such mechanisms were in place to save Michael Mann’s bacon as well. If climate science is perceived as distorting the peer-review process to “keep on message”, it is unlikely that any assessment of research, grant process is safe. As politics has escalated the science validity and process, so the stakes the people who contribute to such escalation place themselves are at risk. Hansen, Mann, Schmidt & others risk the ruination of their reputations, livelihood, and their old age benefits. The political game is an all or nothing game. You are in or out after election day. You are in or out after a game changing scandal. High stakes.
The answer of course is to avoid elevating one’s science into the political arena. At the current state, with so many billions/trillions at stake there is no way the current group of visible climate scientists can do anything other than continue to manipulate peer-review, trumpet minuscule positives, debase people announcing glaring negatives, and hoping that they will outlast the naysayers or a conference (like Durban) or some other conference can “score a home run.” Their hope springs eternal. Ain’t going to happen. The blogosphere has peer-reviewed the peer-reviewers, and found them wanting. No one will want their name attached to a review that will subsequently be demonstrate their ineptitude and conformational bias. There are many in the climate science industry, academia, who would suffer greatly if they were found to display misconduct. What does one do with a degree in climate science when your reputation is trashed. Teach in a community college?
I believe in the peer review process except when it becomes a high stakes game. Slow and steady ultimately wins most races to tenure, grantsmanship, and productive, believable research. Don’t engage reporters. They are seductive, splashy, flashy, and fast out the door looking for the next “Watergate” expose. An academic researcher is best soft peddling their findings to whomever they encounter. And for God’s sake, don’t testify to Congress, it distorts your own self image, which in turn distorts how one reviews manuscripts, which in turn distorts the emphasis one places upon a piece of research which in turn…… When the weather is stormy, retreat to your lab and studying what is most likely.
You remind me of a Bill Clinton quote: “Politics is a blood sport!”
Can I sue Clinton for plagiarism ?
I used that phrase over 30 years ago in direct debate with a then practising politician (I don’t think he was very pleased)
” If Penn State made a series of conscious mistakes to preserve the University’s reputation re its football scandal, such mechanisms were in place to save Michael Mann’s bacon as well.”
Guilt by association?
I apologize for my typos and misspellings.
M. carey: The culture of a university begins with the central administration; what is acceptable behavior and what is not. High moral tones are announced when the stakes are low. Perceptions and reputation become paramount when there are high dollars involved. Budgetary impacts are discussed at the highest levels. Now I admit I was not at the inquiry of Michael Mann. The transcript, for what it is worth speaks to avoidance of asking important questions. Other bloggers have made the point that the inquiry was deliberate and not rigorous. Preserving the Federal dollars stream appears to be the most obvious motivation. I believe, not only Joe Pa, the AD and associates but also the President were canned precisely because of the culture of not looking very hard at serious issues. The Japanese company Olympus also had a culture of deception to hide a series of long ago poor and risky investments that had failed. The Penn State Board of Trustees made a break with the past way of doing things to get a fresh start and put a number of issues of obfuscation behind the University. It may come to pass, that the Michael Mann behavior re: Climategate will resurface. Guilt or guilt by association is still guilt.
Its standard proceedure to revisit previous investigations if those responsible for investigations commit errors.
Ask SFPD
Absolutely not. The BEST papers were pre-published WITHOUT CODE OR DATA. Without code and data, we cannot peer review them in any fashion.
How on earth you have twisted a release without code or data into an “interesting example of peer review” escapes me. You’ll have to explain to me how I could peer review the BEST papers without code or data.
It’s a serious question, Judith. I’m calling BS on your claim about BEST. Either uphold it, or correct it in your head post. You can’t just pass out untruths like that, even if (or especially because) you work for the BEST project.
Consider this to be peer review of your paper. It is unpublishable in its current form, because it contains two unsupported and untrue claims, that the BEST papers were a) capable of being peer reviewed without any accompanying code or data, and b) actually were peer reviewed without code or data.
w.
Willis, I don’t understand what you are saying. The data for the BEST analysis is available at their site (albeit some of the files in the current zip file are corrupted). The code for the central paper (the averaging method) is provided, although not in a form acceptable to some since it is in Matlab and not R.
The UHI, station siting and decadal variation papers are easy enough to understand. Whether or not code is provided for those papers (and they plan to post that code) the methods are explained clearly enough that a competent person should be able to replicate the results. Tamino has for the decadal variation paper.
I downloaded the four papers and a third of a gigabyte of data at the same time. What is Willis asking for? To incorporate the data as an appendix to one or more of the papers? That’d be something to see.
He seems to be running out of things to complain about.
Take a look at the data you so laboriously downloaded, Vaughn. The data is marked “Not for use by third parties”, which means you, so you have no data.
You also don’t have the code for anything but the data manipulations themselves.
What am I asking for?
Code and data. Why is that so hard to understand?
w.
The data is marked “Not for use by third parties”, which means you.
May I see your badge?
What am I asking for? Code and data.
That’s what I downloaded.
Vaughan you did not download the “data”, you downloaded the results of the Code. Without the original data how can you verify the output?
You set an impossibly high bar, A.C. Without J.K. Rowlings’ original manuscripts, how can I read her books?
Willis,
where were you complaining about the lack and code and data for scafetta papers? or the two skeptical papers that Judith highlighted?
If you want to beat the open drum you need to be beating it when skeptics fail to provide data and code.
steven mosher | November 13, 2011 at 12:38 am
Mosh, if you want to beat the open drum, you should do your homework first.
As one of the few skeptical bloggers to critically analyze skeptical papers, I haven’t a clue what you are talking about. Regarding the Scafetta paper I said:
and
Regarding their code, I said:
Where is the code for the spectral analysis indeed? I asked for other code as well:
and
So your claim, that somehow I gave Scafetta a free pass, is total and complete bull. Heck, my co-author Craig Loehle was a co-author on the Scafetta paper, and I still didn’t hold back.
Regarding Scafetta’s latest paper, I didn’t comment on that at all, not one word, because of my previous experience with Scafetta. You can see above the direct questions I asked him, to which I never got a single answer. Rather than play that game again with his newer paper, I just let it go.
I also just did a post very critical of another paper that is favorable to the skeptical cause because it shows much warmer Greenland temperatures in the past. I have shown that their uncertainties are way too narrow.
As a result, I totally and categorically reject your claim that I am not even-handed in these matters. I scream when anyone doesn’t provide data and code, skeptic or not, and I make every effort to provide code and data for my own work (including my discussion of Scafetta’s paper).
As to the “two skeptical papers that Judith highlighted”, I’ve never even read either one. Only so many hours in a day, and I have a day job. Are they worth reading?
w.
steven mosher, did anyone even ask the authors for code/data? I don’t mean in a comment buried amongst hundreds on a blog post either. I mean, have the authors taken any action which indicates they are unwilling to publish it? Heck, do we even know their journal/funding requires they make either available?
I’m all for people releasing data and code, but I’ve seen you compare the situation of those two papers to other situations when they seem radically different to me. That doesn’t mean you no comparison can be drawn, but you should at least mention the distinctions.
A paper is an advertisement for the science. It is not the science .The onus is upon the author to supply the code and data.
Same standards as I had for Jones.
The links in their papers to data were dead.
You cannot even begin to defend them
The increased connectivity and ‘instant reach’ of the internet is changing every aspect of our lives, both personal and professional. Facebook and Twitter were seen initailly primarily as just social but are increasingly used by business. Heck, even UK government departments have twitter feeds. “Likes” have spread into the professional sphere too – look at LinkedIn.
How is this relevant to peer review? Well imagine the pre-publication of all papers and public peer review. Interesting papers and perhaps the more controvercial ones get more people reviewing them. Imagine that both the papers and the reviews can be rated by those reading them – pressure on the reviewers to be thorough, fair and constructive. Perhaps papers that fail to generate reviews or enough interest should sink without a trace.
I like the idea of anonymity for authors as well as reviewers, however should not the reviewer names be published with the paper (without linking reviewers specifically to their review)?
As to a “qualified pool” of reviewers, that’s a difficult one. A friend at a research institute is a journal editor. The best review he ever got was from a technical manager in industry who worked in an entirely different field, but was asked to revew a paper because one aspect was relevent to the industry. The person focussed only on the treatment and understanding of the results, and was entirely constructive and supportive to the authors, despite recommending major changes. It was a case of the least “qualified” person giving the by far the best of the three reviews.
Willis: Absolutely not. The BEST papers were pre-published WITHOUT CODE OR DATA. Without code and data, we cannot peer review them in any fashion.
I downloaded the Matlab code and two sets of data, one an ascii file, and the other the Matlab data. You live in R world not Matlab world, but that does not imply that they did not release their code and data. Supposedly (this is in the second edition of Robert Muenchen’s book “R for User’s of SAS and SPSS”) there is a Matlab interpreter for R. It’s probably somewhere in CRAN.
The ascii file was too big for me to view it in my text editor (epsilon, version 11. I think I have to update), but it was there (a few hours after they acknowledged a problem they made the full data set available.)
So what exactly was it they did not provide?
MattStat | November 12, 2011 at 8:36 pm | Reply
They published four papers. I haven’t been able to find anything in the way of code for anything but their data release, but not for the papers. Where did you find code for say the UHI paper?
In addition, even if we had the code, we don’t have the data. The data they provided was marked not for use by third parties, so it is useless.
Data and code for the four papers, as I said …
w.
. The data they provided was marked not for use by third parties, so it is useless.
Steve McIntyre, Jeff Id and others have performed subsequent analysis, have they not?
I haven’t been able to find anything in the way of code for anything but their data release, but not for the papers. Where did you find code for say the UHI paper?
Ah, so the Matlab is insufficiently documented that you can’t follow the data audit trail back from each paper to the data? That is a problem.
I reread some of the programs, and it is true that you can not relate any particular program to any particular part of the report (e.g. first paper, figure 1.) I believe that is the standard to be aimed for, and I hope that authors can be required to make such available while papers are in review. For my own few papers I did indeed prepare data and programs for the reviewers (and confimed that they did indeed produce all the output in the papers), but they were never asked for.
I know that people who produce data sets and programs feel like owners (or perhaps parents), but for the good of the science, I think they should be required to provide them as a condition of accepting a paper for review.
It is unwise, I think, to be too harsh toward the BEST team on this regard because they released more than anyone else has prior to publishing, that I know of. But you are correct: what is necessary for a careful technical review they still have not provided.
MattStat:
You do not seem to have actually looked at what was provided as compared with what was promised.
Promised:
1. Raw data
2. Algorithms and code for ‘cleaning’ the data – that is correct for various data collection discrepancies and errors.
3. Corrected data for comparison.
4. Auxiliary algorithms and code for studies such as the UHI measurement.
Supplied:
% File Generated: 04-Feb-2011 13:56:55
% Dataset Collection: Berkeley Earth Merged Dataset – version 1
% Type: TAVG – Monthly
% Version: LATEST – Detrended
What was supplied was data file dated February 4th, 2011 and a notice that the the contents are preliminary and not for analytic use. Especially bad is the lack of raw data values. What is supplied is seasonally detrended. That is, it has been run through some a low pass filter to remove seasonal variations. Without the original raw data set and code, the work cannot be verified. It’s that simple.
The fact the data is seasonally detrended is especially important as nobody seems to know how they did that, and it didn’t give “normal” results. This means the “raw” data they released is generated through an unknown process.
If the discussions of Padilla et al, Muller et al, and Luedecke et al have shown anything, they have shown that someone has to monitor (i.e. reject many) the comments posted on blogs.
Who would do that? Would they get career advancement or annual acknowledgements for their time and effort?
Readers will recall that Dr. Tol and I disagreed on the issue of whether the Luedecke papers ought to have been published. Except for the common complaint that reviewers did not even understand the paper, nothing is more common than to have reviewers make conflicting recommendations like that. I doubt that open reviewing with many commentators will improve that part of the review process.
I like the idea that the reviewed papers should be as anonymous as the reviewers. What would improve much review would be if the reviewers had access to code and data. That would require authors to recheck their code and data to remove identifiers.
I can foresee parallel systems of all kinds in place for a while as the kinks of online public reviewing are worked out.
Willis Eschenbach is incapable of communicating in a polite manner. I am sure you’d never use it Prof Curry, but the phrase ‘Suck your own —-‘ is tailor-made for him.
Willis Eschenbach is incapable of communicating in a polite manner.
That is uncalled for. He’s just terse and judgmental.
You, on the other hand, totally wasted our time and attention with junk.
WB | November 12, 2011 at 8:55 pm | Reply
People making certain that they are polite rather than making certain that they are being honest are part of the problem with climate science. I’m sorry if my bluntness upsets you, but I’m not going to blow in Dr. Muller’s ear and rub his stomach.
I will, however, tell the truth as I see it. If only more climate scientists were willing to do that, I wouldn’t have to be so insistent and I likely wouldn’t be so aggro about it.
I do notice, WB, that you have not commented on the scientific issues I raised. Instead, you want to discuss my communication manner … a response I find all too common. Lots of folks like you are more than willing to complain about my style, because that way you don’t have to talk about the substance of the science.
w.
PS—I haven’t a clue what the missing word is in “Suck your own ____”. “Candy Cane“? “Exhaust”? “Lollipop”? “Johnson”? “Elbow”?
I’m reminded of the phrase in “Lord of the Flies”, “Sucks to your asthmer”, but I doubt that is your meaning.
In any case, whatever the word might be that goes in the blank, I don’t know what it is, so I fear you’ll have to suck it yourself. I’d do it, but I don’t understand what you’re referring to, so perhaps if you’d demonstrate it first I could follow your lead …
Rattus Norvegicus | November 12, 2011 at 8:59 pm | Reply
Right, but the data is not provided, and the code is not provided for the other three papers.
Nonsense. Nobody has figured out their UHI paper. Nor should they have to try. Muller was supposed to be uber-transparent. Instead, he has hidden his code and data as well as Mann ever did.
And I am unclear how Tamino has “replicated the results” without data. The data set that they released was clearly marked:
So whatever Tamino has done is meaningless, since he’s not using the dataset that they used, he’s a “third party” employing a dataset clearly marked
THIRD PARTIES DO NOT USE
Maybe that’s peer review on your planet. On mine, trying to do anything when the developers of the dataset have enjoined me from using their data is not peer review … how could it be?
Finally, the dataset they released was not even the raw data. It was processed data, done by an undocumented process, and containing demonstrable errors in the data.
And you think this pile of half-baked data and unbaked papers is ready for peer review?
w.
Willis,
Ever feel you put a nuclear device in the heart of science and it is just a matter of time before detonation?
http://jonova.s3.amazonaws.com/guest/lalonde-joe/world-calculations.pdf
http://jonova.s3.amazonaws.com/guest/lalonde-joe/world-calculations-2.pdf
Willis, could be said that it’s now shown nicely how Rattus talks through his hattus.
you clearly dont understand what that marking means.
it means. If you are going to use this data to do your own independed work, then stop. you better wait for verion 2. IF you want to check our work, then this is the data we used in our work.
Judith,
The Peer-Review process forced me to do an alternative and that was to e-mail and post far and wide.
Every scientist is absolutely silent.
But the activists that are more motivated are stirring.
By the way, that offer is still open.
Having participated in the peer review process extensively on both sides, author and referee, I believe that the process provides some value, particularly in an atmosphere where publish or perish results in a lot of manuscript submissions of low value. I believe the most serious problems relate to this “monetization” of the peer reviewed literature. Pay, promotion, and even retention in many settings are tied to the peer review process. This leads to attempts to game the system to try to inflate ones publication record or its influence. Some of the other problems I see:
1. Proliferation of journals.
2. Increasing specialization of journals and the tendency to rely on “experts” in a narrow field for manuscript review. Most papers in climate science for example should be reviewed by an independent statistician from some other field of specialization. Same goes in medicine. Climate modeling papers should be reviewed by computational scientists in other fields.
3. Personal or even monetary interests of the reviewers can destroy objectivity. I’ve seen this a couple of times prevent publication of very good papers.
4. Very interesting papers that are not fundamentally new in one field but that illustrate critical application in another field where the result is not know can be very valuable. I’ve seen some excellent papers like this rejected (not my papers incidently).
5. Very slow review process. Senior people like myself and others I know often decline to review manuscripts that will take substantial time. This process unfortunately tends to weed out the more conscientious reviewers who feel an obligation to do a thorough job. Less conscientious or biased reviewers can accept the job with no qualms or regrets.
6. There is I think a prejudice against manuscripts that seem to contradict large bodies of already published work. In my view these manuscripts should receive priority for publication. Merely reiterating the prevailing views usually offers little added value.
7. It is always unhealthy in any field to have a small group of recognized “elite” people who can capture the peer review process. This is relatively uncommon, but in climate science I think this is the case. I don’t know what to do about it though other than to try to help dissenting voices get a hearing.
Now that I’ve written this, I actually think some of the latter points are more important, such as 5 and 6. I actually like the idea of open review of manuscripts and/or blog review. Particularly for controversial manuscripts, this leads to very detailed examination often by people outside the field. It’s not an ideal solution, but experimentation with it will determine how to make it work well.
What if a piece of work cannot be disputed by the sheer nature of simple mathematical equations. Yet it does have a tremendous impact on an extremely wide range of areas?
Scientists would view it as a threat as it could undermine their whole careers.
So rather than embracing new science they try to ignore it hoping that “Pandora’s box” was sealed.
http://jonova.s3.amazonaws.com/guest/lalonde-joe/world-calculations.pdf
http://jonova.s3.amazonaws.com/guest/lalonde-joe/world-calculations-2.pdf
@David Young. 5. Very slow review process.
My experience as a junior faculty member at MIT in computer science in the 1970s was very different, for the simple reason that I blew off journals (as well as department committees), mainly publishing in conferences where the deadlines are strict to within days (these days hours). I was warned not to, but I was finding that my conference publications were having their intended impact and I was unimpressed by the promised return of high archival value on the even higher investment of polishing a conference paper that had already achieved its purpose. This risky strategy turned out to work (i.e. I got tenure at MIT and a couple of years later a job offer of full professor at Stanford), in part because of a growing recognition in the 1970s of the role played by flagship conferences in fast dissemination of research in a rapidly growing field. YMMV.
6. There is I think a prejudice against manuscripts that seem to contradict large bodies of already published work.
I have far too few datapoints of my own to comment on this in climate science. On the one hand bloggers like Michael Tobin, Gavin Schmidt, and Grant Foster (Tamino) have all had difficulty with my ideas, whereas I have had just as much difficulty understanding their objections. On the other hand I’ve had what I would consider more “normal” scientific discussions with Jeff Kiehl, Richard Muller, and Mark Jacobson, and to a lesser extent our host Judith Curry. And I’ve had my first attempt at submitting to a geophysics conference accepted.
On that very limited basis I can only say that I’ve encountered more prejudice from the blogosphere, which on both sides of the climate debate gives the appearance of dueling religious zealots, than from scientists seemingly more curious about nature herself than how the blogosphere views her.
7. It is always unhealthy in any field to have a small group of recognized “elite” people who can capture the peer review process.
It is always unhealthy to kick them off the program committee or editorial board, which should leaven the wisdom of its seniors with the vision of its juniors. Wise visionaries are scarcer than hen’s teeth, which is why you need both types.
The peer review process needs to move into the 21st century and use the blogosphere.
The peer review process needs to move into the 21st century and use the blogosphere.
The blogosphere would be enormously improved by borrowing the 20th century’s per review process, suitably adapted to the former’s much shorter time constants.
I reserved a couple of URL’s for a realization of this that’s on hold pending getting several other projects to a satisfactory point.
A friend of mine was the first to chemist to produce pure peroxynitrite and his extinction coefficient is still used. When he submitted his original manuscript the first comment from referee number 1 was :-
“Everyone knows that there is no such thing as peroxynitrite”
I had to rewrite an introduction once as one of the referees insisted I remove a number of references, not references to my own work, but another group.
A friend of mine was the first to chemist to produce pure peroxynitrite and his extinction coefficient is still used.
Beckman 1996?
Martin Hughes, in
Hughes, M. N. & Nicklin (1968) J. Chem. Soc. A 450–452.
But the Hughes-Nicklin paper called it “pernitrite,” with no mention of “peroxynitrite.” So referee number 1 must have said “Everyone knows that there is no such thing as pernitrite.”
Whereas Google returns 530,000 hits on peroxynitrite, it returns a mere 3,242 hits on pernitrite. I don’t know enough chemistry to judge, but is it possible referee number 1 had a valid point? Has there ever been a time when pernitrite “existed” in the sense of being the commonly accepted name? Would the referee have taken the paper more seriously if it had called the anion peroxynitrite?
After reading more about pernitrous acid (intriguing history!) I concluded the referee was far more likely to be referring to the instability of the peroxynitrite anion, whose half-life as pernitrous acid in aqueous solution is on the order of seconds, than to any disagreement over nomenclature (presumably the referee was ok with H&N calling the peroxynitrite anion “pernitrite”).
As long ago as 1816 Thomson and Guy-Lussac were debating whether what Guy-Lussac was calling pernitrous acid was distinct from nitrous acid or merely the latter in vapour form as Thomson believed. Yagil and Anbar’s 1964 paper “Formation of peroxynitrite by oxidation of chloramine, hydroxylamine, and nitrohydroxamate” makes clear that at least some people were calling the anion “peroxynitrite” well before H&N called it pernitrite (the British term?).
Y&A obtained a value of 1.3 mM⁻¹ cm⁻¹ for the extinction coefficient of the anion at 302 nm, the (wavelength) peak of its broad (over 100-nm-wide) UV absorbance spectrum. (This is broad enough to absorb some visible blue, accounting for the yellow colour of the anion, yellow being the complement of blue). However they had a very large error bar due to inability to stabilize the anion.
Presumably the conventional wisdom at the time was that the anion did not have any stable form. Easily stated hypotheses that have been around long enough have an interesting habit of evolving into indisputable facts, which is my current best guess as to where the referee must have been coming from.
What H&N accomplished four years after Y&A was to stabilize peroxynitrite as sodium peroxynitrite, sufficiently for it to take months to isomerize. Once stabilized in this way they were able to refine Y&A’s 1.3 figure considerably, namely to 1.67(5) mM⁻¹ cm⁻¹. This much more precise estimate remained in common use as you say, for over quarter of a century.
In 1996 Bohle et al used a very elaborate method to produce tetramethylammonium peroxynitrite, [N(CH3)4][ONOO], as a pure bright yellow salt very stable in water. This enabled them to measure its extinction coefficient more precisely as 1.705 mM⁻¹ cm⁻¹, superseding H&N’s 1.67 figure.
I didn’t run across any more recent values, though in the process I did learn a lot of chemistry. (I was going to take sophomore chemistry until my girlfriend talked me into taking the statistics honours track instead, a subject I’d never heard of at the time, and I never looked back.)
One other thing about the requirement to publish code and data. This is a good thing for public employees, but for industrial scientists, its often just not possible. If the code and data is valuable, the most you can hope for is to publish the methods and results of some of your computations. To a priori exclude these people from the literature would be a great loss.
“The scientific enterprise is all about building on the results of others – but this can’t be done if the results of others are languishing in the hands of reviewers, or suffering through multiple rounds of peer review. There can be little doubt that this delay slows down scientific discovery […]”
Pretty hard to disagree with that.
I can.
For the sheer fact that what if the area’s of science were incorrect in the first place?
For the sheer fact that what if the area’s of science were incorrect in the first place?
True, except you didn’t say who would judge that. The Tea Party?
It may come down to the students themselves choosing facts or fiction learning.
Yes, the problem is with the education system (particularly functional numeracy) and it will take more than a human lifetime – perhaps a millennium – to solve (provided we maintain sufficient grip on enlightenment).
Joe Lalonde | November 12, 2011 at 10:28 pm | Reply
I can. For the sheer fact that what if the area’s of science were incorrect in the first place?
Peer review does not determine if the science is correct, only if the methods are correct. eg: the data adds up correctly.
Replication (testing) is the only valid proof for the correctness of science.
Only 1 Earth, so pseudoreplication tempts assumers.
Judith,
The only real thoughtful review of any of the BEST papers that I have seen is this two parter on the decadal variations paper from Tamino:
http://tamino.wordpress.com/2011/10/24/decadal-variations-and-amo-part-i/
http://tamino.wordpress.com/2011/10/28/decadal-variations-and-amo-part-ii/
Where he shows that the claims don’t really hold up. I certainly haven’t seen any good analysis of the key paper on the BEST averaging method. Have you?
The only real thoughtful review of any of the BEST papers that I have seen is this two parter on the decadal variations paper from Tamino:
Tamino is a self-confessed statistician. The principal difference between statistics and logic is that the latter acknowledges the notion of an inconsistent theory. Ever encountered the saying, “There are lies, damned lies, and inferences”?
One can prove anything with statistics.
Yes, but this is an essentially statistical problem, therefore a statistician might have something worthwhile to say.
Yes, but this is an essentially statistical problem, therefore a statistician might have something worthwhile to say.
Don’t get me wrong, I would be thrilled if Tamino had something worthwhile to say about BEST’s analysis.
But your argument is like saying that the complexity of factorization is an essentially computational problem, therefore Stephen Wolfram might have something worthwhile to say. I don’t see anything worthwhile on factorization in the 1200 pages in ANKS, what does he know about that?
Tamino’s career experience in statistics amounts to deciding whether a star is variable. It remains to be seen whether that experience transfers to a meaningful critique of the BEST analysis. It would be fantastic if it did. What are the odds?
Yes but he does have a couple of papers out there with over 100 citations. And what is variable star analysis if not time series analysis? He seems to be relatively competent in this area.
he does have a couple of papers out there with over 100 citations.
Is that more or less than the number of citations of either the Miskolczi or Gerlich and Tscheuschner papers? (Afraid I’m not very good at keeping score.) And did he have coauthors?
Besides the obvious global warming in the various temperature records such as HADCRUT3 and GISTEMP, there are also a number of very interesting oscillations at various frequencies, some of which I posted on my website a while ago.
Someone drew Tamino’s attention to them, and he lambasted them at his website. A month later Willard drew my attention to his comments. Never having heard of Tamino or his website before, and not understanding the criticisms, I posted a request for clarification. Tamino declined to engage in any debate, saying that these oscillations were the result of variations in volcanism and that I was an immature child to complain about his criticisms.
Needless to say, this did not endear me to Tamino, and prompted me to wonder whether he was compensating for something.
By far the best way to rehabilitate my impression of Tamino would be to say in one sentence each what he did in those two papers that was worthwhile.
Tamino has some critically weak areas. I once challenged him where his reasoning was fundamentally flawed (false assumptions about variance partitioning in decompositions). First he started snipping parts of my comments to make the remainder appear out of context. Within 24 hours he outright banned me from his site. Based on the comments that offended him, I suspect he would be unwilling to hear any serious challenges from Tomas Milanovic. My impression of Tamino is that (much like Leif Svalgaard) he’s there to tell innocents what to think without willingness &/or ability to learn from those with superior grasp in specific key areas. Can we afford to have a societal structure where those with decent but dull vision & crucial blindspots are allowed to harassingly assert unconditional “scientific” leadership just because they have a lot of knowledge &/or technical skills camouflaging their catastrophic philosophical weaknesses (such as patently untenable assumptions) in the eyes of innocents? It’s a dangerous cocktail, but we do live in a society that values intoxication, so we can sensibly assume people will be drinking.
MattStat | November 12, 2011 at 9:30 pm | Reply
MattStat, if Muller hadn’t made such a big deal out of saying that he was going to be transparent, and if he hadn’t gone to the media before peer (or indeed any) review, I would be less harsh.
But he did claim that he would be transparent, and he did engage in “science by press release” and “science by interview” … so I fear he’s used up the good will he started with.
Indeed they have not, which is why I objected so strenuously to our good hostess saying that there have been two …
Judith knows, or certainly should know, that there has not been any “extended peer review”, or indeed any peer review, of the BEST papers.
How could there be? The data was marked “THIRD PARTIES DO NOT USE”, and last time I looked, everyone here (except Judith) is a third party.
w.
PS—Another reason for the strength of my objection is that Judith was putting the putative peer review (which never happened) out there as though it were a known fact. I have seen this technique used over and over by AGW alarmists, where they take something which is either in contention or totally untrue and refer to it casually as though it were an established fact.
So when Judith did it (likely unintentionally, since it appears she truly was fooled into believing that “extended peer review” had actually occurred with the BEST pre-data and pre-prints), I responded more strongly than I should have.
So … sue me for over-reaction, but while you’re at it, please notice that Judith’s claim was not true, there’s been no peer review of BEST of any kind.
So … sue me for over-reaction, but while you’re at it, please notice that Judith’s claim was not true, there’s been no peer review of BEST of any kind.
I think a jury of your peers would hang, and then we’d have to start over.
Willis,
Is that what you are referring to?
Doesn’t seem like they are putting the restrictions on the data that you claim.
Rattus Norvegicus | November 12, 2011 at 11:10 pm
They say they do not recommend that I (a third party) use their data because there are “known bugs”. Some of the bugs give temperatures 50° in error, so they’re not minor bugs.
In addition, it is not raw data, but has been run through some unknown algorithm. This algorithm appears to be the source of some of the buggy results.
Call me crazy, but I am taking them at their word and not using their data because a) it is buggy and b) contains big known errors and c) has been munged by some unknown algorithm.
If you think buggy, munged data with large errors and a direct recommendation from the authors for us not to use the data does NOT contain significant “restrictions”, I don’t know what to say.
w.
Willis,
There is a huge difference between restrictions on use and caveats about the content.
The files flags.txt and sources.txt seem to contain a traceable record of the source for each data point, although these files are corrupt in the zip that I downloaded a couple of days ago while investigating Steve’s claims about the Longmont, CO station data). Unfortunately, I was unable to continue my investigation because these file were corrupt in the file I downloaded.
Rattus Norvegicus, I’ve heard similar complaints elsewhere. Do you happen to know how corrupted the files were? It often happens that corrupted files contain all the original material, but cannot be opened due to a small error (such a structural error). If that’s the case here, it may be relatively easy to extract the data.
I would try it myself, but I never did download the zip file (even though I have been in “civilization” quite a few times). I found a link to the files for the graphs I was interested in, so I didn’t need the entire thing.
Anyway, if they’re in simple file types, and the corruption is something minor, it should be relatively easy to get the data out.
Willis, it was a bug.
Brandon, the files were extracted successfully with 7-Zip, but contain data consisting almost entirely of zeros.
Ah. That doesn’t sound like recoverable then. I wonder how it happened.
Rattus Norvegicus | November 13, 2011 at 12:33 am | Reply
So was it a restriction or a caveat that stopped you?
In any case, the BEST folks said it was not recommended for use. Let me repeat their statement real slow.
THIS … RELEASE … IS … NOT … RECOMMENDED … FOR … THIRD … PARTY … RESEARCH … USE.
Now, call me crazy, but I’d say that was neither a restriction on its use, nor is it a caveat.
it is a direct recommendation not to use the data. Not a restriction or a caveat on its use, but a clear statement saying don’t use it at all.
So I fear I don’t understand your hairsplitting in the face of their quite clear recommendation.
w.
In addition, it is not raw data, but has been run through some unknown algorithm. This algorithm appears to be the source of some of the buggy results.
Call me crazy, but I am taking them at their word and not using their data because a) it is buggy and b) contains big known errors and c) has been munged by some unknown algorithm.
The algorithm seems to be well-described by the Matlab code. I have to say “seems” because I may be mistaken. Maybe someone has discovered mistakes.
I think the BEST authors’ disclaimer is like the disclaimer you get with software: that it is distributed “as is”, and is not warranted, even for its intended use. Certainly R is not warranted, and if you want to use it for commercial purposes you need to license Revolution R. Every large data set has errors, and every large data set will continue to have errors. When BEST have refined their data, and produce version 2, it also will be distributed with the disclaimer that it has errors.
You can not tell exactly which code produced which table, but you can track a lot of their work.
The BEST team have been the most open of all teams trying to create a composite summary of the temperature record. Maybe during the review process they’ll think to provide more information relating all the programs to all the results in the papers. They have received a lot of suggestions about how to improve their work, and maybe they’l respond positively to this suggestion.
MattStat, what leaves me baffled is why did they release seasonally adjusted data instead of raw data? It seems highly peculiar to release adjusted data rather than raw data with the code to do the adjustment. Even worse, by my understanding, they didn’t release any code on their seasonal adjustment method. This makes it practically impossible to figure out/verify what they did. That wouldn’t be so bad, except the data they published is not what you’d get with any of the “standard” seasonal adjustments made in temperature records.
I don’t mind the idea of them releasing a rough data set like this, but they need to offer more information about it if it’s all they’re going to release for a while.
(By the way, their UHI paper has a completely bogus reference for a claim about an analysis of their whole data set. Their website’s FAQ has at least a couple statements which are, to be charitable, misleading. It all just seems really sloppy.)
Brandon: Even worse, by my understanding, they didn’t release any code on their seasonal adjustment method. This makes it practically impossible to figure out/verify what they did.
They supply the statements that read the raw data sets that they managed. As far as I can tell without more study of Matlab, they the code that they released shows a great deal of how they performed all of their adjustment.
Then maybe there is code provided for the seasonal adjustment. If so, someone should look at it and see just what they did.
I have to say though, it seems weird they would provide the code they used to read the raw data in, yet provide adjusted data. Unless they adjusted each data set as they grabbed it, I can’t think of a reason for it.
Some of the bugs give temperatures 50° in error, so they’re not minor bugs.
By that reasoning a divide-by-zero error would be a really huge bug! If you’ve never committed one of those yourself you’re either extremely competent or extremely inexperienced.
Numerical instability can easily creep into code written by those inexperienced in its management. This was a cornerstone of George Forsythe’s rationale in the 1960s for teaching Numerical Analysis as a subject, before the subject turned its attention to backwards error analysis, QR-factorization, singular value decomposition, and beyond.
BEST release ? Mosher says you do like him and practice with it – like with kissing a blow up-doll. It’s good for that kind of thing, apparently.
The data is unforntunately not set up for amateur hour.
What is the “data” set up for exactly?
Are you referring to the file called “data”?
Beta release. for people of good will and great skill.
not whiners
and foot pump
It shouldn’t be beyond human ingenuity to devise a system where those submitting papers are anonymous unless and until they are accepted. Protects to some extent against irrational biases based on the identity of author or institution.
How many scientific papers stand the test of time? I suspect the vast majority are pretty much worthless. Maybe too many papers are being written, more for the sake of, than the need for, it. Quality, not quantity as they taught me at school.
Not sure how this issue could be addressed. I suppose it goes back, in part, to funding proposals. It’s all linked, from inception of research to final publication. Not sure you can look at peer review/publication in isolation, because it’s an interdependent system.
Maybe we need to go back to examination, from first principles, what we are trying to achieve in scientific research, and engage in some blue-sky research on that!
When AGWists refer to peer review, it seems somehow to come down to “self-review”. Muller’s perhaps one of the BEST examples to date.
Don’t forget Richard Lindzen. Credit where credit’s due.
My opinion. The basic problem with traditional peer review at journals is N (less than or equal to 3). No convincing empirical conclusion is based on N <= 3. Even professors are noisy measuring instruments; even professors have fixed effects different from the population mean. In my field, the number of citations of papers admitted to most of the top journals is extremely skewed: There are WAY too many of those papers whose 10-year citation counts are 3, 2, 1 or 0, in most of the selective journals.
The ultimate peer review is citation, not journal membership. More and more, I see working papers (posted to the websites of the authors) that have more citations–even before publication–than do most papers in the top journals. Yes the top journals.
The whole journal system is, in my own opinion, completely obsolete. It is like many things in academia, a relic of some medieval way of thinking or mode of technology. If I want to know how successful some young scholar is, I find out much more by going to several online resources where I can see how often he or she is being cited and by who. Lines on the vita are almost completely uninteresting relative to that, especially given the inability of the top journals to REALLY select papers with long-run staying power. At least, that is the sad state of the top peer-reviewed journals in my field. Sure, they get a larger SHARE of the papers with staying power; but frankly the SIZE of that share is too small to make their reputations worth my time. Far better, frankly, to visit google scholar (for crying out loud…and there are much better online resources if you want to get picky).
This opinion has nothing to do with climate science. I say this regularly to people in my own field and frankly there is never a very good counterargument. Citations can be gamed… sure. But that doesn't mean the game-theoretic equilibrium of a citation game is going to be less informative than the N=3 samples at journals.
Death to the journals. Long live the personal webpages and citation engines.
Not far from own view. Very close, perhaps. Maybe even identical.
“I say this regularly to people in my own field and frankly there is never a very good counterargument.”
I suspect that you haven’t listened to the counterarguments. There is a very simple one, which is that peer review performs a critical role in elevating the quality of publications before they are broadcast to the public. it is not perfect, but it results in much better than the quality of blog based publications and other forms of personal webpages.
Now I do agree with you that the highest tier publications (PNAS, Science, Nature) are often poor predictors of research quality, but there an enormous number of mid-tier journals which are publishing good quality research, mostly.
Another major contribution of journal articles is that they constitute mini end-goals for the research process and this structure is critical for pushing scientists to excel, and for providing consolidated repositories of work and data. What we are just beginning to see is the requirement of publishing data online along with journal publications in a durable form, and that is going to be a phenomenal development.
“I suspect that you haven’t listened to the counterarguments. There is a very simple one, which is that peer review performs a critical role in elevating the quality of publications before they are broadcast to the public. it is not perfect, but it results in much better…”
Oh come on. How do you know this? I know an editor in my field–a top guy–who took over a journal several years ago. He is convinced that almost all referee reports are almost worthless and amount to referees attempting to rewrite papers. He has instituted a form of review that authors can request: Up or down. In this form of reviewing, the referee can make comments BUT they cannot request revisions. They must make the up-or-down judgment: This paper is worth publishing as is, or reject. This editor believes that this will attract good work in the long run, from people who are tired of the comedy of manners that is contemporary peer review.
“Oh come on. How do you know this? ”
I have been publishing articles in academic journals for well over 10 years. In every article that I have published, the reviewers have caught oversights/errors, or suggested interesting new interpretations.
Furthermore, the revision process always caused me to catch my own mistakes, and I would not have executed this extra level of review had the revision process not required me to.
With all due respect to your friend who is running a journal the up/down process sounds like a bad idea. All work benefits from revision.
I suppose it’s possible that things are fundamentally different in my field than your friends’ field. In Psychology we typically get three reviews per submission, at least one of which is thoughtful and detailed. These reviews rarely try to railroad the paper into the reviewer’s preferred interpretation. In fact, I just suggested resubmission of a paper that was in direct support of my own work because the quality wasn’t high enough.
Then let someone else write a comment and put it up on their own website. It’s an opportunity for someone else to show how smart they are. If there are problems with a paper, you can be sure someone else will exploit them…if of course the paper is worth arguing with in the first place.
There is no need here for a centralized process. No need at all. Commentary and improvement can be a self-organizing process; improvement can be an emergent property of spontaneous order. This is how scholarship works anyway, even with reviewers. Why not cut out the middlemen entirely? Elsevier and Springer and the other robber barons of the publishing world are perfectly happy to act as scholarly intermediaries in a world that no longer needs their intermediation. Referees could instead publish comments on papers they actually care about, rather than the (mostly uninteresting) papers that get shoved down their throat by some editor. I say free the referees and editors too, and good riddance to Elsevier and Springer. Let those guys go pound sand with the rest of us.
“If there are problems with a paper, you can be sure someone else will exploit them…if of course the paper is worth arguing with in the first place.”
That’s not necessarily true. Given the enormous amount of papers which are published, most of them would probably simply be ignored, and thus never receive the possibility to be improved by revision.
“There is no need here for a centralized process. No need at all. ”
There’s not a centralized process in the current system. There are multiple journals, they compete with each other, and new forms of publications (open access journals) are free to arise and compete with them.
I fully acknowledge that the current system is imperfect, and I was griping about the academic publishing industry tonight myself. But that doesn’t make it useless, just in need of repair, and you’d see that repair is in progress if you paid closer attention to the changes that are occurring already. For example check out the “Frontiers in” online journals.
I am firmly on both Brad and NW’s side. Brad is absolutely right that expert feedback from people that haven’t been deeply involved in the research can pick up errors and improve the presentation of the paper. NW is absolutely right that the original authors don’t need to do this, “improvement can be an emergent property of spontaneous order.”
There is nothing to disagree about here.
Brad, maybe authors would be more careful in the first place if they couldn’t depend on referees. Awhile back, someone suggested to me that some people write careless papers because they know that referees are going to take them through revisions anyway, and if it gets rejected… well as you say there’s more than one journal. So it isn’t transparently true that the current system produces better papers. Maybe it encourages people to be less careful in the first place.
Personally, I spend a lot of time on referee reports. I don’t know why: It’s some irrational compulsion I have. I get no credit for it at all. I would much rather write my devastating critique of Curly, Larry and Mo as a comment and put it up on my website. This accomplishes the same thing, and Curly, Larry and Mo don’t get credit for thinking of that mixture model I suggested–I do. I think I like that a lot better. And maybe Curly, Larry and Mo wouldn’t have sent out (or posted up) such a shoddy piece of work in the first place, if they knew that Gumby and Pokey were waiting out there for them.
The current social institutions in empirical sciences do some things. This does not imply that a different, less centralized institution wouldn’t accomplish all that and more. The internet changes everything in this regard. Journals are an expensive organizational model, and perhaps they made sense in a pre-internet world. But they no longer do in my opinion.
I hardly read journals anymore. The only reason I read them is because some poor author was REQUIRED to remove her working paper from her website as a precondition of publication. In other words I read them because the journal has monopoly rights over the paper once it is published. I do not find out about her paper by reading it in the journal. Rather I see it cited in someone else’s paper, or hear about it by word of mouth or at a conference or workshop. Or I do a web search and find it on the author’s webpage. Journals… worthless. Uninteresting. Redundant. Monopolies.
NW, what’s been your experience with how often the Wayback Machine has retroactively blocked access to a formerly available prepublication version?
Re Brad, hopefully he’ll converge to your position in due course.
For comic relief purposes, the (very old) paper I stole Curly, Larry and Mo, and Gumby and Pokey, from (see the bibliography):
http://www.radford.edu/~ibarland/Public/Humor/childhood
Vaughan, I’ve never tried to get an old pdf from the Wayback. I should try! I’m lucky to have contemporary online access to most journals through my institution, but not always.
This illustrates a point often overlooked: there is much variability in editorial practice, between journals within fields, and within each journal over time. Not many journals practice literally the same review process as 40 years ago when I started (actually, my first paper with me as senior author was only 30 years ago.) Furthermore, new journals have been created when there was sufficient complaint about extant journals: in climate Science, we now have E&E. Journal editors, and the members and officers of professional societies are always reviewing and upgrading the review process.
Your friend’s initiative may prove to be an improvement, or it may fail. There is trial and error here as everywhere.
Academic standards change as well: I was instructed by my thesis advisor that from now on the department would require that the computer code be included in the thesis so it could be reviewed; there was debate about whether mine would be the first thesis for which this rule would be enforced, and I prepared an appendix listing all the programs that were used to generate all the tables and graphs. I think that now they require that data be put in a permanent repository. My data were saved on tape, but it was not a requirement for the degree. Now a few journals have adopted these two policies as firm requirements for publication.
I think on the whole we are trailing the discussion here, not leading it.
I agree.
I should mention that each paper of mine and my coauthors that we submitted was improved by the review process. Sometimes it took much work, and sometimes little, but the result was always better. With the exception (already mentioned by me, Brad and others, of posting complete data and code during the review process, it might be that there is not a lot of room for improvement. Or, what amounts to the same thing, the total work load of everyone involved in the review process might double with a negligible improvement in return.
Papers have to be reviewed, no doubt – the review is to ensure that the journal prints the best papers on the market……
But how……? Peer review….. another word: Pal review?
Lets look into the great classic thinkers: Aristotle, Hobbes, Locke and the best: Adam Smith: He calls (Treatise on moral sentiments) for
review and judgement for the “Impartial Spectator”.
Climate science is politicized: Two teams, two sides, the Warmlers vs. the Skeptics…….The Warmlers control the journal gates (since the 90th CO2 hype, and placed their guys at the gates…..) and the Skeptics have a hard time getting in, see Lindzen, after 2 years somewhere in Korea……
(1) Were is the Impartial Spectator ? The Warmler team provides the referees!
(2) Its unimportant, whether reviewers are anonymous are not, they come from one side…… impartial judges according to Adam Smith?
(3) To insure quality of papers………: the IPCC warmlers base their
papers on a so-called: IPCC “very likely probability”, see ar4-wg1……
Missing in this game is the “real global warming mechanism”, which
has nothing to do with AWG “likely”-probability……. and the journals take part in this guesswork…….
Once the real climate change mechanism is more and more accepted, then journals will realize, that they supported the low quality side “science”, the looser team of history and of quality in science…..
JS
General:– Away with them, and place them at the bar!
Enter RUTH. Ruth:–
One moment! let me tell you who they are.
They are no members of the common throng;
They are all noblemen who have gone wrong.
Girls:– They are all noblemen who have gone wrong.
General:–
No Englishman unmoved that statement hears,
Because, with all our faults, we love our House of Peers.
I pray you, pardon me, ex-Pirate King!
Peers will be peers, and youth will have its fling.
Resume your ranks and legislative duties,
And take my daughters, all of whom are beauties.
(I was the e-PK in a 1966 performance at Sydney U, and still free-associate with any mention of “peer review”..)
“You may put ’em on the list — you may put ’em on the list;
And they’ll none of ’em be missed — they’ll none of ’em be missed!”
Amen to that.
steven mosher | November 13, 2011 at 12:43 am | Reply
you clearly dont understand what that marking means.
it means. If you are going to use this data to do your own independed work, then stop. you better wait for verion 2. IF you want to check our work, then this is the data we used in our work.
Mosh, here is their statement:
Here is Mosh’s translation:
First, 1) nowhere does it say that they used that data, and b) that data contains some horrendous errors, sixty degrees or more, and c) the data is not the raw data.
So what makes you think that is the data they used?
Second, they say that their are “known bugs” in the data. However, they have not identified those known bugs. I don’t know what they are.
So perhaps you could explain to us just exactly what those “known bugs” are, and how we should avoid them if we are trying to replicate their results.
Third, they have not identified the data used in their rural vs. urban breakdown. Again, without that we’re out of luck as far a replication.
So no. They have advised us that there are known bugs that could screw up any results. I have no idea if they even know what those bugs are, but I’m sure I don’t. Given their general lack of quality control (e.g. mindlessly repeating known bad station locations) I have no faith that they have even identified the bugs … and without the raw data there is no chance at all of identifying them.
So you can read into their statement about known bugs that it really means that the data is fine for Mosher to use.
Me, I see no point in dicking around with their buggy dataset, it can never prove anything. In addition, they can disavow any results that I might get, simply saying “we told you not to use it”.
I find this releasing, not of the raw data itself (which they easily could have done), but of a munged dataset that is known to be buggy, with no quality control, huge errors in some unknown number of records, half processed by an unknown algorithm, and accompanied by a vague warning not to use the dataset, to be a reprehensible parody of scientific practice.
w.
Willis (quoting BEST): “IF you want to check our work, then this is the data we used in our work.” First, nowhere does it say that they used that data,
You practicing law without a license again, Willis?
I must be tired. Their != their != they’re. I do know the difference.
Thanks,
w.
Not to mention there …
w.
where? ;-)
When was the last time all 10 recent comments were from the same thread? Do threads with obfuscated four-letter words draw more comments? Experiments are in order.
It actually happens somewhat regularly, especially when there’s a new blog post (and almost always when there is a new blog post after days of not having any new posts).
That said, I do think this case is a bit different, but that’s mostly because the interest in the Ludecke et al papers has (largely) died out even though nothing was really resolved. I think it was one of those topics where a lot of people didn’t want to listen to the “other side,” so after an initial flurry of activity, there was nothing more to be said since no discussions were actually happening.
I agree with Willis on this issue. As a software guy, there are few things more irritating than devoting time to track down an issue, only to be told later “Oh, that’s not the we use. So we don’t have the problem.” So, until BEST says “here is the data we use, and here is our code for the data we used to get the results in our paper”, trying to do any replication is an exercise in futility, given the uncertainly involved.
Also, another way to look at BEST’s disclaimer on the data is that even if you COULD replicate the results, the disclaimer says your results could be an erroneous conclusion. What is the point of using data that can’t be used to draw conclusions from? I’m not exactly certain what the use of data in an unknown state is. I suppose if they were to dump the data in XML format, then something like this might be useful, but reading flat data isn’t that hard.
But this does point out one other aspect about the review process (remember Dr. Curry’s original topic? :) ). I think part of the breakdown in the system is that somewhere along the line, (some) science lost what made science work – that is the ability for others to duplicate the work. Without the ability to duplicate that work, you have to introduce ‘trust’ in the system, which is non-arguably in short supply. If I can duplicate what someone has done, I don’t have to care if that person believes in . I can look at the process and make a judgment on it.
If the peer process also required “show all work”, and this is archived with the paper (I believe Steve McIntyre also advocates this), I don’t think things would be as poisonous as they are today…
Shouldn’t the paper just be written as a ‘here’s our data and here’s how we got our results’ briefing anyway? If they spent a proper amount of time on it to get the results, they should be able to show all their working and describe it with ease.
Let’s be charitable and assume that the tradition of only reporting the ‘highlights’ and not showing the working arose because of the technological limitations of ink on paper publishing. That no longer applies and volume of work is not really an excuse any more.
And if it means that academics have to spend an extra few dyas making their work more widely accessible and scrutinsable, then that is just the price the pay for maintaining the relative lack of oversight he enjoy compared with many professions outside the ivory tower.
Being able to show that you actually carried out the work, that other competent workers would use the same methods, get the same results and draw the same conclusions does not sound to me to be very onerous requirement.
But the current system, which is no better than self-certification does not inspire any sort of trust, Remember that Phil Jones- according to his testimony to Parliament – has never been asked to show his data or working for over 200 peer-reviewed papers. This is scandaloulsy lax.
I work part time in a very mundane observation job. Once working I am substantially left on my own. But once every three months my work is covertly checked by an inspector without my knowledge, and every year I must go through a full on-job review of my competence, knowledge and process. This is good because it gives the eventual recipient of our work confidence in what we do, and allows them to build trust in us as observers. And within our little industry it is recognised that those who survive a few years are reputable and trustworthy. Those who aren’t – and there are a few – are usually found out within the first six months and never employed again.
Can academia – as currently constituted – make similar boasts, based on anything other than academic solidarity? I think not. And peer-review does very little to ensure that it could.
I agree the current peer review process is flawed. In my view so would open review on the web (judging from the quality of many responses to posted material). Who is going to moderate? How would Einstein’s General Theory of Relativity have fared in the Blogosphere?
“How would Einstein’s General Theory of Relativity have fared in the Blogosphere?”
A lot like this:
http://conservapedia.com/Counterexamples_to_Relativity
“The theory of relativity is a mathematical system that allows no exceptions. It is heavily promoted by liberals who like its encouragement of relativism and its tendency to mislead people in how they view the world.[1] Here is a list of 39 counterexamples: any one of them shows that the theory is incorrect.”
“1.Despite wasting millions of taxpayer dollars searching for gravity waves predicted by the theory, none has ever been found.[2] Sound like global warming?”
““1.Despite wasting millions of taxpayer dollars searching for gravity waves predicted by the theory, none has ever been found.[2] Sound like global warming?””
Reminds me of dr Evil:
It seems improving peer process would nice, but having information which could inform the public regarding climate in general might good idea.
It seems the use of the term “greenhouse affect” is rather poor.
If you leave a window open on top of a greenhouse, there is no greenhouse affect. The greenhouse affect is entirely about inhibiting convection of heat.
And inhibiting the convective of heat does not have to do increasing global average temperature. So for instance wiki has to explain that greenhouse affect isn’t really like a greenhouse but it’s more like a blanket.
A blanket isn’t much better. But a “blanket affect” would be small improvement on the term “greenhouse affect”. Though a blanket doesn’t do much to keep a dead person warm.
So where a blanket affect would slight improvement, but if is going bother to change the term “greenhouse affect” it seems one should able to do better term than blanket affect.
The other aspect of greenhouse affect is it seems there are two different phenomenons one is increasing warming during sunlight [during the day] and the other has to do with reducing heat loss during the nite.
It seems the most significant factor in terms creating a higher average temperature is the latter. But keeping nites warming seems to not be discussed much.
The Moon obviously has warmer day time temperature than compared Earth- it’s lower average temperature is due solely to lunar surface getting much colder during the night. The moon doesn’t have a “greenhouse affect”.
So, since warmer night time aren’t mentioned much, the focus becomes for the general reader on the idea that daytime temperature is increased. Obviously this is misleading.
So currently have a “greenhouse affect” which isn’t actually anything like an affect caused by greenhouse.
Next, we have mention of black body. We told what earth temperature would be if it’s a black body, but earth isn’t a black body. In fact there isn’t anything which is a black body- it’s a theoretical concept.
We can measure earth’s radiated energy spectrum and it matches a blackbody signature- a Planck curve corresponding to temperature.
What the significant of this is isn’t exactly clear.
One could determine mathematically what a black body’s temperature would be at earth distance. And/or one could measure earth and get it’s Planck curve.
According to:
http://nssdc.gsfc.nasa.gov/planetary/factsheet/earthfact.html
It’s: Black-body temperature (K) 254.3
And moon is: Black-body temperature (K) 270.7
It seems it must be planck curve of earth [earth and moon are same distance from sun].
Over in wiki they say:
“Temperature of Earth
“Substituting the measured values for the Sun and Earth yields:
T_{\rm S} = 5778 \ \mathrm{K},[24]
R_{\rm S} = 6.96 \times 10^8 \ \mathrm{m},[24]
D = 1.496 \times 10^{11} \ \mathrm{m},[24]
\alpha = 0.306 \ [25]
With the average emissivity set to unity, the effective temperature of the Earth is:
T_{\rm E} = 254.356\ \mathrm{K}
or −18.8 °C.
This is the temperature of the Earth if it radiated as a perfect black body in the infrared, ignoring greenhouse effects (which can raise the surface temperature of a body above what it would be if it were a perfect black body in all spectrums, with an albedo of zero”
http://en.wikipedia.org/wiki/Black_body
So they saying the same value is determined theorically- therefore other than Moon’s size the moon should be same value.
Anyways my point is earth isn’t a blackbody- a blackbody is something that adsorbs all radiation and emits it very efficently. Or:
“Because of this perfect absorptivity at all wavelengths, a black body is also the best possible emitter of thermal radiation”
So black body is a theoretical object which very efficent at absorbing and emitting radiation. Earth doesn’t approach a theorical black body in terms of absorption or emission of radiation.
But for strange reasons it’s considered to be similar enough to a theoretical black body. A black body in space would resemble a stealth object- it would appear as black as the black universe- though depending upon distance from the sun [or if it was in the shade of some body in space] it radiate it’s Planck curve [plus what internal heat it generated]. And btw such a theoretical object would cool very fast if not in sunlight- because not only does absorb energy very efficiently it also emits it efficiently.
So our bright planet isn’t a blackbody, nor is our dark moon a black body, neither are very close. And of course to be even approaching a black body all of it it’s surface must evenly emit energy- which doesn’t vaguely describe earth or the Moon.
So here we are, the earth isn’t the same temperature of a black body- and somehow this is wrong. We need theory that explains this difference.
No one explains why such a theory is needed. No one explain why the average temperature of earth needs a greenhouse theory so it could somewhere then to equal it’s black body temperature. Why the obvious- earth isn’t a black body.
Now it is said that earth is grey body rather than actually a black body- though not much definition of grey body, other than is an approximation
of somewhat blackish body.
So black body in which earth is 33°C warmer than [33 C is the greenhouse affect] is like the term greenhouse affect. Earth isn’t a black body in anyway, just as there isn’t actually a “greenhouse affect”.
So if one explaining what a greenhouse affect is one isn’t being helpful talking about things that don’t exist nor are descriptive or vaguely analogous.
Ah, but they assumed that Earth had a 30% albedo and no significant atmosphere! Just because there is potentially a few degrees error should never get in the way of a good theory.
Where do they get their temperature from?
Must be the exact equator of the moon as all the other areas are slightly bent and distance goes further and further as you go to the moons poles.
And that one temperature applies for the whole moon?
Where do they get the Moon’s black body temperature from?
I don’t know.
Ref here:
http://nssdc.gsfc.nasa.gov/planetary/factsheet/moonfact.html
270.7 K is -2.45 C or below freezing- so not from sunlit side
And: “Diurnal temperature range: >100 K to <400 K "
A simple average is 250 K
Add in albedo [somehow??]
Here they give different number:
Black-body temperature (K) 274.5
http://www.lunarrepublic.com/info/resources.shtml
Here we go:
If an ideal thermally conductive blackbody was the same distance from the Sun as the Earth is, it would have a temperature of about 5.3 °C.
http://en.wikipedia.org/wiki/Greenhouse_effect
So I guess they took the 5.3 C [278.4 K] and subtract the Moon's Bond albedo which is 0.11
Since it's a dark surface but not black it lowers the theoretical black body temperature. But if you measured the Moon it wouldn't give you a Planck curve which corresponded to 270.7 K or 274.5 K
So it seems if the moon was snow white they would imagine it would significantly lower black body temperature. Or if wrap it in aluminum foil it would as cold or colder. Of course it wouldn't be.
That sounds about right. Albedo and surface emissivity have to both be considered though. Speaking of emissivity.
http://redneckphysics.blogspot.com/2011/11/building-better-model.html
It seems to require more significant digits that balancing your checkbook.
The solar constant at this distance from the Sun is 1367.6 W/m^2. A sphere has 4 times the area of a cross sectional disc, so the surface of a sphere gets, on average, 342 W/m^2. The Moon’s albedo is about 0.11, so the energy actually absorbed by the Moon is 342*(1-0.11) = 304.3 W/m^2. By energy balance, that is also the energy that the Moon radiates.
The energy radiated by a blackbody is σT^4, where σ = 5.67e-8, so the blackbody temperature is (304.3 / σ)^0.25 = 270.7 Kelvin = -2.5 C
That is the definition of “blackbody temperature”. There’s no implication that it is a blackbody; merely that it radiates the same energy as a blackbody at that temperature.
In fact, the Moon’s surface actually is fairly close to emitting as a blackbody once you take account of reflection (albedo); but it’s non-uniform. Under the “noon” Sun, the surface of the Moon gets the full 1367.6 W/m^2, and if that happens to be a dark patch (low albedo) then you get about 121 C, or 250 F, as the temperature at that spot. (This is the top of the diurnal range given).
“The solar constant at this distance from the Sun is 1367.6 W/m^2. A sphere has 4 times the area of a cross sectional disc, so the surface of a sphere gets, on average, 342 W/m^2. The Moon’s albedo is about 0.11, so the energy actually absorbed by the Moon is 342*(1-0.11) = 304.3 W/m^2. By energy balance, that is also the energy that the Moon radiates.”
Yes, I think this is exactly how they arrive at the black body temperature, but, it’s obviously wrong.
It’s wrong because the Moon isn’t the same as a black body.
Suppose the moon was a black body and you corrupted this black body so as to make it reflect some sunlight [gave it a Bond albedo of 0.11- say lightly dust it with table salt] then it might possibly be closer to being correct.
But to be closer to being correct would depend on how you designed and constructed this black body. One could make a model of this fairly simply [low cost] to get something which was a close approximation. A silver sheet metal which was blacken [and perhaps given some kind of texture]. To be properly designed it would need to be able to receive 1367.6 W/m^2 and emit half
of the energy [682.8 W/m^2] on both side of the sheet [side facing the sun and other side not facing the sun.
Since silver is a good conductor of heat [diamond is better- but we have budget concerns] one also could make a sphere instead of a flat sheet.
One could also do a thought experiment [much cheaper]. If a planet is rotating, it’s rotational speed is acting similar to the planet being made of good conductor of heat. With earth at the equator, it has heat which is being transferred at 1000 mph to towards it’s nite side. The faster a planet rotates the faster heat is being “conducted” to nite side.
Therefore a moon type body which had a 2-4 hour day would more closely approximate a black body. It certainly would absorb much more energy and it would emit much more energy.
What’s humorous about this is that you can get arbitrarily close to modeling the moon’s temperature as it varies empirically. The dynamic range is large so it you can get a good feel for it..
gbaiki, you say:
The definition of “blackbody temperature” is the temperature that a blackbody would have in order to radiate the observed amount of energy. It doesn’t actually matter whether the Moon is a blackbody or not; it still has a well defined “blackbody temperature”.
Yes, a higher rotation speed would give more uniform temperatures; but it would have zero effect on how much energy is absorbed and emitted. It would only change the distribution of emission over the surface.
“In fact, the Moon’s surface actually is fairly close to emitting as a black body once you take account of reflection (albedo); but it’s non-uniform. Under the “noon” Sun, the surface of the Moon gets the full 1367.6 W/m^2, and if that happens to be a dark patch (low albedo) then you get about 121 C, or 250 F, as the temperature at that spot. (This is the top of the diurnal range given).”
It seems that what you mean by black body is synonymous with the highest temperature something can achieve from sunlight being perpendicular to it. And with this you numerically process it, combining with lowest temperature. And some kind of averaging of averages.
Or if the moon was painted as black and white chess board, one could arrive at the answer to it’s black body temperature. The size of chess board squares and the material properties involved is unneeded to arrive at some answer.
And since a mirror has higher albedo as compared to white [a mirrored moon would appear somewhat similar to the sun- if mirrors angled so reflect towards the earth, then very similar to sun- if merely horizon to lunar surface the sun’s brightness would a much smaller area- whereas disffuse nature of white light from white paint may give nearly as much light and more of the “softer” 100 watt light bulb type light], would a mirrors horizontal to lunar surface have a higher or lower black body temperature as compared to the Moon being snow white?
If seems to me that Bond albedo and black body are treated as the same.
A Bond albedo [.01 to .99] is the range of how much something is regarded as black body.
Also the Thermal equilibrium in regards to sunlight is also the same as black body- any material with the highest Thermal equilibrium in regards to the solar radiation is the most black body like thing.
So it’s a very black and white world. And it seems as though transparency
and/or conduction isn’t seen. Very child like.
gbaikie, you say:
No; by blackbody I mean emissivity 1 over the whole spectrum; the usual physical meaning of the term. To emit like a blackbody is to have emissivity close to unity over the main part of the emission spectrum. The moon is pretty much like this.
However, the Moon does reflect about 11% of the high frequency solar radiation; which is why I speak of “after albedo is taken into account”.
Curious aside: a blackbody is the LOWEST temperature you can have for something to emit a certain amount of energy. You can get much hotter in principle with regions where the albedo is very low (all incoming energy absorbed) but the emission is in frequencies which have lower emissivity.
But in practice that does not occur. The Moon’s surface is a lot like a blackbody in those frequencies where thermal emission occurs; and that gives a physical basis for the hottest temperatures likely to be attained at noon.
I showed how you calculate the “blackbody temperature” directly. It doesn’t average things. It simply works from the emitted energy, and calculates a corresponding temperature.
“The definition of “blackbody temperature” is the temperature that a blackbody would have in order to radiate the observed amount of energy. It doesn’t actually matter whether the Moon is a blackbody or not; it still has a well defined “blackbody temperature”.”
I agree one could measure Earth or the Moon’s Planck curve and this it would give you it’s black body temperature. And that would mean something. As I said in above post.
But it’s apparently isn’t done this way.
Instead it’s numerically arrived at.
“Yes, a higher rotation speed would give more uniform temperatures; but it would have zero effect on how much energy is absorbed and emitted. It would only change the distribution of emission over the surface.”
Lunar day is 28 day long. When the sun is at noon for about day, the temperature on the surface doesn’t increase. It reaches a maximum temperature relativity quickly. So during the week of “mid day” the temperature roughly stays the same. But at dawn the temperatures going rise relativity fast starting for roughly 100 K. You would see a temperature change quite dramatic- like nothing seen on earth. in minutes to hours the surface will climb tens of K. Even though the lunar dawn is is about a day in length, you see more surface temperature increase in a lunar dawn than the variation one normally see in a day on earth. So by 9 am on the Moon one should over a 100 K increase in surface temperature.
And during the week of near midnight the surface isn’t going to get much cooler- and isn’t radiating much energy to space.
If speed up rotation, during midnight it will be cooling more and more radiative will going to space.
Or now have Diurnal temperature range: >100 K to <400 K
4 hr day it could instead be about 300k to 400 K
It would emit more energy- therefore as to absorb more energy.
gbaikie, you quoted some fact sheets, from NASA, giving planetary data. All I am telling you is the definition of terms used in those sheets, and how the blackbody temperature is calculated.
You now say:
Actually, you cannot get the blackbody temperature by measuring the spectrum of emission. (Spectrum from where? A sum over the surface? That would give something approaching a Planck curve corresponding to a much higher temperature.) Remember that the surface is not a uniform temperature. If you are troubled by the fact that astronomers speak of blackbody temperature in the way I have described, and get the numbers you see in the NASA page, so be it; but you are not “agreeing” with anything anyone has ever said, as far as I know.
You conclude:
Actually, spinning a body does not let it absorb more energy. If a surface gets to equilibrium temperature rapidly, then it will also cool rapidly. If you spin it fast enough to keep the night time temperatures a bit higher, then you will necessarily be letting daytime temperatures get not quite as high. Remember to that emission goes as T^4, so a large difference in the night temperature is compensated by a smaller emission in the dayside.
It will all boil down, in the end, to the fact that spinning the surface doesn’t change the amount of energy being absorbed.
I was looking for reference I saw earlier, but I didn’t find it.
Instead there is this:
“With the preliminaries out of the way we can look at what the emission from the Earth through the atmosphere looks like at 20 km. ”
http://rabett.blogspot.com/2010/02/another-try.html
Another quote from above:
“If you look to the right, you see that there are parts of the curve that roughly follow the Planck curve for ~290 K which is the temperature of the ground. This is radiation that is going through the atmosphere essentially undisturbed. …”
gbaikie, quite so. This has no bearing on what we’ve been discussing above; but no matter.
Onto this new topic… the Earth’s emission spectrum is much less like a blackbody than the spectrum from the Moon. The emission from the Earth is powerfully affected by its atmosphere. The emission from the Earth looks like a Planck curve with big bites taken out of it. The emission from the Moon, on the other hand, looks like a Planck curve corresponding to the surface temperature at the point where you are taking the spectrum.
The Earth has an overall “blackbody temperature” as well, which is defined in exactly the same way as given previously for the Moon. That temperature is about -18C. It’s less than the Moon’s blackbody temperature, because Earth has a higher bond albedo.
As before, this is not an average temperature, and it is not based on looking at the shape of the emission spectrum.
In my opinion „peer review“ is a necessary process for scientific publications.
However, everyone should be aware that “peer review” is a bucket with holes and a gate-keeper process on the same time.
At least every scientist should know that.
The “peer review” process should and can be augmented and accompanied by pre and post publication discussions in the internet. This certainly will be an improvement.
In the internet the label “peer review is extensively used as a quality label or an appeal to authority argument, which is treacherous and misleading.
In a scientific discussion such an argument is useless.
However, in a political discussion such arguments are typical.
Best regards
Günter
Maybe “peer filtered” would be a more precise label.
Yes. Very inefficient in every way.
No politics and phoniness and much more transparency please!!
The skills needed to assess “significance/impact” and those needed to assess “quality” are very different.
It is often immediately obvious to someone working in the field what is significant or likely to have a high impact, but that impression is often highly subjective and also variously interpreted (significant now, tomorrow, forever?). It would be an easy matter to canvass a range of people to get an impression of how likely a paper is to have high impact. You could even have little ads for submitted papers at the side of the journal website that would count click throughs to assess interest.
The process of vetting the paper to find errors in math, logic, omitted controls, poorly thought out error analyses, missing references and so on takes much longer if done especially thoroughly. This step could be applied only to papers that pass the first ‘significance’ test and crucially it requires somewhat more objective criteria so fewer reviews would be needed.
Two separate tasks, two-step process. Easy. Maybe
Probably not.
Problem is what if a discovery could impact your own career?
Certainly NOT going to allow it through no matter how correct or knowledge changing that could be.
Massive cuts are coming and who will survive?
Open review systems seem like a good idea, but there are a number of caveats.
The first is that most papers don’t get ‘open’ reviews they are reviewed by reviewers chosen the old-fashioned way, by the editor often with recommendations from the authors. Creating open review doesn’t magically make more time for people to review papers.
Second, a small number of papers will attract a large number of reviews. If you think that gatekeeping is a problem, then it’s worth thinking about what would happen to skeptical papers if the ‘gatekeeping’ community all weighed in on an open forum at the same time (provoing an equal and opposite response from the gatebusters). What does that actually tell us about the paper?
Third, different disciplines (even different people within the same discipline) have different standards and everyone likes to think that theirs are the most rigorous. However, engineers applying engineering standards to a scientific paper is bordering on an absurdity, but applying standards from particle physics to biochemistry would be weird too. Who decides what standards a paper needs to adhere to? What comments need to be addressed, which don’t. To some this will seem arbitrary, and it mean that open review is unlikely to satisfy everyone.
I was classed as a “nutbar” and should be ignored.
Snipped many times trying to show the discoveries I made in trying to advance our knowledge base into a correct one of understanding.
Since the velocity mapping…
Still get ignored, but no “nasty comments”.
Have not been snipped once…
Just a distancing in case some nasty knowledge may creep in and implode the current reliable theories.
Proving that not all accused nutbars are falsely accused!
:)
“The values that people primarily ascribe to peer review are maintaining the integrity of the scientific literature by preventing the publication of flawed science”
Is this not the problem? This is, at least as I was taught NOT the role of peer review. The role of peer review is to assure that the paper is fit for publication, nothing else. One may disagree with it, see it as flawed but any suggestion that it is flawed is a debate to be carried out in the literature, not by anonymous peer reviewers with shall we say many reasons to ‘redefine the peer review literature’.
The peer-review system is currently set up to field science through a funnel of keeping fictional science on coarse as they made the mistake of generated an enclosed “we know it all” system.
When in fact, they screwed it all up and do not want to change.
That is an indefensible position Joe. Journals are quite accepting of new ideas in a many fields. In fact it is quite hard to get a journal to find a replication study interesting enough to publish. They want new new new, all the time.
This acceptance of new ideas may dwindle when a field becomes highly charged or politicized, but that is not the fault of peer review.
andy writes “The role of peer review is to assure that the paper is fit for publication, nothing else.”
Well said, andy!!! This is precisely my idea. Different publications have different ideas as to what should and should not be published in their magazines. Phil. Trans.of the Royal Society has a different level of what is worthy of publication; compared with other publications which are prepared to give space to way out ideas.
I certainly see the issue of peer review lies at the heart of what has gone wrong with science. It was never much more than a process that eliminated outlandish mistakes, and ensured that outsiders/newcomers had to publish very orthodox work, before being permitted a little more leeway as they became progressively better known.
How could it possibly achieve much more – nobody was going to repeat a long and complex experiment or calculation, just to respond more authoritatively to an anonymous review request!
Many years ago, when I was in academic science, I was asked to review a few papers, and I always felt embarrassed by the inevitably cursory nature of my response – even though I was thanked by the relevant editors.
The open review process sounds better, with perhaps all papers remaining visible on the internet (including all the data) – with their reviews – for all to see. Who then would want to publish a paper that was (say) full of statistical flaws that would become exposed and impact directly on the author’s credibility?
Excellent point of view!
Science created theories and expanded all around them to ensure that this was absolutely correct and can never be changed as they created them into LAWS.
Every point on this planet is unique.
Not by current theories that lump them into single calculations.
Many areas were missed and NEVER included as technology changed yet science theories did not.
We need to start from the ground up to review all of science. Show where our errors were made and how to correct them.
Instead all we are doing is expanding upon bad theories.
I think the heart of what went wrong with science is grant money.
I see a common theme in this griping about peer review, and it is that a few bad experiences with peer review are over generalized to apply to all peer review everywhere. Can those of you who have had such bad experiences accept the possibility that it’s not rotten and useless everywhere?
Your right about the grant money part.
Now, where would you go to publish a piece of work?
Yes, you are right about grant money, but was peer review ever a really good way to assess papers? Maybe it was the only method available before the internet arrived, but now I think we could do a lot better.
Does anyone think that ‘climate science’ is uniquely awful – I sense that part of the reason that the scientific establishment has kept a lid on this mess, is that there are a lot more cans of worms awaiting opening!
If you think 100,000 people commenting rapidly on each other’s ideas is going to succeed where 10,000 people commenting careful on each other’s ideas has failed, I think your ideas about the power of the internet have gotten away from you.
From my perspective as a publishing scientist, here are some pretty clear facts:
Science is really hard.
It is easy for science to be done poorly because human beings are inherently designed for confirmation bias.
Peer review is a big help because it adds a level of quality control.
Peer review is not perfect.
Some fields of science have more abuses of the system than others.
If you dig deep you will find other areas of rotten science.
These areas of rotten science are due to human nature, not the peer review system.
Any other system of quality control in science will also be vulnerable to confirmation bias, consensus thinking and all of the other things that we spend hours griping about on these threads. There is no magic bullet to convert human beings into bastions of objectivity. We are what we are.
Well said.
Brad,
I think I could sum up your comment as, “Leave the system as it is, it is not too bad!”. I’d say that if it causes our electricity grid to be reconfigured in a hugely expensive and pointless way, it is absolutely terrible!
The problem with your approach, is that when you leave the rot in a system, it tends to grow – just as rot grows in teeth, or wood.
When I speak of the internet, I am thinking of its electronic infrastructure, global reach, and ease of use – not of some particular mechanism such as blogging. I’m not suggesting that peer review be replaced by blogging!
Think for a moment of GOOGLE. It routinely sifts untold petabytes of information, but usually manages to bring the most salient information to the front. Are you really saying that some equivalent couldn’t be created to enhance peer review. Suppose for example people could read one of Mann’s papers, and immediately see a trail of caveats (or more accurately, a tree of caveats) contributed by named individuals (with real digital signatures – not blogging names) – are you really saying it would not help?
Imagine if we could just see the pattern of peer review – visualising the incestuous knots of mutually favourable reviews in certain areas – are you really seeing that would be of no value?
David,
I’m not saying leave the system alone, I’m saying that tearing out peer review and replacing it with something else will not fix the problem. The problem is in the culture of science, not the peer review system.
Fix the problem at its core, not the symptom.
Brad, I don’t think it is rotten and useless everywhere. The problem really seems to be the “prestigious” journals. Just their standards tend to bias toward previously publish material. Why would they want to publish papers that show their own fallibility? That would be bad for their reputation and is not the intent of science I would think.
The IPCC issue seem to spill over to other journals as publishing time draws near.
I should add, there is no way in hades, Stieg et al could have been published in Nature had it not seemed to support published theory. I nearly peed myself laughing at their conclusions and confidence intervals well before the internet warriors were published.
I agree that there are a few highly prestigious journals which have a remarkably high level of bias. But they are dwarfed by the number of mid-tier journals that represent the real meat of the literature.
I’m all for posting papers early, and having crowd review. I think part of the current paradigm is that a paper is static. There’s a stake put in the ground when it is published. I’d rather see a “first edition, second edition”, or other revision system.
I’d sooner put my science up against current theories in front of a class of students and let them pick who is correct.
Since you have never been able to describe ‘your science’ in several years of being asked to on this blog and others, what makes you think you would be more successful in real life in front of a class?
Latimer,
You must have been away for awhile…
Velocity, I mapped our planet and can map any rotating body as long as I have the diameter and rotation speed.
http://jonova.s3.amazonaws.com/guest/lalonde-joe/world-calculations.pdf
http://jonova.s3.amazonaws.com/guest/lalonde-joe/world-calculations-2.pdf
Peer reveiw can certainly do with some tweaking, and the journals need to more rapidly evolve as technology makes dissemination more efficient and easier. Much of the journals structure still reflects their paper origin.
But there’s not much being suggested that isn’t more problematic than the isues they claim to be solving. Some of the solutions would likey lead to more papers being published, when part of the problem in some fields is trying to keep up with the flood of published literature.
Michael,
It is called keeping the tradition alive.
Like much of science is tradition rather than evolution.
Good points.
Michael, That is why we are all here to help you with the tweaking. One thing though, journals don’t evolve. Man seems to and its still a mystery to the rest of us. One thing at a time though. Keep up the good work.
” journals don’t evolve”
Untrue.
http://www.plos.org/
http://www.frontiersin.org/
Brad, but they aren’t animal life forms now are they? When I consider newspaper sales though, I can see your point that it is a variable of:) survival of the fittest. Good luck, by the way.
Of course, there are some ethical issues specific to biomedical science or any other type of research; and also transparent political issues specific to how grants are given out in the U.S. in relation to the extensive influence of members of government on funding allocation (kind of like the military contracts problem).
In general, is pre and post publication (mostly via the internet) in science really a substantive change in how science is done? or is it about how science is published?
These questions are raised and discussed in a couple of the links. As discussed, informal commentary isn’t something new and it exists in many forms at many levels as part of current scientific activity and process. In other words, pre and post publication review are done all the time in some form as part of the current process – via discussion, technical input, correspondence etc etc – so what, exactly, is new?
Maybe lots, maybe nothing much. At least, at this time.
As another link argues, it might presently just amount to a ‘new element in formal scientific publishing’ – part of what anyone would expect as society learns to use the internet – that appeals to popular perceptions but does not in fact add much to either the content or the evaluation of content or decision-making.
As the same link further notes, evaluation of the content of the online discussions has not occurred; and there are pragmatic problems with it, especially for anyone seriously concerned about maintaining science as a critical, potentially radical space for ideas. It is simply not occurring on the internet, for all the same reasons that the internet has (so far) disappointed anyone seeking more potential for grassroots democracy and critical spaces for progressive thought. An example is the extent to which the Right wing American blogosphere is extensively biasing internet discussion of climate science and policy– but that is a different conversation. :-)
The main counter-argument to these criticism remains the potential, not the reality. Online versions of these discussions make the process more accessible to the media and public; however the question would still remain whether this can eventually result in any substantive change in the nature of the process, and what else needs to be in place to enable this.
As those links illustrate, clarifying what we mean when we discuss pre and post publication review is a basic first step; and those who mean ‘open review’ should perhaps use that term instead, if that is what they mean.
I suggest that change in composition of government committees will go much further to reducing bias and promoting open science, than anything you are proposing regarding publication processes. ;-)
cheers
Martha,
Can you pick out what is fiction, what is fact and what is uncertain?
Can you leave all your biases and upbringing behind to be totally objective to the facts?
This means what you were taught in school may not necessarily be actual facts. Many generalizations have tainted and bent science for a certain outcome rather than following the path of science.
Well you might pass peer review, but ain’t going to pass Svalgaard of WUWT as certain dr. Scafetta found most recently.
Are there any medals for the injured pride?
vukcevic,
I’ll take that challenge.
My velocity mapping is facts that cannot be in disputed as incorrect.
“My velocity mapping is facts that cannot be in disputed as incorrect.”
Joe, perhaps you’d like to add to your map the contribution of the Earth’s velocity in its orbit around the sun? Don’t forget to include the effect of the axial inclination.
Facts can always be disputed.
……And the velocity of the entire Solar System relative to the Milky Way..and that of the Milky Way relative to the rest of the Universe….and so on until you reach that still point where you can measure all absolute velocities from.
Let me know when you find it.
Latimer,
Never satisfied until you get that last drop of blood eh???
People who want to do away with peer review in science have to describe the new system, say what it will cost and who will pay for it, as well as how they want to force the change to occur, and enforce it thereafter? The journal system is a multi-billion dollar industry, with over 10,000 journals and one million articles a year. It is highly independent and international.
Saying you want it to undergo fundamental change without saying how that happens is meaningless hand waving.
The Obama Administration is looking for ideas and arguments related to two policy areas that are directly related to this thread. The first policy area is public access to peer reviewed scholarly publications that are based on federal funding, while the second is public access to federally funded scientific data. Two official Requests for Information (RFI) have been issued. Comments are due on the publications RFI by January 2, 2012 and on the data RFI by January 12.
The publications RFI is at http://federalregister.gov/a/2011-28623
The data RFI is at http://federalregister.gov/a/2011-28621
Hello David,
How about everyone has an internet page….I don’t know called ‘Peer review’ etc and any work you want to submit for a journal goes up there to be reviewed by any interested party. You can then acknowledge the help received in the article.
I’m sure I’ve seen this somewhere but I cannot think where at the moment;)
Either that or stop journal articles all together and get folk to write updated webpages/books etc. They could even put up their code and data.
Of course that would screw the hierarchy the institutions of science have placed on researchers to allow allocation of research money. But I think that’s a good thing.
Grant review is not handled through the journal peer review process. The two are completely separate. Whatever ills are present in the current mechanisms of grant allocation would survive the complete dissolution of the journals.
As for your proposed web based solution, how would I, as a new researcher, know which websites to visit? How would they be ranked? Who would store such rankings?
If your suggestion is that google would do this work for you, that would turn the entire scientific process into a pure popularity contest, making it very difficult for a new scientist’s work to be noticed. In contrast, the current peer review system allows a new scientist to publish a work in a major journal (admittedly with some difficulty), and that article then lands on the desk of every scientist in the field.
In fact, I suspect that your proposed solution would make it even more difficult for new ideas to overturn the establishment, and isn’t that exactly what you’re looking for?
Important point: the current state of affairs in the climate sciences, with pal review, gatekeeping, etc, is NOT present in every field. The majority of scientific disciplines enjoy a peer review system that is highly functional.
Another important point: the woes of climate science peer review could certainly be recreated in whatever open review process you are proposing. The problems here are not caused by peer review, they are caused by human nature.
Let me add another complaint about the putative web-based system: it would create an opportunity for ad-farm technologies to be transplanted into the scientific process. in other words, tech savvy researchers could play games with spurious citations to crank up their rankings.
Google keeps this problem mostly in check but it is a constant (and expensive) struggle.
Who would fund this effort to keep technological tricks from perverting the web-based open review system?
Funding is always a problem, but it is important to keep in perspective the waste of resources associated with the present system:
Badly done research still costs money in salaries and materials.
Research that impacts on government policies, costs even more!
Perhaps a better question would be what percentage of research resources should be spent on providing a robust method to assess research.
Andy: How do you then get the millions of scientists to stop submitting articles to the journals and/or the journals to stop publishing them? You can’t just wish a service industry out of existence because you personally don’t like it. That is the silliness that feeds this thread.
M. carey | November 13, 2011 at 4:28 am |
No. The Board was very clear on that. They said they were firing Spanier for taking pencils and paperclips from his office without authorization.
Perhaps a purblind mole might make that interpretation. A reasonable man would conclude that McIntyre was saying that the facts show that Spanier had been as negligent in overseeing the Mann allegations as he was in overseeing the recent scandal. Which, if you bothered to read the facts … you’d already know.
Dude, if you interpret that as the Board of Trustees finding, you need to get your interpreter checked. It’s reading way high.
w.
PS: Since you seem to be unclear about why the Board fired Spanier, consider some facts:
See if your “interpreter” might be able to interpret those facts and come up with a clue as to why Spanier might have been fired … hint: I was kidding about him stealing paperclips.
PPS: Here’s a protip for your interpreter: When Steve or I or anyone is making a claim about what the Trustees said, generally the words “the Trustees said” or some equivalent will be in the sentence or nearby. If those words are not seen, the odds that the Trustees said it drop radically.
Just sayin’ …
Willis, that latest Penn state sprots problem is just one in a long line.
have you seen this article
http://sports.espn.go.com/espn/otl/news/story?id=3504915
It states that “Since 2002, 46 Penn State football players have faced 163 criminal charges, according to an ESPN analysis of Pennsylvania court records and reports. Twenty-seven players have been convicted of or have pleaded guilty to a combined 45 counts. “
It would appear that they do not have much control over their “employees” at all.
Willis, I believe McIntyre interprets things the way he likes to interpret them. So do I. McIntyre interprets Spanier’s firing as evidence Mann may be guilty of academic misconduct. I interpret McIntyre’s interpretation as evidence he is not above trying to exploit the Penn State sex scandal to smear Mann.
Willis, do you too interpret things the way you like?
My guess is y interpret things the way you like.
“McIntyre interprets Spanier’s firing as evidence Mann may be guilty of academic misconduct.”
No, I think he interprets Spanier’s firing as evidence that Spanier has had trouble conducting inconvenient investigations into programs at Penn State that generate money and prestige for the school. He raises a legitimate question as to whether Spanier’s version of ‘investigations’ are really robust.
The climategate emails provide ample evidence of Mann’s academic misconduct.
John, I’m not surprised you like to interpret the e-mails as evidence of academic misconduct.
I am surprised if you think McIntyre’s comments at Climate Audit about the firing of Spanier have nothing to do with Mann.
M. carey, perhaps you misunderstood what I wrote. I think they have to do with Mann as far as how thorough the investigation of Mann’s alleged misconduct was performed. The thoroughness of Mann’s investigation was surely influenced by Spanier to some degree. If Spanier did not demonstrate good leadership skills in an investigation of sexual child abuse, with an eyewitness, and allowed or worse, was instrumental in sweeping it under the carpet….. then how many other high profile investigations (like Mann’s, with clearly readable emails of being complicit with deleting files) got a similar treatment? That is what I believe is at the heart of McIntyres post. If you think I have it wrong, let me know.
M. carey | November 13, 2011 at 11:21 pm | Reply
If you were to add a quote of McIntyre actually saying that, you might get some traction with your claim.
But you haven’t done that. Since it is merely a vague, non-falsifiable claim without a scrap of evidence to back it up, you’ll pardon me if I just snicker behind my hand and pass on by, and wait for you to back up your wild fantasies with some actual quotes or facts.
w.
Willis, if McIntyre does not interpret Spanier’s firing as evidence Mann may be guilty of academic misconduct, what does McIntyre’s following statement mean?
‘As noted above, Spanier had falsely reported to the Penn State trustees and the public that the Penn State Inquiry Committee had properly interviewed critics and had examined the Climategate documents and issues “from all sides.” ‘
Thanks, M. carey. It means that Spanier falsely claimed that a proper investigation had occurred, when nothing could be further from the truth. It says nothing about Mann’s guilt or innocence at all. It simply says that a proper investigation of Mann’s guilt or innocence had not occurred, and Spanier lied when he said one had occurred.
You may not be aware, M., that not only did the Mann Inquiry Committee not get the answers to the relevant questions. They didn’t even ask them.
Spanier’s role in this was widely known already from the time of the Penn State whitewash of Mann. So for those of us who were aware of Spanier’s false report to the Board of Trustees in Mann’s case, it was quite telling (but unfortunately, not all that surprising) when Spanier was discovered and dismissed for doing exactly the same thing, only regarding covering up something much worse.
It proves nothing about Mann, nor did Steve say it did. It definitely proves that Spanier’s cover-up in the Mann Inquiry was not an isolated case.
w.
Willis, perhaps you see Steve McIntyre as a neutral observer, and an even-handed man who doesn’t take sides and wants Michael Mann to get a fair shake. Facts suggest a different picture.
A panel of Penn State faculty and staff released the findings of their inquiry of Michael Mann in February 2010. Collegian Staff Writer Colleen Boyle reported that Graham Spanier addressed the inquiry and the panel’s work during the Board of Trustees meeting on Jan. 22.
“I know they’ve taken the time and spent hundreds of hours studying documents and interviewing people and looking at issues from all sides,” Spanier said.
http://www.collegian.psu.edu/archive/2010/02/01/mann_inquiry_concludes_board_t.aspx
Steve McIntyre took issue with Spanier’s statement, particularly the claim the panel had been “looking at the issues from all sides.” “They didn’t contact me,” McIntyre protested (Climate Audit, Feb 1, 2010 9:46 AM). More recently, he complains “In short, the case for the prosecution is never heard.”(Climate Audit, Nov 10, 2011 4:41 PM).
I think based on McIntyre’s comments it would be fair to conclude he thinks of himself as one of the sides, the prosecution side.
M. carey | November 13, 2011 at 11:21 pm | Reply
I’m still waiting for your quote, M. carey. I see that you are flailing about, but you have not produced a scrap of evidence to back your bull.
Do I think Steve McIntyre is a “neutral observer”? Don’t be daft. He’s on one side. However, the “investigation” didn’t do what any good investigation should do—interview people on both sides of the question.
Why does that seem so hard for you to grasp? Are you sure you understand this “investigation” thing? It’s a time when people, you know, go in and talk to everyone in order to dig out the answers. Not just talk to the Friends of M. carey or to the Friends of M. Mann. Everyone.
But NONE OF THAT HAPPENED, M. You’re in fantasy about some “investigation”. They came in. M. Mann said he was innocent, honest, no kidding, really, really innocent.
They said thanks, sorry to bother you, and wrote their report.
And you actually think that constitutes an investigation? All that shows is you didn’t even read their Inquiry Report. READ THE REPORT, YOU LOOK LIKE A BLITHERING IDIOT WITH YOUR PUERILE CLAIMS AND YOUR CHILDISH FAITH IN PENN STATE HONESTY!
Every time I think that I can no longer be surprised by people’s blind credulity, some genius like yourself jumps up to say Mann has been exonerated. And this happens even after Penn State has shown clearly that they can’t be trusted …
Mann has not been exonerated, M. He has not even been investigated. If you would do your homework on what actually happened, instead of taking Spanier’s word for it (as the Board of Trustees foolishly did … twice …), you’d avoid looking like a total newbie regarding the Penn State whitewash of Mann.
Mann was “investigated” by Penn State in the same way and with the same attention to detail as the sex offender was “investigated” by Penn State—and at least in the latter case, I assume you know the amount of investigation was “not at all”, I assume that are not so stupid as to accept Spanier’s claim that the sex incident was “investigated”.
And yet you blindly accept Spanier’s same claim in the Mann case … you don’t get out much, do you?
w.
Willis, I presume you are referring to the following reports when you shouted “READ THE REPORT, YOU LOOK LIKE A BLITHERING IDIOT.”
RA-10 Inquiry Report: Case of Dr. Michael E. Mann
http://www.research.psu.edu/orp/Findings_Mann_Inquiry.pdf
RA-10 Final Investigation Report Involving Dr. Michael E. Mann http://live.psu.edu/pdf/Final_Investigation_Report.pdf
Willis, the Penn State investigative bodies that prepared these reports exonerated Mann based on the evidence they deemed relevant. You wanted them to find Mann guilty of wrong doing, and since they didn’t, you impugn their integrity and competence.
Unfortunately, there was no similar investigation by Penn State into the allegations of sex crimes by Sandusky. It’s too bad he was not scrutinized as thoroughly as Mann.
This topic is well covered in this month’s Technology Review.
First, is “Public Mea Culpas” which outlines how peer reviewed papers are being retracted in record numbers over the past decade, however, there is scant formal retraction policies at many journals. Unfortunately, you need to subscribe to read the whole thing.
http://www.technologyreview.com/biomedicine/38873/
Second is The Office Next Door, which starts off:
“”Science,” the physicist Werner Heisenberg once wrote, “is rooted in conversations.” As he saw it, scientists are rarely solitary thinkers but people who constantly talk: about ideas, findings, research techniques, and unresolved problems. Some of these conversations last for a few minutes or hours. But others continue for years or decades, shaping careers, disciplines, and even institutions.”
http://www.technologyreview.com/article/38806/
I think that the blogs has the potential to make us all next door and available for conversation. Unfortunately, the signal to noise ratio is quite low as demonstrated by the numerous endless-loop digressions on this thread. Fortunately, the weight of the high value contributions makes this messy process worthwhile.
Thanks for these links
The Technology Review article also refers to an example for Biomedical Journals:
Various. Retraction Watch. Catalog. Retraction Watch. http://retractionwatch.wordpress.com/
andy | November 13, 2011 at 7:35 am | Reply
“The values that people primarily ascribe to peer review are maintaining the integrity of the scientific literature by preventing the publication of flawed science”
—–
Is this not the problem? This is, at least as I was taught NOT the role of peer review. The role of peer review is to assure that the paper is fit for publication, nothing else.
———————–
Agree, 110%. There is a mistaken notion that Peer Review is some sort of guarantee that the paper is sound science. Peer Review in no way establish that the science is correct, only that the methodology is correct.
The problem is GIGO (garbage in – garbage out)
garbage data in ===> perfect methodology ===> garbage conclusion out
Perhaps the statisticians can answer this. If you have 10 numbers with a random component that has a standard deviation of about 0.1 degrees, how certain can you be of a least-squares fit gradient with these numbers?
Sorry wrong thread, disregard. Will re-post on Congressional thread.
David I would do nothing, rather I would envisage those who know peer review is broken slowly moving away from it.
Brad I know grant review is separate but it depends on ones reputation…measured by peer reviewed articles.
I don’t think the objections you present amount to much. Don’t you think a good researcher would know good science when they see it? That is all the measure required.
Fred, I agree, but I would also add that looking at garbage is essential to allow you to see what is not.
“I don’t think the objections you present amount to much. Don’t you think a good researcher would know good science when they see it? That is all the measure required.”
One of the points about peer review is that it elevates the quality of the publication. The analyses are more reliable, the figures are clearer, the text has fewer spurious assertions.
The other point is that peer review acts as a filter which is extremely valuable. I simply do not have time to sort out the good from the bad (on the average).
Andy, very few scientist think it is broken. They do 3 million voluntary reviews a year.
Many – particularly those who think about it – think it’s broken, but it’s all they’ve got.
“once a Nature paper, always a Nature paper.”
If papers would have their primary existence as digital files in a well publicized repository, then any necessary corrections or retractions could be stored with them. Any time someone wants to look up a reference, they go to the repository for that paper and would have access to all the reviews, corrections, etc.
Since this is on peer review, the equation W=alphaln(Cf/Ci) is based on a paper published over 100 years ago that fail post publication peer review. Ten years later, the author retracted his result without publishing a corrigenda. His results changed from 4-5 C to 1.6 (2.1) with water vapor.
The fatal assumption of the paper was that the atmospheric radiant layer would be 255K degrees based on his assumption that the surface without the benefit of greenhouse gases would be nearly -200C colder.
The average radiation layer at 255K resulting in a change of flux/temperature ratio of 4F/T. Which when compared to a surface flux change per degree K results in 5.8F/T yielding 4/5.5 = 0.69 or the ln(2).
First the average radiant layer of the atmosphere does not appear to be 255K. The top of the atmosphere would be ~255K, but it is not the average radiant layer impacting the surface.
Second, dF/dT at the surface in 1896, was not 5.8 but closer to 5.3F/T, They seemed to have had issues with average global temperatures back then as well.
How, about we do a Post- posthumous peer review of the paper assumed to be the definitive work on the atmospheric greenhouse effect?
That’s an excellent idea, Dallas. There are those who think saying “Arrhenius ” wins an argument in the same manner that saying “abracadabra” invokes a spell.
It would be fun. I doubt those that invoke Arrhenius have ever waded through the paper.
I read, somewhere else, that the original paper was in German and no translation readily available. True, or not, I guess that might preclude quite a lot of invokers from actually having read it.
I’ve been on the receiving end of such advice, but can also tell a short Arrhenius-story.
As a Grad student I had occasion to make use of an ‘Arrhenius Plot’ to measure a rotational-bond energy-barrier by experimentally measuring the temperature dependence. Before I synthesized the (new) molecule I performed some fairly extensive computational quantum-mechanical calculations.
End result? Lovely data. Beautiful curves and straight lines in all the right places. But the experimental didn’t match the computational as well as I hoped. Who was right, Arrhenius? or Schrödinger? For good reasons, I didn’t trust either well enough to publish. With hindsight perhaps I was wrong, because I read so many authors who appear not to let such things trouble them. And I now have one less publication on my rather short CV.
A unique identifier (for subsequent internet reference) should be appended to any “hard copy” published document. This document (with that reference number included) must also be made available to all via the internet.
Subsequent references to that document (whether rebuttals or enhancements) will always refer to that original identifier (and may also carry along their own addendum identifier) and should otherwise be treated the same as the original document. (If response/rebuttal is initially hard copy to the published periodical it should also, with its addendum # be placed on the internet.)
The point of this is prevent questionable research from inadvertently being recognized as valid by facilitating easy access to all subsequent feedback related to that document.
Absolutely agree with the article. Journals are really a relic of the 1970’s . Time to undercut them. Universities and/or government could ensure all papers are published and apply pressure.. Probably the most important check that none of the useless journals do is an scientific ethics such as is the data available.
Perhaps this issue belongs under “peer review”, although it is more akin to to “No Pressure”.
Michaels, Patrick J. 2011. Climategate II: An Open Letter to the Director of the National Center for Atmospheric Research. Opinion. Cato Institute: Commentary. December 2. http://www.cato.org/pub_display.php?pub_id=13906
_________________________
“No Pressure”: “Green” turns Totalitarian. Just joking, of course.
thanks for citing my blog so generously. i’ve consolidated these ideas in a paper that just came out here: http://www.frontiersin.org/computational_neuroscience/10.3389/fncom.2012.00079/abstract
it’s part of a collection of visions for the future of scientific publishing we edited in frontiers in comp neurosci: http://www.frontiersin.org/Computational_Neuroscience/researchtopics/Beyond_open_access_visions_for/137
I am truly thankful to the holder of this website who has shared this enormous paragraph at at this time.
It is appropriate time to make some plans for the
future and it is time to be happy. I have read this post and if I could
I wish to suggest you some interesting things or advice.
Maybe you could write next articles referring
to this article. I want to read even more things about it!
The blade is extremely tough and robust and the
nonstick plastic handles provide for a secure grip when cutting.
Residential clients looking to light their own home landscapes can determine the required wattage in
their order by following the same simple rule of thumb that commercial
clients of RLLD follow in choosing transformers for outdoor lighting.
STHOF has taken knife manufacturing to a new level.
My personal favorite is cilantro because it really adds a
kick to the dish but you can also use parsley, chives, mint and basil.
It usually grows indoors and is a result of years of selective
breeding. Others have skin healing properties, such as lavender and vitamin E.