by Judith Curry
Earlier today via email, I received the following list of questions on research integrity and scientific responsibility.
The source of the questions and the reasons for asking them is an interesting story in itself, that I will share with you later next week. Here are paraphrased versions on the questions, which I am throwing open for discussion at Climate Etc. Note the context for the questions is science broadly defined, although the topic of climate comes up specifically in some of the questions. I look forward to your responses to these questions; I would appreciate it if you refer to the question numbers in your response.
1. In the case of the climate models, have all the relevant raw data and code, particularly that which has been publicly funded, been placed in the public domain, in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists? Should all raw data being made available be a pre-condition of publication? Should there any case for exceptions to this, beyond the protection of the privacy, confidentiality and anonymity of human subjects?
2. Is it responsible and ethical to hide research articles, based on publicly funded data, behind paywalls, as in the case of journals like Nature and Science, which are often financially prohibitive even to rich-country universities, let alone those in developing countries? What are the research ethics and scientific responsibilities of the open science movement, versus what is done behind paywalls?
3. How can we address the challenge “aggregating” uncertainty across many researchers and many disciplines: each narrow subspeciality may have different techniques and customs. How are uncertainty estimates to be reached across disciplines and how are the decisions in that regard to be reported?
4. What is the responsible use of grey literature (including blog posts) by scientists, particularly in scientific assessments?
5. What is responsible behavior of scientists in balancing the challenges of rights of free speech and political activism? How can the public and policy makers distinguish between a scientist speaking with the credibility of a subspecialist in a narrow field, and those extending that to the use of the phrase “science says” about very widely ranging matters of political economy in which they are not specialists but activists?
6. Should very large scientific operations arriving at public policy be obliged to have all their decisions about important scientific matters publicly traceable in writing in the way that every engineering decision in the building of a nuclear power station is written down and signed off and logged by two responsible officers, with all participants’ inputs being recorded? With regards to the IPCC, does the IAC recommendation go far enough?
7. How is the internet changing communicating about science? What is the way forward to conducting ethical and responsible scientific conversations in the new media, noting that it gets faster, technically better and continually changes?
8. What are the ethics and responsibility of scientists with regards to press releases and other announcements of scientific findings before peer review and before peer reviewed publication?
9. Is freedom of scientific research unduly impeded by the bureaucratic demands regarding documentation and other proof of compliance with scientific responsibility and ethical rules and funders’ requirements?
Unfortunately, Professor Curry, Climategate emails and documents released in late November 2009 were only the tip of a “dirty” little secret that had grown out of sight in government-funded research since ~1971.
My web site [http://myprofile.cos.com/manuelo09] vanished. Here’s a link to the “Deep Roots of the Global Climate Scandal (1971-2011)”:
http://dl.dropbox.com/u/10640850/20110722_Climategate_Roots.doc
I think I found it.
1. No. Relevant raw data and code for climate models, even that which has been publicly funded, have NOT been placed in the public domain.
2. No. It is irresponsible and unethical to hide research articles, based on publicly funded data, behind paywalls, as in the case of journals like Nature and Science.
3. The federal funding agency and the US NAS should be held accountable for failing to insist that researchers address the challenge of “aggregating” uncertainty instead of encouraging researchers to downplay uncertainty and promote official dogma as scientific fact.
4. Grey literature includes blog posts? If so, it functions to bypass official gatekeepers of scientific knowledge (federal funding agency and the US NAS) when they hide new findings that are of public concern and no threat to national security: E.g., Neutron-repulsion in the Sun’s core, its Fe-rich mantle, and its production of H as He as a cloud of waste products.
5. These issues cannot be solved if leaders of the scientific community promote the same false pride and blind arrogance as leaders of nations and religious organizations. Humility is the “eye of the needle” through which reality can be accessed by spirituality, experimental science, or a combination of experimentation, observation, contemplation, meditation, prayer, etc. Without humility and reverence, scientists, religionists and politicians cannot properly play their roles on the stage of life:
http://www.artofeurope.com/shakespeare/sha9.htm
6. No. More bureaucracy isn’t the answer. Funding agencies should strictly limit most research funding to individual investigators, and retain a few retired scientists to actively seek any indication of poor adherence to basic principles of science in large scientific operations, like CERN, the IPCC, Fermilab, Brookhaven, Los Alamos, Sudbury Neutrino Observatory.
7. The internet is like the release valve on a pressure cooker. Politicians would be very foolish to remove or further restrict the pressure that has been vented by that valve since evidence of mismanagement in science began to accumulate.
8. My view is to “give them enough rope to hang themselves” in press releases, pre-publications, or peer-reviewed papers in prestigious journals like PNAS, PRS, Nature, Science, etc. Reviewers need only caution author(s) that publication might damage their reputation(s).
9. Freedom of scientific research is NOT and should not be unduly impeded by bureaucratic demands regarding documentation and other proof of compliance with scientific responsibility, ethical rules, etc.
Conclusions (for Speed 18 Jan 2012 @1:47 pm):
Arrogance blinds the leaders of nations, religions and
scientific organizations from seeing the Great Reality.
Humility, awe and reverence are required to see the
Great Reality that surrounds and sustains our lives:
1. The energy source is recorded in atomic rest masses
2. Einstein explained the conversion (E = mc^2) process
3. Mass => Energy, as water flows into the valley of the
4. “Cradle of the Nuclides” shown on the front cover*
5. Of a symposium Dr. Glenn T. Seaborg organized
_ http://en.wikipedia.org/wiki/Glenn_T._Seaborg
*http://www.amazon.com/Origin-Elements-Solar-System-Implications/dp/0306465620
The alliance of consensus scientists and propaganda artists that world leaders hired as modern-day “fortune-tellers” have simply confirmed Earth’s climate is cyclic:
1970s: Cooling – Another Ice Age Coming
1990s: Global Warming – CO2-induced
_Now: Global Climate Change for sure
Like breathing, and the expansion and contraction of sea ice:
http://noconsensus.wordpress.com/2012/01/17/full-length-nsidc-sea-ice-data/
1> Should all raw data being made available be a pre-condition of publication?
Yes, if the data was obtained with public funds and if it does not threaten national security.
Everything necessary for replication must be in the public sphere any time the findings of a study are used to inform public policy. One would hope that publications understand that basic necessity. If publications do not, governments should. Depriving people of their lives, liberty or property without giving them the right to question or cross-examine the evidence used to do so is a violation of their basic human rights. It should be regarded as standard due process.
Herman, why would the source of funding matter for publication of data used in a scientific paper? I daresay, given a few months of work on the jargon, I, or anyone else, could come up with a credible scientific paper suggesting that, say, that the regular ingestion of bark from a silver birch, can cure the common cold in 12 hours, but that the ability to extract the health giving part of the bark and make a suitable medicine from it would take 20 years. I could quote surveys and experiments I’d done and would have to be taken at my word if the funding for the research came from a private source because there’d be no need to produce my source data. No, whoever funds the work a scientist should always make the source data for his/her/their conclusions available for scrutiny.
They all seem to have a common underlying theme: money and access to resources.
So the question is: are we living in the nightmarish world described by President Eisenhower almost exactly 51 years ago, not the military industrial complex, but the “domination of the nation’s scholars by Federal employment” risking to make society captive of a “scientific-technological elite”?
http://omniclimate.wordpress.com/2011/01/05/the-ike-nobody-mentions/
I think many people already enslave themselves under the umbrella of that “elite”. So the ethical issues are invisible to them. In the meanwhile every public policy discussion is cloaked by hypocritical “scientific” remarks and especially so when it’s public health that is at stake.
Thanks for your excellent summary of President Eisenhower ‘s warning in Jan 1961 that a federal “scientific-technological elite” might one day hold our society captive. That portion of his speech is here:
http://www.youtube.com/watch?v=GOLld5PR4ts
World leaders were so frightened by the Cuban Missile Crisis in Oct 1962 that they eventually adopted (~1971) a plan to use a federal “scientific-technological elite” to avoid the threat of mutual nuclear annihilation, by uniting nations against an imaginary “common enemy” – Global Climate Change.
That conclusion was based primarily on attempts by federal funding agencies and the Geophysics Division of the US National Academy of Sciences to manipulate, hide or avoid experimental evidence after 1972 that the Earth’s heat source – the Sun – is the remains of the supernova that made our elements and ejected them five billion years (5 Gyr) ago [See above link to Climategate Roots, 1971-2011].
Will you be submitting a written response before giving testimony?
Should all raw data being made available be a pre-condition of publication?
The answer to this is yes and why would anyone in the climate dealing with raw data such as temperature readings and tree ring proxies etc not wish to publish the raw data?
Don’t publish and be damned or ignored?
Stacey, you seem to be laboring under the delusion that all scientists are under the control of whatever country you belong to. Try to imagine if you can that they aren’t. How do you propose to enforce extracting the raw data from countries not under your control? Bomb them to kingdom come if they don’t comply?
What kind of bomb would you suggest starting with? A few conventional bombs as a shot across their bow, or just pulverize them with a tactical nuclear weapon so you can barge in and grab all their climate data?
You talk like a typical one-world imperialist.
Hey Jude
I got in early, please don’t let your blog be hijacked?
+1.
Although it is a) just what happens b) sometimes interesting..
1. No. Yes. Data should go public a few months after publication.
2. No. Open access is enlightened ipso facto
3. Don’t mix apples with pears, just don’t pretend uncertainties disappear. Compute using min, max, expected values
4. Grey lit is good as long as conflicts of interests are declared (and its content is defensible)
5. Activist scientists are good as long as open about it and they accept it means Activism takes precedence over science. They should also show a.grasp of history of their science.
6. Controls have to be proportional to the required changes in policy. In hindsight the IEA was a joke.
7. Paper journals and especially expensive ones should be a thing of.the past already. We will slowly go back to the pre-peer-review era, all inertia being evidence many follow a herd rather than be scientists truly
8. Independent auditing groups within universities should double check if the press releases are correct and especially not overreaching. Just like we have auditors checking the financial results contain the right figures.
9. Funders’ requirements, certainty. Other proofs of compliance are not bureaucracy any more than one’s annual tax declaration is.
Sorry of course I meant the IAC was a joke
“2. Is it responsible and ethical to hide research articles, based on publicly funded data, behind paywalls, as in the case of journals like Nature and Science, which are often financially prohibitive even to rich-country universities, let alone those in developing countries?”
This is not a question for the scientists – this is a question for the journals.
Journals are financial entities that have overhead costs that have to be met. The paywall is supposed to be the equivalent of buying the printed journal, and it’s purpose is to fund the journal so it can continue to do it’s work.
Until an alternate business model is shown to be viable, journals will continue to use paywalls to ensure they stay financial.
The question for the scientists is whether, in addition to publishing in a journal that uses a paywall, should they also make the paper freely available, such as by a personal website/blog? Or, alternatively, should the government provide a site for posting publicly funded research articles so they are freely available without needing to pay to access them?
Good points, Graeme.
But it’s not just paywalls on the internet. Academic libraries pay significant fees to publishers such as Elsevier and Kluwer for publications of research that has, in many cases, been paid for with public funds.
An interesting read on this topic:
http://www.monbiot.com/2011/08/29/the-lairds-of-learning/
Spot on G. You can tell from the framing of this (although It’s Prof Curry paraphrasing so something could have been lost in translation) that the questioner is a partisan about access to data without appreciating any publishers’ business imperative.
1. A) “have” they? I don’t know. B) Yes C) No
2. I don’t think this issue rises to the level of ethics. It is certainly possible to get paywalled articles if you don’t have a subscription. It would be ideal if paywall didn’t exist, but I don’t think it’s an ethical issue.
3. Only response is to have the process (including the actual happenings, as opposed to generic process) fully documented. Could be many ways.
4. Gray literature should probably not be used in scientific assessments such as IPCC reports. If it’s that important, it needs to be published.
5. A) Basically, don’t say the science says this if it doesn’t say this. There are many shades of gray and it won’t always be possible to call. Consider that even in court, it’s up to the judge or jury to decide. B) Intuition? Everyone can be fooled, but lots of people and probably a higher % of policy makers, probably have a good sense.
6. A) Yes B) No
7. Don’t see much difference on a grand scale, but it will affect particularly how academic scientists are regarded for their career performance.
8. Scientists should not issue pre-publication press releases. Again, shades of gray, but the most ethical thing would be to release for “extended peer review” as freely as possible, save the media blitz for publication.
9) No. In particularly, dealing with a university IRB in my experience, no. FWIW some industries are pretty pathetic at this too.
9) “…dealing with a university IRB..” I second that emotion. There’s a phony advertised homogeneity of IRB practice in that many online IRB FAQs (and similar documents) sound similar, almost identical. Legal departments cause this herding together of the public statement of IRB procedures. In fact, the actual practices of IRBs are extraordinarily heterogeneous across universities, to the point where you begin to think caprice and total lack of institutional memory characterize IRB behavior.
Sorry if you serve on an IRB. I am sure everyone means well but after fighting a silly IRB for 18 years I am sooo delighted to have a sensible, practical and unsilly one now.
NW – I’m not on an IRB, just have dealt with one. I’m not sure how much we’re agreeing – except perhaps that one can’t make a blanket statement about “bureaucratic demands requiring documentation” because of the personalities involved.
2. Is it responsible and ethical to hide research articles…
Not if these were funded by the taxpayer.
The answer to all of your questions have this at the core. If I pay for it, it’s mine. There is no reason a journal should profit on my money. Let them profit on their own accord e.g. maybe they offer access to tons of private funded info, and if so, that’s fine. But for what I paid for? No.
1. a) ? b) yes, if it concerns policy decisions, c) yes, to protect the intellectual property right of the researchers, but full disclosure must be made available if referenced for policy making purposes.
2. Yes, but if reference in policy making decisions, free access to all requests should be granted.
3. Comparison of methods to determine uncertainty for each element and the final results. No result can be more accurate than the most uncertain element. “Novel” methods to reduce uncertainty must be traceable to standard methods.
4. It should be used carefully and noted as such with the text, not as a cryptic footnote.
5. a) Publicly declare their advocacy. b) Science has to be presented as science. What is “says” is open to interpretation. “Science says…” should be reserved for game show hosts :)
6. a) That is up to the judgement of the country financing the research. b) The rules should be as strict as required by any government involved in the process.
7. a) It stimulates ethical behavior, light of day. b) Get used to it. It is an evolving process.
8. To announce their intentions. If it is an attempt for expanded prepublication peer review, fine. If it is an attempt to toot ones own horn, fine. If it is an attempt to beat competitors to the punch, fine. If it is an attempt to influence policy, fine. We are all grown up.
9. LOL, of course! That’s is part of the job requirements. If you can’t follow the rules, find softer funding.
Your mother told you – tell the truth. If any of the above conflict with your mother, don’t do them. And telling the truth doesn’t involve saying as little as possible, or speaking in a way that can’t be proven to be untrue.
And of course every math(s) teacher you ever had told you the same thing – show your work!
1. (a) Yes, all data and code in the public domain.
1. (b) Publications identified as research, or proof of concept, or alpha / beta pre-release of production code do not require complete documentation in the peer-reviewed literature. There should be Laboratory reports available, however.
1. ( c ) No exceptions what so ever for production-grade calculations that are used for decision support.
While many claim that sufficient documentation is available in the public domain, I have not yet been shown GCM documentation at the level of detail that is considered SOP for all other activities that affect the health and safety of the public.
2. No, it is not responsible. All aspects of publicly-funded work need to be available to the owners, which is the public. This requirement is easily met by all other research activities: simply check the Web sites, and libraries and public libraries, of National Labs everywhere.
3. No response.
4. Literature that is a part of decision support cannot be grey literature.
5. Scientists should not be associated with political activism that is based on their work. Politics brings disrespect to science.
6. (a) Yes. Why should there be any exceptions at all? It is not necessary to invoke building a nuclear power station: such requirements are SOP for all sorts of activities that impact health and safety of the public.
6. (b) I do not know the IAC recommendations.
7. The internet can be highly effectively used to provide an interface between the public and all aspects of the decision-support activities. The Yucca Mountain Project provides a good example.
8. Scientists should not be associated, in any way, with press releases. Neither before or after peer review publication.
9. No: based on over 35 years of direct experience in meeting extensive documentation and demonstration of compliance requirements. Thousands of people in highly technical areas perform these duties on an on-going daily basis. Why should there be any exceptions for any activities that affect the health and safety of the public.
1. It would be ideal that all information is rapidly available in the public domain, but there are reasons to accept somewhat less, e.g. to allow some rights to limit the availability temporarily. One reason is that the motivation for tedious data collection would be reduced without some protections. The other and perhaps more important is that demanding immediate availability of the data may lead to significant delays of publication as scientists would use such a delay as a way of protecting the data. There are similarities with problems that are well known in other areas of intellectual rights and patents.
2. We must ask, whether journals like Nature and Science are needed. If the conclusion is that they are, then they must be given the possibility of surviving. My conclusion is that they should be accepted, but it’s quite possible that they will gradually lose their status as other ways of publication develop and take over.
3. Perhaps the first thing to notice is that perfect or even nearly perfect solutions don’t exist. Aggregating uncertainties is fundamentally difficult. Most important is that all factors that contribute to the uncertainties are reported openly and well. That allows others to judge the level of uncertainties of the results.
4. The Internet has changed the landscape. A wide spectrum of publication channels has been created. How they will develop further is very difficult to judge. It may be better to be open and let the matter to settle by “natural selection”.
5. Scientists may choose their own balances. They learn soon that they lose on one side, when they weight the other more. Overemphasizing openness may help in being strong in both worlds. Any minor reason to doubt honesty is very detrimental for those, who try being both.
6. The requirement is not realistic. Science is not engineering and suffers seriously from attempts to force engineering standards for all aspects of science. (They are, however, perfectly applicable for the bulk of practical scientific work.) Openness is again a more important part of the solution. It’s the task of the decision makers to proceed from that. IAC was not fully realistic in it’s assessment of possibilities of presenting uncertainties. Thus they proposed practicies that cannot lead to the results they seemed to think.
7. Some of that is discussed already in 4.
8. That’s an area that I have often not been happy with. The press releases have often been misleading simplifying results and implying importance that’s not there. The press releases should not precede public availability of the paper, but in some cases it may be fully justified that both are done before peer review has been done (not always).
9. That must be rare, but on the other hand bureaucratic requirements should be avoided, when their basis is not totally clear.
1. Obviously not. But it’s somewhat irrelevent, as the models are so bad that no sceptic would bother trying to replicate them other than to see all the assumptions and fudge-factors that are used.
2. In the Internet Age it is irritating that much information is hidden behind paywalls. However, I do understand that these journals need to make a buck. Though I’m sure the journals will be redundant before too long, as the internet takes over.
3. No idea.
4. The issue is not whether the information is from a “grey” source. The issue is the standard of scientific integrity used in coming up with the information. There are many many many awful papers in the scientific literature. I’m assuming that the person who posed this question was thinking about the IPCC. There’s a lot of rubbish quoted by the IPCC. Some of it comes from the “grey” literature.
5. I think this is a major thing that will fall out of the global warming debate over the next few years. The only thing of any great interest to come from a scientist is their science. All the rest is of no more importance than the views of your bin man, or you manicurist. This is a lesson for us all, I think.
6. Being written down and logged isn’t the issue. Being entirely open and public, that’s the issue. I don’t want to trust an auditor’s opinion that everything is OK. I just want to be able to see it for myself. As for the IAC, no idea.
7.The internet has changed everything. Vast numbers of people know whats in the climategate emails, for instance, even though the media won’t touch it. This is a truly vast step forward. For the rest of the question – see Richard Betts’ comments in the previous post.
8. Odd question. I think there are serious issues about the press releases that accompany scientific papers. But whether they are pre-publication or post-publication seems to grant too much importance to the publication process.
9. No.
1) Science in part is reproducibility. Provide all the assumptions, data, code whatever so that someone else can replicate what has been done. Then and only then can one be sure that what is being stated is supported.
2) Paywalls have emerged in the era where journals have been steered away from loading up on advertising which in an of itself, co-opted some of what could and could not be printed. Paywalls should have time limits; 3 months after print publications is my guess.
3) Aggregated uncertainty arises when one tries to lump disparate disciplines together; just like lumping tree rings with thermometers; one has to take the largest uncertainty and apply to the aggregate. You have to have a very detailed plan and rationale to lump things together. Lumping should be a very rare event just because of the uncertainties.
4) NO grey/gray literature in a scientific assessment. Take the grey to… the grey literature. This is the visceral appeal of which Schmidt is so fond.
5 The Supreme Court already has decided (by not deciding) that one should follow the money. The employer dictates what use the product can be applied outside of the work environment. Health care systems are already not hiring cigarette smokers; making all health workers in their system to be vaccinated against the flu; etc. In the case of the Hockey Team, the Feds are paying someone to lobby for or against some statute or regulation. You want to be a climate junkie? either retire or find yourself another employer, you are certainly free to do that.
6) Transparency is nice. If you don’t want to show your poker hand after calling the other players, no money. If you want to say, “I am making a decision based largely upon political considerations;” fine, just don’t expect to be making other political decisions in the near future.
7) The internet has and will make science a whole lot better because of more knowledgable, diverse and intense scrutiny; faster too. Just don’t expect “breakthroughs” at a more rapid pace, they are few and far between. The internet ultimately will make us all “smarter”; i.e., more cognizant of nuances we had not considered, which, may make some of us less willing to rush to judgement.
8) Press releases are meant to “get out in front” of an issue to co-opt the talking points with the talking heads. Bad idea. There is no place for prominent retractions, much less interest in the general public sphere.
9) When scientists use the money from one grant to fund portions of another project or develop a separate “pilot project data”, the funding agency gets annoyed although tacitly stays mum. Living from one hard grant money to another leads to all sorts of sleight of hand maneuvers and “shades of gray”. Just like breakthroughs, not every good idea gets positive results, so results and continued funding can be rare events. Careful research is a deliberate process, and we have to eat while we are researching. Along with the hands needed to set up the equipment, wash the test tubes, transfer raw data to a usable format, money for administrative costs has to come from somewhere. Are research universities too big? and if they downsize, will tenure lapse back to the one dead body at a time paradigm? I don’t know. I do know that integrity and transparency resides with the researcher. Fortunately, thinking don’t cost much.
The easy way to deal with the hiding of data is to write a clause in federal contracts that for anyone to be eligible for federal contracts that must agree to release ALL of their data. An additional clause could add that eligibility would require the release of all of their past data underlying climate change studies. Seeing how many people are proposing very expensive solutions to climate change, I can’t possibly see any reasoned opposition to such clauses. The Federal government routinely has similar clauses that condition eligibility for work on compliance with government objectives in the EEOC area.
JD
The easy way to deal with the hiding of data is to write a clause in federal contracts that for anyone to be eligible for federal contracts that must agree to release ALL of their data.
You wouldn’t by any chance be one of the authors of the SOPA bill? You sure talk like one.
How do you propose dealing with the commonly arising situation that “their data” (that of the scientists) is not really theirs?
Climate today is a global issue that involves many countries, on whom scientists are dependent for their data. These countries often don’t grant a free license to further distribute data they make available. How do you propose enforcing the clause that all data be released when the data was obtained subject to not being freely released?
The easy way to deal with the hiding of data by any such country is to invade that country. If they won’t release their climate data, clearly they’re using it as part of their WMD program and therefore must be invaded. Stands to reason, right? The US can’t just sit on its hands when some country is ramping up WMD. To save such a country you have to destroy it.
Vaughn P “How do you propose dealing with the commonly arising situation that “their data” (that of the scientists) is not really theirs?”
What “commonly arising situation that data is not theirs?” Please identify. I have seen many articles stating that the CRU falsely relied on this reasoning to evade FOIA law. For example Wikipedia states: “In a decision announced on 27 July 2011 the Information Commissioner’s Office (ICO) required release of raw data even though permissions had not been obtained or in one instance had been refused, and on 27 July 2011 CRU announced release of the raw instrumental data not already in the public domain, with the exception of Poland which was outside the area covered by the FOIA request.” See link http://en.wikipedia.org/wiki/Freedom_of_Information_requests_to_the_Climatic_Research_Unit Even if there were restrictions on the release of climate data that is not a reason to fund institutions using the data because the integrity of the data would always be at issue.
Your irrationality and unfamiliarity with the facts is confirmed by your ridiculous references to invading other countries and the totally irrelevant Sopa bill.
You ask if “In the case of the climate models, have all the relevant raw data and code, particularly that which has been publicly funded, been placed in the public domain, in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists?”
Speak to John Christy and Roy Spencer. They still have not released the code for their TLT product, one, Eli might add, which has been shown to have major errors in the past by examining the results, but, alas, not the code.
Wiley Wabbit – you seem hung up on one specific case rather than the general question, so let me re-word it for you:
“In the case of the climate models (in addition to the work of Spencer and Christy) have all the relevant raw data and code, particularly that which has been publicly funded, been placed in the public domain, in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists?”
Oh come on Max, don’t be such an a-hole. You know the answer is yes. What are you claiming is still being withheld, other than data from countries that haven’t granted access to their data?
Switzerland apparently is willing to trade for climate data using unclaimed money in their bank faults.
Seriously, I have asked Max about this before, and being neutral, he doesn’t respond.
Eli,
If you examine ONLY what suits your own extremism; honest proofs can be ridiculed. Unless a person is interested in the truth – can be as sleaze manipulator as you. Having lots of skeletons in your cupboards, for supporting extreme Warmist, can that justify in jumping on any chance to manipulate the truth, without doing your homework first?. Should others be lied to for you not doing your homework. Should it be re-established the rule; when blaming somebody unjustly – public apology should follow?
Boy, talk about content-free. Can you say anything coherent at all?
Eli Rabbet
On 99,99999999% of the planet’s surface area the temperature is not monitored. 2] unlike on the moon, on the earth the temperature distribution is 3 dimensional. There is no such a thing as ”raw data” all data is cooked for misleading. Data collected in the selective few places – should never be refereed as ‘GLOBAL” Antarctic / Greenland are large areas as USA; but is monitored on couple of places the temperature. Same as New York + Boston temperature to tell for the whole of USA temp for the day.
Lots of warm winds go to Antarctic – shrink instantly – that air becomes much heavier per volume – because of that, the earth’s centrifugal force picks that very cold air and blows all over Oceania / south Pacific. In mid South Pacific temperature instantly drops by 3-4-6C for few days; then they stop – air get warmer by few degrees; NOT A SINGLE THERMOMETER IN MID SOUTH PACIFIC – 10 time larger area than Europe / USA combined (where most of monitoring for IPCC is) 3] satellite takes OCCASIONALLY temperature photo TWO DIMENSIONAL – in a 50km thick troposphere. Winds in the air never stop and change the temperature every 10-15 minutes – instead they send a balloon ones a month…?!
All temperature data is SKILFULLY cooked; for B/S Merchants as Eli Rabbet and Steven Mosher, to degrade completely the naive Skeptic’s brains. The B/S they sell to the ignorant Skeptics should be on the label; how many kilojoules per shovelful – that top quality B/S makes the Skeptics fat and obese.
Prof. Curry writes: Is it responsible and ethical to hide research articles, based on publicly funded data, behind paywalls, as in the case of journals like Nature and Science, which are often financially prohibitive even to rich-country universities, let alone those in developing countries?
From this Eli concludes that Prof. Curry is a socialist who hates our successful capitalist system with private companies making profits.
Eli
I would reckon that each article I write ciosts me around $200 as there are alweays a number of papers that appear key and need to be bought if there is to be a full complement of current research material from which conclusions can be reached.
It is a very rare day that the pay wall material add anything to what is available free.
tonyb
Eli, these aren’t my questions, they are questions I received in the email
satire.
….and clearly not aimed at you Judith.
Eli,
You wwraskly wrabbit, it is not capitalism to take publicly funded research and selectively give it to insiders or favored persons and allow them to profit from the tax payer’s treasure.
it is not capitalism to take publicly funded research and selectively give it to insiders or favored persons and allow them to profit from the tax payer’s treasure.
Aha, someone on Judith’s blog who knows what capitalism is not. How many on this blog know what it is? (I don’t so I’m all ears.)
Dr. Pratt,
You make an excellent pointin your question. “Capitalism” is a fairly ambiguous term.
The Wiki definition, “There is no consensus on the precise definition nor on how the term should be used as a historical category.[2] There is general agreement that elements of capitalism include private ownership of the means of production, creation of goods or services for profit or income, the accumulation of capital, competitive markets, voluntary exchange and wage labor.[3][4] The designation is applied to a variety of historical cases, varying in time, geography, politics and culture.[5]”
http://en.wikipedia.org/wiki/Capitalism
Is a fairly good definition. Many people would add to that the concept of private risk. IOW, taking public assets (publicly funded research) and allowing select persons to profit from the public effort at no cost to the select person, and to even force the people who paid for the work product to pay again to review it would be an abuse.Since even socialist states use principles of capitlism to justify state expenditures (more or less), perhaps more emphasis needs to be made about the source of the capital and the control of the capital in the definition.
I don’t know it this helps or not. It does lead me to re-state the point which you responded to, but I am pressured to get some real work done, so perhaps later I can find this thread and add a clarification.
Regards,
Hmm, Perhaps Hunter would like to have a conversation with Jamie Dimon about whether it is not capitalism to take public money and selectively give it to insiders or favored persons and allow them to profit from the tax payer’s treasure. Ben Bernanke is another you might ask
Eli.
please re read the assignment. Your effort to say something snide and insulting has gotten in the way of your scholarship.
Joshua, perhaps you would like to comment on Eli’s abilities here. Go ahead, unload on him.
please re-read the comment.
Eli has been eating fermented turnips again.
Eli –
You like to tease. However, I have noticed a mellowing of late. There have been numerous comments directed at skeptics which have been helpful, insightful and humorous – and without the slightest tinge of contempt. That knack for funny, nice jokes clearly reveals that rabetts are not rodents after all.
bi2hs
The ”Bunny droppings” on Eli’s blog turned into dysentery, when he proudly was trying to humiliate me and to deceive the people. Eli Rabbet single handedly tried to silence the ”CLIMATEGATE” Tried to cover the stench with his own bigger stench. Apology is well overdue, shame, shame Eli Rodent
Nah, Rabett are lagomorphs, anonymice are the cute rodents. Stefan OTOH, is no denier, but amusingly confused.
Eli Rabbet no, no, no, you are the confused one; because this time you got the carrot, from the other end. Ridiculing me is easy, because of my limited English vocabulary. But ridiculing me is; ridiculing the truth. I don’t go on maybe, it’s possible; but 100% proofs. Eli Rabbet, I’m an open target. You must have being on my website looking for dirt – instead you have found 24 carats PROOFS that: GLOBAL warming is 100% lie. 2] human can change the climate, for better or for worse. Because H2O changes the climate, on many different ways; not CO2
Warmist / Skeptics believing that GLOBAL warming is possible – they don’t know all the proofs I have. People believing that; human cannot change the climate are ”the common sense deficient people” I call them ”the Ian Plimer’s Smarties” Al Gore will give his kingdom for a packet of those Smarties. Therefore, unless one can understand that the phony GLOBAL warming and the constant climatic changes are two COMPLETELY unrelated, that person is wasting his life consuming only B/S. in his diet
Stefan, you appear to need to read up on the ideal gas law:) Somehow you think that methane molecules attach to others (BTW, a lot of the other stuff is scrubbed out at the well, water vapor and CO2 decrease the heating value, hydrogen sulfide is pipe and people unfriendly and the other hydrocarbons are too valuable to be burned)
http://www.naturalgas.org/naturalgas/processing_ng.asp
But no dear Stefan, the gas molecules are not stuck together.
Eli Rabbet, you are changing the frequency again. I didn’t talk about fraking gas of old methane. New methane is produced together with other substances b] same as H2O molecule attaches itself to dust > makes dust to be heavier and drops down – fresh methane doesn’t come out of a placenta animals by itself, then belch / fart separately the smelly bit. have you ever heard of ” molecular adhesion”. ?! Pure methane, molecule disperse; but as ”fresh” methane is lucky to have the smelly bit – sinks in the ground.
For how long you can dismiss that: creation of extra new methane, prevents depletion of free oxygen in the atmosphere?! It is ALL proven. You used an of-cut to ridicule me on your blog – didn’t inform me as right to reply – was informed by somebody about you gossiping unfairly. When somebody that has read the whole text. on my blog wrote a comment that: Stefan deserve recognition; you stopped comments.
Would be fair for you to paste the whole text ( with improved English – your English is brilliant) then we are even. It’s your credibility. Plus, if you have any respect for people that comment there – they should have the rest of the story. Many of them must be good people, you mislead them. When they hear the truth from somebody else – your credibility will be zero. P.s, another page next to the page on ”METHANEGATE” I made ”the creation of crude oil’ should be read after METHANEGATE as an appendix. They are very related; it’s in the world interest, in your interest to know what’s in both – you will get my respect, you will respect yourself. Please stop comparing apples with old methane, yes fresh methane can be made from apples, with the adhesiveness attached to other impurity. People don’t need to go to Antarctic to find out that fresh methane with its impurity stinks. Have some dignity, be a good sport. .
1. In experimental science, raw data is collected from scratch by other researchers. Climate science based on historical data needs to agree on some principles which overcome the problem of historical data sets – yes these data should be freely available.
2. The traditional model of publication requiring payment is acceptable. Get access to a university library if you need access to published papers. Often one can find copies of papers or prepublication drafts on the net.
3. Good science reduces uncertainty. Politics corrupts science. Publication of bad science appears to trump quiet good science. (had a long discussion with a scientist about this last week)
4. All scientific sources have different weightings and can be used appropriately. The gold standard is a good repeatable experiment, climate science is troubled in this regard – claiming experiments are impossible and being detached from experimental physics.
5. It is the media’s job to present balance rather than bias. Sadly this is not the case in the mainstream science media.
6. Present methods seem acceptable. Bias in IPCC output is evident to anyone reading the complete docemtation. Iac has documented the procedural problems. Push for change iscurrently lacking media traction.
7. The Internet makes possible discussions like this. We are more likely to discover bad science and be able to correct it more quicklt with more brains on board.
8. Media is full of bad science announcements because perception drives share prices. This will not go away while we have a stock market.
9. Scientific research is limited by access to funds. Publication volume rather than quality drives ongoing funding.
4.
2.
The gold standard is a good repeatable experiment, climate science is troubled in this regard – claiming experiments are impossible and being detached from experimental physics.
Too bad you didn’t attend the technical sessions of the annual Fall Meeting of the American Geophysical Union. No one could come away from those sessions with that impression.
The Internet makes possible discussions like this. We are more likely to discover bad science and be able to correct it more quicklt with more brains on board.
Now you’re contradicting yourself. First you ask for experiments, then you propose doing (“correcting’) science by debating it in the modern-day counterpart of the 17th century coffeehouse. You can’t conduct climate science by debating it, you have to get out and observe the climate.
1. In the case of the climate models, have all the relevant raw data and code, particularly that which has been publicly funded, been placed in the public domain, in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists? Should all raw data being made available be a pre-condition of publication? Should there any case for exceptions to this, beyond the protection of the privacy, confidentiality and anonymity of human subjects?
General Circulation Models have no inherent capability of either incorporating the effect (if any) from increased atmospheric CO2 concwentration, or converting their output in energy flux units of W/m^2 to temperature.
If GCM’s are used as evidence for public policy decisions it is an absolute necessity that the parameter used to input the effect from increased CO2 be both fully documented and validated to a scientific standard.
This has not been done and the inmput parameter for CO2 is clearly false because the models predict 0.782W/m^2 decrease in OLR for the increase in CO2 over the past three decades but the satellite measurement of OLR shows a 2W/m^2 increase in OLR refuting the model projection both in sign and amplitude.
There is also no established conversion for model output forcing to temperature with “climate sensitivity” values all over the map. Since Climate Models are the sole basis for the current climate change policies before any policy is made on climate change initiatives climate sensitivity must be established and verified to a scientific standard.
This has not been done which is why the GCM predictions of warming for the past decade based on increased CO2 concentration have not materialized with the past decade of global cooling and for some peculiar reason in spite of the continued cooling with increasing CO2 emissions; climate change policies are still being based on GCM outputs all of which have been completely wrong.
–> 5. What is responsible behavior of scientists in balancing the challenges of rights of free speech and political activism? How can the public and policy makers distinguish between a scientist speaking with the credibility of a subspecialist in a narrow field, and those extending that to the use of the phrase “science says” about very widely ranging matters of political economy in which they are not specialists but activists?” ~Walter Starck
“The leading scientific prophets of this cult are overwhelmingly comprised of young researchers whose entire careers are based on climate alarmism. In contrast, the middleground, balanced (“sceptical”) scientists are overwhelmingly researchers with well established expertise in other fields. The alarmists repeatedly refer to a catechism of highly selective evidence to support their claims. The sceptics cite voluminous other evidence from their own varied fields which contradicts the alarmist’s claims.
For GISS and NCAR at least, the model code has been available for awhile. The public GISS code is updated every day from the central code repository.
How can people above who criticize the model development process not know this? Why haven’t they looked at the code themselves? Are they lazy? Is it too hard for them? More fun to make stuff up on blogs?
1. a) Don’t know (ask Steve Mosher). b) Yes (all data should be made available) c) No exceptions (beyond those listed)
2. No. (hiding research articles)
3. More formal attribution studies that are traceable and consider all alternates; less reliance on “expert judgment” alone
4. No grey literature in summary reports
5. No political activism allowed by scientists; phrase “science says” should not be allowed as it implies “the science is settled” – could be replaced with “some scientific studies suggest”…
6. a) Yes (traceability required). b) No. IAC recommendation did not go far enough (was seen by many as simply a “white wash”)
7. Internet can bring openness and light – as long as it is not simply misused as a mouthpiece for selling a story
8. Questions on pre-review press release should include: what is value of peer review process as it stands and how could it be improved?
9. No (unless these bureaucratic rules are being misused to favor one particular “mainstream consensus” view).
max – you’re not going to be able to get around the “expert judgment” completely (sure you realize this) the key is the documentation including who said what. Maybe in a journal article it is ok to have anonymous expert elicitation. In a summary report for policy making it should not be.
Corollary to question 2: how many critics here have participated in the open review process at the EGU journals? If not, why not?
Follow-up: credit to Nabil Swedan for taking the time to post at ACPD on a Hansen article, whatever the merit of the argument.
1. No.
2. Absolutely, otherwise, it’s not science.
3. Yes.
Re 2. The American Meteorological Society sets a fine example. All their Journals (including Journal of Climate) are available online. There is a paywall; but all articles older than 2-years are available free for download.
This is more than a question of ethics. I have worked a lot with scientists from developing countries who simply have no financial base to subscribe to the major Journals. The American Meteorological Society “free after 2-years” decision has had an enormous impact on their ability to follow the literature.
1. This should be a general question, not restricted to climate models. All relevant data must be made available, and it should be a precondition of publication. Otherwise it’s not science. As for code, I think it’s sufficient to make the logic available. That still enables verification. Very simple example (NB. very simple example): If I publish the result of a least-squares fit then it is sufficient for me to state that it was a least-squares fit. I do not need to provide the computer code for doing that calculation.
2. Basically, no. But the financial realities of journals do need to be taken into account, and there may be scope for some compromise. For example, payment gets instant access, free access comes later. An external example of that in practice would be a 20-second delay on free stockmarket prices. Obviously “later” can’t mean after a millenium.
3. Standards, or implicit standards, develop over time. After a while, work that does not follow the developed standards is not acceptable. Not a good idea to push (legalise) too hard too early.
4. That is developing too.
5. The public and policy-makers often can’t distinguish, initially. First, opponents speak out. After a while, the public gets it, then the policy-makers (in a democracy) eventually have to follow. (“I am their leader, I must follow them.” – Jim Hacker)
6. Yes.
7. The internet is adding accountability when the traditional processes fail. It may go a lot further, for example replacing (some of) the traditional processes. Let it develop, and wait for problems to become apparent before trying to fix them.
8. None, really, as statements made before formal publication lack the credibility that publication provides, and can be seen to be non-scientific. There is, though, the formal ethic that the journal’s (or other place of publication) policy and guidelines as accepted by the scientist in order to publish, must be adhered to. If they don’t like the rules, publish somewhere else.
9. No.
20-minute, not 20-second.
“If I publish the result of a least-squares fit then it is sufficient for me to state that it was a least-squares fit. I do not need to provide the computer code for doing that calculation.”
The “logic” in texts does not go far enough. It is not what is sufficient for you, it is what is sufficient to reproduce your results. If it is cnecessary andlear from the text, then the code makes checking your work all the easier. In short there is no reason NOT to provide the code. Its no extra work if the work is properly done to begin with. Further it eliminates problems that arise when people read descriptions differently.
Yes, what you say is correct. But I was testing the idea that code should always be supplied and was trying to illustrate in not too many words that actual code wasn’t always needed. As you say, “it is what is sufficient to reproduce your results”.
and so many scientists don’t write their own code, not talking about GCMs, but more mundane articles analyzing specific datasets, usually they tell you what software and technique was used, so the key is in the raw data in that case.
My answer above should have b een 1a, 1b, and 1c
My answer to 8 is that while I do think it’s manipulative to try science by press release, the fault here is really with the journalists, who are so lazy they publish press releases.
This is a ridiculous exercise. Each of these questions addresses a large, complex policy issue, so there are no simple answers. Most are already the subject of extensive debate. Most imply problems that probably are not important, yet they suggest major restructuring of basic established scientific practices. Yes, no, or a few sentences are meaningless answers.
The questions come from a rather important organization. I have been asked to comment on these issues. Not a straightforward or simple assignment.
Judith,
Don’t go. Take the fifth. Send josh in your place. He is expendable.
1. Just to play Devils advocate, I don’t think it is necessary to *automatically* post every jot of info into the public domain, as long as it can be retrieved and shown if requested. Most people do not have the time or motive to go through piles of data to check a paper.
However, everything used should be logged and filed, so that later questions can be answered. Publically-funded science should be treated like a tax return, with the possibility of being audited. By audited, I mean ‘show your workings and evidence in full’. If someone can’t produce the data, they fail audit. Btw, sorry to mention tax returns.
Why is there any problem with this? If a scientist has spent months on a paper, wouldn’t a simple sense of propriety suggest he should keep all the gumph he used filed away safely for future reference? This would have saved Steve McIntyre, Ross McKitrick and others years of frustration, and we still don’t know whether some of Phil Jones’ papers reflected the raw data because, apparently, the data went walkabouts.
Geniuses can get away with “I reached this conclusion. I don’t know how I did it. It just came to me, and it seems to be right”. Mann, Jones, and the rest of us have to provide a bit more by way of evidence.
1. In the case of the climate models, have all the relevant raw data and code, particularly that which has been publicly funded, been placed in the public domain, in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists? Should all raw data being made available be a pre-condition of publication? Should there any case for exceptions to this, beyond the protection of the privacy, confidentiality and anonymity of human subjects?
Answer: some of the models have their code available. I don’t believe that all GCMs used by the IPCC have their code in the public domain. “relevant” “raw’ data is subjective and calls for a conclusion about what is meant by relevant and raw. Their operational definition is better. All code and data required to reproduce results is more definitive and verifiable.
Availability of raw data: All data should be made available prior to publication. Exceptions: None.
One issue here is the dependence upon data that is not in the public domain.
The best example I have is Landscan population data. Its used by some as an input to analysis but is not publically available. the other issue is how
public domain is defined. is registration required? what is the license? can it be redistributed? Victoria Stodden is your best source here.
2. Is it responsible and ethical to hide research articles, based on publicly funded data, behind paywalls, as in the case of journals like Nature and Science, which are often financially prohibitive even to rich-country universities, let alone those in developing countries? What are the research ethics and scientific responsibilities of the open science movement, versus what is done behind paywalls?
#########
Research paid for with public dollars should not be behind paywalls for an indefinite period.
4. Grey literature is fine provided it is clearly indicated in the text and conflicts of interest are identified.
6. Traceability, yes. Does the iAC go far enough? no
more later
Let’s not over-look the last part of the first question: ” . . . in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists?”
That requirement demands a degree of documentation of the model specifications that cannot ever be attained in peer-reviewed journal articles.
And in many cases, the code itself is widely available. That so few model-bashers bother to look at it speaks to their true motivation.
of the dozen and a half or so GCMs how many have their code available.
if you say “all’ or “all except the Kazakh one” then everyone on this blog will stop bashing them.
On Question 2 (paywalls):
I hear the bleats about publishers needing to make profit, but they are charging high amounts for tax-funded information (aka rent-seeking)
An INTENDED consequence of this is that hard information is deliberately and constantly denied to the public domain
If I was to subscribe to these multifarious journals (yet another line of defence – “you can buy these issues”), I would be the best-informed bankrupt in the country. Access to a library which stocks these journals is not possible for many millions of people, yet the pay-walled articles are used as a battering ram to reduce their aspirations to improved standards of living
Abuse of power IS an ethical issue
Paywalls are a fact of life. Before the internet journals were available in printed form only, and paid subscriptions were required to access them. It takes resources to set up and manage a scientific journal, even one with electronic access. Perhaps papers based on research conducted with government money should be made publicly available e.g. in an archive operated by the funding agency or by the agency paying the publisher to make certain papers free to the public.
I agree that papers based on data analysis should have the data and algorithms made available. A good example is the Annals of Applied Statistics, which strongly encourages authors to do so, and provides server space for this purpose. Their policy is explained at http://imstat.org/aoas/mansub.html. For example, the paper by McShane and Wyner on temperature reconstructions (2011) has an archived zip-file over 38 MB which contains their data and algorithms. With cheap storage being the norm there really is no excuse not to make this mandatory.
While I tend to agree with David W in that any serious response to the issues covered in these questions would be impossible in this context, my initial thoughts are:
1. Publicly funded research implies that all raw data and code should be placed in the public domain as soon as practically possible, subject to protection of human subjects used in any research program. The enforcement of any rules that may emerge from the examination of this subjects would be difficult.
2. Paywalls are a neccesary part of commercial life and provide for the future viability of sources that provide valuable information. The viability of Wikipedia, for example, is known to be at serious risk due to lack of a sufficient funding model. The fact remains that the articles are available if you pay for them and that it would be unreasonable for all interested persons to expect to get access for free.
3. Uncertainty needs to be clearly addressed in each and every scientific paper that is published, especially the degree of confidence to which any conclusions that are made should be held by the reader. Specific reference should be made to opposing views and some discussion of the degree to which such opposing views may need to be taken into account by any lay reader.
4, The use of material that has not been the subject of appropriate peer review and/or verification and validation processes should always be qualified and the shortcomings of this material adequately acknowledged.
5. Specialists in any field should be assumed to be qualified to express an informed opinion in that field alone. It is inappropriate for scientists from other fields to express any opinions as an “expert” in an unrelated field. The separation of scientific facts from political opinion has always necessarily been the task of the reader, but more particularly for any reviewer of such work prior to publication.
6, Every scientific organisation (large or small) that is involved in public policy decision-making should have transparency in their internal workings so that the decision-making body might seek to clarify and examine the bases of the recommendations that have been put before them.
7. The internet is certainly opening up science and its workings to a more critical if not neccesarily informed analyses. There are many problems attached to having anyone with an axe to grind being able to hijack any debate but again its for the reader to spot and reject such “noise”.
8. Nothing should be published as a scientific body of work until peer review and verification and validation processes have taken place. Otherwise the publication of such work can only be considered as grey literature, without much credence.
9. Freedom of scientific research that is publically funded must neccesarily be constrained so that the public interest can be maximised.
“The fact remains that the articles are available if you pay for them and that it would be unreasonable for all interested persons to expect to get access for free.”
Completely disagree – and that’s another fact of life, chum. Now bleat on. The cost of these papers is simply beyond the means of most people and this is INTENDED
Address the issue. Using hard information made unattainable through deliberate costliness to determine people’s lives is an unethical abuse of power
What evidence do you have on the average cost of articles or that average paywall costs are excessive? My experience has been that these costs are usually in the tens of dollars.
Peter Davies said “tens of dollars”.
I would say $25 – $30 per research article and the problem is that you don’t have a clue to the quality ahead of time.
A large research organization has a blanket subscription, but the individual citizen scientist gets to drain their wallet.
I have paid out $25 a handful of times and got a decent payback once or twice.
So I pay out money for information and a network connection, I synthesize the info, and you get to view the results for free. That is the wonder of open access publishing.
WHT says that $25 – $30 was his experience and this agrees with mine. I find that a free version of any article usually appears somewhere after a while, often appearing in blogs such as Climate Etc in the case of climate articles of importance.
ianI8888 seems to have an agenda about paywalls in general and his statements remind me of that of some of our “conspiracy” sceptic friends who “bleat on” (his choice of words) about their perceived behaviour of mainstream climate scientists.
The paywall dustup hence seems more like left wing freeloader rhetoric and it did seem that Dr Curry was begging this question when she formulated question 2 for comment.
I’m surprised by the number of you who blast through (9) with a dismissive “no.” A recurrent theme is that you deal with compliance with “rules” and why shouldn’t everyone else do the same?
P.J. O’Rourke once pointed out that legal rules need to be specific, clear and predictable: If they are vague, unclear and capricious, this is like having parents, not law.
Where the “rules” are vague federal guidelines administered locally by tens of thousands of IRBs at their whim, with zero institutional memory, “compliance” becomes a bad and maddening joke.
These are excellent questions, and if I wasn’t packing up to go away for a few days I would respond to your request. By the time I get back, I am sure that my responses will have been articulated by others!
Don
A somewhat controversial take on question 6:
“How much openness should be required for scientific organizations informing policy ?”
Little. Scientific organizations should never inform policy. If the subject matter has been explored in sufficient detail that it can inform policy, it is no longer research and instead enters the applied realm. The engineering staff that picks up the results of the work and beats it into applied form with standard and known techniques will make the policy suggestions in an objective format.
If the engineering review can replicate your work and you are capable of actually addressing direct engineering comment concerning your techniques, your work might actually be ready for primetime. Openness would go a long way in helping the engineering staff actually understand what you are doing but since as a scientist you should be concerned with research not the applied effects of your results you have no business in the policy realm in the first place.
On the other hand, if your results require only your special handpicked proxies and only your “special” statistical techniques, you have some work to do before you even present to the engineering team.
There’s much truth in that. Scientific bodies are not organized in a way that makes them capable of informing properly on controversial issues, and it’s easy to get the same advice from other sources, when the issue is not controversial. I feel that it’s often detrimental to the bodies to be directly involved in informing policy.
(It’s, however, natural also for scientific bodies to try to convince others of the importance of their science. In a competitive world, it may be justified that scientific bodies inform on their views on science policy and perhaps educational policy issues. All bodies are allowed to inform on their direct interests, when they are open on that they are discussing their own direct interests.)
That doesn’t mean that scientists should not inform policy. They may try to do it in their own name as individuals or as groups, which don’t pretend to represent anybody else than the members of the group. Scientists have the same right of trying to influence policy as everyone else. Views of specialists of some particular field are often given more weight than views of laymen, but it’s the task of the users of the information to decide what weight they give to each piece of information.
Questions on research integrity and scientific responsibility
On the HOCKEY STICK
http://foia2011.org/index.php?id=4804
What does publishing a hockeystick that is not robust say about integrity and scientific responsibility?
1- AFAIK no, all the data or the detials of the models ahnd adjustments have not been released and yes, they would be- especially if used in any way, shape or form for policy decisions.
2-Interesting. I’m not sure whether it’s a question of ethics, or more a practical legal issue. Technically, (at least in the UK iirc) if research was done on the public pound, and it’s palced behind paywall you can FOI it.
Any research that is publically funded should be available, for free, by the public.
3- difficult. Especially in the context of climate science. The pragmatic route is to take the highest uncertainty for the entire ensemble and use that as the uncertainty for the whole theory- though i think that’s probably ‘unfair’ in this instance.
Individual assessments, including weightings on each subsets importance wrt the theory, tied back to the uncertainties and then perhaps using the method above (though modified as just stated) could work- but i’d hate to be the one setting it up!
4- Grey literature in ‘official’ documents shouldn’t be allowed. period. Blogs can be used as an open ‘review’ source review though- i can see that becomming more useful and practical.
5-You can not be an objective scientist AND an activist at the same time. The two are mutually exclusive as one wholly compromises the other. Any scientist in an official capacity must cease work on any aspect they turn to activism on. It’s a conflict of interest of the worst kind and it amazes me that it’s allowed.
6-Yes absolutely. This is a must.
7-See #5- open review (or at least initial open review stages), imo, is the way forward.
8-CLearly, until a piece of work is ‘approved’ or ‘reviewed’ no press rleeases should happen. Again, it’s akin to biasing a jury prior to a trial and a conflict of interest (arguably can be placed under activism).
9-No. Industry have to operate under these kind of guidelines, it’s about time academia caught up and joined the real world.
Remember- the VAST majority of all science on this planet is carried out by industry NOT academia. All these questions are clearly slanted towards an academic viewpoint completely dismissing the larger industrial one.
1. No doubt here. Authors of publications arising from publicly funded research must as a pre-condition of publication supply all relevant data and code to a publicly accessible archive prior to publication.
2. Again, no doubt here. Initial publication of material arising from publicly funded research must be confined to journals that do not operate a paywall policy.
Somebody called Vaughan has entered some kind of mental loop…if you have something to say you can just say it once, uh?
How do you propose to enforce extracting the raw data from countries not under your control?: simple. Secret data cannot be used for public scientific articles. IOW If they don’t allow you to publish their data, you don’t use their data.
Otherwise anybody could publish any article claiming anything on the basis of “withheld” data from Elbonia.
Of course, then we’d have ‘skeptics’ crying – this isn’t gobal, you didn’t include data from Elbonia!!
Congratulations for not getting my point at all
Oh I did, just that it was ridiculous.
Anybody can reply “It’s ridiculous” to anything. Especially people with no argument at all.
“Otherwise anybody could publish any article claiming anything on the basis of “withheld” data from Elbonia.”
How’s that for ‘no argument at all’? Just pure baseless assertion.
If you publish an article based on raw data and then refuse to provide me the raw data, how can I know if your article is based (a) on flawed computations, or (b) on a misread of the raw data, or even (c) on a manipulation of the original raw data, or finally (d) on “raw data” that didn’t actually exist?
I know English Majors are very popular in certain scientific circles nowadays, still I believe in the absolute importance of Maths.
Some reply to the above saying, I could ask for the same raw data from the original source. But that is not an answer: in the example, I don’t want to check if the data they provide is correct, I want to check if the data and computation you used are correct. Because even with the original raw data, if I can’t replicate your findings I’d be left completely in the dark about the reasons for that.
“show me the raw data”
they cry.
Mosher used to be hung up on the ‘raw data’, but even he got over it.
Maybe there is some kind of support group?
Michael – you still have nothing to say of any relevance to this blog, Vaughan’s comments, or even my comments. Congratulations again.
How’s this – the vast majority of data and code is freely available. What isn’t is a tiny portion.
The data whiners cry over the latter and do SFA with the former.
No need to repeat yourself Michael. I understood already you have not understood at all what the point is.
Vaughan raised the issue of what to do with raw data that the original source is unwilling to publish. I replied that if the source is unwilling to publish the original data, then no scientific article can be published either about that data.
In the meanwhile you lose yourself in comments about Mosher and “he vast majority of data and code is freely available”. Two points that have zero to do with this exchange.
Having worked all day, shopped, cooked, cleaned up, communicated with family, showered, fed the cat and cleaned his box and read posts for 3 hours, I’m too lazy to properly answer these questions or check if further reading is required to do so. I’ll just copy LabMunkey’s answers.
Science and scientists have an enduring but unwritten contract with society.
Society supports them in a quite lavish style compared to the level of the average individual in our society and in doing so, unlike any other sector of society, also gives scientists and science an almost unlimited and unfettered entitlement to explore whatever they may to the limits of the human mind and human comprehension.
In return as a part of that unwritten contract, society expects and demands that science and it’s practitioners create the advances that will take our civilisation to new and ever higher levels of knowledge and provide the ability to apply that knowledge to mankind’s ultimate betterment.
Science and scientists in their responses and actions should and must be made to follow the accepted and legally binding contractual frameworks that exist in all other sphere’s of our society between a contractor ie; the public and society, and the contracted ie; scientists.
From this everything else should and must follow or any breakdown in that contract / agreement will eventually destroy much of science and eventually impoverish society.
Many of the public today are increasingly feeling that science and the scientist’s obligations to the public and society, their paymasters and contractor, are no longer being fully honored.
Judith,
Looks like fingers are about to be slapped!
Question?
Do you think programs like “Mythbusters” have any scientific credibility?
If you do, have I got a story for you on “consensus” gravity and a car dimpled that managed to get 11% more efficiency.
There is a nice essay by Paul Uhlir in DSJ entitled Information Gulags, Intellectual Straightjackets, and Memory Holes: which refrarmes and reposes the arguments in a more reduced way that is both humerous and constrains the information hoarders.eg
I refer to the first of these problems as “information gulags.” By this, I mean the incarceration of large numbers of data resources in dark repositories, to be manipulated and viewed only by their masters. Such inaccessible warehouses of information, which have also been referred to as data cemeteries
http://www.jstage.jst.go.jp/article/dsj/9/0/ES1/_pdf
Hello,
In regard to the first question only: if a result isn’t reproducible, it should not be accepted. Public funding is irrelevant. If the result is dependent on computer models, the result is not reproducible without the all input data and the code. If only the algorithm is given and someone makes is unable to reproduce the results then the only way to tell who is correct is to examine the code. Subtle round-off errors can occur in the implementation of an algorithm and the only way to find them is to examine the code.
In Unix systems common and it is not difficult to create an archive (“tar ball”) of all files used at time of running the program which will include a Makefile that will produce the results. The archive should also contain a README file containing the version numbers of all data sets (if available) and code and the date of the run.jj
If certain files are confidential and omitted from the archive the README file should indicate where the file came from so others may seek the file. In this case the results should be provisionally accepted since the results are not fully reproducible.
It used to be that ability to reproduce results was paramount in science. Results from computer runs are reproducible, so why is the question even asked?
klee12
“Should all raw data being made available be a pre-condition of publication? ” is hopelessly naive. While there are data sets that are compact, some experiments cast off gbytes of incomprehensible garbage every hour. Most of that gets boiled down to a few parameters. Gonna save the gbytes bucky? Archiving this as raw data is like being given a hard drive out of which the directory has been erased. Well, you have the data, go to it.
Eli, you make a good point. If the authors reduce the raw data to a smaller set to support their analyses then this should be made available. If an interested party still wants the raw data they should be able to get it, gigabyes and all, directly from the authors with the former paying the expense of having it provided. Exceptions should apply to protect sensitive or classified data. Generally, authors who claim insights from data that they cannot make available to others should expect that the status of their work will be reduced. The alternative is to base scientific understanding on faith in individuals’ handling of data that others cannot see.
It’s worthwhile to remember that the possibility of making data easily available is very recent. Even lesser amounts of data that could have been made available were kept out of public. That didn’t stop the progress of science.
The wider availability of data is not essential for progress, but the possibilities opened up by Internet should be taken advantage of as that’ll help in speeding up the process. Most important is to get rid of old prejudices against openness, using regulations to achieve result will certainly lead also to unwanted side effects like delays in publication of important results or tricks to circumvent the formal requirements.
Pekka, until recent times data sets usually were much smaller than they are now. It was not unusual to see data sets produced in their entirely in scientific papers. As data sets grew, so did the taste for gaining new insights by mining large data sets. Publication of data sets has become impractical. Something else that has changed in parallel with this development is an emerging positivism whereby insights gained from data become the theory. So I disagree that science can progress without others being able to access data and replicate analyses. The alternative is to base our concept of scientific progress on faith in individual persons. I find that to be unacceptable.
Much of the raw data from natural sources that we collect is so driven by disorder that we can use information theory techniques to compress it into a parameterized function. This is similar to maintaining only a mean or variance, but is more general since the statistics don’t have to follow a normal distribution. The result of this is that the huge amounts of data can be reduced into concise representations, with which you can recreate the data (albeit synthetic) using sampling techniques such as Monte Carlo.
The ultimate amount of compression one can achieve with information modeling techniques is many orders of magnitude.
“Should all raw data being made available be a pre-condition of publication? ”
I agree that the answer is no. But all data used in the computer run should be made available (to the extent possible) so that the results may be reproduced.
klee12
1-4 no answer
5 takes a long answer – one that I may or may not get around to
6 (2 questions) a – ABSOLUTELY; b – no
7 no answer
8 Responsibilities huge! Probably, researchers don’t always control or have say in what goes out in press release. As far as ethics – stupid question. Published science papers should always be objective (ethics should not be an issue). Once objectivity is lost (which would probably occur b4 publication), the press release is a priori subjective.
9 resounding NO
In response to question 1: I do not believe that climate models have been placed in the public domain adequately. The EPA GHG endangerment finding regulatory impact analysis did not provide sufficient information. Environmental regulatory analysis requires a higher standard than traditional peer-reviewed publications because the results have policy implications.. Particularly in the case of publicly funded research, all the relevant raw data and code should be placed in the public domain, in order that independent verification and replication of the models and the policy resulting from them might be made by other scientists. If the analysis is used to justify environmental regulations such as GHG limits then the raw data should be made available as part of the regulatory impact analysis. In this context there are very few for exceptions to this, beyond the protection of confidentiality and anonymity of human subjects in health studies.
In response to question 2: My career has been in the environmental field. If you want to use an analysis to justify regulations then the results have to be available. As a result, I am not comfortable putting those research articles that form the basis of regulations behind a paywall. The open science movement should be encouraged and public funding should generally include that as a prerequisite.
Question 3 is a difficult one. For air pollution regulations, EPA has established the Clean Air Scientific Advisory Committee which ostensibly tries to address uncertainty across different disciplines. I might add that the membership of this kind of advisory committee has to be picked carefully so that “tribal influences” do not lead to the situation where everyone is from the same background and point of view. In the case of GHG regulations you would need to deliberately include skeptics on a “GHG policy scientific advisory committee”.
Question 4 is a problem in environmental science. Too often a grey literature analysis is introduced by press release and picked up by the media without caveats so that the general public accepts the analysis as fact. Therefore, grey literature should clearly be labeled as such in whatever application it is used.
Question 5 blurs the distinction between scientist’s rights and the development of public policy. Policy makers should get advice tailored to their application. My preference is that the model of establishing an advisory committee carefully chosen so that you get both sides of the story be used to address the needs of the policy makers. That leaves the scientists free to do whatever they want to do. Unfortunately, EPA’s reliance on the IPCC documents which incorporate too many advocacy “science says” pronouncements is fully at odds with an independent advisory committee approach. I should also point out that the Inspector General determined failed the White House peer review requirements for its finding that GHGs endanger public health and welfare.
Question 6: I think that my preference for a GHG policy scientific advisory committee is a better approach than “nuclear grade” documentation and would accomplish the same thing. “Nuclear grade” documentation is a disparaging term in the utility business because it generally means twice as much effort to document what is done than is really necessary. I have no comment on the IAC recommendations because I don’t really care about the IPCC process rather I think EPA should follow their own standards and not rely on an approach that does not meet this country’s peer review requirements.
Question 7: The internet is a boon to open transparent science. Make no mistake however, that this will take a change in current work practices because documenting work to the point that it can provided as standard practice will take more time and much more effort. In the long run, based on some unfortunate personal experiences with inadequate documentation, the result is better.
Question 8: I am personally opposed to science by press release and think that it should be discouraged. Very rarely does any scientist’s true description of scientific findings get communicated by a press release. Doing this before the peer review is inappropriate.
Question 9: Funder requirements should absolutely require documentation and other proof of compliance with scientific responsibility and ethical rules. It should be viewed the same way OSHA rules are regarded: it is a pain in the neck to comply with the rules but there is reason. It should be considered just part of the job. Those scientists in the ivory tower of a research institute or university who think it is an impediment should join the real world.
Judith: You may be interested in whether other fields require important data to be deposited upon publication. In biology, the 3D molecular structure of enzymes is established by X-ray crystallography, a process that can require years – and occasionally decades – of effort by multiple scientists. When pharmaceutical companies are interested in the 3D structure of a protein to aid in the discover of drugs that bind to that protein, the 3D-coordinates can be extremely valuable. Many years ago, investigators wanted to receive public funding for finding the structure of such proteins and get scientific credit (and more funding) by publishing such structures, but keep the actual coordinates secret so they could sell them to pharmaceutical companies. Journals wanted disclosure, but cheating was rampant. Today essential all biological journals demand that scientists deposit the coordinates of a protein structure in the Protein Data Bank WHEN SUBMITTING an article for publication. The same is true for sequences of genes. The passage below is from Science’s information for authors and is typical. Note that biologists – upon submitting a manuscript – are required to deposit information in a manner that guarantees public disclosure upon publication, but climate scientists are given far more flexibility. (Note: Scientists retain copyright to coordinates and sequences they deposit and can seek patents before publication. They can seek compensation from those who profit commercially from their work, but they have the burden of proof of proving that other misused their information.)
“Science supports the efforts of databases that aggregate published data for the use of the scientific community. Therefore, appropriate data sets (including microarray data, protein or DNA sequences, atomic coordinates or electron microscopy maps for macromolecular structures, and climate data) must be deposited in an approved database, and an accession number or a specific access address must be included in the published paper. We encourage compliance with MIBBI guidelines (Minimum Information for Biological and Biomedical Investigations).
Details include but are not limited to:
Molecular structure data. Atomic coordinates and structure factor files from x-ray structural studies or an ensemble of atomic coordinates from NMR structural studies must be deposited and released at the time of publication. Three-dimensional maps derived by electron microscopy and coordinate data derived from these maps must also be deposited. Approved databases are the Worldwide Protein Data Bank [through the Research Collaboratory for Structural Bioinformatics, Macromolecular Structure Database (MSD EMBL-EBI), or Protein Data Bank Japan], BioMag Res Bank, and Electron Microscopy Data Bank (MSD-EBI), and for synthetic molecules, the Cambridge Crystallographic Data Centre.
DNA and protein sequences. Approved databases are GenBank or other members of the International Nucleotide Sequence Database Collaboration (EMBL or DDBJ) and SWISS-PROT.
Microarray data. Data should be presented in MIAME-compliant standard format. Approved databases are Gene Expression Omnibus and ArrayExpress.
Climate data. Data should be archived in the NOAA climate repository or other public databases.”
Thanks frank this is useful info. can you provide a web link for the relevant Science policy? thx
Just the type of insight we need to fire up the deabte.
Thanks for the suggestions you are sharing on this weblog
Will you be submitting a written response before giving testimony?
Thanks, I will subscribe to the RSS feed. It’s a pleasure to read this kind of article.
Read fresh business and worldwide news today ! Knowledge is power!
Creation de site internet
Creation de site internet
I truly agree that very rarely a scientific discovery is ever published or bought to the notice by the press. It is qutie disheartening to know as the pain and the endless suffering that these scientist take for a discovery gets unnoticed.
Thanks for a wonderful and useful article
My website is
http://decorat1.blogspot.com/
its about interior design In Arabic
¨”Its always good to learn tips like you share for blog posting. As I just started posting comments for blog and facing problem of lots of rejections. I think your suggestion would be helpful for me. I will let you know if its work for me too.”
Steve McIntyre has insightful comments:
Thanks to sharing with us.
Well If you live in the Dayton Area Check https://daytondonations.com
Hi, Neat post. There is a problem with your site in web explorer, would check this?
IE nonetheless is the market chief and a big portion of folks will omit your excellent writing due to this problem.