Questions on research integrity and scientific responsibility: Part II

I have found no information about this Project online, the IAC website does not have any current activities posted since November 2011 (I urged the committee to make information online at their website).  Hence I am not making available here any further information about this effort, and will focus on my own contribution.  I note the purview of this group is all of science (not just climate science).

In addition to questions listed on the previous post, I was also asked to respond to the following questions:

What do you believe are the 3 most important core values of research? Do you believe that these values are widely shared among scientists across disciplines and countries?

How are the new tools of communication influencing the conduct of research; how are they impacting integrity? Looking ahead, what changes in the research environment do you anticipate over the next 10 years that will affect research integrity? What will be the most difficult challenges?

What are the 3-4 most important actions that could be taken by researchers, institutions, sponsors, journals, societies, and other stakeholders to foster research integrity and discourage misconduct, particularly in the international context?

The text of my presentation is appended at the end of the post.

My discussion with the Committee

I spent 1 hour with the committee via skype (from a hotel room in New Orleans while attending the Annual Meeting of the American Meteorological Society).  I was asked about a dozen questions, I highlight two of them here (paraphrased based on my recollection, with my response (embellished somewhat).

What is your summary message for the Committee?

[see my presentation below for context].  I’ve focused my remarks on the topic of politicized science, or science that is relevant to controversial policies.  From the perspective of the individual scientist, the challenges are more to integrity and responsibility, than to actual research misconduct (e.g. fabrication, falsification). And these challenges can be exacerbated or even be caused by institutions (e.g. govt agencies, UN, professional societies), who are interested in protecting reputations, funding, policies, and/or industrial practices.  I hope that the committee will tackle this more complex problem, which goes well beyond the traditional concerns with fabrication and falsification.

What is the responsibility of scientists in communicating to the public risks that may be related to their research?

To maintain integrity, scientists absolutely have the responsibility to present their research in context of background knowledge and areas of ignorance and disagreement.  With regards to public risks, individual scientists or even particular disciplines, may not have the background knowledge or understanding of societal risks.  So I do not see it as an obligation of scientists to communicate public risks associated with their research.  However, if a scientist or community of scientist does identify a possible risk, it is responsible behavior for these scientists to engage with health professionals or engineers or whoever is appropriate to clarify and better understand the risk.  Alarmism about possible risks is as irresponsible as hiding risks.  The key element is to put the science out there in the appropriate context of uncertainty and possible risks, and let other experts and the public assess the risk.  Public proclamation by scientists about societal risks, without adequate providing a context for the uncertainty in both the science and  the connections of the science to the risks, is not responsible behavior.

—

JC’s presentation

The text from my ppt slides is provided below.

Research Integrity & Scientific Responsibility

A perspective from a climate researcher

Judith Curry

Georgia Institute of Technology

26 January 2012

—

Most important core values of research

Honesty:  avoiding fabrication, falsification, plagiarism

Integrity:  how we deal with uncertainty and disagreement

Responsibility and accountability:  providing objective, impartial, unbiased advice to decision makers

Public trust in science requires honesty, integrity, responsibility

Credibility = trust  X expertise

—

Integrity:  Richard Feynman Cargo Cult Science

“I’m talking about a specific, extra type of integrity that is not lying, but bending over backwards to show how you are maybe wrong”

“Details that could throw doubt on your interpretation must be given, if you know them. . . the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.”

Integrity failures:

• deliberately mischaracterizing inconvenient arguments
• inappropriate generalization
• misuse of facts and uncertainties
• hidden value judgments
• selectively omitting inconvenient results

—

Different challenges for different fields

Pure science (e.g. high energy physics)

• Disputes: purely scientific

Applied/regulatory science (e.g. environmental regulation, pharmaceuticals)

• Honesty, integrity, responsibility
• Disputes: industry/economics versus public health/safety

Politicized science (e.g. climate change, health care)

• Integrity and responsibility
• Disputes: values, economics, winners/losers, complexity, etc.

—

Institutional problems:  Protecting reputations, funding, policies & industry practices

universities

• insufficient oversight of individual honesty; insufficient concern and education about integrity and responsibility

professional societies

• position statements on controversial, policy relevant topics
• journals with biased editorial practices

national academies

• concerns about bias, impartiality, objectivity

national governments and funding agencies

• targeting research areas and picking “winners” to support government policies

international institutions (e.g. UN assessments; IPCC)

• impact of politics on narrow framing of complex scientific issues for scientific assessments
• stacking of advisory boards to support an agenda
• lack of conflict of interest oversight
• insufficient independent oversight
• insufficient uncertainty assessment and management
• insufficient transparency and traceability of conclusions
• fundamental conflict between integrity and consensus building

—

Impact of new tools of communication:  Open Knowledge

Enables:

• Open access to published journal articles
• Online access to data (all levels, including metadata)
• Online access to model code and documentation
• Development of extended peer communities (public auditing) around issues of regulatory or political relevance
• Scientists engaging with the public via social media.
• More diverse sources of information available to the public

Opportunities for scientists:

• public engagement
• public education
• open knowledge
• networking

Challenges to responsible conduct of scientists:

• authoritatively writing on topics outside personal areas of expertise
• conflict of values in discussing politicized science
• concerns about policy advocacy by scientists
• dealing with “noise” and disagreement

—

Solutions for Politicized Science: Accountability                                  (I)  Best practices from regulatory science & engineering

Data quality management and assessment

• accuracy & precision; data cleansing, integrity validation
• information quality, data quality assessment

Independent model verification and validation

• complete documentation, publicly available

Uncertainty assessment and management

• identify areas and types of uncertainty, ambiguity, ignorance
• sensitivity assessment, uncertainty propagation, expert elicitation
• test for reasoning flaws

—

Solutions for Politicized Science: Accountability                                  (II)  Open access to scientific data, models & publications

All scientific materials upon which regulations & policies are based should be publicly available and easily accessible (online)

• Open (internet) access to published journal articles
• Online access to data (all levels, including metadata)
• Online access to model code and documentation

—

Solutions for Politicized Science: Integrity                                          (III)  No explicit consensus building in assessments

Scientific consensus building is not part of the scientific method; it is most often invoked in popular or political debates.

Perils of an explicit consensus building process:

• Explicit consensus building processes can enforce overconfidence  and belief polarization.
• Consensus beliefs tend to serve as agents in their own confirmation
• Dismissal of skepticism is detrimental to scientific progress
• Disagreement is marginalized and hidden
• Overreliance on expert judgment motivates shortcuts in        reasoning and hidden biases

Should all raw data being made available be a pre-condition of publication?

YES.   As part of the publication agreement, authors should sign a statement that they will make their data available upon request, during the review process and for a specified period post publication.

Impossible to enforce something like this with all journals;  making data publicly available is more logically enforced by the funding agencies.

Climate science based on historical data, which should be freely available.  The key issue is how to overcome the problems of historical data sets; the data manipulation and metadata are of essential importance.

Is it responsible and ethical to hide research articles, based on publicly funded data, behind paywalls, as in the case of journals like Nature and Science, which are often financially prohibitive even to universities, let alone those in developing countries?

Scientific journals need to be economically viable to exist.

National and international assessments on policy relevant science should only use journal articles that are available in the public domain.  Such a requirement would influence journal policies towards making articles publicly available after a certain time period (< 6 months).

What are the research ethics and scientific responsibilities of the open science movement, versus what is done behind paywalls?

Let freedom reign on the internet.  Authors of published journal articles should have the right to post their published articles (at least in manuscript form) on the internet.

How can we address the challenge “aggregating” uncertainty across many researchers and many disciplines: each narrow subspeciality may have different techniques and customs. 

Characterizing uncertainty and reasoning about uncertainty is one of the biggest challenges in science.  Inadequate attention has been paid to this issue.

Most important is that all factors that contribute to the uncertainties are reported openly, allowing others to judge the level of uncertainties and the impact on the conclusions.

What is the responsible use of grey literature (including blog posts) by scientists, particularly in scientific assessments?

If grey literature is perceived to be a very important element in a scientific assessment, we should be asking whether this topic is appropriate for scientific assessment.  In the case of the IPCC WG II Report, the answer to this question might very well be no.

Public policy is influenced by a broad range of information, including grey literature and blog posts.  There is unquestionably some valuable information in the grey literature.

The use of grey material should always be qualified and shortcomings of the material adequately acknowledged.

What is responsible behavior of scientists in balancing the challenges of rights of free speech and political activism? 

Scientists have the same right as everyone else to try to influence policy. It is ultimately the task of the decision makers to decide how to weight each piece of information, and who it comes from.

Getting involved in advocacy is the choice of an individual scientist.  The choice to become an advocate brings risks to the scientist’s credibility and raises concerns about scientific bias.  Any reason to doubt the individual’s honesty can be very detrimental to the scientist’s credibility.

Advocacy by scientific institutions (e.g. professional societies; IPCC) is a much bigger problem, and can be very confusing to decision makers.

JC comments

Kudos to the IAC for taking on these issues.

I would like to thank those of you who provided thoughtful responses on the Part I thread, I read these carefully and used some of the sentiments that were expressed on the blog.

I look forward to a continuing dialogue on this topic, particularly as more information about the IAC project becomes available.

p.s.  The reason I was invited to engage with the IAC Project is that one of the Committee members is a regular reader of this blog (!)