by Judith Curry
As many have argued, rigorous scientific research requires dissent, or what Robert Merton called “organized skepticism”. Yet it is increasingly the case that some forms of dissent in pharmaceutical research are either absent or unheard. – Justin Biddle
I recently received an invitation to attend a philosophy of science workshop. As I was reading the list of invited participants, I spotted Justin Biddle, Georgia Tech, philosophy of science. !!! I was hitherto unaware of Justin or that Georgia Tech had hired a philosopher of science. Upon looking at his publications, there are many publications of relevance to topics being discussed at Climate Etc., notably some papers under review.
I picked the following paper for discussion here, which follows the previous thread on Importance of intellectual and political diversity in science. Below are some abstracts from Biddle’s paper.
Institutionalizing Dissent: A Proposal for an Adversarial System of Pharmaceutical Research
Justin Biddle
Abstract. There are serious problems with the way in which pharmaceutical research is currently practiced, many of which can be traced to the influence of commercial interests on research. One of the most significant is inadequate dissent, or organized skepticism. In order to ameliorate this problem, I develop a proposal that I call the “Adversarial Proceedings for the Evaluation of Pharmaceuticals,” to be instituted within a regulatory agency such as the Food and Drug Administration for the evaluation of controversial new drugs and controversial drugs already in the market. This proposal is an organizational one based upon the “science court” proposal by Arthur Kantrowitz in the 1960s and 1970s. The primary benefit of this system is its ability to institutionalize dissent, thereby ensuring that one set of interests does not dominate all others.
Excerpts:
Many now acknowledge that there are serious problems with the way in which pharmaceutical research is currently practiced. These problems include the suppression of undesirable results, bias in the design of studies and in the interpretation of results. These problems can be traced to the influence of commercial interests on research.
As many have argued, rigorous scientific research requires dissent, or what Robert Merton called “organized skepticism”. Yet it is increasingly the case that some forms of dissent in pharmaceutical research are either absent or unheard.
This proposal is an organizational one based upon the “science court” proposal by Arthur Kantrowitz in the 1960s and 1970s. The primary benefit of this system is its ability to institutionalize dissent, thereby ensuring that one set of interests does not dominate all others.
The perspective of this paper is that it does not presuppose that policy-relevant research such as clinical trials can be done in a completely value-free manner. A serious problem with current pharmaceutical research is bias, including bias in the choice of which hypotheses to investigate (and which to ignore), bias in the design of experiments, and bias in the interpretation of results.
Bias in experimental design is also a common form of bias in clinical research. There are at least four ways in which sponsors design trials in order to make preferred drugs look better than they really are. Rather, they are examples of “preference bias,” which occurs when the preferences of researchers unduly influence a study in such a way as to increase the likelihood of obtaining a desired result.
It might be tempting to explain these failures by appealing to the greed of individuals. A better explanation, however, appeals to the broader institutional environment in which individual scientists operate.
The appropriate response to this, in my view, is to design systems that institutionalize certain types of criticism and that, more generally, counteract the power of entities that have large financial stakes in the outcomes of research.
In 1967, Arthur Kantrowitz published an article in the journal Science entitled “Proposal for an Institution of Scientific Judgment,” in which he argued for the establishment of a “science court” where scientific questions that are relevant to public policy debates would be adjudicated. Emphasizing the growing entanglement of science and politics, Kantrowitz argued that we are increasingly forced to make consequential “mixed decisions,” or decisions that have both a scientific and a moral/political component. Whether it is the decision to build an atomic bomb or to enact policies to curb ozone depletion, we are increasingly confronted with science-based public policy decisions that have wide-ranging effects on the social and political landscape.
In a number of essays beginning in 1967, Kantrowitz argued that we do not have the appropriate organizational structures for making mixed decisions effectively. The primary reason for this, he argued, is that individual scientists engaged in cutting-edge research are almost always affected by various biases. For example, scientists who become involved in the policy-making process almost inevitably allow their moral and political beliefs to influence their appraisal of scientific hypotheses. Similarly, scientists who are immersed in researching a particular question for an extended period of time almost always develop cognitive prejudices, including preconceptions about the results of future experiments . In addition, the fact that mixed decisions must be made quickly, typically before a consensus is formed within the scientific community, makes it even more likely that such biases will affect individual scientists.
As argued, privatization is impeding the ability of communities to instantiate the norm of organized skepticism. Yet, as should be clear from the discussion of Kantrowitz’s proposal, one of the primary epistemic benefits of an adversarial system is the way in which it institutionalizes dissent. Given this, it is plausible to think that an adversarial system could help to alleviate some of the problems that privatization is causing. In particular, an adversarial system could help to expose the kinds of bias that are so often found in current pharmaceutical research, such as bias in the choice of hypotheses, bias in the interpretation of results, and bias in experimental design.
The adversarial system of pharmaceutical research that I will outline – Adversarial Proceedings for the Evaluation of Pharmaceuticals (APEP) – retains several of the features of Kantrowitz’s original science court proposal. Two groups of advocates would present arguments for a specific position, and a panel of judges would adjudicate between these two groups. The issues to be discussed should be issues that are controversial; in certain circumstances, there will be previously-existing evidence that is sufficient to close the controversy, but in general, the central questions will be underdetermined. The conclusions arrived at by the panel of judges should not be considered definitively true, but should rather be viewed as provisional and subject to change with the acquisition of further information.
Conclusions
I have argued that there are serious problems with the way in which pharmaceutical research is currently done – epistemic, moral, and socio-economic problems – and that an important cause of these problems is inadequate dissent. As a means of improving this situation, I have proposed that an adversarial system of pharmaceutical research, APEP, be instituted within a regulatory agency such as the FDA. The adversarial nature of APEP represents an acknowledgment that pharmaceutical companies, and the scientists that they sponsor, should not be viewed as disinterested arbiters of research but, rather, as advocates for particular hypotheses. APEP, in other words, is an institution that acknowledges that interests play an inevitable role in the evaluation of pharmaceutical research, and it ensures that the interests of pharmaceutical companies are not allowed free-reign but rather are checked by interests that are diametrically opposed to them. In this way, it enforces organized skepticism.
JC reflections
The analogy with climate science is that the dominant moral and political beliefs of climate scientists are introducing problems and biases into climate research. To use just one example, consider climate models and the bias in experimental design. Until very recently, driven by the UNFCCC/IPCC mandate, climate models have focused on the time series of temperature anomalies (not on getting the absolute temperatures correct), on anthropogenic forcing (with little attention paid to solar forcing and indirect effects), and on global average surface temperatures (not on regional variations or the disposition of heat in the ocean). Preference bias is evidenced in how climate information is displayed graphically – alternative graphical representations would send a different message.
One of the norms of science is organized skepticism. Those working at the climate science – policy interface (including the IPCC) have worked hard to kill organized skepticism by manufacturing a consensus on climate change. The idea of a climate red team has been put forward by John Christy. Kantrowitz and Biddle have thought through how institutionalizing dissent might actually work. Particularly for climate science, implementing something like this wouldn’t be simple, and actually achieving the desired objectives would be quite difficult.
But not impossible. The closest I’ve seen was the APS Workshop to consider its climate change statement. A committee of eminent physicists, each with no particular expertise in climate science or an apparent dog in the public debate, selected 6 scientists (Held, Santer, Collins, Curry, Lindzen, Christy) to address specific questions prepared by the committee. The committee has not completed its deliberations (I’m not expecting to hear anything from the committee before the end of the year) so the end of this story has not yet been written. But this is the kind of thing that is more likely than a manufactured consensus seeking process to move the science forward while at the same time providing decision makers with a better sense of the uncertainties and areas of disagreement and ignorance.
An intriguing aspect of the Science Court is that it makes the debate about advocacy and sources of funding as a pernicious influence essentially moot. It takes away the responsibility from individuals in these matters by institutionalizing dissent.
I’m certainly in favor of this general idea, but I can easily see this falling into the usual rathole for climate science if groups like the NAS are put in charge. Your thoughts on this?
