by Judith Curry
I confess that I prefer true but imperfect knowledge, even if it leaves much indetermined and unpredictable, to a pretence of exact knowledge that is likely to be false. – Friedrich von Hayek
While discussing my recent interview with Russ Roberts, Russ referred me to von Hayek‘s Nobel Prize Lecture in 1974 [link]. While focused on economics, von Hayek provides many insights that are relevant to climate science. Excerpts:
Unlike the position that exists in the physical sciences, in economics and other disciplines that deal with essentially complex phenomena, the aspects of the events to be accounted for about which we can get quantitative data are necessarily limited and may not include the important ones. While in the physical sciences it is generally assumed, probably with good reason, that any important factor which determines the observed events will itself be directly observable and measurable, in the study of such complex phenomena as the market, which depend on the actions of many individuals, all the circumstances which will determine the outcome of a process, for reasons which I shall explain later, will hardly ever be fully known or measurable.
We know: of course, with regard to the market and similar social structures, a great many facts which we cannot measure and on which indeed we have only some very imprecise and general information. And because the effects of these facts in any particular instance cannot be confirmed by quantitative evidence, they are simply disregarded by those sworn to admit only what they regard as scientific evidence: they thereupon happily proceed on the fiction that the factors which they can measure are the only ones that are relevant.
JC comment: Refer to Fig 8.14 in the AR5. The only forcing known with very high confidence is greenhouse gases, which has had very high confidence level since the SAR. Prior to the AR5, only GHG and ozone were known with high confidence (confidence has increased in AR5 for nearly all of the forcings.) External forcing is relatively easy to measure, compared with 3-D ocean circulation patterns and the vertical transfer of ocean heat. Interesting that the only relevant quantity known to very high confidence is judged to the dominant influence on 20th/21st century climate.
The correlation between aggregate demand and total employment, for instance, may only be approximate, but as it is the only one on which we have quantitative data, it is accepted as the only causal connection that counts. On this standard there may thus well exist better “scientific” evidence for a false theory, which will be accepted because it is more “scientific”, than for a valid explanation, which is rejected because there is no sufficient quantitative evidence for it.
JC comment: This is the argument for rejecting (or at least ignoring) various natural variability arguments.
The reason for this state of affairs is the fact, to which I have already briefly referred, that the social sciences, like much of biology but unlike most fields of the physical sciences, have to deal with structures of essential complexity, i.e. with structures whose characteristic properties can be exhibited only by models made up of relatively large numbers of variables.
JC comment: By these arguments, climate dynamics has more in common with with systems biology and economics than it does with experimental physics.
In some fields, particularly where problems of a similar kind arise in the physical sciences, the difficulties can be overcome by using, instead of specific information about the individual elements, data about the relative frequency, or the probability, of the occurrence of the various distinctive properties of the elements. But this is true only where we have to deal with “phenomena of unorganized complexity,” in contrast to those “phenomena of organized complexity” with which we have to deal in the social sciences. Organized complexity here means that the character of the structures showing it depends not only on the properties of the individual elements of which they are composed, and the relative frequency with which they occur, but also on the manner in which the individual elements are connected with each other. In the explanation of the working of such structures we can for this reason not replace the information about the individual elements by statistical information, but require full information about each element if from our theory we are to derive specific predictions about individual events. Without such specific information about the individual elements we shall be confined to what on another occasion I have called mere pattern predictions – predictions of some of the general attributes of the structures that will form themselves, but not containing specific statements about the individual elements of which the structures will be made up.
JC comment: I like the framework of organized complexity, and would like to see it applied more to the problem of climate change. The stadium wave idea arguably falls in this category.
The conflict between what in its present mood the public expects science to achieve in satisfaction of popular hopes and what is really in its power is a serious matter because, even if the true scientists should all recognize the limitations of what they can do in the field of human affairs, so long as the public expects more there will always be some who will pretend, and perhaps honestly believe, that they can do more to meet popular demands than is really in their power. It is often difficult enough for the expert, and certainly in many instances impossible for the layman, to distinguish between legitimate and illegitimate claims advanced in the name of science.
JC comments: This seems an apt description of the IPCC’s efforts to provide information to the UNFCCC.
There are some special problems, however, in connection with those essentially complex phenomena of which social structures are so important an instance, which make me wish to restate in conclusion in more general terms the reasons why in these fields not only are there only absolute obstacles to the prediction of specific events, but why to act as if we possessed scientific knowledge enabling us to transcend them may itself become a serious obstacle to the advance of the human intellect.
JC comment: Along these lines, the IPCC framing and consenus on the climate change problem is arguably an obstacle to making progress in understanding the dynamics of the climate system (see IPCC diagnosis-paradigm paralysis).
The chief point we must remember is that the great and rapid advance of the physical sciences took place in fields where it proved that explanation and prediction could be based on laws which accounted for the observed phenomena as functions of comparatively few variables – either particular facts or relative frequencies of events. This may even be the ultimate reason why we single out these realms as “physical” in contrast to those more highly organized structures which I have here called essentially complex phenomena. There is no reason why the position must be the same in the latter as in the former fields. The difficulties which we encounter in the latter are not, as one might at first suspect, difficulties about formulating theories for the explanation of the observed events – although they cause also special difficulties about testing proposed explanations and therefore about eliminating bad theories. They are due to the chief problem which arises when we apply our theories to any particular situation in the real world. A theory of essentially complex phenomena must refer to a large number of particular facts; and to derive a prediction from it, or to test it, we have to ascertain all these particular facts. Once we succeeded in this there should be no particular difficulty about deriving testable predictions – with the help of modern computers it should be easy enough to insert these data into the appropriate blanks of the theoretical formulae and to derive a prediction. The real difficulty, to the solution of which science has little to contribute, and which is sometimes indeed insoluble, consists in the ascertainment of the particular facts.
This corresponds to what I have called earlier the mere pattern predictions to which we are increasingly confined as we penetrate from the realm in which relatively simple laws prevail into the range of phenomena where organized complexity rules. As we advance we find more and more frequently that we can in fact ascertain only some but not all the particular circumstances which determine the outcome of a given process; and in consequence we are able to predict only some but not all the properties of the result we have to expect. Often all that we shall be able to predict will be some abstract characteristic of the pattern that will appear – relations between kinds of elements about which individually we know very little. Yet, as I am anxious to repeat, we will still achieve predictions which can be falsified and which therefore are of empirical significance.
To act on the belief that we possess the knowledge and the power which enable us to shape the processes of society entirely to our liking, knowledge which in fact we do not possess, is likely to make us do much harm. In the physical sciences there may be little objection to trying to do the impossible; one might even feel that one ought not to discourage the over-confident because their experiments may after all produce some new insights. But in the social field the erroneous belief that the exercise of some power would have beneficial consequences is likely to lead to a new power to coerce other men being conferred on some authority. Even if such power is not in itself bad, its exercise is likely to impede the functioning of those spontaneous ordering forces by which, without understanding them, man is in fact so largely assisted in the pursuit of his aims.
If man is not to do more harm than good in his efforts to improve the social order, he will have to learn that in this, as in all other fields where essential complexity of an organized kind prevails, he cannot acquire the full knowledge which would make mastery of the events possible. He will therefore have to use what knowledge he can achieve, not to shape the results as the craftsman shapes his handiwork, but rather to cultivate a growth by providing the appropriate environment, in the manner in which the gardener does this for his plants. There is danger in the exuberant feeling of ever growing power which the advance of the physical sciences has engendered and which tempts man to try to subject not only our natural but also our human environment to the control of a human will.
JC comment: I find the above bolded text to be a good prescription for climate science and policy.