by Judith Curry
Selection biases in information processing occur when expectations affect behavior in a manner that makes those expectations come true.
Emerging selection bias in large-scale climate change simulations
by Kyle Swanson
Abstract. Climate change simulations are the output of enormously complicated models containing resolved and parameterized physical processes ranging in scale from microns to the size of the Earth itself. Given this complexity, the application of subjective criteria in model development is inevitable. Here we show one danger of the use of such criteria in the construction of these simulations, namely the apparent emergence of a selection bias between generations of these simulations. Earlier generation ensembles of model simulations are shown to possess sufficient diversity to capture recent observed shifts in both the mean surface air temperature as well as the frequency of extreme monthly mean temperature events due to climate warming. However, current generation ensembles of model simulations are statistically inconsistent with these observed shifts, despite a marked reduction in the spread among ensemble members that by itself suggests convergence towards some common solution. This convergence indicates the possibility of a selection bias based upon warming rate. It is hypothesized that this bias is driven by the desire to more accurately capture the observed recent acceleration of warming in the Arctic and corresponding decline in Arctic sea ice. However, this convergence is difficult to justify given the significant and widening discrepancy between the modeled and observed warming rates outside of the Arctic.
Citation: Swanson, K. L. (2013), Emerging selection bias in large-scale climate change simulations, Geophys. Res. Lett., 40, 3184–3188, doi:10.1002/grl.50562. [Link] to complete manuscript.
From the Introduction:
Here we suggest the possibility that a selection bias based upon warming rate is emerging in the enterprise of large-scale climate change simulation. Instead of involving a choice of whether to keep or discard an observation based upon a prior expectation, we hypothesize that this selection bias involves the ‘survival’ of climate models from generation to generation, based upon their warming rate. One plausible explanation suggests this bias originates in the desirable goal to more accurately capture the most spectacular observed manifestation of recent warming, namely the ongoing Arctic amplification of warming and accompanying collapse in Arctic sea ice. However, fidelity to the observed Arctic warming is not equivalent to fidelity in capturing the overall pattern of climate warming. As a result, the current generation (CMIP5) model ensemble mean performs worse at capturing the observed latitudinal structure of warming than the earlier generation (CMIP3) model ensemble. This is despite a marked reduction in the interensemble spread going from CMIP3 to CMIP5, which by itself indicates higher confidence in the consensus solution. In other words, CMIP5 simulations viewed in aggregate appear to provide a more precise, but less accurate picture of actual climate warming compared to CMIP3.
From the description of Figure 1:
Figure 1. Changes in mean surface air temperature are the standard metric used to assess climate change. Temperature anomalies are shown for the decade 2002–2011 relative to the 1979–2001 mean. Values for the ERA-Interim reanalysis and HadCRUT4 are shown for comparison. Panel C shows these surface air temperature anomalies as a function of latitude for the CMIP3 simulations (red curves), as well as for the ERA-Interim reanalysis (heavy black curve). Panel D shows the same but for the CMIP5 simulations, with the CMIP3 and CMIP5 mean simulation curves inserted for reference.
The latitudinal structure of the warming shown in Figures 1C and 1D provides insight into the unusual behavior exhibited by the CMIP5 ensemble. In the CMIP3 ensemble, the largest deviation between observed and simulated warmings is in the Arctic, where the observed warnings are roughly 1C larger than the CMIP3 simulation ensemble mean. The CMIP5 ensemble successfully reduces this deviation in the Arctic (Figure 1D), with differences in the warming pattern between the CMIP5 and CMIP3 ensemble means outside of the Arctic consistent with diffusion of the enhanced CMIP5 warming in the Arctic into the Northern Hemisphere midlatitudes. However, the enhanced CMIP5 ensemble mean Arctic warming unveils offsetting errors in the CMIP3 ensemble mean warming (not enough warming in the Arctic, too much warming almost everywhere else), leading to the poorer overall CMIP5 ensemble mean consistency with the observed warming relative to CMIP3.
This description provides a reasonable explanation for why the CMIP5 ensemble mean performs poorly relative to CMIP3. However, the issue of the reduction in the CMIP5 simulation spread still remains. One way to approach this problem is to ask what subset of the CMIP3 ensemble has statistics most like the CMIP5 ensemble. To this end, consider a subensemble comprised of those CMIP3 simulations that warm more than the ensemble median CMIP3 simulation (hereafter CMIP3+). Curiously, the statistics of this CMIP3+ subensemble are indistinguishable from those of the CMIP5 ensemble using Student’s T-test (Table 1; p ’ 0.15 for both tropics and extratropics). This contrasts with the behavior of the entire CMIP3 ensemble, which differs from theCMIP5 ensemble in a statistically significant fashion in both the tropics and extratropics (T > 3.25; p < .002). This indicates the possibility of a selection bias based upon warming rate (either globally or regionally in the Arctic), with only those model configurations that warmed more aggressively ‘surviving’ in an appropriate sense to be included inCMIP5, while those that did not warm as aggressively were more significantly modified. This statement is of course highly speculative; the actual rationale for this convergence is likely to be more complicated.
JC comments: I find this paper to be very illuminating. The latitudinal variation of observed temperature anomalies clearly shows that most of the recent warming occurs in the Arctic. How much of the recent arctic warming is associated with external forcing (e.g. predictable) versus natural internal variability is hotly debated.
Latitudinal comparisons of climate model simulations with observations is very revealing. Climate models are not accurately simulating the hemispheric gradients in temperature anomalies, with too much warming in the lower latitudes and too little warming in high latitudes. Owing to the large multi-decadal natural internal variability in the high latitudes, the over prediction of warming in the lower latitudes may be the more telling model deficiency in terms of AGW sensitivity.
And finally, the convergence of CMIP5 models towards a common solution does give credence to Swanson’s thesis of selection bias. Gavin Schmidt is adamant that climate modelers (well at GISS anyways) don’t tune to observations. However, it is abundantly clear (see also this previous post Climate model tuning) that ‘expectations’ of model simulations do influence which model versions and combinations of parameter choices are used in the final model version for production runs.
The idea of model selection bias based on simulating a large amount of arctic warming and a sea ice decline is an intriguing one. If this is the case (again, subjective biases are not easy to unambiguously identify), then it is even more important to sort out how much of the recent Arctic warming is from AGW versus natural internal variability.