by Greg Goodman
**UPDATE at end of thread**
The effect of the adjustments introduced in Met. Office’s HadSST3 release are compared to the original ICOADS data to evaluate their effects on the frequency content of the data. The relative merits of making a simple adjustment for the war-time glitch in ICOADS are also investigated. It is demonstrated that the various adjustments made in preparing Hadley SST versions combine to effectively removing long term variations from the climate record. Frequency analysis shows the adjustments generally disrupting, rather than improving the data.
The U.K. Meteorological Office Hadley Centre’s sea surface temperature records HadSST2 and hadSST3 are based on data from the ICOADS project which claims “ICOADS is probably the most complete and heterogeneous collection of surface marine data in existence. ” The global average monthly time series can be downloaded from the JISOA project. Temperatures are given as deviations from local seasonal averages, calculated for the period 1950-79.
These differences from the locally-averaged, monthly variations are referred to as “anomalies”.
Since the real meteorological shipping data are very non-uniform and unevenly spaced, ICOADS have done some complex interpolation and extrapolation in both time and space to provide this information as a 2×2 degree grid covering most of the global sea surface.
The Hadley Centre have reprocessed this data into a 5 x 5 degree grid, calculated a similar grid of local seasonal variations and produced a gridded database of monthly average temperature deviations from this seasonal climatology. In addition a number of adjustments are applied that aim to remove supposed systematic biases due to changes in sampling methodologies.
As well as the gridded data, HadSST2 is made available as a global mean temperature time series. The 6.6 MB HadSST3 data-set (released in July 2011) was released as 100 different “realisations” of the adjusted time series, using variations to the parameters of some of the corrections. The median average of all 100 versions is provided as a monthly, gridded dataset but do they not seem to provide a global mean time series as was done for HadSST2. To get a global average from HadSST3 it is necessary either to calculate the average from the gridded median data-set or to take the median value for each month from the 100 time series realisations.
The latter approach seemed the simplest and the least prone to possible differences in method and is what is used in this study.
One notable feature of the ICOADS data-set is a jump during the second world war. Obviously world wars do have a huge impact on shipping practices and hence on ship based data collection. During that period most merchant shipping was reduced to limited, protected convoys and the proportion of military vessels, with very different zones of activity, greatly increased. As well as the difference in shipping, the profile of the records that were preserved from that period was markedly different to the periods immediately before and after. Thompson et al  note:
“Between January 1942 and August 1945, ~80% of the observations are from ships of US origin and ~5% are from ships of UK origin; between late 1945 and 1949 only ~30% of the observations are of US origin and about 50% are of UK origin.”
Thousands of merchant ships were lost each year. It is perhaps surprising that the magnitude of this disturbance to shipping patterns and record-keeping did not have a larger effect on the temperature record.
The wartime glitch is clearly visible in the SST data shown in figure 1a.
A closer look at the detail shows an upward jump in December 1941 and a similar downward drop in 1946, red highlight in figure 1b.
It is accepted that these two dates correspond with entry of the USA into the war after the attack on Pearl Harbor and the demobilisation of US Navy after the war.
A simple way to deal with this is to subtract an offset from the data in this short interval. Such an adjustment is shown in figure 1b, above.
The Met. Office adjustments
From the papers documenting HadSST3: Kennedy et al [Met Office](2011c) [3c]:
Historical records of sea-surface temperature (SST) are essential to our understanding of the earth’s climate. Data sets of SST observations are used to detect climate change and attribute the observed changes to their several causes. They are used to monitor the state of the earth’s climate and predict its future course. They are also used as a boundary condition for atmospheric reanalyses and atmosphere only general circulation models (IPCC 2007).
Clearly, any adjustments made to the SST record will have far reaching implications and must be founded on sound science.
Attempts by the Hadley Centre to remove certain supposed biases from the SST data have a long history.
Folland et al (1984)  notes a difference between the unadjusted sea surface temperatures (SST) and the already adjusted marine air temperatures (MAT) around December 1941, as reported in Folland and Parker (1995) :
However, because Jones et al. (1986) used COADS summaries, they were unable to separate NMAT from day MAT which are affected by historically varying, on-deck solar heating: their corrections therefore differed from those of Folland et al. (1984). In both these early studies, about 0.5 °C was subtracted from MAT for 1942-5, a period of non-standard measurement practices owing to war.
[Note: NMAT refers to nocturnal marine air temperature. Emphasis added ]
Folland et al. (1984) applied corrections to NMAT to compensate for the historical increases of the average height of ship’s decks. These rose from about 6 m before 1890 to 15 m by the 1930s and 17 m by the 1980s. The corrections, based on surface layer similarity theory, removed a spurious cooling of about 0.2 °C between the late nineteenth century and 1980.
Parker and Folland (1995)  ( emphasis added ):
Folland et al. (1984) explained this as being mainly a result of a sudden but undocumented change in the methods used to collect sea water to make measurements of SST. The methods were thought to have changed from the predominant use of canvas and other uninsulated buckets to the use of engine intakes. Anecdotal evidence from sea captains in the marine section of the Meteorological Office supported this idea. However, it is known that engine intakes did Provide some SST data as far back as the 1920s or before (Brooks 1928).
So the adjustments to the data were based “anecdotal evidence” and “undocumented change”, ie. unfounded, hypothetical speculation.
Folland and Parker (1995)  :
A nineteenth-century oak ships’ bucket covered in iron bands has been studied though there is no indication that it was used for taking sea temperatures.
Yet a presumed change-over from wooden to uninsulated canvas buckets was the basis for the adjustment that reduced the late 19th century cooling trend.
Again, Parker and Folland (1995) :
The correction method is based on:
(i) The observation that the earlier SSTs, expressed as anomalies from recent averages, are not only too cold relative to NMATs similarly expressed (Barnett, 1984), but also, outside the tropics, show enhanced annual cycles, presumably because more heat is lost from uninsulated buckets in winter when stronger, colder winds blow over relatively warm water (Wright, 1986; Bottomley et al., 1990);
The possibility that annual cycles may actually vary for climatic reasons is not even considered before “presuming” more bucket adjustments are needed. One could equally say: “presuming skies were generally clearer during the pre-war warming period, summers were warmer and winters colder. But such speculative presumptions seem highly inadequate in science.
Folland and Kates (1984) improved on the uncorrected SST analysis of Paltridge and Woodruff (1981) by applying an adjustment which was 0.15 °C up till 1930 and decreased linearly to -0.1 OC in the 1970s (their reference period was 1951-60). However, comparison with NMAT suggested that the change in instrumentation took place rather suddenly around the Second World War, so Folland et al. (1984) added 0.3 °C until early 1940, 0.25 °C thereafter through 1941, and nothing subsequently.
So having diagnosed a short-term problem with NMAT which was resolved by a fixed war-time adjustment, as had been done in earlier SST studies, Folland et al decided to apply a one-sided adjustment to SST. This resulted in a significant post-war drop in temperature relative to the pre-war period that was not in the original data. This was the largest of the adjustments applied in hadSST2 and has remained in place since it was published in 1984. It is difficult to understand why the post-war drop, that was previously noted and corrected by Jones et al, was retained uncorrected.
The resulting adjustment can be seen by plotting the difference between the HadSST2 and ICOADS time series. This is shown in figure 2. Note the significant warming adjustment in the earlier half of the record that “corrects” for a change-over from wooden buckets that were never known to have been used for temperature sampling in the first place.
Study of the papers detailing these changes shows that the meta-data (supplementary information) concerning the type of buckets or other equipment used, and data collection practices are inconsistent and often missing. Absence of reliable documentation showing the periods of application of various techniques and equipment used, make any attempt to apply a quantifiable bias adjustment highly speculative. To a large degree, all these adjustments, including more complicated recent work, are founded on gross approximation, speculation and pure hypothesis. Parker et al (1995) has 43 occurrences of “assumed” .
Before making adjustments to the data, in any scientific study, it is necessary to have solid evidence of a bias, not just hypothesis and speculative reasoning.
It is obvious that the 1946 drop, from one monthly datum to the next, is anomalous and not of climatic origin. The following analysis shows a simple correction to this period removes anomalies in the time series, its derivatives and its frequency content. The supposed instrument biases proposed by Hadley are neither obvious in the data nor is their removal beneficial when the effects are analysed.
The resulting spurious post-war drop in temperatures is one of the reasons that climate models have difficulty in reproducing temperatures going back more than 50 or 60 years. It is also disruptive to any attempt to analyse the nature and magnitude of any natural cycles in climate: the spurious drop introduces significant frequency signals that are not there in reality and almost certainly disrupts those that are there.
The magnitude of this spurious adjustment is about half the size of the total variation over whole period of the ICOADS record. It constitutes a significant rewriting of the climate record of the 20th century. HadSST2 is still the basis for the marine component of the combined HadCRUT3 land and sea record offered by the Met Office.
This issue finally got some recognition in Thompson 2008  and an attempt was subsequently made to improve the adjustments to the data. This culminated in the release of HadSST3 in July 2011.
Figure 2b shows the difference between each HadSST version and the original data, displaying the adjustments that are applied in each case.
The war-time glitch seems to be more reasonably dealt with in HadSST3. However, what is rather surprising is that the buckets-and-pipes adjustment, originally introduced to explain the 1941 increase, is retained. Despite the recognition of the post war drop that cancels the earlier rise, a further reduction is still applied and oddly ends up making the same overall adjustment as before.
All that HadSST3 does differently in this respect is to round off the edges and slide the change in over 40 years. Most of the defects and speculative assumptions included in the earlier work have been retained. This makes the post-war drop less obvious to the eye and presumably less disruptive to attempts to make climate models match the historic temperature record, though it may equally be argued that this is an attempt to make the record better match the assumptions implicit in the models.
Kennedy et al 2011c [3c] goes into some detail about how the duration of the change was determined.
If a linear switchover is assumed which started in 1954and was 95% complete in 1969, the middle of the James and Fox study period, then the switchover would have been completed by 1970. Based on the literature reviewed here, the start of the general transition is likely to have occurred between 1954 and 1957 and the end between 1970 and 1980.
However, this assumption seems at odds with figure 1 one from the same paper that shows a significant proportion of buckets readings in 1970. A proportion that rose from 1955-1970 and only declined from then to the end of the record. Figure 3 reproduces figure 1 from K2011c [3c]
Neither does this hypothesised linear change-over from 1954 onward correspond to the bulk of the adjustment actually applied, as seen in figure 2b above, where the cooling adjustment clearly starts as early as 1920 and has already achieved 2/3 of it’s final extend before 1954.
Steve McIntyre at Climate Audit had criticised pre-HadSST3 studies on a number of occasions. Since much of HadSST3 is based on earlier studies and the result retains much the same form (as shown in figure 2b), many of the issues he raised have not been resolved and are, unfortunately, still relevant.
Further, it was noted in a detailed study of the available meta data by Kent et al (2006)  that as late as 1970 fully 90% of temperatures, where the meta-data stated the nature of the measurement, were still done by bucket. Yet the Hadley correction is fully applied by this date assuming, incorrectly, that bucket sampling had been phased out by this time. Figure 3b shows figure 2(f) excerpted from that paper.
[EDIT] As noted by John Kennedy in comments the switch-over refereed to
in the paper was from canvas buckets to insulated buckets, not buckets
to ERI, so this particular criticism was incorrect. Apologies to Kennedy
et al for the error.
From K2011c, [3c] section 4.2 (emphasis added):
It is likely that many ships that are listed as using buckets actually used the ERI method (see end Section 3.2). To reflect the uncertainty arising from this, 30 ± 10% of bucket observations were reassigned as ERI observations. For example a grid box with 100% bucket observations was reassigned to have, say, 70% bucket and 30% ERI.
Some observations could not be associated with a measurement method. These were randomly assigned to be either bucket or ERI measurements. The relative fractions were derived from a randomly-generated AR(1) time series as above but with range 0 to 1 and applied globally.
Clearly the timing and magnitude of the Hadley correction is determined by something other than recorded data.
Method: data processing.
Fourier and other frequency analysis techniques were performed on several climatic temperature data series in both the time series and the first and second derivatives. Derivatives can be helpful in identifying data sampling and data processing errors, secular changes and in isolating periodic variations. For example, and exponential rise in temperature will remain an exponential rise of the same duration in the derivative. However, a 64y cycle will shift back 16y, a 200y with shift 50y; a linear trend will become a constant offset. The dip in temperature due to a volcanic event will become a dip followed by a positive peak. This means that different effects that are superimposed and confound one another in a time series, may be distinguishable by studying the derivatives.
The monthly climate data were downloaded from the sources cited in the reference section. HadSST3 was processed by taking the median value of the 100 realisations for each month in the published data. This was then used as the global average time series for comparison with the other datasets.
The HadSST3 data runs from 1850-2006; ICOADS from 1845-2008 and HadSST2 from 1850-2011. For the frequency analysis, the data were truncated to a common end-date (Dec. 2006) to give a more direct visual comparison of results. Data are published as a list of monthly averages in the form yyyy month , this was converted to a decimal date based on the middle of the month (ie January = + 0.4/12 ; Feb=1.5/12 ; etc. )
Time derivatives were calculated on a month-to-month basis by dividing the incremental change in temperature by the time period. Derivatives are indicated on the graphs by the elements such as “-diff1″ meaning the point differential taken over one data interval (typically one month). Each point is logged with the date of middle of the interval to avoid introducing a time shift.
Gaussian frequency filtering, where required, was achieved by calculating weighted mean of neighbouring points for each point in the record. A three-sigma gaussian filter was used and is indicated in the graphs by its sigma value: “gauss-24m” signifies a three-sigma gaussian filter of sigma=24 months. This method requires a full window of neighbouring points and thus shortens the records by three-sigma data points at each end. Each result is logged with the date of middle of the sample to avoid introducing a time shift.
The dates of the step changes during the war-time period can determined by inspection. A simple correction for the war-time glitch would be to subtract a fixed amount form the monthly averages over that period. A range of values between -0.2 and -0.5K were tested. A value of 0.4 was found to best remove the disruption seen in the first and second derivatives. Considering the level of noise and variability in the signal, the probable range the offset was determined to be 0.45 +/- 0.05K. This is an approximate solution since the war in Europe started in Sept 1939 and global maritime traffic would already have been disrupted before the U.S. became involved in Dec 1941. This probably accounts for the early rise being less sharply defined than the later fall: part of the change had already occurred before 1941. This results in a small negative then positive glitch between ’39 and ’41 but this is considered insignificant in this context. A value of 0.5K seemed to be slightly better in removing disruption from the FFT periodogram. A value of 0.4K was retained as the best overall correction.
A better, more graduated correction could be made with a more rigorous examination of the precise causes but, in view of the data quality, the stated level of precision seems appropriate.
The effects of this correction on the time series and its time derivatives were examined both directly and in the frequency domain by Fourier analysis. Similar consideration was given to the correction to this period applied in HadSST3. The two were compared.
The Fourier analysis was done with software using the well-reputed, open-source FFTW discrete Fourier transform library. The method was checked against other open-source software with it’s own FFT implementation. Results were identical to within calculation error limits.
There are well-known problems with the discontinuity created when FFT effectively joins the start and end of the data in a continuous loop. A common solution to this that is standard practice in engineering and digital signal processing is to multiply the data by a windowing function that progressively reduces both ends to zero. However this will notably affect longer frequencies and could give the impression of a long-term cycle where one does not exist in the original data. Here we restrict the use of FFT to dT/dt where the difference in level at the two ends is less marked than in the temperature time series. However, there is a rising trend and this is seen in the residual non-cyclic term. Further study of the profile of this part of the FFT will give further useful information about the nature of non cyclic rise but that is not explored here.
Since a discrete Fourier transform only gives discrete frequencies that correspond to sub-multiples of the data window, it is necessary to repeat the FFT with different window lengths in order to get detail at longer periods. This technique can also be used to get several evaluations of intermediate frequencies when different sub-multiple results covers the same period. For example the 50y period component can be evaluated by looking at the last 50, 100 or 150 years of data. If the data has a reasonably homogeneous frequency structure and little error, the three results will be close; if not, comparison can give information about how the more recent section of the data compares to the fuller, longer windows. Dividing the period by the number of the sub-multiple allows overlaying of the these different evaluations for direct visual comparison. Where this has been done is indicated in the graph’s title and the individual plot lines are labelled “half-window period” in the legend. This approach allows more detail but is also affected by the different sub-section of the data being used at each point. A peak or dip can be due to local changes at the window end as well as true frequency patterns. Punctual disturbances, such as major volcanoes or data collection changes, can be seen to cause short range anomalies. These can be usefully identified by comparison of the full window and half-window plots since these will affect all traces at the same point and will not scale as cyclic variations will. Other, longer, non-cyclic variations such as long term secular trends and data collection errors will also manifest in this way.
In ideal data with clearly defined, stable frequencies, any periodic cycles would be represented by well-defined peaks. In noisy data from a largely chaotic system, peaks will be spread and may even change from one part of the sample period to another. Here it is only possible to determine a broader range of frequency where power of the spectrum is concentrated. Other more advanced frequency analysis techniques that tolerate such changes in the data structure, such as wavelet or entropy methods, may give more precise results. However, this approach will allow some useful insight into the effects of the various adjustments on frequency and provide an informative comparison of the global datasets. It is thus possible to evaluate whether an adjustment improves the dataset or disrupts it.
Since cyclic terms will be largely unchanged in dT/dt a comparison with the frequency spectrum of the derivative is equally useful in separating secular and cyclic variations. There is the added advantage that a linear trend will be come a constant in dT/dt and the wrap-round discontinuity is less marked. As a consequence the distortion due to any windowing function will be less pronounced. There has been a recent focus in climatology on ocean heat content, ie. the total thermal energy. In the same way that the temperature reflects the thermal energy of a body, the rate of change of temperature reflects the power entering or leaving a body. Inspection of dT/dt may give useful insight into the net power entering or leaving the ocean surface layer.
Curve fitting was done using Gnuplot, an open source plotting utility. It uses an implementation of the non-linear least-squares (NLLS) Marquardt-Levenberg algorithm. Unlike linear least squares, NLLS methods require initial parameter values be supplied as a starting point. If the initial values are poorly chosen it may fail to converge on a result. If this is the case, it is indicated by the software. Where the model function is successfully fitted detailed statistics on the quality and confidence of result are given.
The data from 1940 to 1946 were given zero weighting in model fitting, so the results are not perturbed by the unreliable war-time period. This was done identically in analysing both the original and the adjusted versions.
The original ICOADS global mean sea surface temperature is shown in figure 1 along with the simple war-time adjustment analysed in this study. This is similar to the 0.5 K adjustment applied by Jones et al (1986), as previously noted above. This adjustment was used prior to the opening of the Hadley Centre in 1990.
Expanded detail of the war-time period showing an abrupt upward jump in December 1941 and a similar downward drop in 1946 (red) can be seen in figure 1b.
Figure 2 shows the magnitude of the adjustments applied by HadSST2 and HadSST3 to the original data (note these are not the adjusted time series themselves but adjustments that are applied in each case). It can be seen that HadSST3 fully retains the step change of SST2 but phases it in over an extended period and smooths the transitions. The adjustments are mostly neutral after 1970.
The overall form of the adjustment is rather surprising in view of its supposed origin in changes to data collection practice. There seems to be a strong term cyclic element in the adjustment. Its general character can be modelled by fitting two cosine functions. This raises the question of whether HadSST3 processing is inserting or attenuating some long term cycles in the original ICOADS data. The fitted curve is shown in Figure 4.
It can be seen that the adjustment, which is deemed to result from the study of changes in data collection methods, is remarkably close in form to the variations in the original data itself. Its magnitude being about 50% of the variations before 1920 and around 67% between 1920 and 1970. The only variation that is not attenuated is the post 1980 rise (which actually gets gently increased towards the end). In effect, a major proportion of the long term variations over the majority of period of available data are being removed on the basis of “correcting” the hypothesised data sampling biases.
The magnitude of the adjustment is comparable in size to the total warming of the 20th century, ie. the “correction” deemed necessary is almost as big as the effect being observed.
Figure 6 shows the Fourier analysis of the rate of change of temperature in the ICOADS data (24 month gaussian low-pass filtered) with the simple war-time correction. The x-axis shows length of window of data used for each point. The subset of data used always includes the most recent data and increases in length backwards in time. Thus this is a periodogram ( period plot ) rather than a frequency plot.
The energy in the spectrum is concentrated around 60 years, 10 years and towards the longest periods available in this data, all three being of roughly equal magnitude. The presence of a longer periodic signal running strongly to the end of the graph, where it converges with the non-cyclic residual shows there is a cyclic variation of more than 160 years in duration. [The marked trough in the blue line is due to the short window coinciding with a marked change around 1975. Analysis of the sub-20 year frequencies by conventional FFT frequency plot shows this is not a frequency feature. The full window plot better represents these frequencies.]
It is noted that there is a strong similarity between the blue and green lines. The blue line represents the shorter periods found to have two cycles in the window, the green one, those with a period equal to the window. Their similarity demonstrates a consistent distribution of energy between the more recent half of the data and the record as a whole. This would strongly argue against the need for substantial corrections to the data.
The red line shows the residual, non-cyclic component. The regular occurrence of small peaks every ten to eleven years reflects the window’s transition through the periodic variations commonly attributed to the “solar cycle”. As the length of data used in the FFT window increases, their impact on the data decreases. Also, with window length greater than 80 years, the longer term variations begin to be detected and hence the residual generally decreases. This shows how selectively restricting any analysis to only the most recent portion of the available data opens up the likelihood of confounding cyclic and non-cyclic trends leading to false diagnosis and attribution.
A similar frequency analysis of HadSST3, shown in Figure 7, shows that the long term cyclic variation, greater then 100 years, has been severely attenuated. Also that the scaled secondary frequency plot (blue) is no longer in agreement with the green line, indicating that the later half of the record is no longer in accord with the full record as it was when applying a trivial correction. Removal of the supposed biases has destroyed the homogeneity of the data. On the positive side, the two lines agree more closely with window lengths around 60y where the window end falls within the problematic war years. This suggests that HadSST3 is dealing with the detail of war-time glitch more precisely than the simple adjustment.
A lot more can be gained from this kind of analysis and that will the subject of further study but these two main features show the ways and the extent to which the Hadley adjustments fundamentally change the nature of the data.
Figures 8, 9 and 10 show the time series and the first and second time differentials for original ICOADS data, HadSST3 and the simple 0.4K adjustment respectively. Due to short term variation and the noise being amplified by the differentiation, a 24 month gaussian filter has been applied after differentiation (48m in the case of 2nd differential.) Without this measure, the noise dominates and obscures the rest of the signal. Due to the filtering reducing the length of the available data, the fitting period was started at 1860 to make the results more directly comparable. Since the recent data is a lot less noisy and more certain the fit period runs to the end of the data in each case. The fitted data is shown in dark blue.
Figure 8 ICOADS triple plot
Figure 10 ICOADS_adjusted triple plot
It is noted that the simple adjustment is consistent in the time derivative: the two primary components detected in the time series remain similar in period, phase (reference year) and magnitude in the first derivative (see appendix). Equally, the disruption of the war-time glitch is clearly seen in all three plots of the original data but it does not prevent similar values for the cosine fit.
It seems unlikely that any error due to sampling methods and unrelated to climate would introduce a cyclic variation that is consistently found in the time derivative. Errors and biases would likely be non-cyclic and become more obvious by comparison with the derivative. In contrast, the century scale component in HadSST3 changes completely in period, phase and magnitude in the derivative. This is not necessarily wrong in itself because non-cyclic variations will be different in the derivative, but it indicates a fundamental change is being made to the structure of the data. A time series that has a strong cyclic feature looses that quality as a result of a correction deemed to simply correct data sampling errors.
It was also noted that the cosine model fitted to the simply adjusted ICOADS data, closely follows the mid-points of the oscillations in dT/dt and does not have any post-war anomalies. There seems to be a more natural coherence to the data globally, whereas HadSST3 seems more chaotic. This needs quantifying mathematically.
The broad, positive war-time peak in dT/dt of HadSST3 is unique in the record. This feature seems anomalous in both duration and form.
Also, studying the second differential of the simply adjusted ICOADS record shows solar cycles over the last century to be regularly grouped in pairs. Since the pseudo 11y cycle is known to be two halves of circa 22y magnetic cycle this feature seems to be a coherent feature and gives credence to the data quality. This feature is destroyed by the Hadley processing, although it would be worth investigating whether some of the individual realisations preserve it.
Figure 12 shows part of the Wang et al  Total Solar Irradiance reconstruction overlaid on the d2T/dt2 plot from figure 10(c).
It is not the object of this study to suggest or refute any particular link between climate and TSI, nor to suggest that TSI would necessarily be the appropriate solar parameter to study. However, there is more than a coincidental similarity in the timing of variations of the two quantities over the last century and TSI can at least be regarded as an indicator of solar activity. It seems improbable that an error with such a similarity could be erroneously introduced by the sampling bias. Any data processing that removes or distorts such a signal must therefore be rejected as flawed.
Removing climate signals from SST data which is used as a primary reference for climate study will not aid scientific understanding of climate. In fact it will confound it.
Studying the second differential reveals other important changes. Notably the non-cyclic constant term ‘c’ in the fitted parameters represents an accelerating increase of the temperature with time. The fitted values are shown as annotations on the triple plots in figures 8(c), 9(c) and 10(c):
ICOADS v2.5 : 1.26 K/c/c
HadSST3 : 6.63 K/c/c
ICOADS WWII adjusted : 1.19 K/c/c
The idea that temperatures changes are accelerating at over 6K per century per century is beyond even the extreme end of the projections of the IPCC, yet this is what is found in the Hadley modified data set. The unmodified and the simply corrected ICOADS data show a more reasonable 1.2 K/c/c.
While such a simple analysis is not intended to fit a definitive model to the data, it again show how hadSST3 is fundamentally changing the nature of original data.
The convergence statistics for the various cosine fits show that the asymptotic standard error of the fitted parameters are generally very good, though notably higher for HadSST3 than for the simply adjusted ICOADS dataset(see appendix). The equivalent sine functions fitted to dT/dt are equally certain and the pairs of results are consistent: the period and reference year derived in each case are in agreement to within (or close to) the statistical fitting error reported by the fitting algorithm. The one exception is the longer cycle’s period which is shortened from circa 200y in the time series to around 150y in the derivative. This likely due to non cyclic variation in the time series that is not accounted for in the pure cosine model. The long periods found in dT/dt are different since a linear trend is accommodated by the constant rate of change ‘c’ in the model.
All the analyses find the reference year of this cycle to be around 1995. This is a peak for the temperature time series and the transition from positive to negative rate of change in dT/dt.
They are also all (including HadSST3) in general agreement about a circa 11y period peaking in 2001/2002; a circa 21y cylcle peaking in 2005/2006; and a circa 64y cycle peaking in 2010-2016 (more likely the end of that interval.)
In contrast, the longer term fitted to HadSST3 is twice that found in the original data (circa 350 years) with a somewhat larger uncertainty and parameters incompatible with the results for ICOADS. In the derivative the NNLS algorithm converges to a much shorter, incompatible period circa 120y, this indicates that the longer cycle was a fortuitous fit to a non cyclic variation and probably does not reflect a true cycle in the data. The parameters for the short term periods are similar to those in the original data and are consistent in the time derivative.
This confirms what was seen in the FFT analysis: HadSST3 is removing (or totally disrupting) the clear, long term variation found in the climate record.
Similar processing of the difference of the adjusted and non-adjusted time series (HadSST3-ICOADS) ie. the modifications being applied, shows the adjustment itself has a cycle close to the circa 165 year and 64 year cycles found in the data.The values are annotated in figure 4
However complex the methodology claims to be, the result is surprisingly simple. HadSST3 selectively removes the majority of the long term variations from the pre-1960 part of the record. ie. it removes the majority of the climate variation from the majority of the climate record. Examination of the time differentials also show it is distorting rather than improving the data.
Comparison to non-SST measurements
The Gomez Dome is an ice dome at the base of the Antarctic Peninsula, its climate is dominated by the surrounding ocean. A study by Thomas, Dennis et al 2009  derived a high resolution temperature proxy record from oxygen isotope ratios from the ice core. This oxygen isotope ratio is generally regarded as being a reliable proxy for temperature of the water at the time of evaporation (ie. in this case SST in Bellingshausen Sea). Figure 13 is the d18O graph excerpted from that paper.
Figure 13 Gomez Dome d18O
Authors’ original caption:
Gomez annual average #18O (blue), running decadal mean (red) and nonlinear trend (black). The running decadal mean is derived using an 11-point Gaussian window filter.
The paper explains the derivation of the non-linear trends thus:
The EMD approach is used to extract physically meaningful modes and nonlinear trends from nonlinear and non-stationary time series that cannot be captured by a linear least-square fit.
The figure shows a non-linear trend that appears to peak around 2000 AD and has a trough around 1890. This is be consistent with a circa 200 year cycle similar in period and phase to that found in ICOADS SST in the present study.
Camp and Tung  recently reported finding a correlation of 0.64 between air surface temperature the solar TSI index, with a magnitude of 0.18 ± 0.08 K per W/m2 linked to the nominal 11y cycle. The present study found that evidence of a similar correlation with SST was disrupted by the HadSST3 processing.
HadSST3 contains a series of adjustments. With the exception of the war-time glitch, they are not obvious from study of the record. Their existence is based on speculation and hypothesis. Calculation of the biases involves inverting a significant portion of written record’s meta-data for the period of the principal adjustment and ignoring detailed studies on the proportion and timing of changes in data sampling methods as well a speculation as to the magnitude of the various effects.
The principal effect of these adjustments is to selectively remove the majority of the long term variation from the earlier 2/3 of the data record and to disrupt circa 10-11y patterns clearly visible in the data. These changes are fundamentally altering the character of the original data.
The strong similarity in form between the variations in the original ICOADS data and the corrections deemed necessary to correct sampling biases is remarkable. All the more so in view of the lack of documentary information on which to base the estimated magnitude and timing of the adjustments.
The analysis presented here indicates that, outside the immediate war-time period, these adjustments are distorting and degrading the data rather than improving it.
A number of different analyses suggest that a simple correction to the war-time period (as was used before the creation of the Hadley Centre) provides a more coherent and credible result.
Comparison to studies of non SST data suggest that much of the variation in ICOADS is quite possibly due to real climate signals, not instrument bias. These variations require proper investigation, not a priori removal from the climate record.
[EDIT] The ICOADS data provided by JISAO, used in this study, was v 2.4, not version 2.5 as stated. There is little change between the two and
this does not make a material change to the analysis and arguments
presented in this study.
Thompson, D.W.J., Kennedy, J.J., Wallace, J.M. & Jones, P.D. (2008) A large discontinuity in the mid-twentieth century in observed global-mean surface temperature. Nature 453, 646-649
Chris E. Forest and Richard W. Reynolds (2008)
Hot questions of temperature bias Nature 453, 601-602
Kennedy, J., R. Smith, and N. Rayner (2011a), Using AATSR data to assess the quality of in situ sea surface temperature observations for climate studies, Remote Sens. Environ. (in press) http://www.metoffice.gov.uk/hadobs/hadsst3/RSE_Kennedy_et_al_2011.doc
Kennedy J.J., Rayner, N.A., Smith, R.O., Saunby, M. and Parker, D.E. (2011b). Reassessing biases and other uncertainties in sea-surface temperature observations since 1850 part 1: measurement and sampling errors. in press JGR Atmospheres http://www.metoffice.gov.uk/hadobs/hadsst3/part_1_figinline.pdf
Kennedy J.J., Rayner, N.A., Smith, R.O., Saunby, M. and Parker, D.E. (2011c). Reassessing biases and other uncertainties in sea-surface temperature observations since 1850 part 2: biases and homogenisation. in press JGR Atmospheres http://www.metoffice.gov.uk/hadobs/hadsst3/part_2_figinline.pdf
Folland, C. K., D. E. Parker, and F. E. Kates. 1984. Worldwide marine temperature fluctuations 1856-1981. Nature 310, no. 5979: 670-673.
C. K. Folland* and D. E. Parker (1995) CORRECTION OF INSTRUMENTAL BIASES IN HISTORICAL SEA SURFACE TEMPERATURE DATA
Q.J.R. Meteorolol. Soc. (1995), 121, pp. 319-367
D. E. Parker, C. K. Folland and M. Jackson (1995)
MARINE SURFACE TEMPERATURE: OBSERVED VARIATIONS AND DATA REQUIREMENTS*
Climatic Change 31: 559-600, 1995 Kluwer Academic Publishers.
A Study of Six Operational Sea Surface Temperature Analyses
C. K. Folland, R. W. Reynolds , D. E. Parker (2003)
 Steve McIntyre, Climate Audit
 E.C. Kent S.D. Woodruff D.I. Berry 2007
Metadata from WMO Publication No. 47 and an Assessment of Voluntary Observing Ship Observation Heights in ICOADS
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY vol 24.
 Wang, Y.-M.; Lean, J. L.; Sheeley, N. R., Jr. 2005
Modeling the Sun’s Magnetic Field and Irradiance since 1713
The Astrophysical Journal, 625:522-538, 2005 May 20)
C. D., and K. K. Tung (2007), Surface warming by the solar cycle as revealed by the composite mean difference projection,
Geophys. Res. Lett., 34, L14703, doi:10.1029/2007GL030207.
Table 1. Sample of fitting errors for cosine model.
hadSST3_median time series results Final set of parameters Asymptotic Standard Error ======================= ========================== p1 = 344.302 +/- 44.7 (12.98%) p2 = 66.0627 +/- 0.8661 (1.311%) a1 = 0.361004 +/- 0.05316 (14.73%) a2 = 0.127095 +/- 0.004566 (3.593%) yz1 = 2052.23 +/- 19.29 (0.94%) yz2 = 2010.63 +/- 1.169 (0.05814%) p3 = 21.3518 +/- 0.1659 (0.7772%) a3 = 0.0415534 +/- 0.004699 (11.31%) yz3 = 2006.16 +/- 0.7091 (0.03535%) c = 0.0107957 +/- 0.05011 (464.2%) icoads_monthly_adj0_34 time series results Final set of parameters Asymptotic Standard Error ======================= ========================== p1 = 198.429 +/- 4.095 (2.064%) p2 = 61.6491 +/- 0.7408 (1.202%) a1 = 0.463026 +/- 0.006512 (1.406%) a2 = 0.138481 +/- 0.004394 (3.173%) yz1 = 2001.13 +/- 1.72 (0.08596%) yz2 = 2011.54 +/- 1.013 (0.05035%) p3 = 21.1679 +/- 0.2235 (1.056%) a3 = 0.0293529 +/- 0.004655 (15.86%) yz3 = 2004.16 +/- 0.9152 (0.04566%) c = -0.189096 +/- 0.007575 (4.006%)
Biosketch: The author has a graduate degree in applied physics, professional experience in spectroscopy, electronics and software engineering, including 3-D computer modelling of scattering of e-m radiation in the Earth’s atmosphere.
JC comment: The HadSST is generally regarded as the best of the global SST data sets. The substantial improvements to HadSST3 were discussed on this previous post, which included comments from John Kennedy. I am particularly interested in this mid-century period, since it is an important period in the context of understanding the 20th century climate change attribution.
I have been discussing this topic with Greg for several months, and I invited him to do a guest post on this topic. I did some light editing and suggested some shortening. The views expressed in this post are those of GG, and not my own.
Moderation note: this is a technical thread that will be moderated strictly for relevance. Apologies for the glitches in the first version of the post, a few comments were lost.
The author has had quite some interesting discussions with John Kennedy
(leading author of the three papers in the reference section above that present the HadSTT3 dataset).
As a result we were able to agree on the main points raised in the article:
1. That HadSST3 removed the majority of the variation from the majority
of the record.
2. The these adjustments are based on hypothesis rather than being
The following link will help locate the salient part of the discussion:
otherwise use the browser search faciltiy to locate either “Goodman”
or “Kennedy” (usually control-F or “Find” on the Edit menu)
We also discussed something that I had not entered into in the article
because I wished to remain focused of the central issue. That is the question of the claimed “validation” of HadSST3 adjustments by other studies.
I maintain that the claimed validation by comparison to computer model
outputs and extremely geographically limited data, are totally inappropriate and do neither validate nor disprove anything. John did not agree with my position on that but was not able to show I was wrong. He did provide some useful
and interesting information on how the models are optimised to the period 1960-1990, which underlines the problems with their use to “validate” the earlier adjustments.
He suggested that I should compare the time series of the Hadley
adjustment to the residual “corrected climate” as well as comparing to the original data as I did in the article. I agreed it would be interesting and the result can be seen here: http://i39.tinypic.com/a2gjv8.png
The two plots are on the same scale and this confirms my intial
observation the Hadley adjustment is basically cutting the variation of the data in half before 1910, ie. they are suggesting that bias is equal to the “true” climate signal and varies in a similar way over this period. I find such a result surprising and improbable and it is worth seeing this clearly displayed since it is not presented in this way in the papers. In a similar way it can be noted that the 1920-1940 trend in the adjustment is very similar to the trend in the residual.
John was also good enough to send me the icoads data after the remapping
to their 5×5 degree gridding but before the application of the bias adjustments. This should be useful in examining which aspect of the Hadley processing is responcible for the various changes I noted. I am still looking at that in more detail.
I would like to take this opertunity to thank John for taking the time
to discuss some of the issues raised in this article and to engage in serious dialogue. We are continuing discussions by email and if anything significant comes out of this I hope to post further updates.