reanalyses.org

by Judith Curry

How do you estimate the state of the global atmosphere and ocean when observational data sets are incomplete, imperfect and noisy?

Because dynamical process states in a large and complex dynamical system are only partially accessible by measurements, most quantities must be determined via model-based state estimation.  This is accomplished by inverse modelling.

The Wikipedia describes inverse problems in the following way:

The inverse problem can be conceptually formulated as follows:

Data → Model parameters

The inverse problem is considered the “inverse” to the forward problem which relates the model parameters to the data that we observe:

Model parameters → Data

The transformation from data to model parameters (or vice versa) is a result of the interaction of a physical system with the object that we wish to infer properties about. In other words, the transformation is the physics that relates the physical quantity (i.e. the model parameters) to the observed data.

The mathematics of dealing with inverse modelling of large scale state estimation problems is rather hairy, using various methodologies including variational, Kalman filter, maximum likelihood ensemble filter and other ensemble methods.  The Wikipedia article on data assimilation provides a reasonable introduction to the basic technical application of this to weather forecasting.

So skipping over the technical details of all this, lets discuss climate reanalysis products, which are products of model-based state estimation.

Reanalysis

Reanalysis  is a climate or weather model simulation of the past that includes data assimilation of historical observations.  The rationale for climate reanalysis  is given by reanalyses.org:

Reanalysis is a scientific method for developing a comprehensive record of how weather and climate are changing over time. In it, observations and a numerical model that simulates one or more aspects of the Earth system are combined objectively to generate a synthesized estimate of the state of the system. A reanalysis typically extends over several decades or longer, and covers the entire globe from the Earth’s surface to well above the stratosphere. Reanalysis products are used extensively in climate research and services, including for monitoring and comparing current climate conditions with those of the past, identifying the causes of climate variations and change, and preparing climate predictions. Information derived from reanalyses is also being used increasingly in commercial and business applications in sectors such as energy, agriculture, water resources, and insurance.

Overviews of climate reanalyis are found at this page, under Meeting Presentations.  I refer specifically to useful presentation by Kevin Trenberth on atmospheric reanalyses.  Some excerpts from the text:

Data Assimilation merges observations & model predictions to provide a superior state estimate. It provides a dynamically- consistent estimate of the state of the system using the best blend of past, current, and perhaps future observations.  Experience mainly in atmosphere; developing in ocean, land surface, sea ice.

[Using a weather prediction model]  The observations are used to correct errors in the short forecast from the previous analysis time. Every 12 hours ECMWF assimilates 7 – 9,000,000 observations to correct the 80,000,000 variables that define the model’s virtual atmosphere. This is done by a careful 4-dimensional interpolation in space and time of the available observations; this operation takes as much computer power as the 10-day forecast.

Operational four dimensional data assimilation continually changes as methods and assimilating models improve, creating huge discontinuities in the implied climate record. Reanalysis is the retrospective analysis onto global grids using a multivariate physically consistent approach with a constant analysis system. 

Reanalysis has been applied to atmospheric data covering the past five decades.   Although the resulting products have proven very useful, considerable effort is needed to ensure that reanalysis products are suitable for climate monitoring applications.

The remainder of the presentation described problems associated with the first generation of atmospheric reanalysis products.
.
From a recent summary in the June 2011 issue of the Bulletin of the American Meteorological Society:
.
Atmospheric reanalyses are useful tools for examining climate processes because they provide greater coverage than observations alone, combine information from a diverse range of soucres, and adjust for biases that may exist in any specific set of obseervations.  however, they must be used cautiously because the quantity, quality and type of observations being included in the reanalyses have changed over time, which can produce artificial trends.  (note the recent paper paper discussing trends in the Arctic is an example of this problem).
.
2nd generation reanalysis products

At reanalyses.org, the second generation of reanalysis products is described, here are the main products that cover the longest period of time:

ECMWF Interim Reanalysis (ERA-Interim):  1979-present.  ERA-Interim was originally planned as an ‘interim’ reanalysis in preparation for the next-generation extended reanalysis to replace ERA-40. It uses a December 2006 version of the ECMWF Integrated Forecast Model (IFS Cy31r2). It originally covered dates from 1 Jan 1989 but an additional decade, from 1 January 1979, was added later. ERA-Interim is being continued in real time. The spectral resolution is T255 (about 80 km) and there are 60 vertical levels, with the model top at 0.1 hPa (about 64 km). The data assimilation is based on a 12-hourly four-dimensional variational analysis (4D-Var) with adaptive estimation of biases in satellite radiance data (VarBC). With some exceptions, ERA-Interim uses input observations prepared for ERA-40 until 2002, and data from ECMWF’s operational archive thereafter. See Dee et al. (2011) in the references below for a full description of the ERA-Interim system.

NASA Modern Era Reanalysis for Research and Applications (MERRA): 1979-present.  MERRA is a NASA reanalysis for the satellite era using a major new version of the Goddard Earth Observing System Data Assimilation System Version 5 (GEOS-5) produced by the NASA GSFC Global Modeling and Assimilation Office (GMAO). The Project focuses on historical analyses of the hydrological cycle on a broad range of weather and climate time scales and places the NASA EOS suite of observations in a climate context.

NCEP Climate Forecast System Reanalysis (CFSR):  1979-Jan 2010. The National Centers for Environmental Prediction (NCEP) Climate Forecast System Reanalysis (CFSR) was completed over the 31-year period of 1979 to 2009 in January 2010. The CFSR was designed and executed as a global, high resolution, coupled atmosphere-ocean-land surface-sea ice system to provide the best estimate of the state of these coupled domains over this period. The current CFSR will be extended as an operational, real time product into the future.

NOAA CIRES 20th Century Reanalysis V2 (20CR):  1871-2008. The 20th Century Reanalysis version 2 (20CRv2) dataset contains global weather conditions and their uncertainty in six hour intervals from the year 1871 to 2008. Surface and sea level pressure observations are combined with a short-term forecast from an ensemble of integrations of an NCEP numerical weather prediction model using the Ensemble Kalman Filter technique to produce an estimate of the complete state of the atmosphere, and the uncertainty in that estimate. Additional observations and a newer version of the NCEP model that includes time-varying CO2 concentrations, solar variability, and volcanic aerosols are used in version 2. The long time range of this dataset allows scientists to examine better long time scale climate processes such as the Pacific Decadal Oscillation and the Atlantic Multidecadal Oscillation as well as looking at the dynamics of historical climate and weather events. Verification tests have shown that using only pressure creates reasonable atmospheric fields up to the tropopause. Additional tests suggest some correspondence with observed variations in the lower stratosphere.

Atmospheric reanalyses comparison table is found [here].

Reanalyses intercomparison and plotting tools are found [here].

JC comments.  I’ve used the first generation reanalysis products from ECMWF and NCEP quite extensively, they are invaluable tools albeit with  significant limitations.  I’ve also started using the ECMWF Interim Reanalysis, which is a substantial improvement, and am also looking at the MERRA, which looks quite good especially for clouds and hydrological cycle.  These products are tremendously useful for a variety of applications, but the new shouldn’t be used for trend analysis without more assessment of their capabilities.

Those of you who don’t like models probably won’t like reanalysis products.  But this is the best alternative  in the face of incomplete and inconsistent data sets.  Such state estimation using inverse modeling and data assimilation is far preferable  to statistical “homogenization” and use of EOFs to fill in for missing data.

134 responses to “reanalyses.org

  1. J Storrs Hall

    Thanks for the pointer!
    Another piece of interesting work on pulling models out of data can be found at
    http://creativemachines.cornell.edu/eureqa

  2. “Those of you who don’t like models probably won’t like reanalysis products. But this is the best alternative in the face of incomplete and inconsistent data sets. ”

    I found independent evidence that the NCEP reanalysis is probably better than previously thought. It is evidence which raises a very interesting question about what it is that controls humidity in the upper troposphere:
    Enhanced GHE or Solar cycle amplitude?

    • No disrespect meant but does “better” here mean it confirmed what you hoped, or expected it would confirm?

    • So do you accept the radiation models the re analysis employed?

      • Dunno, I haven’t studied them. If I found I needed to tweak them to make them acceptable to me, would that make the correlation disappear? I suspect not, but can’t be certain.

      • You are missing the point. You like the data. That data was produced by physics models. Those physics models are inconsistent with your view of how radiation works. So, you kinda have to choose

      • Tell me Steve, how do you think I think radiation works?
        I don’t have any disagreement with the mainstream on how radiation works that I’m aware of, but do please give me the benefit of your interpretation of my view. Maybe I’ll learn something about how I’ve communicated what I think. :)

      • Sorry, but you are completely missing the issue. Please read moshers other points.

        You appear to love the resulting data without regard to the process by which the conclusion was accomplished. In English- it is not the responsibility of someone pointing out a model is unsuccessful to develop a good model.

        Completely different skill sets.

      • Who is saying which model is unsuccessful, and on what grounds?

  3. Bruce Cunningham

    I didn’t follow each argument point made by everyone in the previous thread, but it seemed that some were saying that you divide the total surface area of the Earth by 4 to get the average over the surface, since the area of a sphere is 4 x pi x radius squared, and pi x radius squared for the area of the circle. But since the sun only shines on half the area of the sphere, that is wrong by a factor of 2. Your eyeball estimate seems correct to me Dr. Curry.

    • John Vetterling

      Bruce,

      Your number is correct if you are talking about the solar radiance during daylight hours. But for the entire 24 hrs you divide by 4.

      The math is as follows:
      total solar radiation at 1 amu unit is multiplied by the area of a disk with the radius of the earth – area = pi x Re^2. That gives the average radiation in W for the disk. To convert that to the effective radiance on the surface divide by the surface area 4 x pi x Re^2.

      But (pi x Re^2)/(4 x pi x Re^2) = 1/4.

      • Bruce Cunningham

        Judith’s comment seems to have disappeared. However, I was not sure of from which direction the discussion was coming (ie was the radiation measured at the surface looking up, or from space looking at the sun), but someone was off by a factor of 2. Dr. Curry said she had checked and found what she thought was the answer. I was agreeing that one of the sides was off by a factor of 2. This type of error happens all the time (even to me!). I don’t know if it is worth going back and reading all the back and forth in detail to find out who was right. Probably not.

        Regards

      • Bruce,

        Interesting mechanical differences from inverting a power generation turbine to the problems that the current system has with speed and the interference from centrifugal force.
        Inversion actually allowed the mechanics to work with centrifugal force compared to the friction of centrifugal force interfering.

    • There’s another piece to the puzzle. The sun shine does not fully impact the regions around the edges of the 1/2 of the Earth where it is daytime. There, the “shine” is estimated by the angle (the shine is less powerful at sunrise & sunset) and the approximation cuts the total power by 1/2. The result is a crude approximation that solar power averaged over the globe is 1/4th of the power at the tropics at noon.

      • Gene,

        Add in planetary tilting to spice up the complexity.

      • I am starting to believe in the para-normal, as the world lives thru an
        updated version of Groundhog Day,
        too. Scientists abound & star in this moving drama—‘The Guest Scientists Who
        Refused To Leave’, a long film, all about that incredibly-dangerous gas,
        CO2…

  4. Judith
    I am at a bit of a loss as to how you expect discussion on this thread to proceed.

    Yes , the process described can help create a model that can replicate the past as best as it is know by the modeller. As more of the actual variables and relative weightings are known the models will become theoritically more accurate.

    Ultimately, isn’t the validation of a model the ability to accurately forecast future events or conditions with a high degree of accuracy if variables remain as they were in the model? (Example-the model did not take into consideration an event, say a volcano; and was inaccurate due to a major unpredicted variable).
    Ultimately, there are many ways to develop models, but validating that they are accurate BEFORE being used for policy decisions seems like something that has not been done in the case of climate modeling.

    • Agreed.

      There seems to be a confusion about what is ‘validation’ and what is ‘in-process optimisation’ in the climate (model) community.

      A fully accept that they can be used to infer some useful information; to give clues about interesting aspects of research. However i am still yet to see anything that re-assures me about their capabilities.

      For example, constantly fine tuning a model to meet emerging discrepancies works, to a point. If this is a continual and on going process then it suggests a more fundamental issue with the model, rather than something that needs optomising.

    • Ultimately, isn’t the validation of a model the ability to accurately forecast future events or conditions with a high degree of accuracy if variables remain as they were in the model?

      A theory (and models are simply theories with number crunching) should both predict the future and explain the past — it should be consistent with previous observations as well and present and future observations.

      While in a perfect world, we would test all of the predictions of theory against observation many times, unfortunately the nature of the phenomenon being investigated and the human cost sometimes limits how much predicting we can do. For example, cosmologists have various ideas, based in physics and observations, about how the world is going to end. Obviously no one can test them.

      This is a big problem in medicine, because a lot of things that are potentially harmful to people or just rarely rare events are hard to investigate.

      • Robert
        Using your medical example-if someone was developing a model to simulate the growth of a certain type of cancer in humans, wouldn’t you compare how well the model predicted the cancer would spread vs. what actually happened in order to determine if the model was valid?
        If the model’s prediction did not match the actual change in the cancer over time would the results of the model be publicized? If the model could accurately predict the condition of the cancer two weeks into the future, but was inaccurate after that, would the conditions the model predicted for 10 years from now be of great value?
        IMO model developers (as they do in other engineering disciplines) need to state preciously what criteria their models can be expected to accurately predict over what period of time. Only then, can the results of the models be periodically checked to determine if the model is still valid for its stated purpose. Today’s climate models do not meet this simple criteria and are of limited value for policy makers.

      • R Starkey. Thank you for today’s laugh. I do think these model developers state preciously what can be expected in terms of accuracy, considering how raarely they do as you suggest. And I do think it sohould be precise and accurate enough for the determination of usefulness can be made. After all, the conclusion of Santer, et al, that leveling or cooling is not inconsistant with the models begs the question “Then what use are they (the models)?”

      • John
        I am not trying to communicate that models have no value, or that models should not be used. I am trying to point out that it appears the current climate models have been “oversold” regarding their capabilities.
        All models clearly have goals of what they are attempting to accurately forecast or predict. A climate model is no exception but should not be expected to be able to forecast all the variables accurately all the time. What is expected is for the modeler to declare what the model is expected to accurately predict over what timeframe.
        So called climate scientists do not seem to like to have this very basic discussion and evaluation of the models they propose to rely upon for policy decisions.

      • Rob, there is good reason to communicate that models MAY have no value. That reason(s) is based on Browning and Kreiss’s findings of using parameterizations such as convective adjustments and hyperviscosity to enable reducing intractable partial differential equations in an open water- multicomponent/multi-reacting air-energy system in order to get the model to converge. Browning and Kreiss showed that GCM’s are extended past the timeframe that are expected to be accurately predicted. Re-analysis MAY suffer, not as much, but that the re-integration of known and the predicted , the 7k-9k giving 80K, MAY reflect more the assumptions of the parameterizations than reflect reality and thus their use is of very limited value and SHOULD NOT be fit for extrapolation into the future without verification andvalidation including sensitivity testing.

      • Using your medical example-if someone was developing a model to simulate the growth of a certain type of cancer in humans, wouldn’t you compare how well the model predicted the cancer would spread vs. what actually happened in order to determine if the model was valid?

        Certainly, but you don’t forgo treating the disease, according to the best understanding you have, while you are refining the model.

        I don’t think anybody is talking about not comparing models with the world, are they? The lukewarmer critique, as I understand it, is more that we should sit on our hands and allow our society to radically alter the climate until our scientific model which elucidates exactly how hard the consequences of that are going to suck reaches a higher level of perfection.

        You might compare it to the problem of what antibiotic to start to treat an infection. When an infection presents to a caregiver they typically do not know what bug it is, how fast it is going to progress, how severe the consequences will be if it is left unchecked, or what drugs the bug is resistant too. The preferred approach is to send cultures (improve the model with more evidence) while simultaneously giving several powerful antibiotics that cover the most resistant versions of any of the likely culprits. Then, once you have more information, you can narrow antibiotic coverage or stop it altogether.

        On the other side, for lukewarmers, it seems, that they view climate change rather like a sore throat — mostly viral, a few bacterial, the bacterial one rarely serious except in the case of a few long-term sequela. In that case, you either forgo the antibiotic altogether, or you give a pretty narrow, junior-league antibiotic.

        Of course, conservative treatment like that is only practical when the model is very good, not when it is very bad. Conservative treatment is possible with strep throat because it has been studied to death and the odds it will be harmless vs harmful have been very carefully studied. In medicine, if you don’t have a good understanding of what is going on, you assume the worst until you can prove otherwise. This is what critics of lukewarmism are getting at when they say that the critique of our poor understanding and the conviction that we should react passively until we know more do not really mesh well with one another.

      • Robert wrote: The preferred approach is to send cultures (improve the model with more evidence) while simultaneously giving several powerful antibiotics that cover the most resistant versions of any of the likely culprits. Then, once you have more information, you can narrow antibiotic coverage or stop it altogether.

        Simultaneously you have to be alert to the possibility that the antibiotics do more harm than good, buy killing an allergic patient or killing the body’s natural biota when the infectious agent is already drug resistant, or threatening other harm to the patient when the problem is not even bacterial. Part of the skeptic’s critique is that, with knowledge as limited as it is now, we risk causing more harm than good by trying to shift time, effort and money into fighting CO2. We could depress global economic output 20% with no effect on temperature worth the cost.

        Put differently, on current knowledge, there is no known “no regrets” strategy.

    • Reanalysis = tweakin’ and fudgin’. Necessary hairy evils. To be acknowledged in detail when done, and avoided wherever possible.

  5. Judith,

    Pekka sent me off a few weeks ago to ensure that those who had made SST estimates had done so carefully and with regard to the problems. Your thread here assumes that the data we have are accurate. For SST, there is the uncomfortable fact that about 40 per cent of the oceans are hardly measured at all, and the rest of the sea has well-worn tracks, so to speak, rather than gridded data-points. I can see that Monte Carlo methods will give you a feel for the patterns, but my feeling is still that what we know we know very roughly indeed.

    What sort of models are we going to have when the data they are drawn from are so rubbery?

    • Don, this is a big problem in the pre satellite era, which is why many of the reanalyses start in 1979. The extended CIRES reanalyses use the SST with the fishy interpolation and EOF schemes, which is a major flaw in those reanalyses. Ocean reanalyses are beginning to be attempted, to try to address this problem; ultimately coupled ocean/atm models in data assimilation mode might give is the best historical sst fields prior to satellites.

      • Thanks, and indeed I’m happiest when we are dealing with post-1979 data. But they do not much help discussion about whether or not the rate of warming has been fast, or unprecedented, and thse two propositions have been at the core of the AGW scare.

    • Setting aside the measurement accuracy issue, the habit of creating data where none exists is appalling. There are some cases where interpolation and extrapolation are appropriate, and others where those methods are not. Where I live, the temperature varies by 10F across a 100 miles quite frequently. Perhaps, if, one models in the temperature variations across jet-stream boundaries, the model could “guess” where that weather driver was on a particular day & where the steep temperature gradient was.

      To make a pretty picture, interpolating & smoothing temperatures across the globe is fine. The resulting picture should not be treated as data. At the very least, coastal temperature anomalies should not be used to guesstimate inland anomalies, nor vice-versa.

      • Interpolation is a, troubling issue.

        However it seems that the ‘we don’t know enough to make a conclusion’ position is one that is simply not possible in this field.

      • I seem to recall a long CE post and thread giving lip service to acknowledging ignorance as a source of uncertainty (and vice versa). Much easier said (though not all that easy) than done, obviously.

    • Don Aitkin,

      I cannot recollect the case that you mention, but I certainly don’t claim that it didn’t happen.

      What I remember about the discussion on SST determination is that I argued on some factors that are likely to help in obtaining useful information from sparse data on SST and, but I didn’t claim to know, how far they do help quantitatively or what the accuracy would be.

      The accuracy of the reanalysis should be relatively well known, when it’s used to fill missing data point’s or perform interpolations on variables, on which extensive data exists from other times or nearby sites, because the reliability of that part of reanalysis can be checked against existing data. When the reanalysis is extended to cover persistent gaps in data that kind of cross checking is not possible and the results must be less reliable.

      The SST values are certainly a good example of a situation, where the reanalysis may provide reasonably consistent and plausible results, but in a way, where the risk of systematic errors is significant.

      • Pekka,

        It happened on the tonyb essay on ‘unknown and uncertain SST’ on 29 June. I agreed with Tony that some of the early data have to be, by their very nature, so unreliable and invalid that it makes no sense to use them to ‘show’ anything about trends.

        Your comments were to the point, and their core went like this (quoting you): ‘You may complain about the availability of information on the details of scientific work, but you have a very weak position, if you just assume that the scientists are stupid or dishonest and have not done their work properly. The original posting of TonyB is along these lines, when it appears to assume that everything is impossible, when he doesn’t know the resolution, and some of your comments give the same impression.’

        I had no answer to this, other than neither Tony nor I had suggested stupidity or dishonesty. Nonetheless, I felt that I had to go back to read the main papers to see what the scientists concerned had done to deal with error. I’m trying to do much the same work for land temperatures as well, so it was a reasonable expenditure of time.

        But I have to say that it is not always clear what people have done, and when they have done something like use a Monte Carlo simulation, the outcome is not accompanied by error estimates. And for missing data in the far southern and northern oceans, there is very little at all. People assume that one or two measurements can stand for vast swathes of sea (if sea can have swathes).

        To use your phrase above, ‘the risk of systematic error is significant’ — and if it is, how much faith can we place on the ‘data’? And if these data are then used to construct models which provide us with new ‘re-analysed data’, have we learned anything at all?

        I know there are those who feel that poor data are better than none at all, but I’m not one of them.

        Alas, that is where I came in, and now, weeks later, I am no more convinced than I was then that this stuff is much good at all. I would regard the SST data as relatively decent only in the latter 20th century.

      • Don,
        I comment on one point.

        Whether poor data is better than no data depends on the case.

        Sometimes it’s fully obvious that everybody is willing to use poor data, when nothing better can be obtained.

        In other cases including poor data can contaminate better data. The geographic coverage of the surface temperature measurements is a good example, when we are interested in the “global warming” defined as the change of average surface temperature.

        One argument says that to get the right result, we must cover the whole sphere. Therefore we must use best available data on areas with almost no measurements. That leads to giving single measuring stations very much weight and applying some model to extrapolate. That results in significantly increased statistical error in the result and to a unknown model error on top of that.

        The other alternative is to discard those areas and use a average of lesser coverage. That’s not a perfect approach, if we really want to know, what has happened to the global average, but it’s quite possible that it’s a better estimate of the change in the temperature even for the global change. Whether it is or not, it’s certainly a more accurate and reliable value for the average over the reduced coverage, and that may, indeed, be more useful than the global average with greatly increased uncertainties.

      • Pekka,

        I can’t disagree with what you say, but I infer that you will accept the current data with all their imperfections, and hope that in time we will have better. Until then, you accept what we have. Is that fair?

        For my part, I am doubtful that anyone can produce accurate estimates of temperature for points where none were ever taken, simply through the use of models and extrapolations. Estimates can and will be made, of that there is no doubt, but I doubt their validity and reliability. My doubts increase in company with the growth in the size of the area in question. Until the satellite era (1979 on), I feel like saying that we cannot have a sensible global average temperature based on SST, because too much of the oceans have too little real data about them.

        Thank you for engaging: I do learn a lot from your comments.

      • Don,

        I don’t really know, what you mean by “accept”.

        When science is being done it’s normally not necessary to specifically accept anything. When some value is needed, some value is used, but the uncertainty of the value should be recognized as a factor that makes the further results more uncertain.

        When practical decisions are taken, they must be based on something. That may mean that such factors are are significants, on which only poor data exists. Then that must be accepted, but of course taking the uncertainty into account.

        Accepting or not accepting particular data is not a single decision, but repeated with different criteria, whenever the variable has significance. Often the answer is clear, but even the most accurately known values may be considered too uncertain in some specific situations. That’s not uncommon in science.

      • In practical political and policy terms (and even in the process of constructing and concocting ‘climate science’ hypotheses and theories, apparently) “taking the uncertainty into account” seems to drop away and get short shrift as the process gets more distant from the data-gathering stage.
        Along the way, it often seems that the ‘reset’ button on uncertainty gets pushed to get around the math of multiplication of error ranges vs. addition or overlap, etc

  6. Dr. Curry,

    In your comments, you mention the uses of these products and their limitations. Could you expand a bit on those?

    • Some examples for I have used these products that pop into my mind: provide large-scale meteorological context for field measurements, provide surface flux forcing data for sea ice models, doing bias corrections for daily and seasonal weather forecast models, diagnosing the large scale environment associated with hurricane formation, evaluating regional climate model simulations.

  7. So now the climate community claims to know the global weather conditions on 6 hour intervals ever since 1871. Dare we call this hubris? When will it end?

    • David, what is done is to use weather forecast models in data assimilation mode. 4 times per day, operational weather models produce global analyses of all the weather variables at every point on the globe, assimilating the latest observations into the most recent 6 hour forecast. This produces the initial condition for the next integration of the weather forecast model. The reanalyses redo this using a consistent version of the model and cleaned up data (and data that came in too late to be assimilated in the forecast cycle). So the reanalysis products are really a cleaned up version of the analyses (initial conditions) used several times per day to initialize the weather forecast models.

      • They are not analyses, they are fancy interpolations of bad data. This has nothing to do with weather forecasts. We are talking about global weather every 6 hours in 1871, 1872, etc. Even worse they claim to know what the uncertainties are! This is arithmetic disguised as science.

      • There is a whole literature on inverse modeling and state estimation, coming from computer science, engineering etc. Google the words inverse modeling state estimation and see what pops up.

      • If you model the past ten thousand years and project it forward. you will project a stable temperature.

      • I understand what they are doing. It is applying it to climate and claiming the results are knowledge of the world that is wrong.

      • Dr curry, with respect, you cannot compare climatic modelling to engineering models (or indeed the processes used within). Not without serious ramafications for the climate models.

      • Judith, there is a difference between inverse problems widely used in science, and this climate science concept of ‘reanalysis’. In an inverse problem, you have some data, y, that is usually insufficient to find out what you want to know, x, and you employ various methods to make a best estimate of x from y (if I knew x I could easily find y). An example would be, if I have two photographs y of an irregular asteroid from two different angles, can I construct the 3D shape, x, of the asteroid. At all stages in this process the distinction between data and model analysis is clearly maintained.
        This is quite different from climate ‘reanalysis’ which appears to be an opaque muddling together of real data with numerical models.

      • There is a subfield of inverse modeling associated with large-scale dynamical, which involves models. These techniques are widely applied in a range of fluid dynamics problems, among others

      • Judith

        I don’t want to claim that EVERY bit if historic data we have is worthless, but surely David Wojick is right, and on the whole -especially in the example being quoted of forecasts back to 1871- not only are they ‘hubris’ but they are ‘oversold’ hubris,

        Our grasp of historic ‘global’ Land Temperatures is poor. Our grasp of Historic ‘global’ Sea levels is poorer, whilst we appear to have no grasp at all of historic SSTs. We can’t use such flimsy data in order to make far reaching pronouncements on public policy to deal with a ‘problem’ -AGW- which remains so intangible and ill defined.

        tonyb

      • Nebuchadnezzar

        “Our grasp of Historic ‘global’ Sea levels is poorer, whilst we appear to have no grasp at all of historic SSTs.”

        Yes we do. It’s not perfect, but no one ever said it is.

      • As much as I love science I see no way of stopping this rampage of exaggerated confidence, short of zeroing the climate research budget. Expecting meaningful reform is hopeless.

      • Zeroing in on the climate reaserch budget in the US would be an excellent idea. We are not currently getting appropriate return for the funds expended.

      • JC, that seems fine for a couple of days but how do you extend that very far beyond the period for which you have observations?

  8. I did a lot of work on inversion models for transient EM (TEM) in the early and mid ’90s.
    We used the term “forward model” for what AGW folks just call a model.
    We found that the quality of an inversion model is limited by the quality of the forward model it uses. The inversion model would run the forward model many times tweaking the input parameters until the forward model output resembled the observed data. If the forward model does not accurately describe the real world then the inversion will find a solution, but it will be meaningless.

    My point? Reanalysis of data using inversion will only produce useful results if we trust the basic (forward) models. I am not sure that they *are* trustworthy.

  9. Steve Fitzpatrick

    Judith,

    For certain using models to estimate missing data is a useful tool (eg, modeling historical changes in ocean heat content based on historical changes in ocean surface temperature and recently measured changes in ocean heat content; something, as I have done myself). However, I think that considerable care is needed to rationally evaluate the uncertainty level in reanalysis data, and especially to critically examine the credibility/accuracy of the model used in the reanalysis. The danger seems to me to place too much faith in the model; even a model that appears reasonably accurate with known data could very well be “right for the wrong reasons”….. and lead to terribly incorrect reanalysis data.

    • Steve, this is certainly correct. The three main 2nd generation reanalysis products – merra, ecmwf, ncep – do give some different results, and the differences are some measure of the uncertainty in all this.

  10. Alexander Harvey

    There was a get together at the Isaac Newton Institute for Mathematical Sciences about one year ago on the subject:

    Mathematical and Statistical Approaches to Climate Modelling and Prediction

    Presentations with video are archived here:

    http://sms.cam.ac.uk/collection/870907

    Amongst these are a handful that dealt specifically with Data Assimulation, Kalman Filters, Wavelet Analysis, the ECMWF was represented.

    Also there are presentations on the development of parameterisation schemes, FDT, MEP, statistical emulation,and statisitcal anaylsis of weather and climate in general.

    Some may find some, even all of it, interesting, but there is a huge amount of it (80+ videos ~60hours), and I am not recommending any particular videos, just advertising their existence.

    I found it useful in many details but also in general as a matter of gaining insight into the breadth of the subject, and from hearing first hand what modellers, analysts, and theorists, think are the strengths and weaknesses in their fields.

    Alex

  11. Are these models really models or are they fits?
    Were the constants used in the models determined independently of the model, or were they chosen by allowing them move until they fitted the known data?
    If it was a case of the latter, then these are not models, they are fits.

    • Doc, they are dynamical models, integrating the navier stokes equations, first law of thermodynamics, with subgrid parameterizations to deal with unresolved processes (e.g. clouds). Some previous threads describe climate models, check the climate model category https://judithcurry.com/category/climate-models/

      • Could there not be a better way of modeling these processes than relying on data grids? I.e., wouldn’t it make sense to model the various processes using spherical harmonics? I would think this could lead to drastically reduced model size, complexity, and computational demands.

        As an example, my structural engineers typically give me a NASTRAN model which includes flexible modes up to maybe 100 Hz. I go in and pick out the 3 or 4 dominant ones, and these become my model for the filters and feedback I will implement.

        Often, space engineers will use a truncated expansion of zonal harmonics, especially the J2 term, for predicting the path of an object in earth orbit accurately for weeks at a time.

        You wouldn’t need all these gridded points, once you have determined the dominant mode shapes and associated amplitude functions. And, the interpolation of the model over uncovered areas becomes merely an issue of resolution, instead of all these hand-waving, ad hoc, and somewhat shady methods I have seen discussed in the literature.

  12. Maybe we should end an investigation at the point at which observation limits leave us rather than turning to synthetic data churned out by models. Otherwise someone will say you’re making stuff up via modeling to satisfy an agenda. And yes – I think scientists make s***f up.

  13. Stephen Pruett

    Missing or unreliable data (outliers) are common in biological studies, and there are a number of methods to impute reasonable values for the missing values. This allows a wider range of analysis, such as machine learning methods. However, all studies I have seen of this type in biology include validation in which a portion of the data are not used in training the machine learning method but are held back for validation. The value of the model is then judged by how well it predicts the outcome for the validating data set. Is such a process routine in climate studies?

  14. Stephen Pruett

    Missing data or faulty data (e.g., outliers) are common in biology. There are methods to impute reasonable values for the missing values. This is useful because some modeling methods, like machine learning algorithms, do not respond well to missing data. However, in all the biological studies of this type I have seen, a portion of the data are held back for validation purposes. So, 90% of the data would be used to train the machine learning method and then 10% would be used with the model to predict the outcome. Models are then judged by the accuracy with which they predict the outcome for the validation data set. Is something like this routine in climate science?

    • An experiment like this could be much more useful then just determining how accurate the models are as a whole. By selecting for different topographical features you could make a determination of where the models were least reliable and this could help lead to information on systemic errors in the assumptions.

  15. “Those of you who don’t like models probably won’t like reanalysis products.”

    On the contrary you will find skeptics who use re analysis data to prove a point— Then they discover that it was created by a model.
    There is an example now on WUWT where the author uses NCEP data.
    Not sure if he knew what it was.
    And they are even more shocked to find out that weather models also have radiation code in them. Same thing with satillite images.

    • Steve

      Would you argee that it is also very possible to greatly appreciate the use of models, but believe that climate models are OFTEN over hyped and under validated.

      • That is my view as a modeler. One day while going over results that everybody loved and digging a bit deeper i found a huge “oh no, opps”
        but everybody like the answers and didnt care much that a bug produced the right answer. opps. So ya, for problems like climate you have the tools you have. Same with all big problems. You dont avoid using models like some dolts on here suggest. and model results are not suspect on their face, as some suggest. There is no bright line between observation and model cause its models all the way down. However, some models are more useful than others. Neither do you believe in your own bullshit. no matter how many lines of code you wrote or how many things it gets right.

        Requires judgment, not emotion. particulars, not generalities.

        except my generalities, which are ok. hehe

      • Steve Fitzpatrick

        Steve Mosher,

        Requires judgment, not emotion.

        For sure, but that might be hard for those who are active advocates, based, at least in part, on model projections. Gaining the concurring judgment of truly disinterested parties seems to me a very prudent step.

        Besides, most everyone loves their own work a bit too much. ;-)

  16. The problem is that the degrees of freedom are unknown. The primary utility of this artful view of conceiving of the world is to comfort researchers with the notion that they are being objective and not mere witchdoctors. Of course, the beliefs of mystics are not always false. But, they’re usually false.

  17. Dr. Curry,
    Thank you for the post and the links. While the approach is conceptually interesting, I’m certain you will find that many people will be skeptical of this tool in the hands of climate scientists. The “reanalysis” data of NCAR sounds a lot like the “value-added” data of CRU. It will take a great deal of time to dig into the adjustments and “cleaning up” done by people that contributed to CRU email exchange. At first glance, this does not inspire confidence.

    In addition, you write:
    Reanalysis is a scientific method for developing a comprehensive record of how weather and climate are changing over time. In it, observations and a numerical model that simulates one or more aspects of the Earth system are combined objectively to generate a synthesized estimate of the state of the system.

    When you write the model “simulates one or more aspects of the Earth system,” are we to understand one of the aspects modeled is the level of atmospheric CO2 or climate sensitivity to CO2?

    If so, shouldn’t the reanalysis products point out that based on observations climate sensitivity is not very high? The point was made by the recently published Lindzen and Choi paper being discussed now on WUWT. See http://wattsupwiththat.com/2011/08/16/new-paper-from-lindzen-and-choi-implies-that-the-models-are-exaggerating-climate-sensitivity/

  18. Dr Curry,
    The more we get into this subject, the less at ease i am with the modelling.

    For example, why are the models constantly fine-tuned? I can understand it up until a certain point; this is normal optimisation and all fine and above board. But if the fine-tuning is continuous, surely that means that a base assumption/parameter in the model is incorrect.

    Extrapolating that further, we KNOW we don’t fully understand the climate and all it’s interconnected processes, so modelling it is tricky, which then means any results generated are suspect also.

    This whole validation process really puzzles (and worries) me. for example, are an ensemble of climatic parameters used, evaluated, screened and then improved? or are all models operating on the co2=bad meme?

    Also, how large are the modifications? is there a cut off on the size/sign of a modification before the model is canned or is it a continuous, almost untraceable process?

    • For numerical weather prediction, an individual simulation starts to diverge from reality after 2-5 days, owing to imperfect initial conditions and imperfect models. Hence numerical weather prediction initializes an ensemble of simulations 2-4 times per day, incorporating the latest weather and satellite observations. This is how we get the best possible weather predictions with our our modeling and observing capability. The most sophisticated type of data assimilation used in weather models is 4DVAR, which is described on the ECMWF web page http://www.ecmwf.int/products/forecasts/guide/The_4DVAR_analysis_procedure.html

      This whole topic is pretty complicated, i’m not sure how to talk about it on a blog, but i think it is important to introduce this topic and try to generate some understanding about it.

      • Thanks for the response. I understand that this is not an easy subject to discuss.

        Ok, that’s how i understood that weather models worked, which is good (means i’m not way off). Though i’m still stuggling (not least to articulate) with the climatic versions.

        With a weather model, we’d know quite quickly how ‘bad or good’ they were and ditch the bad ones and concentrate on the good. You’ve then got a ‘stable base’ for the model to which subsequent fine tuning or optimisation can occur. Then of course you have the constant updating of information to try to keep the simulations accurate.

        With climate models, which are based on weather models, we don’t have this ‘check’.

        If a weather models accuracy is 2-3 days (5 at a push), what is a climate models? How does it improve the predictability? The cycles involved are longer, so is this a function of the shortest inputted cycle?

      • For climate models, some of them have the same core as the weather model (e.g. Hadley, ECHAM). They are then intialized at one specific time, with some small perturbations in initial condition to create an ensemble. On the longer time scales of the climate models, the climatology is kept reasonable by the boundary conditions, including the annual cycle of solar radiation. But climate model simulations don’t have any meaning in terms of a specific weather situation at a specific location, only in the context of statistics for a region, season.

      • Dr. Curry,
        Thank you for the links.
        In my industry, a statistically significant number of data points is a fairly large number. How big are the numbers of events considered statistically meaningful in climate science?

      • Ah, i wasn’t suggesting that the climatic models would predict weather Dr Curry- just the approximate climatic states.

        I like the idea of the ensemble runs- seems sensible. What’s the next step then. You run an ensemble and they all go ‘wide’. Is the model discarded/overhauled or are adjustment factors put in to bring it ‘in line’.

        If the latter, how are the adjustments applied? are they arbitary to meet the observational results? are observational results even used in this context?

        Also, i take it all the (mainstream) climate models take the IPCC view on C02?

      • only in the context of statistics for a region, season.
        If the climate models have all over-estimated the temperature increase for a decade, can they be presumed to have improper boundary conditions? Two decades? How long does it take to invalidate a climate model?

      • That’s a great link Dr Curry, thanks.

        I guess i’m just trying to understand how the model validation process works. I don’t see anything that is akin to an engineering validation (though of course i would never expect somthing SO strict here).

        I’m worried that the models, rather than being changed/re-written upon failure, are simply adjusted or retro-fitted.

    • Lab,

      It’s cool to see your light bulb come on.
      Takes time when much of the process is wrapped in confussion.

      • It’s very difficult to get to specifics. Dr Curry’s doing her best and i must admit that my questions could be clearer (i’m often writing as i think as it were- hence the horrendous typo’s at times).

        I’m an industry scientist so am a BIG fan of defined, controlled and RECORDED procedures. I of course know they are not always applicable of possible, but even in the research environment procedures (especially for experiments) have to be in place. With models i don’t see any.

        It is entirely possible that this is simply because i’m looking in the wrong places, but as models are so integral to the cAGW meme, i’m dissapointed that it’s not more available.

        AS it stands i’m just deeply worried about the models and i’m trying to get to the bottom of that.

      • Climate science has generated much confusion.

        Question:
        Why was BTU’s not separated from the CO2?
        Because then CO2 statements would look awfully stupid.

  19. Hi Judy and everyone,
    I am a purist when it comes to data and model output. The danger with reanalysis output is that the source gets lost and scientists and students start treating it like data. It is very nice working with these products. No data gaps! No time wasted with image making software to fill in the empty boxes. It is all done for us. The danger, as those before me point out, lies in interpreting them for purposes other than their original intent. Changes in ocean temperature come to mind.

    In my own research, I am comparing NCEP data to satellite measured data (which also includes models in the processing, so there is no such thing as “pure” data, unless it is a thermometer). If there is a difference, it might be useful to quantify it and determine if or how it affects the rest of the products.

    I found the words “synthetic data” almost appropriate, but it is still deceptive.
    Rose

    • Rose, good point. In the 1990’s, there was a certain segment of the World Climate Research Programme that wanted to declare the reanalsyses as the definitive “data”, fortunately that was hammered down. But too many people use these reanalyses like real data, particularly to look for trends in the climate, something for which at least the first generation reanalysis products were very ill suited for.

      • It probably should not be called data at all, as data in science normally means measurements. These are theoretical constructs. I would call it fabricated data, with all that this term implies. It sounds like data is being fabricated on a massive scale. What is the ratio of the number of real data points (measurements) to fabricated data points? A hundred to one or a billion to one? (And we thought the actual data was a problem.)

        So basically we have models generating data which is then fed into models to tell us how the world works and what a great threat we are to it. Sounds circular to me.

        Mind you I understand why you are doing it. Many forms of analysis require full fields of data. The fact that we do not have this data is overcome by fabrication. Algorithmic fabrication is still fabrication. Ironically, Rick Perry just made headlines by claiming that climate scientists are manipulating the data. Turns out he is right far more than he knows. http://www.star-telegram.com/2011/08/17/3297488/perrys-skepticism-over-global.html

      • David

        How about calling it ‘post modern data’ :)

        tonyb

      • Nebuchadnezzar

        David,

        If I want to measure the speed of a car I can measure the time it takes to travel a measured distance. If I then divide one by the other have I haven’t ‘fabricated’ the average speed: I’ve measured it. If I also have a measurement of the frequency shift of the car’s horn due to the doppler effect, I’ve measured a change in frequency, but, with the help of the speed of sound, I can use it to estimate the car’s speed and hence the time it takes to travel a particular distance. Using physics we can measure things without directly measuring them (my thermometer is a means of converting the length of a mercury column into a temperature to take another example). If we know the parameters within which the car’s engine and brakes work we can even make an estimate of the speed the car was moving between measurements.

      • And regardless of the method that you used to develop your model, there are very specific things that your model is designed to predict. Why aren’t those criteria fully published for each model and we can measure how well they did and then determine if we should use that model for decision making??

      • Is there any reason for extending these reanalyses back in time (all the way to 1871) besides looking for trends in climate?

        Reanalyses are basically models partially corrected by observations. When you change the number and quality of observations that are used to correct the model, you certainly create biases. Do any reanalyses address this problem by using a constant set of observational inputs?

      • Nebuchadnezzar

        Weather records are useful for all sorts of purposes Extending our knowledge of what happened back as far as possible gives us a window onto a much wider range of weather events, which in principle helps our understanding of what can happen and is likely to happen in the future.

        Granted, reanalyses don’t have quite the same appeal as direct observations, but if used with appropriate care, there’s no doubt they can improve our understanding of the climate, the models and the observations that we do have.

        The CIRES 20th century reanalysis uses observations of surface pressure and fields of sea-surface temperature in the reanalysis. This gives some consistency through time although there are still changes in data coverage. By using an ensemble approach they can assess the uncertainty in the reanalysis where there are few observations because the ensemble rapidly diverges when there are no observations to constrain it.

        As I understand it, other reanalysis centres are performing tests where they run the reanalysis with and without different observing systems to assess what biases are caused by their popping in and out of the record.

  20. Dr Curry, on a model-related topic, I would be interested in your view on the relationship between model developers and the ‘owners/developers’ of data, as discussed here:

    http://progcontra.blogspot.com/2011/08/i-love-my-model.html

    • Prog, nice article. Actually, many of the developers of climate models are pretty realistic about their limitations. It is the users of the models that may love them a bit too much.

      • Thanks. I’m interested in the levels of scrutiny of how the models are constructed? Do the users view them essentially as black boxes? I understand that the model outputs can be peer reviewed, but what about the code? How is that scrutinised to expose the underlying assumptions implicit in the model and the correctness of the code?

      • Judith: Skim the AR4 chapter on Climate Models and Their Evaluation. Most of the chapter is devoted to comparisons between models. For comparisons to observations, readers are referred non-specifically to three other chapters. There is a short discussion about how effectively some weather phenomena (ENSO, Rossby waves, etc) are reproduced by models. There’s no discussion that limits the “users of the models” – who use them to make scary predictions – from making over-confident “projections”.

      • Yes, this is discussed at length in my previous post on confidence in climate models. recent efforts at validation are discussed in my previous post on climate model verification and validation.

  21. Judith,

    I still find it interesting that heat and carbon were married up together rather than looking at it as two separate processes.

  22. Leonard Weinstein

    Dr. Curry,
    My ScD is in fluid mechanics and thermal science. I studied fluid flow for almost 50 years, with emphasis on turbulent flows. Even the relatively more simple (i.e., not any mixed phases or storage or clouds, etc.) unsteady 3-D flows for which the basic NS equations are known are generally not able to be fully solved. The temporal and spatial resolution of initial and boundary conditions needed for such flows is not even close (and probably never will be), and there is not enough math known for a closed form solution. For this reason, even these flows are treated approximately, and these approximations can not be in general be extrapolated very far. When you include plugs for mixed storage and phase details (oceans and currents, evaporation, etc.), clouds (cosmic rays?), aerosols, etc., you are truly not going to be able to extrapolate very far and expect valid results. The plugs allow any degree of fit internally (for known data), but this is of NO use for extrapolation a distance away. The Physics is missing on much of climate prediction, and these are chaotic systems. Even if all of the pieces of Physics were determined, and resolution of initial and boundary conditions greatly improved, the extrapolation would still not be possible to extend very far before it failed. This includes trends as well as specific predictions.

  23. We can all hardly wait until science will be able to model the ‘true batting average’ (TBA) of all the ‘great’ juicing sluggers baseball has created over the past twenty years. But will scientists now be able to tell us who will win the pennant races every year? We are now all indebted for the tickets, why not let science have it’s fun. What’s fun; you don’t see…?

  24. My reservations about climate models are only increased by comments here. Leonard Weinstein’s comments concerning the problems in modelling complex systems, Prog Contra’s criticism of the incestuous process of climate modelling, the same people building the models, testing the models, confirmation bias built in and no compulsion to be brutally objective.

  25. Dr. Curry,
    I did a brief search on bing and ClimateAudit to see if Steve McIntyre or any of the other auditors had performed any audits on these reanalysis efforts. I was unable to locate anything. Do you know if Steve, JeffId, Lucia or anyone has undertaken any work on this? Also, do you know how open these groups are to explaining what they have done? Do they provide data, methods and code? Or are there outstanding FOIA requests trying to get them to share their data?

    • Indeed, imagine telling your auditor that most of your financial data has been “reanalyzed,” meaning fabricated using various formulas. This practice is ripe for abuse.

    • Ron, as far as i can tell, reanalyses have not been mentioned much in the blogosphere. on the reanalyses.org site, there is a ton of information, and also links to the individual groups doing the reanalyses. The model codes are not publicly available (certainly not ECMWF). the volume of data is HUGE, well beyond free lancers capability to deal with. The methods are publicly documented. In terms of auditing, i would recommend comparing the different reanalyses with each other and with actual observations.

      • Dr. Curry,
        I think you underestimate the freelancers. They have consistently bested the climate scientists in head to head disputes.

        Big datasets can be a problem. I am working on one now with more than 12 million rows and over 300 columns. It does present problems but they can be overcome.

        Comparing different reanalyses is not helpful – somewhat akin to comparing CRU with GISS with NOAA temp series. They may all somewhat agree and still be wrong. I am eager to see Richard Muller’s temp series to see how that changes the picture.

        I wonder who will be the first to FOI the model codes they have not archived?

      • “Ron, as far as i can tell, reanalyses have not been mentioned much in the blogosphere. ”
        My bet is that will change very soon.
        ;^)

  26. Three years ago at DotEarth I said the modellers were trying to keep their little trains on circular tracks on the ceiling.
    ==============

  27. I can see how inverse modeling might be used when the model was well established as accurate. This is precisely not the case with climate. Once again the modelers are leading the science.

    • Weather prediction models are certainly capable of making 6 hour forecasts, which is what is done in data assimilation for the reanalyses. The atmospheric models used in the reanalyses are the NCEP and ECMWF models used for operational weather forecasts.

      • Dr. Curry: Why do you keep referring to 6 hour forecasts when we are talking about reanalysis reconstructions going back to 1979 and 1871? People are using these reanalyses to do trend analysis for these periods. 6 hour forecasts have nothing to do with this. Are we talking past each other.

      • http://www.agu.org/pubs/crossref/2011/2011GL047995.shtml is an example, a trend analysis from 1871 based on “reanalysis.”

      • Because the reanalyses are constructed by go back and making short reforecasts in the context of the data assimilation cycle. So each time period for the reanalysis is associated with the forecast model run for a short time initialized in data assimilation mode. Once you assemble the entire set of daily reanalysis products, then you can do trend analyses. Reanalysis is not like a climate model simulation that is initialized once and then integrated for years. During the reanalysis the model is initialized (via data assimilation) several times per day and integrated for a short period to produce the reanalyses for that day. And then on to the next day.

      • Surely the values attributed to 30 or 130 years ago do not change every 6 hours.

  28. The best model for the future of our climate is the record of our past climate. The past ten thousand years has been extremely stable in a narrow range. We will continue in that same narrow range. Water and ice on the surface of earth and water in the atmosphere in the form of water vapor, water drops and ice crystals is responsible for keeping earth in this pleasant state. CO2 is a trace gas and has a trace effect. Water in all its forms and the geometry of the continents and oceans is what keeps our temperature regulated. When we are warm it snow more. When we are cool it snows less. It is that simple.

  29. The only proven usefulness of parameterization is that they enable a complex model to be resolved; otherwise, there is not enough computing power on Earth to resolve GCMs—they’re too large: the limitations of the computing power that Earthlings have at their disposal is obstacle that we cannot realistically hurdle, even if we knew about and understand all of the variables that play a role in climate change. And, of course, because there is so much that we do not understand, GCMs are not large enough to actually simulate real-world conditions with any degree of accuracy and in any event the grid blocks that are used to reconstruct the world are too large.

    Weathermen know this: they know which way the wind blows. Weathermen understand that GCMs are not capable of realistically capturing the effects of as thunderstorms, hurricanes and other natural processes that transfer huge amounts of energy from the surface of the Earth to the stratosphere. As humans we cannot even begin to imagine what is going on around us. So, we end up using simple approximations of real-activity to represent actual physical processes.

    Our use of parameters is our way to fill in the gaps that result from our inability to realistically conceptualize nature. Moreover, our use of parameters allows us to invent anything we want; we can make GCMs say anything we believe. That is why climatology has been likened by some to the science of ancient astrology and to the casting of chicken bones or numerology and the of Tarot cards to foretell the future.

  30. Judith

    Looking through the various sea papers you cited on the postma thread, I couldn’t help noticing that you seem to make a habit of researching warm tropical seas and routinely quote temperatures well into the 20’s Centigrade. Your research into much colder waters (such as the chilly English Channel just outside my house) seems to be noticeably lacking. Smart person. :)

    Having been involved in Sea level measurements, I think the satellite data has serious potential flaws and would much prefer to use tide gauges. Consequently I am somewhat sceptical at the idea of a satellite accurately measuring the skin temperature of the ocean, then extrapolating/adjusting the results in order to come up with accurate temperatures for 0.5 and 1 metre.

    Subsequently, any attempt to compare all that satellite collected adjusted data to the data from historic SSTs is mixing apples and oranges. Surely its far better for such as Hadley (although they are by no means the only cuplrits) to admit that making modern comparisons with historic data is highly problematic and the certainties they express (e.g. tenths of a degree) are somewhat illusory.

    Consequently any ‘reanalysis’ based on the sort of sketchy real world data available from the past must be somewhat problematic.
    tonyb

  31. This is a repeat of a post up above, because the sub-thread appears to have gone stale. If anyone has info on whether any investigators are doing this already, I would be interested in knowing.

    Bart | August 20, 2011 at 4:14 am | Reply

    Could there not be a better way of modeling these processes than relying on data grids? I.e., wouldn’t it make sense to model the various processes using spherical harmonics? I would think this could lead to drastically reduced model size, complexity, and computational demands.

    As an example, my structural engineers typically give me a NASTRAN model which includes flexible modes up to maybe 100 Hz. I go in and pick out the 3 or 4 dominant ones, and these become my model for the filters and feedback I will implement.

    Often, space engineers will use a truncated expansion of zonal harmonics, especially the J2 term, for predicting the path of an object in earth orbit accurately for weeks at a time.

    You wouldn’t need all these gridded points, once you have determined the dominant mode shapes and associated amplitude functions. And, the interpolation of the model over uncovered areas becomes merely an issue of resolution, instead of all these hand-waving, ad hoc, and somewhat shady methods I have seen discussed in the literature.

    • Bart, many climate models do use spherical harmonics. It is only recently that grid-point methods are becoming more popular for global models.

      • Then, why are Hansen et al. extrapolating over the poles from the nearest stations? You would not need to do that to get a spherical harmonic fit. You would not need gridded data at all.

  32. Incidental Note: Trenberth.pptx, referenced above, can be opened with Open Office 3.3 “Impress”.

    I did not find “talking points” (Notes) for some of the slides (e.g., 23, 25, 26 …)