by Judith Curry
This proposal by John Bates of NOAA NCDC nails what is needed in terms of climate data records.
The material discussed here is from a ppt presentation dated April 2011, by John Bates, entitled Assessing Climate Data Record Transparency and Maturity. Excerpts from the text:
Overview
- Climate variability and change are having profound effects on society
- Society deserves full and open access to the data and methods used to produce climate products
- Scientists often use nomenclature and methods that are difficult for non‐ specialists, and sometimes even specialists, to understand
- •Scientists must do a better job in standardizing their nomenclature and methods, and more clearly communicate with the public
Motivation
What is at stake?
History shows that weather observations did not become useful for society until a lexicon was agreed to
- The Beaufort scale did this for wind climatology and maritime commerce in the 19th century
For The Climate Service to benefit society, it must adopt a lexicon that sets expectations for openness, process and transparency that are accessible to the public
- How might we define a climate record lexicon useful to both scientists and the general public in the 21st century?
Climate Observation Business Practices
What common preservation information do we need and how do we capture and communicate this?
- Reference information: where can products be easily found?
- Provenance information: what original observations were used in the product?
- Context information: what methods were used to create the product?
- Fixity information: how do we ensure the authenticity of the product?
Maturity Matrix
Let’s define a Maturity Matrix (1=low; 6=high) that sets expectations and assesses progress in the following areas:
- Collection methods: how was the data collected, sensors, surveys, etc.?
- Algorithm stability: are algorithms under configuration management and how mature?
- Metadata and QA: how full and complete are the metadata and quality assessment?
- Documentation: is the operational algorithm description full, complete, and peer reviewed?
- Validation: how complete is the validation?
- Public release: are the data, algorithms, and software open and available to the public?
- Science & applications: how extensive is the peer reviewed literature and how varied are the applications?
- Initial
- Experimental
- Provisional
- Demonstrated
- Sustained
- Benchmark
What is the role of IT?
- The Maturity Matrix sets expectations, but needs enabling information technology to be broadly useful
- Climate Services, to be relevant to society, needs to be accessible to both expert and non‐expert communities
- We need what has been dubbed ‘Knowledge provenance’* – that is, for the content of the Maturity
- Matrix to be truly useful it must be enriched with semantics and semantically-aware tools
JC comments: I look forward to seeing this more fleshed out. I wish John Bates well in getting this implemented, it is certainly what is needed for climate data records.
More information can be found at the NOAA CDR web site.