Example pipeline for ERA5

[alxmrs]

Source Dataset

• Link to the website / online documentation for the data
  • https://www.ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-v5
• The file format (e.g. netCDF, csv)
  • netCDF, Grib2
• How are the source files organized? (e.g. one file per day)
  • Data is accessible by querying the MARS API from ECMWF (docs here).
  • Data is accessible from the MARS API via a selection DSL (examples, syntax)
• Any special steps required to access the data (e.g. password required)
  • ECMWF keys are required (docs).
  • Copernicus (CDS) provides fast access to ERA5.
  • If data is not listed in CDS, ECMWF provides slow access to the data, which is stored on tape drives. Care needs to be taken to access data in this archival format. Consider reviewing the retrieval documentation, especially the "retrieval efficiency" section.
  • ERA5 files can be quite large (~1GB in size per query). Downloading jobs should be partitioned (select smaller subsets of the overall data).

Transformation / Alignment / Merging

Output Dataset

Ideally, the datasets would be converted into a single Zarr with a 10-100MB chunk size.

[rabernat]

Thanks for sharing this @alxmrs! ERA5 is definitely on our roadmap. We have a formal partnership with ECMWF around this. But it is a big job! We will keep you updated.