Download notebook Run this notebook

Climate Adaptation Digital Twin: climatological analysis of temperature in Germany#

This notebook will provide you guidance on how to access and use Climate Adaptation Digital Twin (Climate DT) datasets on Earth Data Hub (EDH). Access to these datasets is granted via Destination Earth’s Standard API keys, but restricted to users with Upgraded Access.

In this tutorial we will learn how to use Xarray to access the Climate DT standard resolution hourly data on single-level or surface, IFS-NEMO, SSP3-7.0 dataset on EDH. However, you can easily replicate the procedure for any other dataset belonging to the same collection.

If you are running this tutorial in your own environment, you can download the convenience modules costing.py and display.py from DESP-UserWorkflowService-Templates/EarthDataHub

What you will learn:#

  • how to access the dataset

  • select and reduce the data

  • plot the results

In this notebook we set two goals:

Our first goal is to plot the average projeceted 2 metre temperature in 2026 over Central Europe (Germany).

Our second goal is to compute the 2 metre temperature climatology (monthly means and standard deviations) in Berlin for the 2020-2049 reference period.

Working with EDH data#

Datasets on EDH are typically very large and remotely hosted. Typical use imply a selection of the data followed by one or more reduction steps to be performed in a local or distributed Dask environment.

The structure of a workflow that uses EDH data typically looks like this:

  1. Data access

  2. Data selection

  3. Data reduction (optional)

  4. Data download

  5. Further operations and visualisation

Xarray and Dask work together following a lazy principle. This means that when you access and manipulate a Zarr store the data is in not immediately downloaded and loaded in memory. Instead, Dask constructs a task graph that represents the operations to be performed. A smart user will first reduce the amount of data that needs to be downloaded and explicitly call compute() on it. Once the compute() operation is complete the data is loaded into memory and available for subsequent fast processing.

1. Data access#

[ ]:

# Optional: ignore warnings
import warnings
warnings.filterwarnings('ignore')

To access any data on EDH, you need to obtain an API key. To obtain an API key you need to register to the DestinE platform. Then, you can go to Earth Data Hub Account Settings - Quota & API keys where you can find out your default API key or create others.

Finally, you need to instruct your tools to use your API key when downloading the data. One way to do it, is to incorporate the API key into the dataset’s url. If you wish to follow this method, cut and paste your personal API key below: ⤵

[ ]:

EDH_KEY = "your_EDH_API_key"
#e.g. EDH_KEY="edh_key_MHOQO_Y_h4ywpaMQe4cndtydvbpU6ySBNARWprm5tgz"

url = f"https://edh:{EDH_KEY}@api.earthdatahub.destine.eu/climate-dt-2/IFS-NEMO-SSP3-7.0-sfc-hourly-standard-v0.zarr"

There are more ways in which you can provide your API key to Xarray, especially if you plan to use the system frequently. Please refer to https://earthdatahub.destine.eu/getting-started for a complete overview.

We are now ready to access the data with Xarray:

[ ]:

# Optional: ignore warnings
import warnings
warnings.filterwarnings('ignore')

[ ]:

import xarray as xr
xr.set_options(keep_attrs=True)

ds = xr.open_dataset(
    url,
    chunks={},
    engine="zarr",
    zarr_format=3,
)
ds

<xarray.Dataset> Size: 22TB
Dimensions:       (time: 306816, latitude: 512, longitude: 1025)
Coordinates:
  * time          (time) datetime64[ns] 2MB 2015-01-01 ... 2049-12-31T23:00:00
  * latitude      (latitude) float64 4kB -90.0 -89.65 -89.3 ... 89.3 89.65 90.0
  * longitude     (longitude) float64 8kB -180.0 -179.6 -179.3 ... 179.6 180.0
Data variables: (12/36)
    avg_ie        (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    avg_iews      (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    avg_inss      (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    avg_ishf      (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    avg_sdlwrf    (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    avg_sdswrf    (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    ...            ...
    tciw          (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    tclw          (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    tcw           (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    tcwv          (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    u10           (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
    v10           (time, latitude, longitude) float32 644GB dask.array<chunksize=(1440, 64, 64), meta=np.ndarray>
Attributes:
    GRIB_edition:            2
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          1003
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts

⚠ At this point, no data has been downloaded yet, nor loaded in memory.

2. Data selection#

First, we perform a geographical selection corresponding to the Germany area. This greatly reduces the amount of data that will be downloaded from Earth Data Hub.

[ ]:

t2m = ds.t2m
t2m_germany = t2m.sel(**{"latitude": slice(47, 55), "longitude": slice(5, 16)})
t2m_germany

<xarray.DataArray 't2m' (time: 306816, latitude: 23, longitude: 31)> Size: 875MB
dask.array<getitem, shape=(306816, 23, 31), dtype=float32, chunksize=(1440, 23, 31), chunktype=numpy.ndarray>
Coordinates:
  * time       (time) datetime64[ns] 2MB 2015-01-01 ... 2049-12-31T23:00:00
  * latitude   (latitude) float64 184B 47.03 47.38 47.73 ... 54.07 54.42 54.77
  * longitude  (longitude) float64 248B 5.273 5.625 5.977 ... 15.12 15.47 15.82
Attributes: (12/20)
    GRIB_paramId:                    167
    GRIB_dataType:                   fc
    GRIB_numberOfPoints:             196608
    GRIB_typeOfLevel:                heightAboveGround
    GRIB_stepUnits:                  1
    GRIB_stepType:                   instant
    ...                              ...
    GRIB_shortName:                  2t
    GRIB_units:                      K
    long_name:                       2 metre temperature
    units:                           K
    standard_name:                   air_temperature
    last_ingested_time_index:        306815

Second, we further select the year 2026. This is again a lazy operation:

[ ]:

t2m_germany_2026 = t2m_germany.sel(time="2026")
t2m_germany_2026

<xarray.DataArray 't2m' (time: 8760, latitude: 23, longitude: 31)> Size: 25MB
dask.array<getitem, shape=(8760, 23, 31), dtype=float32, chunksize=(1440, 23, 31), chunktype=numpy.ndarray>
Coordinates:
  * time       (time) datetime64[ns] 70kB 2026-01-01 ... 2026-12-31T23:00:00
  * latitude   (latitude) float64 184B 47.03 47.38 47.73 ... 54.07 54.42 54.77
  * longitude  (longitude) float64 248B 5.273 5.625 5.977 ... 15.12 15.47 15.82
Attributes: (12/20)
    GRIB_paramId:                    167
    GRIB_dataType:                   fc
    GRIB_numberOfPoints:             196608
    GRIB_typeOfLevel:                heightAboveGround
    GRIB_stepUnits:                  1
    GRIB_stepType:                   instant
    ...                              ...
    GRIB_shortName:                  2t
    GRIB_units:                      K
    long_name:                       2 metre temperature
    units:                           K
    standard_name:                   air_temperature
    last_ingested_time_index:        306815

Due to the chunked structure of the DataArray, xarray must download every chunk that includes a portion of the selected data.

To estimate the size of the download, we can use the costing.py module. This must be done before we apply any reduction operation.

[ ]:

import costing

costing.estimate_download_size(t2m_germany_2026)

Estimated download size: 0.035 GiB (36.0 MiB)

3. Data reduction#

Before downloading the data, we can perform further lazy operations. Here we compute the 2 metre temperature yearly average:

[ ]:

t2m_germany_2026_mean = t2m_germany_2026.mean(dim="time")
t2m_germany_2026_mean

<xarray.DataArray 't2m' (latitude: 23, longitude: 31)> Size: 3kB
dask.array<mean_agg-aggregate, shape=(23, 31), dtype=float32, chunksize=(23, 31), chunktype=numpy.ndarray>
Coordinates:
  * latitude   (latitude) float64 184B 47.03 47.38 47.73 ... 54.07 54.42 54.77
  * longitude  (longitude) float64 248B 5.273 5.625 5.977 ... 15.12 15.47 15.82
Attributes: (12/20)
    GRIB_paramId:                    167
    GRIB_dataType:                   fc
    GRIB_numberOfPoints:             196608
    GRIB_typeOfLevel:                heightAboveGround
    GRIB_stepUnits:                  1
    GRIB_stepType:                   instant
    ...                              ...
    GRIB_shortName:                  2t
    GRIB_units:                      K
    long_name:                       2 metre temperature
    units:                           K
    standard_name:                   air_temperature
    last_ingested_time_index:        306815

3. Data download#

This is the phase where we explicitly trigger the download of the data. To do so we will call compute() on the previously averaged temperature. The result will be small enough to easily fit into memory. Remember to assign the return of the compute() function to a new variable, so that the data is kept in memory.

We can measure the time it takes:

[ ]:

%%time

t2m_germany_2026_mean_computed = t2m_germany_2026_mean.compute()

CPU times: user 1.08 s, sys: 311 ms, total: 1.39 s
Wall time: 2.32 s

The data was very small, this didn’t take long!

5. Visualisation#

Finally, we can plot the 2026 average temperature on a map:

[ ]:

import display
import matplotlib.pyplot as plt
from cartopy import crs

[ ]:

t2m_germany_2026_mean_computed_degC = t2m_germany_2026_mean_computed - 273.15
t2m_germany_2026_mean_computed_degC.attrs["units"] = "°C"

[ ]:

display.map(
    t2m_germany_2026_mean_computed_degC,
    projection=crs.Miller(),
    vmax=30,
    vmin=-20,
    cmap="RdBu_r",
    title="Mean Surface Temperature, 2026"
);
../../_images/notebooks_insula-code-lab_tutorial-climate-dt-climatological-analysis-of-temperature-in-germany_27_0.png

2 metre temperature climatology in Berlin, 2020-2049#

The power of EDH is better showned when working with timeseries. We will now show how fast it is to compute the 2 metre temperature climatology (monthly mean and standard deviation) in Berlin, over the reference period 2020-2049.

With legacy data distributon systems you would need to download the entire world temperature for the reference time period, extract the Berlin data and do your calculations. Alternatively the data provider might do a crop and merge operation for you, but chances are that this is very slow (internally the data provider still accesses the entire world temperature, you just don’t see it!). Thanks to Earth Data Hub this is not needed anymore. You only need to download the relevant chunks.

Here, we select the closest data to Berlin:

[ ]:

t2m_Berlin_2020_2049 = t2m.sel(**{"latitude": 52.5, "longitude": 13.4}, method="nearest").sel(time=slice("2020", "2049"))
t2m_Berlin_2020_2049

<xarray.DataArray 't2m' (time: 262992)> Size: 1MB
dask.array<getitem, shape=(262992,), dtype=float32, chunksize=(1440,), chunktype=numpy.ndarray>
Coordinates:
  * time       (time) datetime64[ns] 2MB 2020-01-01 ... 2049-12-31T23:00:00
    latitude   float64 8B 52.66
    longitude  float64 8B 13.36
Attributes: (12/20)
    GRIB_paramId:                    167
    GRIB_dataType:                   fc
    GRIB_numberOfPoints:             196608
    GRIB_typeOfLevel:                heightAboveGround
    GRIB_stepUnits:                  1
    GRIB_stepType:                   instant
    ...                              ...
    GRIB_shortName:                  2t
    GRIB_units:                      K
    long_name:                       2 metre temperature
    units:                           K
    standard_name:                   air_temperature
    last_ingested_time_index:        306815

This is already small enough to be computed:

We estimate the cost of the download with de costing.py module.

[ ]:

costing.estimate_download_size(t2m_Berlin_2020_2049)

Estimated download size: 0.809 GiB (828.0 MiB)

We explicitly trigger the download of the data. Remember to assign the return of the compute() function to a new variable, so that the data is kept in memory.

[ ]:

%%time

t2m_Berlin_2020_2049_computed = t2m_Berlin_2020_2049.compute()

CPU times: user 11.6 s, sys: 1.41 s, total: 13 s
Wall time: 55.8 s

Now that the data is loaded in memory we can easily compute the climatology:

[ ]:

t2m_Berlin_2020_2049_computed_degC = t2m_Berlin_2020_2049_computed - 273.15
t2m_Berlin_2020_2049_computed_degC.attrs["units"] = "°C"

[ ]:

t2m_Berlin_climatology_mean = t2m_Berlin_2020_2049_computed_degC.groupby("time.month").mean(dim="time")
t2m_Berlin_climatology_std = t2m_Berlin_2020_2049_computed_degC.groupby("time.month").std(dim="time")

We can finally plot the climatology in Berlin for the 2020-2049 reference period

[ ]:

plt.figure(figsize=(10, 5))
t2m_Berlin_climatology_mean.plot(label="Mean", color="#3498db")
plt.errorbar(
    t2m_Berlin_climatology_mean.month,
    t2m_Berlin_climatology_mean,
    yerr=t2m_Berlin_climatology_std,
    fmt="o",
    label="Standard Deviation",
    color="#a9a9a9"
)

plt.title("Surface Temperature climatology in Berlin, 2020-2049")
plt.xticks(t2m_Berlin_climatology_mean.month)
plt.xlabel("Month")
plt.ylabel("Surface Temperature [°C]")
plt.legend()
plt.grid(alpha=0.3)
plt.show()
../../_images/notebooks_insula-code-lab_tutorial-climate-dt-climatological-analysis-of-temperature-in-germany_40_0.png