Note

To run this notebook locally, follow the setup instructions in the GitHub repository.

Download notebook

Getting started with the Earth Data Hub#

This notebook reproduces the Getting Started page of the Earth Data Hub.

Earth Data Hub offers an innovative and super-efficient way to access data. Here is what you need to know to start working.

Datasets are published as Zarr stores and every user has a monthly quota of 500,000 allowed requests, so downloads must be authenticated. See the Quota & API keys page to check current consumption.

We will show how to obtain and use your authentication credentials later, let’s start by connecting to a test dataset that doesn’t need authentication.

Setup the environment#

If you are using the suggested way to run the notebook via uv run jupyter lab you are good to go. In other cases you may need to install the following Python packages:

  • xarray, zarr (version 3 or later), dask and aiohttp to access the data

  • jupyterlab and matplotlib to run the notebook and plot the data

Open your first dataset (without authentication)#

The easiest way to get started with Earth Data Hub is by using Python and Xarray, on our public test dataset.

[1]:

import xarray as xr

[2]:

test_dataset_public = xr.open_dataset(
    "https://data.earthdatahub.destine.eu/public/test-dataset-v1.zarr",
    chunks={},
    engine="zarr",
)
test_dataset_public

[2]:

<xarray.Dataset> Size: 4GB
Dimensions:               (latitude: 720, longitude: 1440,
                           age_band_lower_bound: 14, year: 71)
Coordinates:
  * latitude              (latitude) float64 6kB 90.0 89.75 ... -89.5 -89.75
  * longitude             (longitude) float64 12kB 0.0 0.25 0.5 ... 359.5 359.8
  * age_band_lower_bound  (age_band_lower_bound) int64 112B 0 5 10 ... 55 60 65
  * year                  (year) int64 568B 1950 1951 1952 ... 2018 2019 2020
Data variables:
    demographic_totals    (latitude, longitude, age_band_lower_bound, year) float32 4GB dask.array<chunksize=(180, 180, 14, 2), meta=np.ndarray>

We can process and plot a portion of the data in a few lines of code.

[3]:

demographic_totals_2020 = test_dataset_public["demographic_totals"].sel(year=2020)

world_population_2020 = demographic_totals_2020.sum("age_band_lower_bound")

world_population_2020.plot(vmax=500_000)

[3]:

<matplotlib.collections.QuadMesh at 0x115dc5550>
../../_images/notebooks_edh-learning_00-getting-started_5_1.png

Note that we performed several operations:

  • selection: we choose the variable demographic_totals and the year 2020

  • aggregation: we summed over all the age classes

  • display: we plotted the data with simple configuration options

Setup your credentials for authenticated data access#

To access most datasets on Earth Data Hub you simply need to obtain an API key and instruct your tools to use it when downloading the data.

How to obtain an Earth Data Hub API key#

  1. Register an account on the DestinE Platform.

  2. Visit the Quota & API keys page.

  3. Copy your default API key (or generate a new one).

Now we are ready to access the same small test dataset, but this time from a URL that is authorization protected: https://data.earthdatahub.destine.eu/private/test-dataset-v1.zarr.

Adding the API key to the URL#

The easiest way is to pass the API key as a password in the Zarr store URL, for example:

[4]:

import os

edh_key = os.getenv("EDH_KEY", "<write your API key here>")

assert edh_key.startswith("edh_key_"), (
    f"Your API key is a long string starting with 'edh_key_', not {edh_key!r}"
)

[5]:

test_dataset_url = xr.open_dataset(
    f"https://edh:{edh_key}@data.earthdatahub.destine.eu/private/test-dataset-v1.zarr",
    chunks={},
    engine="zarr",
)
test_dataset_url

[5]:

<xarray.Dataset> Size: 4GB
Dimensions:               (latitude: 720, longitude: 1440,
                           age_band_lower_bound: 14, year: 71)
Coordinates:
  * latitude              (latitude) float64 6kB 90.0 89.75 ... -89.5 -89.75
  * longitude             (longitude) float64 12kB 0.0 0.25 0.5 ... 359.5 359.8
  * age_band_lower_bound  (age_band_lower_bound) int64 112B 0 5 10 ... 55 60 65
  * year                  (year) int64 568B 1950 1951 1952 ... 2018 2019 2020
Data variables:
    demographic_totals    (latitude, longitude, age_band_lower_bound, year) float32 4GB dask.array<chunksize=(180, 180, 14, 2), meta=np.ndarray>

We can now test plotting the private dataset.

[6]:

demographic_totals_2010 = test_dataset_url["demographic_totals"].sel(year=2010)

world_population_2010 = demographic_totals_2010.sum("age_band_lower_bound")

world_population_2010.plot(vmax=500_000)

[6]:

<matplotlib.collections.QuadMesh at 0x1179974d0>
../../_images/notebooks_edh-learning_00-getting-started_11_1.png

Configuring the API key in the netrc file#

A more convenient way to set up the API key if you plan to use the system, is to configure the netrc file (~/.netrc on Linux and MacOS, C:\Users\<username>\_netrc on Windows) as follows:

machine data.earthdatahub.destine.eu
  password <write your API key here>

machine api.earthdatahub.destine.eu
  password <write your API key here>

Earth Data Hub has two data access entry points, so the file needs an entry for each one. You can also retrieve your netrc file directly from the Quota & API keys page using the ⓘ button.

Once this is set up you can use the URL above directly and similarly, the other URL that you find in the catalogue:

[7]:

test_dataset_netrc = xr.open_dataset(
    "https://data.earthdatahub.destine.eu/private/test-dataset-v1.zarr",
    storage_options={"client_kwargs": {"trust_env": True}},
    chunks={},
    engine="zarr",
)
test_dataset_netrc

[7]:

<xarray.Dataset> Size: 4GB
Dimensions:               (latitude: 720, longitude: 1440,
                           age_band_lower_bound: 14, year: 71)
Coordinates:
  * latitude              (latitude) float64 6kB 90.0 89.75 ... -89.5 -89.75
  * longitude             (longitude) float64 12kB 0.0 0.25 0.5 ... 359.5 359.8
  * age_band_lower_bound  (age_band_lower_bound) int64 112B 0 5 10 ... 55 60 65
  * year                  (year) int64 568B 1950 1951 1952 ... 2018 2019 2020
Data variables:
    demographic_totals    (latitude, longitude, age_band_lower_bound, year) float32 4GB dask.array<chunksize=(180, 180, 14, 2), meta=np.ndarray>

And we are ready to test plotting on the private dataset.

[8]:

demographic_totals_2000 = test_dataset_netrc["demographic_totals"].sel(year=2000)

world_population_2000 = demographic_totals_2000.sum("age_band_lower_bound")

world_population_2000.plot(vmax=500_000)

[8]:

<matplotlib.collections.QuadMesh at 0x117b3b9d0>
../../_images/notebooks_edh-learning_00-getting-started_15_1.png

Keep in mind that some tools do not use the netrc file by default, but can be instructed to do so, for example, the storage_options={"client_kwargs":{"trust_env":True}} option is needed by Xarray / Zarr.

Access your first real dataset#

How to use the catalogue#

Datasets are offered in the Earth Data Hub catalogue as Zarr stores, even if a dataset is identified by a single URL its size may be very large, easily hundreds of terabytes.

You will find the code snippet to access the data with Xarray on the dataset page and that will work out of the box after you set up the netrc file as described above.

Be careful, do not try to download an entire dataset, chances are that you will exceed your quota well before you come even close! Instead, we show how to access and work with large datasets in our tutorials.

ERA5 monthly averages from 1940 to present#

A good dataset to start with is ERA5 monthly averaged data on single levels: every variable is about 4GiB in size, small compared to the 2.9TiB of an hourly ERA5 variable, and it is relatively easy to handle.

If you managed to set up your netrc file you can just copy the code snippet in the dataset page:

[9]:

era5_monthly = xr.open_dataset(
    "https://data.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-monthly-v0.zarr",
    storage_options={"client_kwargs": {"trust_env": True}},
    chunks={},
    engine="zarr",
)
era5_monthly

[9]:

<xarray.Dataset> Size: 860GB
Dimensions:         (valid_time: 1041, latitude: 721, longitude: 1440)
Coordinates:
  * valid_time      (valid_time) datetime64[ns] 8kB 1940-01-01 ... 2026-09-01
  * latitude        (latitude) float64 6kB 90.0 89.75 89.5 ... -89.75 -90.0
  * longitude       (longitude) float64 12kB 0.0 0.25 0.5 ... 359.2 359.5 359.8
Data variables: (12/213)
    alnid           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    alnip           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    aluvd           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    aluvp           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    anor            (latitude, longitude) float32 4MB dask.array<chunksize=(240, 240), meta=np.ndarray>
    asn             (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    ...              ...
    vithen          (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    vitoe           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    viwve           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    viwvn           (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
    z               (latitude, longitude) float32 4MB dask.array<chunksize=(240, 240), meta=np.ndarray>
    zust            (valid_time, latitude, longitude) float32 4GB dask.array<chunksize=(120, 240, 240), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2026-10-08T14:35 GRIB to CDM+CF via cfgrib-0.9.1...

Let’s compute the temperature change over an area of interest defined as a latitude-longitude rectangle and a time period, for example Europe from 1979 onward.

ERA5 longitudes go from 0 to 360 degrees, as Europe crosses the 0 meridian we first convert them to the -180 to 180 range:

[10]:

era5_monthly = era5_monthly.assign_coords(
    longitude=(era5_monthly.longitude + 180) % 360 - 180
).sortby("longitude")

aoi_selection = {
    "valid_time": slice("1979", None),
    "latitude": slice(70, 35),
    "longitude": slice(-10, 40),
}

Plot a single month to check the geographic selection:

[11]:

aoi_temperature = era5_monthly.t2m.sel(aoi_selection)

_ = aoi_temperature.isel(valid_time=-1).plot()
../../_images/notebooks_edh-learning_00-getting-started_23_0.png

Now define the geographic average by weighting the latitude / longitude grid points by the cosine of the latitude.

[12]:

import numpy as np

lat_lon_weight = np.cos(np.deg2rad(aoi_temperature.latitude))

aoi_mean_temperature = aoi_temperature.weighted(lat_lon_weight).mean(
    ["latitude", "longitude"]
)

Until now we didn’t download any data (except one month for the plot above), the .compute() performs both the download and the computation step.

The following cell may run for a few minutes, depending on the speed of your network.

[13]:

%%time

aoi_mean_temperature_computed = aoi_mean_temperature.compute()

CPU times: user 1.74 s, sys: 506 ms, total: 2.25 s
Wall time: 23.3 s

Finally we compute the temperature change as the difference from the 1981-2010 average of the same calendar month, and plot the resulting time series:

[14]:

reference_period = aoi_mean_temperature_computed.sel(valid_time=slice("1981", "2010"))
climatology = reference_period.groupby("valid_time.month").mean()

temperature_change = (
    aoi_mean_temperature_computed.groupby("valid_time.month") - climatology
)
temperature_change.attrs["long_name"] = "temperature change since 1981-2010"
temperature_change.attrs["units"] = "°C"

_ = temperature_change.plot()
../../_images/notebooks_edh-learning_00-getting-started_29_0.png

Again! Be careful, do not try to download an entire dataset, chances are that you will exceed your quota well before you come even close! Instead, we show how to access and work with large datasets in our tutorials.