# Getting started with the Earth Data Hub This notebook reproduces the [Getting Started page](https://earthdatahub.destine.eu/getting-started) of the Earth Data Hub. Earth Data Hub offers an innovative and super-efficient way to access data. Here is what you need to know to start working. Datasets are published as Zarr stores and *every user has a monthly quota of 500,000 allowed requests*, so downloads must be authenticated. See the [Quota & API keys](https://earthdatahub.destine.eu/quota-api-keys) page to check current consumption. We will show how to obtain and use your authentication credentials later, let’s start by connecting to a test dataset that doesn’t need authentication. #### NOTE To run this notebook locally, follow the setup instructions in the [GitHub repository](https://github.com/bopen/edh-learning). ## Setup the environment If you are using the suggested way to run the notebook via `uv run jupyter lab` you are good to go. In other cases you may need to install the following Python packages: - `xarray`, `zarr` (version 3 or later), `dask` and `aiohttp` to access the data - `jupyterlab` and `matplotlib` to run the notebook and plot the data ## Open your first dataset (without authentication) The easiest way to get started with Earth Data Hub is by using Python and Xarray, on our public test dataset. ```python import xarray as xr ``` ```python test_dataset_public = xr.open_dataset( "https://data.earthdatahub.destine.eu/public/test-dataset-v1.zarr", chunks={}, engine="zarr", ) test_dataset_public ``` ```text Size: 4GB Dimensions: (latitude: 720, longitude: 1440, age_band_lower_bound: 14, year: 71) Coordinates: * latitude (latitude) float64 6kB 90.0 89.75 ... -89.5 -89.75 * longitude (longitude) float64 12kB 0.0 0.25 0.5 ... 359.5 359.8 * age_band_lower_bound (age_band_lower_bound) int64 112B 0 5 10 ... 55 60 65 * year (year) int64 568B 1950 1951 1952 ... 2018 2019 2020 Data variables: demographic_totals (latitude, longitude, age_band_lower_bound, year) float32 4GB dask.array ``` We can process and plot a portion of the data in a few lines of code. ```python demographic_totals_2020 = test_dataset_public["demographic_totals"].sel(year=2020) world_population_2020 = demographic_totals_2020.sum("age_band_lower_bound") world_population_2020.plot(vmax=500_000) ``` ```text ``` ![image](https://earthdatahub.destine.eu/docs/_images/notebooks_edh-learning_00-getting-started_5_1.png) Note that we performed several operations: - selection: we choose the variable `demographic_totals` and the year `2020` - aggregation: we summed over all the age classes - display: we plotted the data with simple configuration options ## Setup your credentials for authenticated data access To access most datasets on Earth Data Hub you simply need to obtain an API key and instruct your tools to use it when downloading the data. ### How to obtain an Earth Data Hub API key 1. **Register an account** on the [DestinE Platform](https://earthdatahub.destine.eu/login). 2. **Visit the** [Quota & API keys](https://earthdatahub.destine.eu/quota-api-keys#my-personal-access-tokens) page. 3. **Copy your default API key** (or generate a new one). Now we are ready to access the same small test dataset, but this time from a URL that is authorization protected: [https://data.earthdatahub.destine.eu/private/test-dataset-v1.zarr](https://data.earthdatahub.destine.eu/private/test-dataset-v1.zarr). ### Adding the API key to the URL The easiest way is to pass the API key as a password in the Zarr store URL, for example: ```python import os edh_key = os.getenv("EDH_KEY", "") assert edh_key.startswith("edh_key_"), ( f"Your API key is a long string starting with 'edh_key_', not {edh_key!r}" ) ``` ```python test_dataset_url = xr.open_dataset( f"https://edh:{edh_key}@data.earthdatahub.destine.eu/private/test-dataset-v1.zarr", chunks={}, engine="zarr", ) test_dataset_url ``` ```text Size: 4GB Dimensions: (latitude: 720, longitude: 1440, age_band_lower_bound: 14, year: 71) Coordinates: * latitude (latitude) float64 6kB 90.0 89.75 ... -89.5 -89.75 * longitude (longitude) float64 12kB 0.0 0.25 0.5 ... 359.5 359.8 * age_band_lower_bound (age_band_lower_bound) int64 112B 0 5 10 ... 55 60 65 * year (year) int64 568B 1950 1951 1952 ... 2018 2019 2020 Data variables: demographic_totals (latitude, longitude, age_band_lower_bound, year) float32 4GB dask.array ``` We can now test plotting the private dataset. ```python demographic_totals_2010 = test_dataset_url["demographic_totals"].sel(year=2010) world_population_2010 = demographic_totals_2010.sum("age_band_lower_bound") world_population_2010.plot(vmax=500_000) ``` ```text ``` ![image](https://earthdatahub.destine.eu/docs/_images/notebooks_edh-learning_00-getting-started_11_1.png) ### Configuring the API key in the *netrc* file A more convenient way to set up the API key if you plan to use the system, is to configure the *netrc* file (`~/.netrc` on Linux and MacOS, `C:\Users\\_netrc` on Windows) as follows: ```none machine data.earthdatahub.destine.eu password machine api.earthdatahub.destine.eu password ``` Earth Data Hub has two data access entry points, so the file needs an entry for each one. You can also retrieve your *netrc* file directly from the [Quota & API keys](https://earthdatahub.destine.eu/quota-api-keys#my-personal-access-tokens) page using the ⓘ button. Once this is set up you can use the URL above directly and similarly, the other URL that you find in the catalogue: ```python test_dataset_netrc = xr.open_dataset( "https://data.earthdatahub.destine.eu/private/test-dataset-v1.zarr", storage_options={"client_kwargs": {"trust_env": True}}, chunks={}, engine="zarr", ) test_dataset_netrc ``` ```text Size: 4GB Dimensions: (latitude: 720, longitude: 1440, age_band_lower_bound: 14, year: 71) Coordinates: * latitude (latitude) float64 6kB 90.0 89.75 ... -89.5 -89.75 * longitude (longitude) float64 12kB 0.0 0.25 0.5 ... 359.5 359.8 * age_band_lower_bound (age_band_lower_bound) int64 112B 0 5 10 ... 55 60 65 * year (year) int64 568B 1950 1951 1952 ... 2018 2019 2020 Data variables: demographic_totals (latitude, longitude, age_band_lower_bound, year) float32 4GB dask.array ``` And we are ready to test plotting on the private dataset. ```python demographic_totals_2000 = test_dataset_netrc["demographic_totals"].sel(year=2000) world_population_2000 = demographic_totals_2000.sum("age_band_lower_bound") world_population_2000.plot(vmax=500_000) ``` ```text ``` ![image](https://earthdatahub.destine.eu/docs/_images/notebooks_edh-learning_00-getting-started_15_1.png) Keep in mind that some tools do not use the *netrc* file by default, but can be instructed to do so, for example, the `storage_options={"client_kwargs":{"trust_env":True}}` option is needed by Xarray / Zarr. ## Access your first real dataset ### How to use the catalogue Datasets are offered in the [Earth Data Hub catalogue](https://earthdatahub.destine.eu/catalogue) as Zarr stores, even if a dataset is identified by a single URL its size may be very large, easily hundreds of terabytes. You will find the code snippet to access the data with Xarray on the dataset page and that will work out of the box after you set up the *netrc* file as described above. **Be careful, do not try to download an entire dataset, chances are that you will exceed your quota well before you come even close! Instead, we show how to access and work with large datasets in our tutorials.** ### ERA5 monthly averages from 1940 to present A good dataset to start with is [ERA5 monthly averaged data on single levels](https://earthdatahub.destine.eu/collections/era5/datasets/era5-single-levels-atmosphere-monthly): every variable is about 4GiB in size, small compared to the 2.9TiB of an hourly ERA5 variable, and it is relatively easy to handle. If you managed to set up your *netrc* file you can just copy the code snippet in the dataset page: ```python era5_monthly = xr.open_dataset( "https://data.earthdatahub.destine.eu/era5/era5-single-levels-atmosphere-monthly-v0.zarr", storage_options={"client_kwargs": {"trust_env": True}}, chunks={}, engine="zarr", ) era5_monthly ``` ```text Size: 860GB Dimensions: (valid_time: 1041, latitude: 721, longitude: 1440) Coordinates: * valid_time (valid_time) datetime64[ns] 8kB 1940-01-01 ... 2026-09-01 * latitude (latitude) float64 6kB 90.0 89.75 89.5 ... -89.75 -90.0 * longitude (longitude) float64 12kB 0.0 0.25 0.5 ... 359.2 359.5 359.8 Data variables: (12/213) alnid (valid_time, latitude, longitude) float32 4GB dask.array alnip (valid_time, latitude, longitude) float32 4GB dask.array aluvd (valid_time, latitude, longitude) float32 4GB dask.array aluvp (valid_time, latitude, longitude) float32 4GB dask.array anor (latitude, longitude) float32 4MB dask.array asn (valid_time, latitude, longitude) float32 4GB dask.array ... ... vithen (valid_time, latitude, longitude) float32 4GB dask.array vitoe (valid_time, latitude, longitude) float32 4GB dask.array viwve (valid_time, latitude, longitude) float32 4GB dask.array viwvn (valid_time, latitude, longitude) float32 4GB dask.array z (latitude, longitude) float32 4MB dask.array zust (valid_time, latitude, longitude) float32 4GB dask.array Attributes: GRIB_edition: 1 GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 Conventions: CF-1.7 institution: European Centre for Medium-Range Weather Forecasts history: 2026-10-08T14:35 GRIB to CDM+CF via cfgrib-0.9.1... ``` Let’s compute the temperature change over an area of interest defined as a latitude-longitude rectangle and a time period, for example Europe from 1979 onward. ERA5 longitudes go from 0 to 360 degrees, as Europe crosses the 0 meridian we first convert them to the -180 to 180 range: ```python era5_monthly = era5_monthly.assign_coords( longitude=(era5_monthly.longitude + 180) % 360 - 180 ).sortby("longitude") aoi_selection = { "valid_time": slice("1979", None), "latitude": slice(70, 35), "longitude": slice(-10, 40), } ``` Plot a single month to check the geographic selection: ```python aoi_temperature = era5_monthly.t2m.sel(aoi_selection) _ = aoi_temperature.isel(valid_time=-1).plot() ``` ![image](https://earthdatahub.destine.eu/docs/_images/notebooks_edh-learning_00-getting-started_23_0.png) Now define the geographic average by weighting the latitude / longitude grid points by the cosine of the latitude. ```python import numpy as np lat_lon_weight = np.cos(np.deg2rad(aoi_temperature.latitude)) aoi_mean_temperature = aoi_temperature.weighted(lat_lon_weight).mean( ["latitude", "longitude"] ) ``` Until now we didn’t download any data (except one month for the plot above), the `.compute()` performs both the download and the computation step. **The following cell may run for a few minutes, depending on the speed of your network.** ```python %%time aoi_mean_temperature_computed = aoi_mean_temperature.compute() ``` ```text CPU times: user 1.74 s, sys: 506 ms, total: 2.25 s Wall time: 23.3 s ``` Finally we compute the temperature change as the difference from the 1981-2010 average of the same calendar month, and plot the resulting time series: ```python reference_period = aoi_mean_temperature_computed.sel(valid_time=slice("1981", "2010")) climatology = reference_period.groupby("valid_time.month").mean() temperature_change = ( aoi_mean_temperature_computed.groupby("valid_time.month") - climatology ) temperature_change.attrs["long_name"] = "temperature change since 1981-2010" temperature_change.attrs["units"] = "°C" _ = temperature_change.plot() ``` ![image](https://earthdatahub.destine.eu/docs/_images/notebooks_edh-learning_00-getting-started_29_0.png) **Again! Be careful, do not try to download an entire dataset, chances are that you will exceed your quota well before you come even close! Instead, we show how to access and work with large datasets in our tutorials.**