Core Concepts#
Understand how Earth Data Hub makes massive datasets simple to use.
Built on ARCO#
Earth Data Hub is built on ARCO (Analysis-Ready Cloud-Optimised) data. Our solution combines proven, cutting-edge technologies to make massive datasets fast, accessible, and analysis-ready.
Ingredient |
Technology |
Purpose |
|---|---|---|
Object Storage |
Cloud object storage designed for scalable access. |
|
Data format |
Stores data as small chunks that can be streamed on demand. |
|
Catalogue |
Makes datasets easy to discover using a standard catalogue. |
Traditional Earth data workflows often mean downloading large files, even when you only need a small part of the data. Earth Data Hub replaces heavy downloads with direct cloud streaming, delivering only the data you need, when you need it.
Simplified access#
Earth Data Hub is designed to fit into your existing workflows, not replace them. That’s why we rely on open standards and familiar tools, so you don’t have to learn new technologies or install bespoke software just to access Earth data.
To access any dataset, you only need two things:
Your credentials for authenticating access to protected datasets.
The address of any dataset in the catalogue.
Use your API key and dataset URL with your favourite programming languages and libraries, or build your own tools.
Chunking schemes#
Zarr stores datasets as small chunks that can be streamed on demand. The size and arrangement of these chunks have a major impact on performance.
These two extreme chunking schemes optimise for opposite use cases:
Best for viewing maps, loading large regions, and comparing snapshots in time.
Best for analysing how variables change over time at specific locations.
No single chunking scheme fits every use case. We optimise each dataset for its most common access patterns and, when needed, publish multiple versions for different workflows.
Hint
The Climate DT high-resolution datasets are available with both map-optimised and time series-optimised chunking, so you can choose the version that best fits your workflow.