Space data platforms

A satellite that collects data nobody can find, open or afford is not producing value. The infrastructure between downlink and use is unglamorous and is frequently the difference between a dataset that is used and one that is not.

The scale

At these volumes the traditional model, in which a user downloads files and processes them locally, stops working. Moving a petabyte is slower and more expensive than moving the computation to it, which is the structural change that has reshaped the field.

Standards

Interoperability is what allows data from different providers to be used together, and it is achieved by a small number of standards.

STAC deserves particular note because it solved a real problem cheaply. Every provider previously published a different catalogue with a different API, so combining sources required custom integration per provider. STAC gives a common description of what an asset is, where and when it covers, and how to access it, which makes a multi-provider search a single query. Its adoption was rapid because it is simple and was designed by the people who needed it.

Cloud-native formats

The distinguishing property of a cloud-native format is that a client can read part of a file over HTTP without downloading the whole thing.

A Cloud Optimized GeoTIFF is an ordinary GeoTIFF with its data arranged in internal tiles and reduced-resolution overviews, and its layout described in the header. A client wanting a small region at low resolution issues a couple of range requests and transfers a few hundred kilobytes from a file of many gigabytes.

This sounds like a minor optimisation and is architecturally decisive. It is what allows an application to display a global mosaic interactively without any preprocessing, and it removes the need for every user to maintain a copy of the archive.

Analysis-ready data

The processing chain in Satellite image processing produces correctly calibrated products, and a user still faces work before analysis: reprojecting to a common grid, applying cloud masks, harmonising between sensors, and stacking by time.

Analysis-ready data moves that work upstream. Products are delivered already corrected to surface reflectance, tiled to a fixed grid, cloud-masked, and consistent across acquisitions so that a time series can be differenced directly.

The value is that this work is done once by the provider rather than repeatedly and inconsistently by every user. The cost is loss of flexibility, since a user with different requirements must go back to the lower-level product, so well-designed platforms publish both.

Discovery

The branch at the bottom is the structural change of the past decade. Public cloud hosting of major archives, with the Sentinel and Landsat collections available in commercial object storage, means a user can run analysis adjacent to the data and transfer only results.

This has genuine democratising effects, since a researcher without institutional storage can process a continental time series. It also creates a dependency on commercial cloud providers for access to publicly funded data, and egress pricing means moving data out can cost more than the analysis, which is a lock-in mechanism worth naming.

Licensing and cost

Access policy determines use more than technical quality does.

The Landsat open data policy is the clearest natural experiment in the field. Imagery that had been sold at hundreds of dollars per scene was made free in 2008, and use increased by orders of magnitude, producing scientific and commercial value far exceeding the forgone revenue. It is routinely cited in policy discussions, and the Copernicus programme’s open policy followed the same reasoning.

The general lesson is that for publicly funded observation, the value created downstream substantially exceeds anything recoverable by charging for access. That argument does not transfer automatically to commercial data, where the collection itself must be funded.

The archive as an obligation

Space data has a property most data does not: it cannot be recollected. An image of a place on a date is unique, and if it is lost the observation is gone.

This makes archiving a permanent obligation rather than a project task, and it implies requirements that outlive any programme.

The Landsat archive is the model, and its value comes precisely from being long and consistently calibrated. A fifty-year record of Earth’s surface is scientifically irreplaceable, and it exists because successive institutions kept funding curation between the interesting moments. See Ground segment.

References

• CCSDS recommended standards for space data systems, particularly the space link and file delivery protocols.
• STAC Specification, radiantearth.github.io/stac-spec.
• Open Geospatial Consortium standards for WMS, WCS and coverage access.
• Wulder, M. A. et al. “The global Landsat archive: Status, consolidation and direction”, Remote Sensing of Environment, 185:271, 2016.
• Gorelick, N. et al. “Google Earth Engine: Planetary-scale geospatial analysis for everyone”, Remote Sensing of Environment, 202:18, 2017.
• Abernathey, R. P. et al. “Cloud-Native Repositories for Big Scientific Data”, Computing in Science and Engineering, 23:26, 2021.