Processing a country's worth of raster
The problem
Earth-observation data is big, messy, and never stops arriving. A single satellite scene can be hundreds of megabytes of imagery, tagged with the time it was captured, the footprint on the ground it covers, and the spectral bands it recorded. Cover a whole country over months and you're looking at terabytes of these scenes, most of which you don't actually need for any given question. The naïve approach — download everything, process everything — melts the machine and the budget. The real work is answering spatial questions over that mountain without touching most of it.
This is my day-to-day: building the pipelines and interfaces that turn raw imagery and boundary data into clean, queryable products teams rely on. Kept general here — the shape of the work, not any specific dataset.
The principle: filter before you compute
Every expensive step should run on the smallest possible amount of data, and the cheap filters come first. Before a single pixel is read, scenes are narrowed by the three things that are cheap to check and eliminate the most: does the scene's footprint even intersect the region I care about, does its timestamp fall in my window, and is it too cloudy to be useful? A catalog (I work with the STAC standard for indexing imagery) lets me ask exactly that, so I search by area of interest and date range instead of by hand, and throw out the majority of scenes before any heavy processing starts.
Only the scenes that survive that funnel get the expensive treatment: reprojected to a common coordinate system, clipped to the exact region boundary, and reduced to the handful of values that actually matter downstream. Results land in a spatial database (PostgreSQL / PostGIS) with the right spatial indexes, so the front end can later ask "what's inside this boundary?" or "what changed over this period?" and get an answer in milliseconds instead of re-reading imagery.
What makes it hard at scale
- It must be resumable. A run over thousands of regions will be interrupted — a node dies, an upstream source times out. So every step is idempotent: re-running it produces the same result and skips work already done, instead of starting over or double-counting.
- Bad data must never win silently. Scenes go missing, geometries are malformed, a boundary has no imagery at all. The pipeline treats these as first-class outcomes with explicit checks, so a gap gets recorded and surfaced rather than quietly corrupting an output that looks fine.
- Region IDs aren't geometries. A lot of the real work is the unglamorous plumbing: turning an identifier for a region into an actual shape the pipeline can query on, and doing it consistently across many administrative levels and sources.
- Raster and vector have to meet. Clipping imagery (raster) to boundaries (vector) and reducing it correctly is the heart of it — done with the standard Python geospatial stack, run as repeatable batch jobs rather than one-off scripts.
The last mile
None of it matters if the result is unusable, so the processed data is exposed through service APIs that return map-ready GeoJSON, and rendered in React map dashboards where people filter and drill down by region and time. The goal across the whole chain is to make something genuinely heavy — terabytes of imagery — feel light and queryable to the person at the other end.
What good looks like
The result I'm chasing is that a question over a huge area returns in milliseconds, and adding more imagery makes the archive richer without making queries slower — because the heavy lifting already happened at ingest and the query only ever touches the pre-computed slice it needs. The other half of "good" is trust: every output should be traceable back to the scenes it came from, with gaps and bad data flagged rather than silently averaged away. A fast answer nobody trusts is worthless.
What I'm looking into next
- Pushing more of the filtering into the catalog query itself, so even fewer scenes ever get downloaded.
- Better boundary resolution — turning region identifiers into geometries consistently across administrative levels and sources is where a surprising amount of the real work lives.
- Making the quality checks louder: a dashboard of what's missing and why, so gaps are visible before anyone downstream trips over them.