← All articles

Welcome to Lambda Pipeline

September 11, 2026

Most data engineering portfolios are static: a diagram, a paragraph, a GitHub link. This one is different — every number on the dashboard is real, generated by containers running right now, not a screenshot.

Why Lambda Architecture

The pattern splits the same problem into two paths that trade off differently:

  • A speed layer (Redis) that favors latency over perfect accuracy — data lands in under a second, but it's approximate.
  • A batch layer (DuckDB + dbt) that favors correctness over latency — it runs every five minutes, but every row passes schema tests before it's trusted.

The serving layer merges both: recent data from the speed layer fills the gap while the batch layer catches up, and the batch layer's results become authoritative once they land.

What this article format is

This page is rendered from a plain Markdown file committed to the repo — no database, no admin panel, no authenticated write endpoint to secure. To publish, I write a .md file, commit it, and push. The same deploy pipeline that ships code changes ships new writing.

If you're reading this, the pipeline setup worked.