Making Materialized Views Actually Fast with DBSP (with Lalith Suresh)

Описание: There's a problem that's bugged the database industry since the 1980s: you run an expensive query over millions of rows, cache the result, and then a single new row arrives. Logically that's one small update, but most engines throw the cached answer away and recompute everything from scratch. Some will handle changes incrementally, but only for "simple" queries - and the rules for what counts as simple are arbitrary and brittle. So can you incrementally maintain any SQL query, no matter how complex? For decades the answer was no. Then an award-winning paper called DBSP proved that the answer is yes - all queries are simple enough.

Joining me to explain how that works is Lalith Suresh, CEO of Feldera, the company built on top of DBSP. We start with the problem itself, then trace how a group of VMware researchers arrived at it from the unlikely direction of Kubernetes and network control planes. Lalith walks through Z-sets, the weighted data structure that turns database changes into something you can add and subtract, and the four DBSP operators - including one borrowed straight from digital signal processing - that let you compile any SQL program into an incremental version deterministically. Along the way we get into which operations need state and which don't, how the delta join falls out for free, building a standalone query engine with its own storage layer and Calcite front-end, backfills as the real Achilles heel, and how this all differs from stream processors like Kafka Streams and Flink.

If you've ever fought with materialized views that won't refresh, watched a nightly batch job recompute three years of data to capture last night's changes, or you're just curious how one elegant bit of maths unifies batch and stream processing, Lalith has some genuinely satisfying answers. There's an MIT-licensed open source edition and a sandbox at try.feldera.com if you want to play along.

---

Support Developer Voices on Patreon:   / developervoices  
Support Developer Voices on YouTube:    / @developervoices  

Feldera: https://www.feldera.com/
Feldera Sandbox (try it online): https://try.feldera.com/
Feldera on GitHub (open source): https://github.com/feldera/feldera
DBSP Rust crate: https://crates.io/crates/dbsp

DBSP Paper - "Automatic Incremental View Maintenance for Rich Query Languages" (VLDB 2023 Best Paper): https://arxiv.org/abs/2203.16684
Mihai Budiu - "Streaming Queries Without Compromise" (Current 2024):    • Streaming Queries Without Compromise (Miha...  
Mihai Budiu - DBSP talk at CMU Database Group: https://db.cs.cmu.edu/events/dbsp-inc...

Differential Dataflow: https://github.com/TimelyDataflow/dif...
Apache Calcite (Feldera's SQL front-end): https://calcite.apache.org/

Kafka Streams: https://kafka.apache.org/documentatio...
Apache Flink: https://flink.apache.org/
ksqlDB: https://ksqldb.io/
Apache Spark: https://spark.apache.org/
Snowflake: https://www.snowflake.com/
Databricks: https://www.databricks.com/

Kris on Bluesky: https://bsky.app/profile/krisajenkins...
Kris on Mastodon: http://mastodon.social/@krisajenkins
Kris on LinkedIn:   / krisjenkins  

---

0:00 Intro
3:24 The Problem With Materialized Views
10:30 Beyond Databases: Kubernetes and Control Planes
17:00 What Are Z-Sets?
24:20 The Four Operators of DBSP
31:04 Which Operations Need State, and the Delta Join
40:00 Building a Standalone Query Engine
45:30 Incremental Compute at Scale and Backfills
51:00 Kafka Streams, UDFs and Determinism
57:00 From Paper to Production
1:00:30 Getting Started With Feldera and DBSP
1:04:14 Outro