Generalized Consensus & ​Native Top-K Joins in ParadeDB

Описание: These talks were given by Sugu Sougoumarane and Stu Hood as part of the South Bay Systems meetup on March 31st, 2026.

== Generalized Consensus
​Existing consensus protocols have rigid limits, and a new approach to consensus will be presented that accommodates more flexible, alternative implementations. Using Multigres as a case study, it will be demonstrated how this approach enables a robust High Availability solution for Postgres while maintaining the rigorous safety features of traditional consensus systems.

References: https://multigres.com/blog/generalize...

​​=== Speaker Bio
​Sugu is currently the Head of Multigres at Supabase. As a co-creator of Vitess and a co-inventor of FlexPaxos, he has focused on increasing the flexibility of consensus algorithms and adapting them to bespoke environments.

== Native Top-K Joins in ParadeDB
​Recently, some have argued that modern analytical formats have evolved "beyond indexes," relying entirely on data layout and coarse metadata. This narrative breaks down against the high-cardinality, late-materialized reality of full-text search. However, search engines have their own blind spots. While they excel at low-latency retrieval, they typically lack the sophisticated query planning required to efficiently process complex, normalized data. To achieve sub-second search across relational schemas, search architecture must learn from analytics.

​This talk explores that convergence through the implementation of native Top-K joins. By fusing Tantivy’s inverted indexing with DataFusion’s analytical execution framework, we will examine the "selectivity sweet spot" where posting lists fundamentally outperform columnar dictionary scans. We'll explore how embedding an analytical optimizer into a search database allows ParadeDB to combine dynamic score filtering with a vectorized pipeline.

References:
Balancing vectorized query execution with bandwidth-optimized storage -- https://pdfs.semanticscholar.org/a43f...
Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Parquet Queries -- https://datafusion.apache.org/blog/20...
A Practical Dive Into Late Materialization in arrow-rs Parquet Reads -- https://arrow.apache.org/blog/2025/12...
Faster top-k document retrieval using block-max indexes -- https://dl.acm.org/doi/10.1145/200991...
Fast integer compression: decoding billions of integers per second -- https://lemire.me/blog/2012/09/12/fas...
Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask -- https://www.vldb.org/pvldb/vol11/p220...
The Quest for One Million IOPS: Benchmarking Storage at LanceDB -- https://lancedb.com/blog/one-million-...
BtrBlocks: Efficient Columnar Compression for Data Lakes -- https://www.cs.cit.tum.de/fileadmin/w...
Change default block size from 128 to 256 -- https://github.com/apache/lucene/pull...
Selective Late Materialization in Modern Analytical Databases -- https://www.vldb.org/pvldb/vol18/p461...
Supporting top-k join queries in relational databases -- https://www.cerias.purdue.edu/assets/...

​​=== Speaker Bio
​Stu Hood is a software engineer at ParadeDB, focusing on database internals in Rust. His background is rooted in distributed systems, having previously worked on Twitter's distributed databases and contributed to Apache Cassandra. He is currently working on low latency columnar execution for search.