Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework

Qianrui Zhang | Sr Software Engineer, Logging Platform
Kanchi Masalia | Software Engineer II, Stream Processing Platform
Liang Mou | Sr Staff Software Engineer, Logging Platform
Yi Pan | Principal Engineer, Agent Platform

Introduction

This is the third post in our series on Pinterest’s next-generation database ingestion framework. Part 1 introduced the DB ingestion framework built on Kafka, Flink, Spark, and Iceberg, and Part 2 covered automated schema evolution. This post tackles another challenge in migrating downstream customers to the new ingestion framework: knowing when data is complete enough to read.