Contributing to Apache Flink
Series (ongoing) · April 13, 2026
Introduction
In the world of data processing and pipeline building Apache Flink is a tool that shows up everywhere (similar to how it's of the utmost importance to understand how to use Kafka as well in these scenarios). I come across mention of this tool often on job sites for various data involved roles, particularly Data Engineering.
Apache Flink is an open-source framework and distributed processing engine used for stateful processing over both unbounded datasets and bounded datasets. Understanding how it works is paramount if you want to operate in this space.
For further context, unbounded data is simply ever-flowing, non-stop data streams while bounded datasets can best be described as “batch” data.
This series follows a real open-source contribution to the Flink project. Specifically, I'm working on Jira Issue: FLINK-25672, which addresses a known limitation in Flink's DataStream filesystem connector. When using an unbounded file source, the enumerator keeps track of every file path it has already processed and that state can grow indefinitely. For long-running jobs watching directories with high file throughput, this becomes a real problem. The Flink docs themselves acknowledge the need for a compressed form of tracking already processed files, and that's what this contribution aims to build. This series will be me tackling this issue from scratch, making mistakes and learning more along the way
Along the way, we'll dig into the parts of Flink's architecture that are relevant to this fix, how the source enumerator works, how file discovery happens, and why unbounded state growth is a problem worth solving. You'll see the full process: reading the codebase, understanding context, writing and testing a fix, and submitting it upstream.
Welcome!!!
Posts in this series
- Problem Familiarity (April 13, 2026)
- Decide on a Solution Path (May 4, 2026)
- Understand the Codebase (coming soon)
Following the problem write-up, this post digs into Flink's FileSource internals tracing how the enumerator discovers files, where processed-path state lives, and which classes we'll need to touch in order to fix FLINK-25672.