Rerouting the Stream: How Lyft Moved to the Apache Flink Operator

Written by Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar at Lyft.

Why We Migrated: Technical Debt Meets Modern Streaming Demands

Over the past several quarters, Lyft’s Streaming Compute team retired our internally developed Flink Kubernetes operator and moved our entire streaming fleet onto the open-source Apache Flink Kubernetes operator. This post is about why we made the switch, how we pulled it off incrementally without disrupting users, and the follow-on work it took to actually get the benefits we were after.

Back in 2020, when we first architected the Lyft Flink Kubernetes Operator, it was exactly what we needed. At that time, the open-source community hadn’t yet built a dedicated control plane, so we built our own to manage all streaming applications on Kubernetes. It worked well for our initial workloads, but as our real-time data needs increased and our engineers’ scope of ownership grew in both breadth and complexity, the cracks started to show.

First came the maintenance burden. Our operator had become a relic of Lyft’s early Kubernetes days, kept alive by a growing pile of custom code. Every Flink version upgrade meant carefully picking through layers of accumulated technical debt and hoping nothing broke on the way through.

Second came the feature gap. Streaming tooling kept moving, and our engineers kept asking for capabilities that had become table stakes elsewhere, such as autoscaling to right-size jobs, automatic rollbacks on failed deploys, and an end to hand-tuning CPU and memory. Each request left us with two options, neither of which was ideal. We could explain why we couldn’t support it yet, or spend weeks rebuilding something the open-source community had already shipped.

Third was the dependency problem. We were pinned to outdated libraries. That doesn’t break anything today, but it almost always creates new issues down the line. Security patches lagged, modern Kubernetes features stayed out of reach, and every quarter we waited made the eventual escape messier.

Meanwhile, the Apache Flink Kubernetes Operator had matured into production-grade infrastructure, and its feature set read like our wishlist: intelligent job-vertex autoscaling to kill backpressure, automatic memory tuning, safe upgrade mechanisms, and a robust reconciliation loop. Additionally, it was battle-tested by companies at massive scale. The decision in front of us became obvious. We needed to migrate Lyft’s fleet, hundreds of critical streaming jobs, from our internal operator to the Apache one. However, we had to do this migration without disrupting existing deployment patterns, altering our team’s workflows, or causing massive outages in production.

This is the story of how we pulled it off, and why sometimes the best code you can write is the code you delete.

Our Legacy: The Lyft Flink Kubernetes Operator

Before we dive into the migration itself, it’s worth understanding what we were migrating from. The Lyft Flink Kubernetes Operator served us well for years, and being familiar with its architecture helps illustrate both why the migration was necessary and why it was nontrivial.

At its core, Lyft’s operator follows the standard Kubernetes pattern by continuously watching custom resources and reconciling their state. We defined Flink applications using a FlinkApplication custom resource that looked something like this:

apiVersion: flink.k8s.io/v1beta1
kind: FlinkApplication
metadata:
  name: my-streaming-job
  namespace: production
spec:
  image: lyft/my-flink-job:v1.2.3
  flinkVersion: "1.16"
  jarName: "my-streaming-job.jar"
  entryClass: "com.lyft.streaming.MyJob"
  parallelism: 8
  
  jobManagerConfig:
    replicas: 1
    resources:
      memory: "2048Mi"
      cpu: "1.0"
      
  taskManagerConfig:
    taskSlots: 4
    resources:
      memory: "4096Mi"
      cpu: "2.0"
      
  flinkConfig:
    state.backend.fs.checkpointdir: s3://lyft-flink/checkpoints
    state.savepoints.dir: s3://lyft-flink/savepoints
    execution.checkpointing.interval: 60s

Lyft’s operator takes this spec and creates the necessary Kubernetes resources: a JobManager (JM) deployment, a TaskManager (TM) deployment, any external runtime configurations, and optionally an ingress for the Flink UI. Simple enough, in theory.

The State Machine: Where Things Got Interesting

The Lyft operator’s real complexity lived in the state machine. Deploying a stateful streaming application isn’t like deploying a stateless web service; you can’t simply kill the old pods and start new ones. If you did, all in-flight data and state would be lost. Our operator walked through a careful choreography of states to ensure zero data loss.

Dual Deployment Mode (our default) had minimal, but non-zero downtime. When a user updated a FlinkApplication, the operator would spin up a completely new Flink cluster alongside the running one. Once the new cluster was healthy, it would trigger a savepoint on the old job, wait for it to complete, cancel the old job, tear down the old cluster, and finally submit the job to the new cluster from the savepoint.

BlueGreen Mode took it a step further for truly zero-downtime deployments. Both clusters would run simultaneously (“dual running”), and you would manually switch traffic by setting a tearDownVersionHash field after verifying the new version was working correctly. This gave users a manual kill switch, which worked well for critical jobs where one would want to inspect the new version before committing.

The state machine tracked states like ClusterStarting, Savepointing, SubmittingJob, Running, DeployFailed, and our favorite debugging companion, Recovering (for when savepoints failed and we needed to fall back to externalized checkpoints). Each transition was carefully orchestrated with retry logic, timeout handling, and event logging.

Problems: The Cracks in the Foundation

Beyond the maintenance burden and the feature gap, two specific pain points made this decision unavoidable. Both stemmed from early design choices that suited a smaller fleet, but broke down as our workloads grew to be more varied over time.

Savepoint Fragility

Savepoints were the only first-class mechanism for preserving application state during upgrades. If a savepoint failed, the operator fell back to looking for an externalized checkpoint. However, the checkpoint fallback was a last-resort recovery path rather than a strategy users could deliberately choose. The savepoint trigger had no retry logic and no idempotency guarantees. If no recent checkpoint was available, the deploy either failed or restarted the job with no state at all. For large-state jobs where savepoints were slow or occasionally timed out, a transient failure escalated into a full recovery cycle. The Apache operator introduced last-state as an explicit upgrade mode, which uses high-availability metadata or the latest checkpoint by design. This was exactly the robustness we needed, particularly for jobs where an unhealthy JobManager couldn’t cooperate with a savepoint request.

Rigid Resource Model

Our operator exposed a single systemMemoryFraction knob. This represented a fraction of container memory reserved for non-JVM use. For pure Flink Java applications, this overhead was often unnecessary. For Apache Beam workloads using the Python SDK harness, it was vital. Unfortunately, how much memory those Python processes actually needed was essentially guesswork, since usage happened inside the Flink memory model. There was no observability into the utilization and no way to tune it systematically. If we set it too low, TMs would get OOM-killed. If we set it too high, we wasted resources.

Migration: Our Blueprint for a Seamless Switch

Setting up a new system is the easy part. The real nightmare of any migration is unwinding everything that still points to the old one. At Lyft, all Flink deployment configs start as Jsonnet files. We use a shared module to set up operational defaults, and individual teams layer their own application configs on top:

common.sp.FlinkApplication("myflinkapplication", "staging", "myproject") {
  application+: {
    spec+: {
      parallelism: 8,
      flinkConfig+: {
        "state.backend": "filesystem",
        "restart-strategy.fixed-delay.delay": "10s"
      }
    }
  },
}

In the above example, an individual app extends the base factory with its own parallelism and Flink configuration. The entire system was built around the legacy operator’s FlinkApplication custom resource definition (CRD). Rewriting every library and application overlay to natively produce the Apache operator’s FlinkDeployment CRD would have been a massive and risky undertaking.

Instead, we chose to translate at the boundary, so teams could continue defining their applications exactly as they always had. The translation from FlinkApplication to FlinkDeployment happens within our deploy API. When the deploy API encounters a legacy resource, it converts the resource into something the Apache operator is familiar with. This handles the structural mapping:

• Job Spec: We assemble flat fields like jarName and parallelism into the Apache operator’s nested spec. jarName becomes a jarURI.
• Upgrade Mode: We default to last-state to restore state from the last checkpoint if possible, and map legacy configs like savepointDisabled to stateless.
• Pod Templates: Inline container configs on the legacy spec like node selectors, tolerations, and sidecar annotations get promoted into proper PodTemplateSpec objects.
• Backwards Compatibility: We inject legacy environment variables like APP_NAME and HOST_NAME so application code doesn’t break.

Crucially, the migration was incremental. Dynamic configuration allowed us to migrate one application at a time, starting with staging. If an app hit issues, we could instantly roll it back to the legacy operator. To the developer, the change was largely invisible.

From the Trenches: Additional Work to Integrate the Operator

While the translation layer got us onto the Apache operator, the migration was not yet done. We still had to address a few follow-on problems, and do additional work to make sure we saw all the benefits the Apache operator had to offer. Two of those benefits are autoscaling, which dynamically adjusts a job’s parallelism, and autotuning, which adjusts the memory resources each TaskManager gets.

BlueGreen Deployments

Once we migrated apps onto the Apache operator, every deploy cost us an average of 3–6 minutes of downtime, and closer to 20 minutes for our largest jobs. Lyft’s operator had managed JobManager and TaskManager lifecycles together through a complex graph of operations, which kept downtime to a minimum. The Apache operator took a different approach; it spun up only the JobManagers, configured them with the relevant savepoint and resource specs, and let them coordinate the TaskManagers. Each deploy walked through the full sequence of taking a savepoint, triggering the JM pods to tear down TM pods, tearing down the JM pods, and then starting the new FlinkDeployment’s JMs using the recently taken savepoint, which then brought up the new TM pods. For a real-time business like Lyft, this was unacceptable for several of our most important use cases.

The fix for this was FlinkBlueGreenDeployments, a newer CRD that maintains underlying FlinkDeployment objects so a new version can come up alongside the old one before any cutover. This feature was still under development when we started testing it internally; we hit a bug in how the operator renamed Flink configuration fields under BlueGreen, and contributed a minor upstream fix. Benefiting from the open-source community’s efforts and being able to contribute back further reinforced our decision to move to an open-source solution.

Autoscaling

The Apache operator’s autoscaler uses metrics like operator busyness, backpressure, and record backlog in Kafka and Kinesis to adjust each job’s parallelism. Traffic across most of Lyft’s use cases is highly variable and stream processing is expensive, so autoscaling was, in theory, an obvious win. In practice, our Flink version (v1.17) required apps to restart in order to rescale, and the apps that would benefit the most from autoscaling were precisely the ones that couldn’t afford restart-induced downtime.

Upgrading to Flink v1.19 unlocked two important changes. First, it unlocked in-place autoscaling. Jobs could change parallelism without restarting, which both preserved data freshness and avoided the spiky catch-up traffic we used to see when jobs resumed from offsets that were a few minutes old. Second, it enabled an upgrade to the new Amazon Kinesis Source Connector, which emits record backlog metrics for Kinesis streams the same way the Kafka source does. This gave the autoscaler the signal it needed to make scaling decisions on Kinesis-bound jobs, which had previously been a blind spot.

Autotuning

Autotuning appealed to us for two main reasons. It promised dynamic right-sizing to cut cloud costs, and it spared our devs from tuning resource configurations by hand. However, we ran into an unexpected problem after we rolled this feature out.

Our Apache Beam workloads run user code in a Python SDK harness, which lives outside the Flink memory model. The Apache operator didn’t register that we were reserving memory for those Python processes inside the JVM overhead allocation. We had been tucking it there for years because most of our jobs barely touched their JVM overhead anyway. This hack worked until we enabled autotuning. The operator saw that these memory pools were not really used by Flink, and reclaimed them, OOM-killing TaskManagers across our Beam fleet. We ultimately moved the Python harness out of the Flink container entirely, running it as a sidecar in the same pod with its own resource limits.

Another awkward property of autotuning is that it doesn’t play well with in-place autoscaling. To tune a job’s resources, the operator has to restart it. We weren’t willing to give up either feature, so we settled on a two-tier approach where business-critical applications that power pricing, routing, and similar real-time decisions disable autotuning and keep in-place autoscaling. Less critical applications that could tolerate some downtime use both autotuning and autoscaling with restarts.

Karpenter

These features also made our node infrastructure obsolete. Before autotuning, we bin-packed pods onto AWS nodes efficiently by enforcing 1:4 or 1:8 CPU:memory ratios with power-of-two values. Autotuning broke this. The same application could have very different ratios from one tuning cycle to the next, and a static node pool could not keep up. We adopted Karpenter, which provisions EC2 nodes on demand based on what each pod actually needs, optimizing for cost or utilization as appropriate.

This two-tier autotuning model produced an additional unexpected efficiency win related to unused spend (the share of compute resources we are paying for that ends up unallocated). If we are paying for an 8-core node with 32GB of memory, but only allocate 4 cores and 16GB, half of the cost of the node is unused. We had already deployed a custom Kubernetes pod scheduler on our EKS cluster that uses a “most-allocated” strategy, packing pods onto busy nodes first. When autoscaling restarted apps to rescale them, the pods had a chance to move from under-allocated nodes to densely packed ones, letting us shed underutilized nodes.

Conclusion

This migration was a multi-quarter journey that often felt like changing the engines on a plane mid-flight. It took longer than expected to completely sunset our in-house operator, as there was a mess of state-machine quirks, memory model mismatches, and CRDs that didn’t translate cleanly. However, the difference in the platform after completing this migration is hard to overstate; the operator maintenance that used to eat engineering time on every Flink upgrade has been minimized, and the autoscaler is doing what we had hoped, which is right-sizing a fleet that had been overprovisioned by a few million dollars per year.

The important, but hard-to-quantify benefit is that we stopped being the only contributors to our operator and infrastructure. When we noticed the BlueGreen bug, we filed an issue and a fix landed upstream within days. Our team’s posture changed from “we maintain this thing” to “we use this thing, and sometimes improve it for everyone,” which is a much better place to be. What’s next is the work we actually wanted to focus on from the start. We’re building faster feedback loops for marketplace pricing, handling autonomous vehicle telemetry, and scaling streaming primitives for Lyft’s next chapter. The Apache operator finally gave us the breathing room to work on these challenging problems.

Appreciation

Special thanks to Anirudh Adkar and Ravi Magham, who provided guidance to us throughout the project, and Janice Lee, Rohan Varshney, and Yigal Kassel for their contributions to the team’s operations. The migration would not have been possible without their efforts.

Finally, thanks to our many internal customers of Lyft’s Streaming Compute platform, who bore with us and provided ample helpful feedback throughout the migration, including but not limited to Kyle Bilton and Taylor Wilson.

Lyft is hiring! If you love the idea of solving ambiguous problems in technology and infrastructure, visit Lyft Careers to see our openings.

Rerouting the Stream: How Lyft Moved to the Apache Flink Operator was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: eng.lyft.com