Written by Savelii Kolesnikov, Jay Patel, and Ivanna Sivak
Background
Lyft processes billions in payments annually, and that volume continues to grow. Agentic coding has drastically accelerated feature development, but it raises the stakes for the one phase that cannot be rushed: ensuring the highest standards of reliability and correctness in critical payment systems.
Every payment platform eventually hits the same wall. Unit tests cover business rules, integration tests validate boundaries, and load tests push upstream limits, but standard testing stops the moment a request reaches a third-party processor.
Processor test cards are built for simple flows: a card either passes every payment action, from authorization to capture to refund, or fails them all. Real payment flows often have one step fail while the others succeed.
In this post, we’ll explain why test cards break down and how we built our own fake payment provider to simulate any processor behavior on demand. Unlike a typical mock, it runs safely in production, so we can load test our payment stack and trial new workflows on simulated riders before real ones.
Test cards don’t get you there with rideshare
Unlike paying for a product on an e-commerce website or your favorite streaming platform, Lyft, as a rideshare company, has a unique set of challenges with payments. While we can put a hold on the card on a ride request, the final charge does not always match the original hold amount on the card. We can use test cards provided by processors; however, they either succeed with all charge actions (authorization, increment, capture, direct charge, refund, etc.) or fail them all. We hit a wall if we want to test the following scenarios:
- A card that succeeds auth, fails capture, and then succeeds on the direct charge fallback
- A card that succeeds every charge flow but fails on refund
- A card that succeeds auth and increment, fails capture at the incremented amount, and succeeds when we fall back to capturing the original auth amount
- The processor itself is down
- The processor is still processing the charge and returns a non-terminal (pending) response
- The processor returns a response, but our services have timed out
None of the above scenarios are exotic. These are the exact paths our retry and fallback code was written for, and they were hard to exercise before production.
What the limitations of test cards cost us
These limitations cost us across the company. Without realistic failure paths, validating our own retry and fallback logic was hard. Anything touching intermediate payment failures was difficult to validate. Our end-to-end acceptance tests could only cover all success or all failure, never the middle. And because test cards are shackled to processor sandboxes, our simulated rides load testing framework had to bypass the payment flow entirely. We’ll come back to this.
Every one of those costs points at the same root cause: the thing standing between us and confidence lives behind a processor’s API, on a release cycle we don’t control. The first instinct is to ask the processors themselves to fix it — better sandboxes, more test cards, more scenarios. That doesn’t scale. We work with several processors, each with different sandbox coverage, and sandboxes aren’t built to model our retry and fallback logic. We’d be maintaining a compatibility matrix instead of a test suite, one column per processor.
Instead of continuing to work around the sandboxes provided to us, we decided to take full control of our testing environment by building our own solution — a Fake Payment Provider designed specifically to meet the complex needs of rideshare testing.
Architecture
Why build our own Fake Payment Provider
The constraint that had been making our lives hard turned out to be the thing that saved us. Working with several processors forces you to hide all of them behind one interface, and an interface is something we could implement ourselves.
They are different companies with different APIs and their own quirks. Strip all of that away and they do the same handful of things. Tokenize. Authorize. Increment. Capture. Charge. Refund. Void.
So adding a Fake Provider is easy to describe. Implement those methods. Route some users to it. Done.
And that really is most of the design. The router resolves a provider and calls it. Nothing above the provider layer knows that this one never opens a socket. Because every processor’s response is already translated into one internal representation, the fake doesn’t have to mimic any processor’s API. It only has to produce outcomes our platform already understands. That boundary cuts both ways. Because the fake sits behind the provider interface, it never exercises the code that talks to each processor: building requests, parsing responses, and mapping their errors into ours. Those paths still rely on processor sandboxes, and closing that gap with record and replay is next on our list.
Easy to describe is not the same as easy to build. Translating this concept into practice required addressing three key questions:
◾ Access Control: Who is allowed to reach the fake provider?
◾ State Persistence: Where does the fake keep its scenario behavior across calls?
◾ Scenario Modeling: How is that test behavior actually defined and configured?
Access Control: Only simulated users get to touch it
Everything to do with the fake payment provider stays inside a fenced-off group of test users. In our system, they are called simulatedusers. The fake Low Value Token (LVT) is generated by us and accepted only for simulated test accounts.
Notice what is not in that picture. The router has no test mode and no special case. It has rules, and for a real rider the fake never becomes a candidate in the first place. Multiple independent checks across our services ensure fake payment methods can only ever be used by the internal simulated test accounts.
State Persistence: The scenario rides inside the token
A processor that is not real still has to remember how it intends to behave. Which scenario are we reproducing right now? What should the next action do? What comes back for each one?
That is state, and state has to live somewhere. There were three places to put it.
◾ A new service. More infrastructure than the milestone justified. Worse than that, it would have been a data path production does not have. We would have been testing a system that does not exist.
◾ The request. A magic amount. A header. A field called test_behavior. This is the obvious one, and it breaks on the second call. Captures and refunds are started by other services. Retries arrive hours later and read their state from the ledger. Nobody is carrying your header by then. Making it work would mean every service on the path forwarding a test-only field, which is exactly the parallel test-only path we were trying not to build.
◾ The token. We already had something that travels the whole path and gets persisted on the way. So the scenario went into the token.
A short detour, because the rest of this section leans on it. Payment systems try hard never to hold a card number, so every card gets swapped for a token, a stand-in string. Think of a hotel. Your booking reference can’t open a single door, but the front desk swaps it for a key card that opens yours for the rest of the stay and says nothing about who you are. Card tokens work the same way. When you add a card, the app receives the booking reference: an LVT, which on its own cannot charge anything. The tokenization service plays the front desk and swaps it for the key card: a high value token, or HVT, which gets stored and used for every charge after that.
You hand that to the wallet service the same way you would hand over a real card LVT. The tokenization service decodes it, mints a high value token, and stores it exactly as it stores a token from a real processor.
Two nice properties came out of this. The outcome became a property of the instrument rather than the transaction. That is how a real card behaves. And follow on operations read the scenario from the parent authorization rather than from the token, so the scenario cannot change halfway through a charge lifecycle.
Scenario Modeling: A scenario is just a dictionary
At charge time the provider decodes the scenario name and looks it up. A scenario is a map from action to an ActionSpec. The spec says whether the action succeeds, how slow it is, and what error comes back when it fails.
Failures also carry a network decline code, a two-digit number like 51 for insufficient funds or 05 for do not honor. The platform reads it to decide what to do next, including whether a charge is worth retrying on a different processor.
Charge.Action.AUTH: PayinActionSpec(
success_predicate=ACTION_FAILURE,
error_code="insufficient_funds",
error_message="Auth declined",
network_decline_code="51",
delay_ms=100,
retry_specs=...
),
Success is a predicate and not a flag.
ACTION_SUCCESS: Callable[['PayinActionContext'], bool] = lambda ctx: True
ACTION_FAILURE: Callable[['PayinActionContext'], bool] = lambda ctx: False
ACTION_CHAOS: Callable[['PayinActionContext'], bool] = lambda ctx: random.random() < 0.5
Two of those are boring on purpose. The third one is not. ACTION_CHAOS flips a coin on every call. A cleanly broken processor is easy to write code for. The one that fails half the time is what actually hurts you at three in the morning, and now we can summon it whenever we like.
High latency is a scenario, too. Set delay_ms past the caller’s timeout and you get the scariest case of all: the processor succeeded, but we gave up waiting. Now we can test what happens next. The fake doesn’t try to be clever about it. It sleeps, succeeds, and forgets. Everything after that, the caller retrying and the platform reconciling what actually happened, runs through our real payment code, which is exactly the code we wanted to test.
The part we like most is retries. An ActionSpec can carry a map of ActionSpecs keyed by attempt number. The model refers to itself, and that recursion is the whole retry mechanism.
Below is a complete card. It authorizes cleanly. It fails the capture once. It succeeds when the platform tries again.
SOFT_DECLINE_RETRY_SUCCESS_SCENARIO = PayinFakeScenario(
name="card:fake_provider:soft_decline_retry_success",
actions={
# Step 1: AUTH succeeds
Charge.Action.AUTH: PayinActionSpec(
success_predicate=ACTION_SUCCESS,
delay_ms=100
),
# Step 2 & 3: CAPTURE fails on first attempt, succeeds on retry
Charge.Action.AUTH_CAPTURE: PayinActionSpec(
success_predicate=ACTION_FAILURE,
error_code="transient_error",
error_message="Temporary processing error",
network_decline_code="91",
delay_ms=80,
retry_specs={
2: PayinActionSpec(
success_predicate=ACTION_SUCCESS,
delay_ms=100
)
}
)
}
)
Read it top to bottom, and you have the entire card. Reproducing that on demand in a processor sandbox is not something sandboxes are built for. In our case, it is a few dictionary entries.
The provider decides nothing. It decodes the token, looks up the scenario, picks the spec for this action, and checks which attempt this is. Note where that attempt number comes from. It is read off the stored charge and not passed in by whoever is calling.
So that is the architecture. A provider anyone can plug in. A fence only simulatedusers cross. A dictionary that decides what happens next.
The interesting part is what people started doing with it.
What does Fake Provider unlock for us?
Staging: Pick your fake scenario from a menu
Here is the part that changed how we work day to day. You open the rider CLI. You make yourself a simulated rider. You run one command and a list comes up.
Three checkmarks and you are done. That rider now has a card that declines every authorization, and you did not have to ask anybody for it. No sandbox credentials. No waiting on a processor. No pinging someone on our team to find out which test card number does the thing you want.
So what do you do when the one you need is not among them?
◾ You are an engineer working on the retry path. You need a capture that fails once and then succeeds. That one is on the list. You need one that fails twice and then succeeds. That one is not.
◾ You are a quality engineer testing some edge case where the authorization is declined and the direct charge goes through anyway. Very specific. Almost certainly not on the list.
◾ You are a product manager and you want to see your own flow the way a rider sees it when the card is refused. Or a marketer who wants to test an offer tied to a specific card issuer.
Before AI, the answer was to go and write the scenario yourself. Define the scenario. Register it. Add a card payload with a fresh number. Generate the token. Add it to the menu. Five steps in a repository you have probably never opened.
Now the answer is a skill. Open the repository that holds the scenarios. Run the skill that adds one. Describe the card you want in a sentence.
> /fake-provider-add-scenario
I need a card where the authorization succeeds but the capture fails twice
with a soft decline and then goes through on the third attempt.
It writes all of it! The AI generated scenarios still go through a human code review.
Your part of that takes a minute. One sentence is the whole contribution. Everything after it is the agent walking the repositories that Fake Provider is spread across and writing the same five things you would have written by hand.
That is a good place to be, but it is not where we want to stop. All of that work only exists because scenarios ship as library code. Turn Fake Provider into a service and a new card becomes data instead of a config change in multiple repositories.
Production: Load test your services
How do you stress-test a payment system when you can’t charge real credit cards?
Payment services were previously outside the scope of our load tests. Simulated rides short-circuited before they ever reached a real charge. No authorizations fired. No charges processed.
Fake Provider changed that entirely. By provisioning simulated users with a fake card, simulated rides now flow through the entire stack. Fake Provider transactions are tagged and kept separate from real payment activity.
This approach can answer questions we simply couldn’t before:
◾ Does the charge process slow down or start timing out once real load hits it?
◾ Is there a sane, deliberate limit on how long the system waits before giving up on a slow request?
◾ Does something that works in a developer’s local setup behave the same way once real production traffic routes through it?
◾ Where is the charge-lifecycle latency ceiling?
◾ Does auth/capture keep pace, or do retries start stacking under load?
◾ Does a fraud/risk check become the bottleneck, or does a rate limiter just mask the real capacity ceiling?
But how do you set up hundreds or thousands of users in advance for load testing? Well, every test needs the same starting point: a user who already has a payment method that works. Create a brand new one for every single run and you’re testing account setup, not payments. So there’s a pool instead — a set of test users, reused run after run.
Pools drift, though. A card expires. A test leaves a user mid-scenario and never puts them back. Something changes upstream and a card that used to work stops working. Left alone, a pool slowly rots, and the first symptom is usually a test failing for a reason that has nothing to do with the code anyone just shipped.
Here comes the Resource Manager service that we mentioned in the previous article. It constantly runs health checks and automatically fixes issues. Since we want load tests and other scenarios to actually succeed, we configured the fixer to make a fake-success card the default payment method for these users. Still, we take a selective approach: we can leave this check off for pools meant to test failed transactions or other corner cases. The health check’s configuration works less like one global rule and more like a per-pool setting — different pools, different defaults. It lets us get rid of old cards gradually and replace them with fake ones for simulated test users only.
Adding payments to simulated rides meaningfully increased load on our critical payment services in production. This let us verify that both our application tier and our datastores scale smoothly under the surge: autoscaling responded to demand without throttling for DynamoDB, and connection pools stayed healthy for Aurora Postgres. For the flows we pushed the hardest (authorization, capture, and direct charges), scaling kept pace end to end.
Production: Test new workflows
A new payment workflow — a new retry strategy, a new fallback path, a new integration with a processor — can pass every test in staging and still behave differently in production, where the surrounding infrastructure isn’t the same. Because simulatedusers can be routed through Fake Provider even in production, we can point a new workflow at simulated riders first and let them exercise the exact new code path end to end — same services, same config, same scaling behavior real riders will hit, all without a real card or a real dollar involved. If the new capture-retry logic has a bug, our simulation platform finds it, not a rider.
This gives us a few things we didn’t have before:
◾ A canary population to test our new workflow on in production before we’re confident enough to expose it to real traffic.
◾ The ability to pick the scenario the simulateduser rides through — a soft decline, a slow processor, a chaos card — so the canary isn’t just “does it work”; it’s “does it work under the specific failure mode this workflow was built to handle.”
◾ A gate before graduation: a new workflow only gets promoted to real riders once it’s been proven out against simulated ones running the same code in the same environment.
In practice, this changes how we roll out significant payment changes. We exercise the new path with simulated riders and load test it at scale first, so by the time real riders ramp on, the workflow has already handled production-level load.
Looking ahead
Fake Provider has already changed how we test payments: broader failure coverage in staging (manual and automated), payment services included in load tests, and a safe place for new workflows to prove themselves before real riders see them. But every scenario today lives as a dictionary entry in library code, spread across a handful of repositories. That’s been fast to build and easy to reason about, and it’s also the ceiling. As the list of scenarios grows and more teams want to define their own, “a few dictionary entries in a repo you have to know about” turns from a feature into a bottleneck. The next step is turning Fake Provider from a library into a service — a new scenario becomes a piece of data submitted through an API, not a change reviewed and deployed across multiple repositories. That shift is what opens up the rest of this list:
◾ Record and replay: Capture real request/response pairs from the processor’s sandbox environment and replay them as scenarios, so Fake Provider’s behavior tracks what processors actually do, including the quirks a hand-written approximation tends to miss.
◾ Async completion notifications: Fire the delayed webhook that carries a processor’s real outcome on a schedule we control (on time, late, out of order, or never), so we can exercise the code that has to handle all of it.
◾ New payment methods: Simulate payment methods beyond cards.
◾ Disputes: Simulate the full chargeback lifecycle (chargeback, evidence submission, resolution), the one flow we still can’t fully test outside a real processor sandbox.
◾ Card updates: Simulate a card-updater feed landing mid-lifecycle, the way real processors tell us about an expired or reissued card.
◾ Rate limits and outages: Simulate a processor rate limiting us or going down entirely, so fallback routing and circuit breakers get validated under real pressure.
Stay tuned as we build out the next iteration of payments resilience!
Acknowledgements
We would like to thank the DevEnvs team (Maura Killeen, Levi Meng, Roy Tang, Chelsea Xu, and Alex You) for their huge support in turning simulated rides into real load on the payment stack.
Lyft is hiring! If you love the idea of solving ambiguous problems in technology and infrastructure, visit Lyft Careers to see our openings.
Beyond Test Cards: A Fake Payment Provider for Testing at Scale was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source: eng.lyft.com
