See it live

Case study · Platform engineering · October 2026

Rebuilding a live API, one key at a time.

The product I work on is an HTTP API that customers send work to around the clock. I designed its next platform, V2, to run beside the current one, with a router in front that moves customers across one API key at a time, and a way back that takes seconds.

Jamal Aldeen · Backend engineer and acting team lead

10,000
requests a second the routers are load-tested to carry, with zero errors
≈80M
new tasks a week behind it
10.3M
requests in the router's first hours live, 0.001% upstream errors
≤10 s
to move a customer to V2, or back
Scroll to follow one task

The problem

The current platform carries every customer, every minute.

It works, and it can't stop. But the next stage needs things it can't prove, and a "big bang" switchover would put every integration at risk in the same minute. So V2 had to keep four promises:

  • Exact limits. Never admit one task more than the customer's concurrency plan.
  • No lost results. Every accepted task ends exactly once: done, failed or timed out.
  • Honest capacity. Grow the fleet with demand, inside a budget, counting only proven workers.
  • Reversible moves. Move customers one key at a time. Moving back is instant and has no conditions.

Follow one task · 1 of 8

A customer sends work to the same endpoint as always.

No new SDK and no code change. The request reaches the load balancers customers already use. Everything new sits behind them.

Follow one task · 2 of 8

A small Go router decides who answers.

It hashes the API key and looks it up in a route map it refreshes every 10 seconds. A migrated key goes to V2. Every other key goes to the current platform, exactly as before. The map holds only SHA-256 hashes, never the keys.

Follow one task · 3 of 8

V2's edge picks the least-busy API server.

Caddy ends TLS and balances across four Laravel Octane servers. A compatibility layer accepts the old request format, so answers come back in the shape the customer's code already parses. The key, the task type and the payload are checked before anything is stored.

Follow one task · 4 of 8

One atomic step decides whether the task gets in.

Each customer's plan allows a set number of tasks at once. A Lua script in Redis counts the customer's live leases, compares them with the limit and, if there's room, creates the task and takes a 300-second lease. All in one step, so four API servers can race and the limit still holds exactly.

Follow one task · 5 of 8

Handed to its queue, with a receipt.

Each of the 14 task types has its own RabbitMQ queue. The publish waits for the broker to confirm it. If the broker refuses, a second script rolls the admission back and frees the slot, so retrying is always safe.

Follow one task · 6 of 8

A worker does the work and touches nothing else.

Workers read only their own type's queue and publish results back. They never connect to the database. The fleet grows and shrinks with demand, and it only counts workers that have proven they are healthy.

Follow one task · 7 of 8

The first result wins. Duplicates are ignored.

Queues redeliver and workers crash, so V2 expects every result to arrive twice. The result consumer validates the message, takes the worker's identity from the authenticated connection, and finalizes through a script guarded by a once-only marker. The customer's slot is released in the same step.

Follow one task · 8 of 8

The answer goes back the way the customer expects.

The customer polls for the result or gets a signed webhook. Webhooks retry after 30 and 120 seconds, then stop in a dead-letter queue where an operator can see them. Nothing retries forever, and nothing disappears quietly.

Now break it

The happy path is easy. Pick a failure.

—

—

—

Design decisions

Six choices that carry the guarantees.

Most of the reliability doesn't come from clever code. It comes from deciding where each rule lives and refusing to let it live anywhere else, then writing down what that choice costs.

How do we never over-admit?

Limits live in one atomic Redis script

Each customer's running tasks are a sorted set of leases, scored by expiry time. One script counts the live ones, compares them with the plan and admits the task, so no two servers can race past the limit.

The cost A lease is a fixed window. If nothing finalizes a task in time, a reconciler that runs every 15 seconds finalizes it as a timeout and frees the slot.

How do we never finish a task twice?

Assume every message arrives twice

Finalizing is one script guarded by a set-if-not-exists marker. The first result wins. A duplicate is acknowledged and dropped, and a result that arrives after a timeout can't overwrite it.

The effect A worker can crash halfway through a task, and the task still ends exactly once.

How much can a worker touch?

Workers publish. They never write.

Workers only read their queue and publish results. V2's own consumer checks each result and takes the worker's identity from the broker connection, never from the message body.

The effect A misbehaving worker can't corrupt task state or another customer's data. Invalid results are quarantined, not finalized.

What if V2 fails mid-migration?

Fail toward a retry, never a double run

The router never re-sends a failed V2 request to V1, because the task may already be admitted on V2. It answers with an error the customer's client already retries. One command sends every key back to V1 in about a second.

The cost The customer retries once, instead of a task being run and counted twice.

When should the fleet grow?

Count only proven capacity

The autoscaler buys a server only when a shortfall holds across two evaluations, and never past its budget. New workers count as zero until their health check passes, silent workers count as zero, and "unknown" blocks a decision instead of reading as zero.

The effect No buying sprees on a blip. It ran in shadow mode first, recording every decision without acting.

How do failures stay visible?

Every retry is bounded. Every dead end is a queue.

Results that can't be finalized yet retry after 5, 15 and 30 seconds, then land in a dead-letter queue that can be replayed. Failed messages are kept for 14 days.

The effect Operators can see every stuck message, and nothing loops forever.

The router

Three switches, one decision.

Every live request to the API passes through a pair of these routers. Flip the conditions and follow the branch the router takes. The rules are the real ones.

Autoscale

A control loop with a budget.

A working model of the decision engine for one task type. It runs every minute, waits for a shortfall to hold across two evaluations, counts booting workers as zero and stops at the budget cap. Change the demand or silence a worker. One simulated minute passes every 0.7 seconds.

Decision log

    Migration playbook

    One customer at a time, with the exit always open.

    A customer moves by API key, not by task type. The router sends a whole key to one side, so a customer who uses any type V2 doesn't support simply stays on V1. Three gates sit between measuring and switching.

      Rollback, from any stageNo conditions and no approval step. The key is back on V1 at the routers' next refresh, within 10 seconds.

      Where it stands · October 2026

      • Router: in production since September 2026. Every live API request passes through it.
      • V2: running in production on its own nine servers, with the autoscaler switched on for all 14 task types.
      • Customers: moving next, one at a time, starting with a single canary and a seven-day watch.

      How the work was done

      "Done" means evidence, not a checkmark.

      I own the architecture and wrote about 96% of the commits, much of it with AI coding agents that I directed through written specs and reviewed. That only works with a contract in the repository that every agent and every human must follow. The rule that mattered most is the "no fake completion" rule.

      Not evidence of done

      • "It compiles." Run the test.
      • "Tests pass locally," with no gate run on the exact commit.
      • "Looks fine," with no log, request or screenshot.
      • "Will add tests later."

      Evidence of done

      • A test that proves the new behaviour. Every bug fix gets a regression test named after the bug.
      • A green release gate on that exact commit: lint, static analysis, the full suite of 4,200+ tests, and scans.
      • A smoke run on a separate certification environment before production.
      • An entry in the evidence index, and a logged decision for any non-obvious choice.
      If pressed for time, scope down. Don't skip evidence.

      Lessons

      What I'd take to the next system.

      None of these are new ideas. The work was in applying them everywhere, with no exceptions.

      1. Give every rule one home

        Concurrency accounting lives only in six atomic scripts. Application code can't drift from rules it never implements.

      2. Design for the second delivery

        Queues redeliver and workers crash. Finalizing exactly once turned a whole class of incidents into non-events.

      3. Make the way back cheaper than the way forward

        Moving a customer has three gates. Moving them back has none. That difference is what makes moving them safe at all.

      4. Treat "unknown" as a blocker, not a zero

        The autoscaler counts only capacity that has proven healthy. Missing data stops a decision instead of skewing it.

      5. Make the emergency path one command

        A single switch sends every key back to V1 in about a second, with no deploy and no restart, so using it is a routine step and not a gamble.

      • Laravel 12
      • Octane · FrankenPHP
      • Redis 7 + Lua
      • RabbitMQ
      • MySQL 8.4
      • Go
      • Caddy
      • Filament
      • Docker
      • Hetzner Cloud
      • Prometheus · Grafana
      14
      task types, each with its own queue and worker pool
      52
      queues, defined as code and reviewed
      6
      atomic Lua scripts own all task and concurrency state
      55
      alert rules and 5 dashboards, written as code

      About

      Jamal Aldeen

      Backend engineer in Cairo and acting team lead. I build the routing, payment, queueing and server systems products run on, designed so failures stay small and reversible.

      Portfolio ↗LinkedIn ↗GitHub ↗