Choosing a Workflow over a Queue

TL;DR. A long, metered inference call on this site was deferred off its request and canceled at thirty seconds on every real run, and no test could see it. Cloudflare documents a Queue as the remedy, and a Workflow was chosen instead, the second on this site: another was already running the site’s eval suites on a schedule. A Queue is still the right tool for many small, independent events, and this site still runs one. For a single expensive unit of work that is mostly waiting, that someone will ask about by id, and that your tests need to hold, a Workflow gives you three properties a Queue would leave you to build by hand. This is why, told through both Workflows, with the decision rule I would hand another engineer and the three defaults I would override on the first day.

The run that never finished

The call in question is one metered request to a frontier model, measured at 78,222 ms. It sits behind a route that only holders of a scoped token can reach, and it ran inside the request that asked for it. That worked: the browser waited about eighty seconds and the run produced a result. What it did not survive was the connection. A refresh, or a dropped connection, anywhere in those eighty seconds lost the run entirely, and there was no record of it to come back to.

So the work was deferred. The request now opens a record, answers with a permalink in about a second, and the long call finishes in the background.

The background, at first, was ctx.waitUntil, and its documentation is exact about the budget:

For HTTP-triggered Workers, ctx.waitUntil() can extend execution for up to 30 seconds after the response is sent or the client disconnects.

and exact about what happens past it:

If any Promises have not settled after 30 seconds, they are canceled.

78,222 ms is more than double that budget. Every real run was canceled mid-call, the write that closes the record never ran, and the notification that a run had finished was never sent, including the one meant for a run that failed. I confirmed it against production rather than inferring it: the start request answered in 0.98 seconds, and the record still read pending with every result field null at 25, 40, 60, 85, 110 and 150 seconds. The only trace anywhere was the runtime’s own line, waitUntil() tasks did not complete within the allowed time after invocation end and have been cancelled.

No test caught it, and none could have. No test in this repository may spend inference, so the harness replaces the engine with one that declines after 500 ms, sixty times inside the budget. The test that watched the record move from pending to closed passed either way. A waitUntil promise has no handle, no status and no name, so there was nothing for a test to hold onto even in principle.

ctx.waitUntil is still in that route, and it was never the thing that broke. What it holds now is the call that creates the Workflow instance, which settles in milliseconds. The rule the defect taught is narrower than avoiding an API: nothing whose duration is a measured number may be scheduled on a budget that is not.

The documentation recommended a Queue, and it was not wrong

The same passage names the fix a paragraph later:

If the work cannot finish within the waitUntil() time limit, send messages to a Queue and process them in a separate consumer Worker. Queues provide reliable delivery and automatic retries.

The issue I filed for the defect recommended exactly that, and it was a sound recommendation. The design that introduced the deferral had rejected a Queue because a message would have to carry the visitor’s credential and the input they had submitted. Neither was true of the shape by then. The credential is checked once, when the run starts, and is never needed again, and the input is already stored on the record. A message carrying the record’s id and nothing else answers the objection completely.

So both would have worked. That is what makes the choice worth writing down.

Three things a Queue would have left me to build

A paid result is not paid for twice. Queues deliver at least once, and Cloudflare’s guidance for the rare duplicate is to generate a unique id and use it “as the primary key on database inserts and/or as an idempotency key to de-duplicate the message after processing.” That is correct, and for this work it is a system you write yourself. A redelivered message with no record beside it re-runs the whole seventy-eight seconds. In a Workflow, step.do persists its return value, and an instance that is evicted or restarted replays the step from that record instead of calling the model again. That is the platform’s primitive rather than my table.

A run has an address and a terminal state. Every Workflow instance has an id, and I create each one with the permalink id as the instance id. The question “what happened to this link?” becomes one command, wrangler workflows instances describe <workflow> <id>, which answers with a terminal status, and the error if there was one. A message that has been consumed, retried or sent to a dead letter queue has a history, but not one you can ask for by the id a person is holding.

A test can hold the run itself. This is the reason that decided it, and it is not a claim that a Queue is untestable: Queues are simulated locally, and under this repository’s harness a message really is delivered to its consumer. The difference is what a test can hold. A consumer’s run has no handle, so a test of a Queue-based fix would have watched the record, exactly as the waitUntil test did, and passed either way. While building the eval Workflow described below, I declared a probe workflows binding and booted the test harness with it. It started cleanly and create() returned a real instance id locally. That is not true of every binding here: the ones with no local simulator make the harness fail at boot. So the new suite gets the instance by the run’s id, polls instance.status() until it reaches complete or errored, and asserts on both the status and the record the run closed.

That last claim has a limit. The tests still cannot see the eighty seconds, because they still cannot spend inference, and the thirty-second budget is never crossed under the harness. What they pin is the shape that budget applied to: the long call is no longer something a request’s execution context is holding open, and the thing holding it now has a status a test can read. That is less than proof that production works. It is a great deal more than the nothing the earlier shape allowed.

The shape keeps the input out of durable state

This is the pattern with the domain taken out. It is close to what runs here and deliberately not identical to it.

import { WorkflowEntrypoint, type WorkflowEvent, type WorkflowStep } from 'cloudflare:workers';

type Params = { id: string };
type Outcome = { ok: true; result: Result } | { ok: false; code: 'refused' | 'errored' };

export class DeferredRun extends WorkflowEntrypoint<Env, Params> {
  async run(event: WorkflowEvent<Params>, step: WorkflowStep) {
    const { id } = event.payload;

    // Outside every step, so the input is re-read on a resume and never persisted.
    const row = await this.env.DB.prepare('SELECT input FROM runs WHERE id = ?')
      .bind(id)
      .first<{ input: string }>();
    if (row === null) throw new Error(`no run is stored under ${id}`);

    // The expensive call: no retries, and a verdict rather than a throw.
    let outcome: Outcome;
    try {
      outcome = await step.do(
        'work',
        { retries: { limit: 0, delay: 0 } },
        async (): Promise<Outcome> => {
          try {
            return { ok: true, result: await expensiveCall(this.env, row.input) };
          } catch (error) {
            return { ok: false, code: error instanceof Declined ? 'refused' : 'errored' };
          }
        },
      );
    } catch {
      // The step machinery itself failed: a timeout, or an exhausted retry.
      outcome = { ok: false, code: 'errored' };
    }

    // The cheap write: one retry.
    await step.do('close', { retries: { limit: 1, delay: '5 seconds' } }, async () => {
      await closeRow(this.env.DB, id, outcome);
    });

    // Outside every step, so a failed send cannot re-run the write above it.
    await this.env.EVENTS.send({ kind: 'run-finished', id, ok: outcome.ok });
  }
}

// At the route that starts it, after the row is written:
await env.DEFERRED_RUN.create({ id, params: { id } });

Four decisions in there would each be undone by a well-meaning tidy-up.

The params carry an id and nothing else. Params are durable state, so anything placed there is copied into a store you do not sweep. The run reads its input back from the record it is named after, which also anchors it: a run whose record is gone throws before it spends anything.

The read sits outside every step. A step’s return value is persisted, so wrapping the read in step.do would write the input into exactly the store the params were kept out of. Outside a step, a resumed instance simply reads the record again, at the cost of one database read against a call of seventy-eight seconds.

The expensive step returns a verdict instead of throwing one. When this site’s two Workers pass an error across the service binding between them, the far side receives a copy, and instanceof no longer recognizes the copy’s class. A step boundary also serializes an error, into durable state, so I did not rely on instanceof surviving one. The step catches its own errors and returns which kind of failure it was, and that verdict is persisted like any other result, so a resumed instance replays a refusal instead of asking the model again. What can still throw is the step machinery itself, a timeout or an exhausted retry, and the try around the step closes the record as errored so it never stays pending.

The notification is a Queue message, sent last. More on that below.

The second Workflow runs the evals

This site’s prompts are gated by eval suites, and until recently they ran only by hand, from my shell, before a merge. That worked well at first. The record showed what happened when I forgot: the private-tier disclosure suite, the one whose failure is supposed to block a branch, sat red for a few days, and the public operations page published that result under a heading reading “Latest run per suite”, beside another suite’s failure that had been fixed a day after it was recorded and never re-run.

The obvious home for a scheduled job is a CI cron, and it would have cost the credential boundary. A run from outside Cloudflare needs a signing key to mint a token that reaches the model, and an account-level login to record its results. CI here holds exactly one Cloudflare credential, scoped to object read and write on a single storage bucket, and that narrowness is the point. So the suites run inside the Worker that already holds the inference binding. A binding cannot leave Cloudflare, so nothing has to.

A scheduled() handler could hold it, just barely. A Cron Trigger gets fifteen minutes of wall time, and a weekly run is fifteen inference cases paced twenty-five seconds apart, because AI Gateway’s Unified Billing carries a wholesale rate limit set by Cloudflare that could be easily tripped in one run. That is five minutes of deliberate waiting before any model time, spent holding one invocation open against a ceiling the run would creep toward with every case added. A Durable Object alarm chain was the other option, but I chose not to pursue that even before measuring anything. An alarm chain would have needed its own cursor and its own store of results, which is the same “build it yourself” argument as the Queue, one level down. step.sleep suspends the instance between cases, and step.do records each case as it finishes.

One thing the schedule deliberately does not skip is the front door. The run reaches the public endpoints over a service binding the Worker holds to itself, so every case passes through the same request handler and the same single authorization check any outside client does. What that cannot exercise is the edge: a request over the binding never touches DNS or the route, so if the public hostname stopped resolving tomorrow, every scheduled suite would stay green.

The evals code makes three decisions a tidy-up would undo

These are excerpts from the running code, with the long comments trimmed.

The default retry policy is a decision, whether or not you make it. A step.do with no configuration gets the defaults Cloudflare documents:

const defaultConfig: WorkflowStepConfig = {
  retries: {
    limit: 5,
    delay: 10000,
    backoff: 'exponential',
  },
  timeout: '10 minutes',
};

That would be a third retry layer. This AI Gateway instance already retries a call up to four times on its own, and the client makes one more attempt past a transient refusal. A step that retried on top of both would pay for a failing case up to five more times, each attempt fanning out at the gateway. So a case gets no retries, and the one write that records a suite’s result gets one, because repeating it costs a database write rather than a model call:

const CASE_STEP: WorkflowStepConfig = { retries: { limit: 0, delay: 0 } };

const RECORD_STEP: WorkflowStepConfig = { retries: { limit: 1, delay: '5 seconds' } };

The pacing sits between them, and it is the reason this is a Workflow at all:

async function runPaced(step: WorkflowStep, units: Unit[]): Promise<CaseResult[]> {
  const results: CaseResult[] = [];
  for (const [index, unit] of units.entries()) {
    if (index > 0) await step.sleep(`pace before ${unit.name}`, PACE_MS);
    results.push(await step.do(unit.name, CASE_STEP, unit.run));
  }
  return results;
}

The credential is minted outside every step. Each run mints its own token with a one-hour life, scoped to what the suites need, and revokes it in a finally. The mint happens at the top of run() and not inside step.do:

async run(event: Readonly<WorkflowEvent<EvalsRunParams>>, step: WorkflowStep): Promise<void> {
  const env = this.env;
  const suites = suitesOf(event.payload);
  // ...
  token = await mintToken(await signingKey(env), claims);

Almost everything else in that run is a step, so moving the mint into one would look like a consistency fix. It would also write a bearer credential for a frontier model into the instance’s persisted state, and every test would stay green. Minting outside a step means a resumed instance mints a fresh token and the stored state never holds one. The cost is that cases before and after a resume ran under different tokens, and the audit trail records which one was live for each.

A run that could not run says so. A suite whose cases throw has not failed. It did not run, and the record has to be able to tell those apart:

} catch (error) {
  const reason = messageOf(error);
  console.error(`evals: the ${suite} suite did not complete: ${reason}`);
  await record(
    env,
    step,
    incompleteRow(suite, `the suite did not complete: ${reason.slice(0, 200)}`, ranAt),
  );
  continue;
}

The results table gained a status column for exactly this, and the operations page renders an incomplete row as “did not run” rather than letting an older pass stand in for the current state. That was the precise failure the schedule was built to end, so a scheduled run that failed silently would have rebuilt it.

A Workflow keeps what it persists for thirty days by default

None of this is free, and the cost is retention. Params and step results are durable state, kept after an instance completes for 3 days on the Free plan and 30 days on Paid. This site’s retention sweep is driven by a table over its own database, and as written it never touches that store. Cloudflare lets you shorten it, with default_retention on the binding or retention on each instance, and neither Workflow here sets either yet, so both keep state for the full thirty days. Until one does, anything a Workflow persists sits outside the one mechanism that enforces the site’s published retention windows.

That is why the input stays out of the params and out of every step. I cannot claim it stays out entirely. The expensive step’s result is derived from the input and can restate parts of it, and a step has to persist its result to be replayable. I accepted that deliberately, on a bound I can state. The persisted copy is a subset of what the run is about to publish anyway, and it lives for thirty days, a default nobody here has lowered yet, where the published copy lives for a year. The line I would not cross is persisting the verbatim input, or a result that is not going to be published, because either would turn a bounded duplicate into a second original that never gets swept.

A step’s return value can also persist at most 1 MiB unless it is a stream. Neither Workflow here comes close, but a design that returns large payloads from steps should store them elsewhere and return a reference.

Each tool is right for a different unit of work

Here is the rule I would hand someone else, drawn from the two cases above.

Use a Workflow when a single unit of work is expensive per attempt, so replaying a persisted result beats doing it again; is mostly waiting, on a model, a rate limit or a sleep; is something a person will ask about by the id they are holding; and needs a status your tests can read by that id.

Use a Queue for many small, independent events, where batching, backpressure, a dead letter queue and at-least-once delivery are the point, and repeating one is cheap. This site’s EVENTS queue is exactly that: small labeled events, batched, one digest email per batch, three retries and then a dead letter queue. It stayed a Queue on purpose, and eval failures deliberately do not travel on it, because a channel that also carries housekeeping stops being read.

The two also compose. The deferred run’s last act is a small message onto that Queue, sent outside every step so that a failed send cannot re-run the database write before it. The cost is that a resumed instance replays its steps and reaches the send again, so one run can occasionally produce two messages. The place to absorb that is the consumer, not the Workflow. Even a send that could never repeat would not guarantee one delivery, because the Queue itself delivers at least once, so deduplication belongs where the message is read: the message carries the run’s id, and a consumer that records each id before acting can skip one it has already seen. That is the idempotency Cloudflare recommends for every Queue consumer. This site’s consumer does not do it yet, because a duplicate here is an extra notification rather than a second charge. The Workflow does the expensive, addressable work, and the Queue does the fan-out it was always right for.

If you adopt Workflows, these are the three defaults I would override before the first one ships.

1. Set retries on every step, deliberately. The default is five retries with exponential backoff, and it stacks silently on whatever retries your client and your gateway already make. For a step that calls a paid model, the right number is usually zero.
2. Treat params and step results as a retained store, and set its retention. Keep credentials and user input out of both, read sensitive inputs outside any step, and set default_retention to match the shortest window you have promised anywhere else.
3. Give “did not run” its own recorded state. A background job that fails quietly leaves the last good result on display.