Verifying a change no test can reach

TL;DR. Measured against the live index, reranking fixed the search on this site: durable objects, a phrase in ordinary prose on five of its pages, went from no results to the right ones, and hyperdrive, a word on none of them, went from five results to none. Reranking was switched on, and the page did not change, because the code turned it off again on every request. Four separate checks said the change was in place, and none of them looked at what a person typing into the search box gets back. This post walks through what each one confirmed, and ends on the checks I now run instead.

The measurement worked

The search on this site runs on a managed retrieval service. It was returning bad answers, so it was measured: ten queries against the live index, with reranking off and then on, checked against grep over the indexed pages as ground truth. The result was not close.

Without reranking, every score landed between 0.40 and 0.56, and a word that appears nowhere outscored the one correct page for turnstile. No threshold could separate the two. Reranking changes that, so I decided to turn it on. The instance configuration was diffed before and after to confirm the write, and the diff was one line: reranking: false became reranking: true.

Later that evening, durable objects on the live page still returned nothing.

The request overruled the instance

The retrieval call in the handler passes its own options on every request, and one of them was reranking: { enabled: false }, written when the decision had been to leave it off. For that option the request value wins over the instance value. The setting was applied and recorded, it had no effect, and nothing reported a conflict between the two.

It is not a general rule. On the same call, the result cap went the other way: the request asked for 20, the instance carried 10, and the instance won. The instance was raised to match, so the two levels no longer disagree, but the precedence has to be found out one option at a time.

A write can be confirmed by a command that reports the old value

The instance change was verified with a before-and-after diff of wrangler ai-search get --json. It showed the one line that changed, which is what a good verification looks like.

The same check was run again after raising the result cap in the dashboard. It reported the old value and the old modified_at, twice, and the write looked like it had failed. It had not. wrangler ai-search list --json reported the new value and the new timestamp on the first call.

# disagrees with the instance for some period after a write
wrangler ai-search get NAME --json | jq '{max_num_results, modified_at}'

# agreed immediately, on every call
wrangler ai-search list --json | jq '.[] | {id, max_num_results, modified_at}'

get is the command whose job is to report what the instance carries, and for a field written moments earlier it reports what the instance used to carry. The earlier diff worked because the field it checked had already propagated, and nothing about the method guaranteed that.

A cache key needs a version

The search results are cached, keyed on the query text alone, for a day. A query searched before a configuration change keeps serving its old answer until the entry expires, because nothing in the key says which index or which configuration produced it. The fix is to version the key, and to bump that version when the content of the answer changes, not only when its shape does.

The reason first written down for that fix was a query that kept its old results after the flip. The cache could not have caused that: with the request override in place, a completely cold query would have answered the same way. The symptom was the override, and the fix is right for the reason above.

The guard read the wrong field name

There is no test for any of this. The harness stubs the retrieval binding, because no test in this repository is allowed to reach a paid inference service. So the guards are structural: they read the handler’s own source and assert on the shape of the request it builds.

One of them asserted that the source did not contain reranking_model, to prove that no specific model had been pinned. reranking_model is the field name on the instance configuration. The per-request field is model, nested inside the reranking object, so reranking: { enabled: true, model: '...' } satisfies the assertion and is the exact thing it exists to prevent. The helper behind it also read the whole file with comments stripped, although its documentation said it read only the one call.

A code review caught both. The assertions now match the whole object literal, over a slice of the source containing just the call, and they were watched failing against a deliberately broken call before anyone trusted them.

What each check confirmed

The instance said reranking: true. That is a read of the setting.

The CLI said the write had landed. That is another read of the setting, through a different command.

The cache explanation fit the symptom. That is a read of a model of the system, not of the system.

The test said no model was pinned. That is a read of the source text, against the wrong field name.

None of them asked what a person typing into the search box would get back. That takes one query, and the answer was durable objects returning nothing.

The checks are not worthless. The instance read is how you find configuration drift, and the source assertion is how you stop a regression from shipping. Both catch things a live query cannot. What they have in common is that each confirms a cause is in place, and none shows that the effect happened. Most of the time, on most systems, those are the same thing. They come apart for retrieval, ranking and model output, where correctness is a property of the running system rather than of a file, and no test here reaches any of them. So the rule is to check the surface, not the setting.

What the surface says now

The change deployed and the page was queried first. durable objects returns five results led by the case study that carries the phrase, at 0.9711. hyperdrive returns nothing. turnstile returns the single page containing the word, alone, at 0.7368. The scores run between 0.49 and 0.97, where the same four queries used to run between 0.40 and 0.49, which is the shape a reranked answer has and an unreranked one does not.

turnstile is the row that shows the cache fix earning its place. It was the query run most often while measuring, so it was the one most certain to have an old entry under the old key. Without the version bump it would have answered 0.4099 out of the cache, on the first query anybody would think to check.

For a change to a managed service that no test can reach, these are the checks I run now:

1. Run the live query after every configuration change, against the public surface a person uses, and compare it with the answer the change was meant to produce.
2. Confirm a write with list, not get, or with whichever read the service keeps current.
3. Look for per-request options that override the instance, and find out option by option which level wins.
4. Version the cache key on content, not only on shape, and bump it with any change to what the answer should be.
5. Assert on the per-request field name, and break the call once to watch the guard fail before trusting it.