I asked for a confidence score from 0 to 1. The model returned 10.

I spent 30 days building four AI-native DevOps services — a log analyser, a RAG copilot over our runbooks, a tool-calling agent that diagnoses Kubernetes alerts, and a security triage service that grades scanner output. Four services, four ports, four bearer tokens. On the last day I put one door in front of them.

The idea was simple enough that I wrote it down on Day 22 and didn't think about it again: POST /ask, you send a plain-English question, a model works out which of the four can answer it, and the answer comes back attributed to whichever one produced it. A router. Day 2's structured-output lesson and Week 3's tool-selection lesson, applied one layer up.

Then I read the four request models before writing any code, and all three of my example questions died.

Only one of my four services can answer a sentence

Here is what they actually take:

"Why did checkout start 500ing at 3am" routes to the log analyser perfectly well, and then there is nothing to send it. It reads log text it is handed. It cannot fetch, search or tail anything. "Is this Dockerfile safe to ship" is the triage service's question, and triage reasons about scanner output rather than running scanners.

One of my four services answers a bare question. The other three need material that no amount of English contains.

The tempting fix is to have the router fill in the body — extract an alert dict from the prose, synthesise a plausible log line. I didn't build that, and refusing to is the entire design. The self-healing agent has write tools behind that alert field. A router that invents evidence and hands it to a thing that acts on evidence is the exact failure I'd spent four weeks building approval gates and blast-radius limits to prevent.

So the router never fabricates a body. It names a service, and it says what you still have to attach.

Two ways to refuse, and one of them can't be wrong

That gave me two refusals with different owners, which turned out to be the part of this I'm happiest with.

"I don't know which of these you want"    <- the model's call
"I know, but you didn't send the data"    <- a dict lookup

The first is a judgment. The model emits service: "none", and there's a confidence floor behind it as a backstop.

The second is if backend.needs and not request.attachment. It never calls a model. It never guesses. It is right every single time. Pushing a decision out of the model and into a lookup was available here and it wasn't my first idea — I had to go looking for it after the plan fell apart.

The schema is the guard. Half of it wasn't.

I wrote that line in the service README with some confidence: the schema is the guard, not the validator. The service field is a JSON-schema enum built from the backend table, so the model cannot name a fifth service — that's structurally impossible, not validated away afterwards. Across four eval runs of 24 questions each, service never once came back as anything but one of the five legal values.

I put confidence next to it as float = Field(ge=0.0, le=1.0) and assumed the same thing was happening.

It wasn't. The first small model I tested returned this, four times:

{ "reason": "The question asks for an analysis of an Alertmanager alert...",
  "service": "self-healing-agent", "confidence": 10 }

Grammar-constrained decoding enforces structure. The sampler physically cannot emit a token outside an enum. A numeric range is a different kind of claim — the grammar permits any number and Pydantic rejects the value after generation, which is a 502 rather than a guard. Two constraints in the same Field(...) call and only one of them load-bearing.

My first reaction was that this disqualified a bad model, and that keeping the 502 was honest. Then the model I actually ship did it too, once in 24. So it was never a bad-model story. It was about 4% of production requests failing for a reason unrelated to routing quality, on a field that — I'd checked by then — had changed no outcome at all.

I found the same bug a third time while testing something else. reason had max_length=200, and a live call came back with a 199-character reason. One character. That one now truncates instead of rejecting, because reason is prose for a human to read and a failed route is a much worse outcome than a clipped sentence.

Changing the type changed what the model said

confidence became Literal["low", "medium", "high"]. Structural, like service. I expected that to stop the 502s and nothing else.

With the float, the model answered 1.00 or 0.00 across 48 graded routes and nothing in between. Every 0.00 came attached to service: "none", so the confidence floor I'd built never changed a single outcome. Zero times. It was a boolean wearing a float's clothing.

With three named levels, the same model on the same 24 questions used all three: 15 high, 2 medium, 7 low.

Nothing had ever stopped it returning 0.6. Asking for a number got me a number it treated as a switch. Asking for one of three named judgements got me a judgement.

I'll be honest about the limits of that. It was one design decision but two edits — the schema field and the prompt sentence describing it, which went from "how sure you are" to "high if the question plainly belongs there, medium if it probably does, low if you are guessing". Which half did the work, I don't know. And it's one run.

The bias was mine

Six misses in the first eval run, and three of them went to the same service. knowledge-copilot got picked 8 times out of 24 and was wrong on 3.

The model's stated reason for the worst of them — a bare "something is wrong with checkout" that should have been declined outright — was two words:

operational question

That's a quotation. The description I'd written for that service opened "Answers operational questions from the team's own runbooks". The broadest phrase in the whole catalogue, sitting in the entry for the one backend that needs no attachment, and closing with "Needs nothing but the question." I wrote a default and then recorded surprise that the model chose it.

I narrowed that one description and changed nothing else. It now says the service "knows nothing whatever about the running system" and ends by pushing back: "If the answer is not already written in a runbook, this is the wrong service."

knowledge-copilot: 8 picks / 3 wrong, down to 5 picks / 0 wrong. Score 18/24 → 20/24. Then the confidence change took it to 22/24 and the copilot bias stayed fixed, which is what makes it a fix rather than a good run.

One variable per run, so each step is attributable. That discipline came from the triage service, where I'd already learned the hard way that changing two things means learning nothing.

I should say the score movements are each +2, and I'd measured the noise band at ±1–2 before any of this. So treat 18 → 20 → 22 as suggestive. The 3 → 0 on a single service's misses is the part that isn't ambiguous.

What deployment taught me in about four hours

The gateway went live behind nginx and TLS and answered correctly the first time — answered, attributed to knowledge-copilot POST /ask-runbook, with the copilot itself declining: "Not covered in the runbooks." Two refusals stacked properly. That's the whole design working.

It took 192 seconds.

The same call, run again immediately, took 9. I had set the timeout to 120, which sits exactly between those two numbers — so the first call after any idle period was a guaranteed 504 and every call after it was fine.

120 came from a principle. "This is the only model call a human waits on synchronously, so it should be short." That's true. I never checked it against a cold start. Worse: my eval has a warm-up call that I added for precisely this reason, so I'd fixed the symptom in the measurement tool and left it in production.

The gap is model load, and it isn't a fixed cost. Here is the machine hosting Ollama, after I'd set a 30-minute keep-alive and confirmed it applied:

Mem:  15Gi total   10Gi used   764Mi free   5.1Gi available
Swap: 8.0Gi total  2.7Gi used
ollama ps  ->  empty, immediately after a successful call

One laptop, four services, three distinct models. A 7B at 4-bit is about 4.7GB against 5.1GB available with 2.7GB already swapped out. It can just about load and it cannot stay, because keep-alive can't hold memory the kernel is reclaiming.

So 300 seconds stops the error and doesn't fix anything. Nobody waits three minutes.

The thing I actually got wrong

POST /ask returns the answer inline. That's the shape the RAG copilot uses, and the copilot answers in seconds.

This endpoint is 9 seconds warm and 190 cold. That's the triage service's profile — and triage returns 202 plus a poll URL, for a reason written in a docstring I had read: "a run is one model call per batch, minutes each, so a synchronous endpoint would time out on every real request and be retried."

The right pattern was already in my own repo and I copied from the wrong sibling.

It has a consequence past latency, too. Cloudflare fronts the subdomain and caps how long an origin can take, so a cold /ask comes back as error code: 524 at 125 seconds no matter what nginx and the gateway allow. I measured that one the slow way.

Where it stands

86 tests on the gateway, 598 across the repo, five services deployed. /ask routes a plain-English question to whichever of four services can answer it, tells you when it can't place the question, and tells you when it can but you haven't given it the material. It's live, it's correct, and its first call after lunch takes three minutes.

The gap I keep thinking about is a different one. All five of these services feed attacker-influenced text to a model — scanner output, log lines, alert annotations — and the gateway made it worse by design, because /ask puts 200 characters of caller-supplied attachment straight into the routing prompt. Nothing in this repo defends against prompt injection yet.

That's next, and it's a bigger piece than a gateway.

All five services are in crypticani/autonomous-infra-labs. The gateway is services/gateway — its README carries the eval numbers, the nginx block, and the deploy notes in more detail than I've put here. It runs at aiops.crypticani.dev, behind a bearer token.