LLM agent SLA and deadline enforcement systems explained

What SLA deadline enforcement does

An SLA deadline enforcement system guarantees that a run respects a customer-visible latency contract from HTTP ingress through final token or terminal error — not just the LLM forward pass. It answers: how much time remains for this entire agent trajectory, and what should happen when the budget is exhausted?

Three layers often get conflated:

• Admission control — whether a run starts now or waits in a priority queue (queue wait is part of SLA unless excluded explicitly).
• Per-step timeouts — max duration for one model call or tool HTTP request.
• End-to-end deadline — wall-clock budget for the whole run, decremented by every step including queue wait, retrieval, tools, and subagent delegation.

Without layer three, per-step timeouts sum to far more than the marketed SLA. Three tools at 12 seconds each can consume 36 seconds even when each step “passed” its local timeout check.

Tiered SLO profiles

Not every run deserves the same budget. Production systems define SLO profiles attached at ingress from API key, route, or signed client metadata:

Each profile specifies: total deadline, whether queue wait counts, minimum reserve for final synthesis, and which degradation ladder applies at 50%, 80%, and 100% budget consumption. Profiles must align with tenant plan tiers so enterprise keys cannot inherit free-tier budgets by mistake.

Deadline tokens and propagation

At admission, the gateway mints a deadline token: { run_id, deadline_unix_ms, profile, remaining_ms }. Every downstream hop — model router, RAG retriever, tool executor, subagent worker — receives the token and must:

1. Read remaining_ms before starting work.
2. Set local timeout to min(step_default, remaining_ms - reserve).
3. Return unused budget or charge actual elapsed time back to the parent.
4. Refuse new tool calls when remaining_ms < tool_floor (e.g. 800 ms is not enough for a CRM round-trip).

Subagents spawned via delegation receive a sub-budget carved from the parent, never the full parent clock. Parent synthesis reserves typically 20–35% of the total budget so the orchestrator can still draft an answer after tools return or fail.

Propagation must be cooperative: hard kills mid-tool without cleanup orphan CRM writes. Pair deadlines with cancel tokens from the cancellation FSM so tools abort in-flight HTTP and roll back partial side effects where possible.

Soft vs hard cutoffs

Soft cutoff (warning threshold, often 70–85% budget): stop scheduling new tools, skip optional retrieval, switch to a smaller model tier, or begin streaming a partial answer with a visible “still checking…” footer. Users perceive progress; SLA clock may still be running but perceived latency improves.

Hard cutoff (100% budget): cancel pending tools, finalize with best-effort context, return structured DEADLINE_EXCEEDED with whatever tokens were already streamed. Never leave the client hanging until TCP idle timeout.

For streaming delivery, soft cutoff should flush buffered tokens immediately; hard cutoff sends a terminal SSE event with finish_reason: deadline and links to async continuation if the profile allows webhook completion.

Degradation ladder

A practical ladder for interactive profiles:

• 50% budget — drop low-value tools (sentiment scoring, duplicate search); cap RAG chunks to top-3.
• 70% budget — route to faster model; disable reflection loops from self-critique pipelines.
• 85% budget (soft) — stream draft answer from current context; mark tools as skipped in the trace.
• 100% budget (hard) — stop all side effects; append honest limitation text; offer async follow-up for P1/P2.

Ladders should be deterministic per profile version, logged in the audit trail, and covered by golden tests so marketing SLAs match runtime behavior after prompt or tool registry changes.

Measuring SLA correctly

Harbor Desk's false 94% compliance came from measuring the wrong interval. Correct metrics:

• SLA start — request received at edge (or job admitted for async), not when worker picks up the run.
• SLA end — last user-visible token or terminal error frame, not last model log line.
• Attribution — slice misses by queue wait, retrieval, tool name, model tier, and subagent depth.
• Exclusions — document client disconnects and user-initiated cancels separately; do not count them as passes.

Emit OpenTelemetry spans with sla.profile, sla.budget_ms, sla.remaining_at_complete, and sla.degradation_step so SRE can distinguish “need more GPUs” from “need shorter tool chains.”

Harbor Desk refactor

Root causes beyond the single-interval metric:

1. No parent deadline — tools used static 12 s timeouts.
2. Sequential tool fan-out — planner always called three CRM tools even when the first returned enough context.
3. No streaming until final — users waited for the full chain before seeing text.
4. Queue wait excluded — P0 bursts added 4–9 s invisible delay.

Shipped fixes:

• Edge gateway mints deadline tokens; queue wait debits budget on admit.
• Tool planner receives remaining_ms and selects a minimal tool set; parallel fan-out when budget allows.
• Soft cutoff at 6 s streams draft reply; hard cutoff at 8 s for P0.
• Async P1 webhook path for runs that hit hard cutoff with continuation_token.
• Dashboard rebuilt on end-to-end span; model-only chart retained as diagnostic, not SLA source of truth.

Customer-measured p95 latency dropped from 34 s to 7.2 s. SLA miss rate fell from 51% to 6.4%. Partial streaming cut perceived wait by 40% even on runs that eventually used the full budget.

Technique decision table

Common pitfalls

• SLA measured on model only — tools and queue wait hidden; false green dashboards.
• Sum of per-step timeouts > SLA — each step passes locally while the user waits 30+ seconds.
• No synthesis reserve — tools consume 100% budget; orchestrator has zero ms to draft.
• Hard kill without cancel propagation — orphaned writes and duplicate retries.
• Same budget for all tenants — enterprise SLA on free-tier clock.
• Degradation ladders not in CI — prompt deploy re-enables expensive tools silently.
• Ignoring client RTT — mobile regions need edge streaming earlier than datacenter lab tests suggest.

Production checklist

• Define SLO profiles with total budget, queue-wait policy, and reserves.
• Mint deadline tokens at ingress; propagate to every tool and subagent.
• Set each step timeout to min(default, remaining - reserve).
• Implement soft and hard cutoffs with streaming partial results.
• Log degradation step and remaining budget on every run completion.
• Measure SLA on end-to-end user-visible latency, not model latency alone.
• Align profiles with tenant plan tiers and signed API metadata.
• Pair hard cutoff with cooperative cancellation and idempotent tools.
• Offer async continuation for profiles that allow webhook completion.
• Alert when p95 remaining budget at tool start < 20% for P0.
• Load-test chained tools under P0 budget, not isolated model calls.

Key takeaways

• SLA enforcement is an end-to-end wall-clock contract, not a per-model timeout.
• Deadline tokens propagate remaining budget across tools, retrieval, and subagents.
• Soft cutoffs improve perceived latency; hard cutoffs prevent hung clients.
• Admission control and rate limits do not replace deadline enforcement.
• Harbor Desk cut SLA misses from 51% to 6.4% by measuring and enforcing the full trajectory.

Related reading

• LLM agent run priority queue explained — P0–P3 lanes, admission gates, and queue wait as SLA input
• LLM agent cancellation and timeout explained — cooperative shutdown, run FSMs, and tool cleanup
• LLM agent streaming delivery explained — SSE token deltas and terminal deadline events
• LLM agent observability and tracing explained — spans, latency attribution, and SLA dashboards