Runner troubleshooting

Start on the project's Slices view. Filter to failed, open the slice, and read the runner error, import error, exit code, and report count.

Setup says the project is not ready

A new Cloud project should not hit these. Setup (and the dashboard) seed all four profiles and enable xai/grok-4.6. A blocker means something was later removed, or the project predates that provisioning.

Run thespider-agent status from the checkout. It reports the same sentences as this table.

Readiness is server state. Local CLI / login problems show up separately in check-models.

Setup says opencode models timed out

error: `opencode models` timed out after 60.0s (default limit 60s); raise the limit with THESPIDER_OPENCODE_MODELS_TIMEOUT=<seconds>; cannot confirm the OpenCode model id `zhipuai/glm-5.3`; to bind it without the check, rerun with --skip-model-check; file and server state left unchanged

Setup checks an OpenCode model id against opencode models before writing anything, and that command can be slow when many providers are configured. Time it with time opencode models, then either:

• raise the limit: THESPIDER_OPENCODE_MODELS_TIMEOUT=180 thespider-agent setup ... (whole seconds, 1 to 3600), or
• skip the check when you know the id: thespider-agent setup ... --model-id zai-coding-plan/glm-5.3 --skip-model-check.

check-models reports the same timeout on the binding's line and exits 1. It reads the same variable.

A model code is refused

When a model is enabled (setup --model, models enable, or Settings → Models), the server checks its code against the server's copy of the models.dev catalog. A refusal is HTTP 400 with the error code invalid_model_code, and setup prints it as error: HTTP 400: <message>. The message says which case you hit. An instance whose catalog is empty (never synced, or sync turned off) checks only the <lab>/<model> shape.

A reseller's listing. models.dev lists the model under a provider that serves it. A code names the lab that made the model, never the provider that serves it:

model code "azure/gpt-5.6-sol" is azure's listing of openai/gpt-5.6-sol on models.dev, not a model code. Codes name the lab that made the model, so enable openai/gpt-5.6-sol; your agent can still run it through azure.

Enable the code it names. To keep running the model through that provider, pass the provider's id with --model-id (details). When the listed model can't be audited with either, the second sentence says so and gives the reason:

model code "azure/gpt-4o" is azure's listing of openai/gpt-4o on models.dev, not a model code. Codes name the lab that made the model, and TheSpider can't audit with openai/gpt-4o either: it isn't a reasoning model.

Pick another model.

An alias that fails a check. A lab's own alias is a code of its own, and it is checked on its own models.dev entry, which can differ from its target's:

model code "cohere/north-small-translate-09-2026" is a models.dev alias of cohere/north-small-translate-1-0, but TheSpider can't audit with it: its context window is under 200,000 tokens.

Pick another model. The target is checked on its own entry, so search for it before ruling it out.

A model that fails a check.

model code "openai/gpt-4o" is listed on models.dev, but TheSpider can't audit with it: it isn't a reasoning model.

Pick another model. The reason after the colon is one of these:

The first rule a model fails is the reason shown, and deprecation is checked last.

Not in the catalog.

model code "openai/gpt-5.6-sool" is not in the models.dev catalog — see https://models.dev for the full list

Check the spelling against models.dev. The code is <lab>/<model> with the model's id, not its display name. The server syncs the catalog every 12 hours, so a model models.dev added since the last sync is accepted after the next one. Admins see the last sync time under the search box in Settings → Models.

Setup refuses the code before calling the server

When this machine has a cached catalog, setup checks the code first and refuses the same cases with the same reasons, ending ; file and server state left unchanged. --refresh re-downloads the catalog before the check. An agent released before alias support refuses every alias this way, with is not in the models.dev catalog on this machine. Run thespider-agent update and try again.

run says "no claimable slices" but the campaign still has work

A slice belongs to exactly one model, and run can only claim the models this checkout binds in .thespider/config.toml. A remainder made entirely of a model you do not bind is invisible to the claim, so the run stops while the campaign still reports slices left.

The run prints one reason per selected model, then names the models the remainder does belong to:

no claimable slices for models [xai/grok-4.6]
  xai/grok-4.6: nothing claimable — 8 of 8 completed
  claimable work remains, but for other models: openai/gpt-5.6-sol (7)
  nothing in `.thespider/config.toml` binds openai/gpt-5.6-sol — add one with `thespider-agent setup --model <code>`, or let an agent that already binds it run
run summary: leases=0 accepted=0 reports=0 failed=0
campaign #3: 17 of 24 completed; 7 remain (running)
  nothing this checkout can run will drain the remaining 7: 7 claimable by models this checkout cannot run: openai/gpt-5.6-sol (7, not bound)

Each selected model gets exactly one reason:

When the remainder belongs to other models, the lines after it say whether this checkout can run them: a model with no [[model]] binding (bind it, or let an agent that binds it run), a binding whose command is not on PATH (install that agent CLI), or a binding not enabled for the profile. Under --models, it only suggests widening the selection toward models this checkout can actually run.

A bounded run (--once, --max-slices) prints a continue: line only when the command it names can claim something: the selected models' claimable count, or another model runnable here (named with --models). When nothing this checkout can run will drain the remainder, there is no continue: line — rerunning the same command would claim nothing. An older server that does not report the per-model split still gets a continue: line for the whole remainder, with a note that some of it may belong to models this checkout cannot run.

thespider-agent progress (or the campaign: block of status) shows the full per-model table — total, claimable, leased, completed, failed, skipped, and whether each model is runnable here. To stop a model nobody will run from holding the campaign open, disable it:

thespider-agent models disable xai/grok-4.6 --profile security --dry-run   # see the slices it would skip
thespider-agent models disable xai/grok-4.6 --profile security --yes

--profile limits the change to the campaign's profile; without it the model is disabled for every profile of the project and its local [[model]] block is removed. The hints run, status and progress print always carry the campaign's --profile. Slices on a live lease are left to finish. The skipped slices no longer count as pending, so the campaign completes and wait and the gate return a verdict (degraded, exit 4, because that coverage was never audited). See Turn a model off.

The run also totals the selection's slices held by a live lease and those that have spent their attempt budget (repair gives them a fresh one). An expired lease is never "held": with attempts left its slice is claimable again, and on its last attempt it counts as failed.

The remaining slices are for another effort

A slice belongs to one model at one effort (a participant). When a project enables anthropic/claude-opus-5-5 at xhigh, those slices wait for a binding of that model at xhigh. A binding of the same code with no effort, or at low, never claims them, and the agent does not fall back to another effort. The pair stays pending, and the campaign stays open.

run, status and progress name such a pair as anthropic/claude-opus-5-5@xhigh. The fix is one of:

• bind it: thespider-agent setup --model anthropic/claude-opus-5-5 --agent claude --effort xhigh, which adds a second [[model]] block beside the one you have;
• or stop planning it: thespider-agent models disable anthropic/claude-opus-5-5 --effort xhigh --profile security --yes (--dry-run first shows the slices it would skip), or switch the pair off in Settings → Models.

The campaign was cancelled

If the campaign was cancelled, run --campaign N, status and progress do not say "no claimable slices" and do not suggest repair, models enable or releasing leases. They print the cancellation instead:

campaign #18 was cancelled by Colin at 2026-10-14T12:00:00Z: runner environment was broken
  start fresh: `thespider-agent run --profile bugs --isolated --redact` creates a new campaign at the campaign's commit (its slice claims are billed again; a cancel refunds nothing)

A cancelled campaign is terminal. Nothing revives it: not repair (it prints this message, changes nothing and exits 4; the server refuses with 409 campaign_cancelled), not a late result, not enabling a model. Start fresh the way the campaign was created, with the same profile and options: a full campaign with thespider-agent run --profile <profile> at the same commit (the line above repeats the options of the run that printed it); an incremental (pull-request) campaign by pushing again or redelivering the webhook, because run creates a full audit instead; a retest with thespider-agent refine. Cancelled campaigns are never offered to daemon and never used as a gate baseline.

Findings that slices produced before the cancel stay in the project, and stay triageable.

The remaining slices are held by leases

When the selected models' slices are held, the run lists each lease on a current server:

  2 slice(s) are held by a live lease:
    slice #213  lease #90  anthropic/claude-sonnet-5-5  claimed 28m ago, last renewed 27m ago, expires in 32m  [this token]  attempt 1 of 3
      local log: no process with pid 48121 on this host
    slice #221  lease #94  anthropic/claude-sonnet-5-5  claimed 3m ago, last renewed 30s ago, expires in 4m  [another token]  attempt 1 of 3
  (a local pid observation does not show whether the runner is alive elsewhere — see #27)
  they return to the queue when the lease expires (unless it is the last attempt)
  if you know their runner is gone: thespider-agent lease release 90 --dry-run

• A recently renewed lease (last renewed 30s ago) has a runner that is still working. Wait.
• A lease claimed by an agent from this version on a current server lasts 5 minutes and frees itself soon after its runner dies. Wait for expires in.
• A lease that went quiet long ago and still expires much later (last renewed 27m ago, expires in 32m) was claimed by an older agent with an hour-long lease. If you know its runner is gone and the line says [this token], thespider-agent lease release 90 --dry-run shows whether the server would release it, and thespider-agent lease release 90 releases it.
• [another token] leases, and leases that were never renewed, free themselves at expiry.
• then fails (last attempt …): when that lease expires the slice is retired to failed; run thespider-agent repair --campaign <id> afterwards to give it a fresh attempt.

The local log: line is only what this machine can see about the writer's process id. A runner on another machine or in a container looks gone from here, and a reused pid can look alive, so it is never used to choose or allow a release. thespider-agent lease list shows every lease of the campaign. An older server cannot list leases; the run then says so, and the slices free themselves when their leases expire. See Leases.

409 prompt_hash mismatch

The result's hash does not match the prompt that was claimed. Something rewrote the prompt.

Discard it, claim again, and pass {prompt_path} through unchanged. Do not invent a hash and retry.

413 output too large

Stdout or stderr exceeded the server cap (default 10 MiB). Ask the model for concise findings, keep transcripts off stdout, and try --redact (caps stdout at 1 MiB). Then claim again. A rejected payload is not imported.

Lease expired while the model was running

On a current server the agent holds a 5-minute lease and renews it at least every 60 seconds, so the lease only runs out when the agent could not reach the server for about five minutes (network outage, a sleeping laptop, server downtime). The model's own timeout_seconds no longer matters for this. What happens next depends on what happened to the slice meanwhile:

• Nobody took it: an expired lease that nobody reclaimed is simply renewed by the next heartbeat that gets through, and the run carries on.
• Superseded: another runner claimed the slice, or another lease completed it. While the model is still running the agent stops it and submits nothing: slice #S (pass x bucket) -> <model>: lease #N lost after Xms (another lease holds the slice, or an older server ended it); model killed, results not submitted. The slice is worked by the other runner, so the cost is the duplicate model spend, never a duplicate import. If the model had already exited, the loss only ends the heartbeat: the result POST still goes out (a late result gets 202 and is stored, not imported twice, when someone else already completed the slice).
• Ended, nobody else has it (lease #N no longer holds its slice (released, retired or closed server-side), but no other lease holds it): the lease was released, retired on its last attempt, or closed by a result or a repair. The model keeps running and its late result is still accepted. The agent keeps trying to renew, so it still notices if someone claims the slice later.
• Last attempt: an expired lease on the slice's last attempt (3 by default) is retired to failed as soon as anything reads the campaign, even though a late result can still be accepted. Run thespider-agent repair --campaign <id> to give it a fresh attempt.

Each reclaim spends one of the slice's attempts. Keep the agent online. Against an older server the agent keeps the old hour-long lease, and a lost lease always stops the model.

Daemon cannot clone

The daemon shells out to git. It does not manage forge credentials.

1. git ls-remote REPOSITORY_URL as the daemon's OS user.
2. SSH: right key in ssh-agent, host key trusted.
3. HTTPS: credential helper (or a token URL you do not store in TheSpider).
4. Confirm the campaign has a remote URL and a commit.

Managed checkouts live under ~/.local/share/thespider/checkouts unless you pass --workdir.

Slice failed, or campaign coverage looks too high

A missing marker, unclosed marker, empty marker body, malformed JSON, a value that is not an array of objects, or a nonzero runner exit used to complete the slice. It no longer does: the server returns 422, the slice is failed and claimable again, and thespider-agent increments failed and exits 1.

A valid empty array ([]) is still successful coverage — that slice really found nothing.

Retries are bounded. A slice is offered at most three times (MAX_SLICE_ATTEMPTS); after that it stays failed but is left alone, so a permanently-broken binding cannot loop run or daemon forever on the same paid model call. The campaign then reaches a terminal status and the gate a verdict — degraded, never clean. repair gives such a slice a fresh budget once you have fixed the cause.

Historic campaigns that already stored those invalid completions stay wrong until you repair them:

thespider-agent repair --campaign ID --dry-run   # exits 1 when there is something to repair
thespider-agent repair --campaign ID

You no longer have to guess that there is something to repair: when a campaign holds invalid coverage, run says so on its way out and prints the exact command —

note: 6 slice(s) in campaign #33 produced invalid coverage (nonzero exit, missing marker, or unparseable output).
Inspect: `thespider-agent repair --dry-run --campaign 33` — requeue: `thespider-agent repair --campaign 33`

thespider-agent status prints the same count for the newest campaign with claimable work.

Then run again. --dry-run shows valid / invalid / unverified without changing anything, and exits 1 when invalid > 0 so it works as a CI check — under set -e, run the two commands separately or guard the first with || true. Valid empty-array slices are not requeued.

Repair also takes back what an invalid run produced: its reports are deleted, and a finding left with no report behind it goes too. A finding you have triaged, fixed or dismissed is never deleted — it is reported as stranded so you can decide.

OpenCode returns early or keeps a session

Usually a timeout or a stream that never closed the marker block.

Use the recovery command printed to stderr (it reads the retained session). Raise OPENCODE_CAPTURE_TIMEOUT but keep it below the model timeout. Confirm occtl can reach the OpenCode host.

A slice reported as parse_error: marker <…> not found together with runner_exit: exit_code=1 is not a lost marker — the CLI itself failed before it could emit one, and the marker is simply absent from a short error message. run now says which stream holds the cause and prints the exact command:

slice #36 (config-supply-chain x access-control) -> zhipuai/glm-5.3: exit 1 (2140ms) -> rejected (parse error)
  the model command exited nonzero, so its own error text — not the missing marker — is the cause: `thespider-agent log show 36 --lease 44 --stream both`

The local lease log keeps the complete stderr; the server stores only its last 4000 bytes. A two-second failure is the agent CLI refusing to start at all (bad arguments, a model id the provider does not serve, an expired login); a failure near the timeout is a context or rate limit.

Network, 429, 5xx

The first-party agent retries those with backoff. It does not retry 409 or 413. If retries keep going, check /healthz, the token, and that it is scoped to this project.

status says the token is not verified

thespider-agent status asks the project's server to accept the stored credential, and the token: line names exactly one of five states. Only two of them are fixed by logging in:

The line after it carries the underlying error, and the one after that the fix. Exit code: 0 only when verified. Agent CLI probes (harness: lines) are reported in every case, because they do not depend on the server. status --json carries the same state as auth.state; see Agent CLI contracts.

check-models or status says auth unknown

An agent CLI's auth probe failed in a way TheSpider cannot classify. It never blocks — but the probe's own reason is shown, with ANSI codes stripped and secret-shaped values masked, plus a cause: line when the output names a failure class or a file:

harness opencode: available; auth unknown: Error: Unexpected error — Unknown: FileSystem.open (/home/dev/.local/share/opencode/log/opencode.log) (probe: `opencode auth list`)
  cause: filesystem error at /home/dev/.local/share/opencode/log/opencode.log — the CLI could not use that path; make it writable for this user, or run where the CLI's data directory is writable

This one is OpenCode failing to open its own log file, typically inside a sandbox or container that mounts the home directory read-only. Run the printed probe command yourself to see the CLI's full output. check-models --json carries it as probe_error.{message,class,path}.

log list shows (project not recorded)

Lease logs written by an agent older than 0.5.0 have no project field in their meta.json, and no agent version has ever recorded a campaign id. What such a record does have is the server, the slice id and the prompt hash, and a slice belongs to exactly one project. So when another attempt of the same slice (same server, slice id and prompt hash) recorded a project (a retry, or a slice repair requeued), log list uses it and shows where it came from:

acme/payments-api (from lease #31)  slice #88 lease #12  xai/grok-4.6  p1 x b1  exit 0 · accepted · 3 reports · 61s   17.8 KB

Anything else keeps (project not recorded), and a footer on stderr says how many rows were resolved, which agent version wrote the rest, and which fields are missing. A newer lease of a different slice in the same campaign is deliberately not used: nothing in the old record ties it to that campaign. The prompt hash is part of the match because slice ids restart when a server's database is reset behind the same URL (a local dev server, an on-prem reinstall); the prompt text embeds the project's name, commit, pass and bucket, so an unrelated slice never matches. Unresolved rows stay visible under any project filter rather than being hidden (a resolved row is filtered by its project like any other); log clean --older-than-days N removes them when you no longer need them.