Runner troubleshooting

Use the project Slices view first. Filter by failed status or search the error text, then open a slice to inspect the runner error, import error, duration, exit code, report count, and artifact metadata.

409 prompt_hash mismatch

Symptom: The agent prints rejected (prompt_hash mismatch).

Cause: The prompt_hash echoed with the result does not match the exact rendered prompt bytes served for that lease. A proxy, custom host, or runner changed the prompt after claim.

Fix:

1. Discard the modified prompt and lease payload.
2. Claim the slice again.
3. Pass the temporary {prompt_path} to the model without rewriting it.
4. If you implement an external host, hash the served UTF-8 prompt bytes before execution and compare them with prompt_hash.

Do not retry the rejected result with a fabricated hash.

413 output too large

Symptom: The agent prints rejected (output too large).

Cause: stdout or stderr exceeded the server's per-artifact limit. The default MAX_ARTIFACT_BYTES is 10 MiB.

Fix:

1. Configure the model to emit concise findings rather than transcripts.
2. Keep progress output out of stdout when the runner supports a separate diagnostic channel.
3. Run the first-party agent with --redact. It masks secret-shaped strings and caps stdout at 1 MiB with an explicit truncation marker.
4. Claim and run the slice again. A rejected oversized payload is not imported.

The server does not support presigned result uploads.

A lease expires while the model is running

Symptom: A running slice returns to claimable state, or two workers run the same slice.

Cause: The lease reached its expiry time. The first-party agent requests a lease based on the largest configured timeout and sends best-effort heartbeats every 15–60 seconds while the process runs. Network loss can prevent heartbeats.

Fix:

1. Set timeout_seconds high enough for the model binding.
2. Keep the agent online when possible so heartbeat requests can extend the lease.
3. Let the original worker submit its result. A late result is accepted if no other lease completed the slice.
4. If another lease already won, the server returns 202 and stores the late output as a superseded provenance artifact without importing it twice.

Lease expiry requeues work; it does not automatically invalidate the original worker.

The daemon cannot clone or fetch

Symptom: thespider-agent daemon logs a Git clone, fetch, or checkout failure before it claims slices.

Cause: The daemon shells out to the local git command. It does not manage forge credentials.

Fix:

1. Run git ls-remote REPOSITORY_URL as the daemon's operating-system user.
2. For SSH URLs, load the correct key into that user's ssh-agent and trust the forge host key.
3. For HTTPS URLs, configure a credential helper or a token-bearing URL appropriate for your environment.
4. Confirm the campaign has both target_remote_url and target_commit.
5. Restart the daemon after Git authentication succeeds.

The managed checkout lives below ~/.thespider/checkouts unless --workdir overrides it.

A process exits successfully but imports no reports

Symptom: The slice is completed, stdout exists, and imported_reports is zero.

Cause: The parser did not find a complete marker-delimited JSON array, the array was invalid JSON, or every item failed report validation.

Fix:

1. Open the slice and note its output artifact key for operator correlation.
2. Run the command manually with the same prompt.
3. Confirm both opening and closing marker tags exist.
4. Confirm the first non-whitespace character after the opening marker is [.
5. Validate the array as JSON.
6. Compare each object with the runner contract.

The OpenCode runner returns early or keeps a session

Symptom: The runner produces empty output, times out, or reports that it kept an OpenCode session.

Cause: The capture did not observe a complete result block before OPENCODE_CAPTURE_TIMEOUT, or the OpenCode stream failed.

Fix:

1. Use the recovery command printed to stderr. It reads assistant messages from the retained session.
2. Increase OPENCODE_CAPTURE_TIMEOUT while keeping it below the model binding timeout.
3. Confirm occtl can reach the configured OpenCode host and port.
4. Confirm the assistant output includes a complete marker block, not only prose that mentions the marker.

Network, rate-limit, and server failures

The first-party agent retries network failures, 429, and 5xx responses with exponential backoff and jitter. It does not treat 409 or 413 as transient. If retries continue, check /healthz, token validity, project scope, and the server logs.