Introduction
Learn what TheSpider is and why the brain lives server-side
TheSpider helps you audit a codebase for security vulnerabilities and bugs using the coding-agent / LLM CLIs you already run locally, while a central service handles config, prompts, parsing, dedupe, triage, and collaboration.
Setup in one prompt: tell your AI agent
Get me started with TheSpider: https://thespider.xyz/startup.md and it runs the whole setup for you — see the AI Setup Wizard. These docs are AI-native throughout: an MCP server (/mcp) and automatic llms.txt generation, compatible with AI tools out of the box.What is TheSpider?
TheSpider is a multi-tenant security & bug audit platform built around a strict split:
- A thin Rust agent (
thespider-agent) runs on the machine that has the source tree. It claims audit slices, runs your model CLIs locally via a POSIX shell, and POSTs the raw model stdout back. - A Bun brain-server owns everything else: orgs, projects, RBAC, the versioned methodology corpus, prompt rendering, parsing, fingerprinting, dedupe (heuristics + AI adjudication), triage, reporting, billing, and the web UI.
The differentiator versus cloud scanners: your source code stays on your machine; only the findings the model chose to report — with the evidence it chose to cite — sync to the server.
thespider-agent (Rust) HTTPS / bearer server (Bun + Hono)
• enroll + token (keychain) ── claim / results ──▶ • orgs / projects / RBAC
• model → command bindings ◀── rendered prompts ── • Postgres + object storage
• git detect + local models • Nuxt SPA (same origin)
source stays local never sees the source tree
What you can do
- Audit a whole repo — kick off a full campaign; the brain renders prompts, the agent runs your models slice by slice, and findings are parsed, deduped, and triaged centrally.
- Scan a pull request — a forge webhook creates an incremental campaign scoped to the changed paths; the runner daemon runs it and CI polls a cheap gate.
- Triage as a team — multi-user web UI, project-level finding identity that carries across branches and merges, statuses with actor attribution.
- Run on-prem — the same image runs single-tenant inside your network. Bring your own Postgres, S3-compatible storage, and model keys.
When to use TheSpider
Use TheSpider when you need to:
- Keep source code private. A cloud scanner can't see your tree; TheSpider's agent never sends it.
- Use your own model credentials. Your API keys and CLI auth are used by the local runner and never leave the host.
- Collaborate on audit data across a team and across branches. Project-level finding identity + branch attribution means "show only NEW issues on this PR" is first-class.
- Gate merges without putting model runtime or secrets into CI.
- Run fully air-gapped. The on-prem image validates its license against embedded public keys; no phone-home required.
The two deployment modes
| SaaS (multi-tenant) | On-prem (single-tenant) | |
|---|---|---|
| Server | thespider.xyz, run by us | customer-run Docker Compose (bundled Postgres or BYO) |
| Orgs | many | exactly one, created at first boot |
| Entitlements | subscription + plan limits | claims in the signed license JWT |
| Object storage | our S3/R2 | your S3-compatible creds (MinIO works) |
| AI dedup key | platform key | your key/endpoint (env) |
| Client config | server = "https://thespider.xyz" | server = "https://spider.corp.example" |
The agent is mode-agnostic — the server URL in config.toml is its only coupling. There is no second client code path. See Deployment modes.
Where to go next
- AI Setup Wizard — let your AI agent do the entire setup from one prompt.
- Installation — prerequisites and how to boot the server + build the agent.
- Quickstart — run your first audit end-to-end in minutes.
- Data handling — exactly what crosses the boundary and how secrets are protected.