Author an audit methodology

Each project has separate Security, Bugs, Hygiene and Pipeline methodology histories. You pick a profile, then author a version. Versions are immutable. Models are enabled per profile (Settings → Models, one switch per profile).

You need admin or owner. New Cloud projects already ship a starter methodology, so you only do this when you want your own coverage.

The matrix

Work is buckets × passes × enabled models, where a model enabled at two efforts counts twice (participants).

• A bucket is a review area (authentication, data integrity).
• A pass is a perspective with a path scope (HTTP surface, persistence).
• A category is an optional label a bucket can include.

Start with two or three buckets and two focused passes. A bigger matrix means more model calls, often with the same evidence twice.

Author a version

1. Open the project, pick a profile (Security, Bugs, Hygiene or Pipeline), then Config.
2. Author new version.
3. Replace config_json with something like:

{
  "name": "Web API review",
  "buckets": [
    {
      "code": "auth",
      "name": "Authentication and authorization",
      "description": "Identity, session, and permission boundaries",
      "categories": ["authn", "authz"]
    },
    {
      "code": "data",
      "name": "Data integrity",
      "description": "Validation and persistence",
      "categories": ["validation"]
    }
  ],
  "categories": [
    { "code": "authn", "name": "Authentication" },
    { "code": "authz", "name": "Authorization" },
    { "code": "validation", "name": "Input validation" }
  ],
  "passes": [
    {
      "code": "http",
      "name": "HTTP surface",
      "description": "Trace untrusted requests through auth and validation",
      "scope": "src/http src/routes"
    },
    {
      "code": "storage",
      "name": "Persistence",
      "description": "Queries, transactions, tenant boundaries",
      "scope": "src/db migrations/**/*.sql"
    }
  ]
}

Every bucket and pass needs a code and name. Every pass needs a scope. Category codes a bucket lists must exist in categories.

Scopes

A scope is a space-separated list of paths or globs.

• src/http matches files under src/http/.
• * matches across /, so src/*.ts also matches src/sub/app.ts.

Incremental campaigns only run passes whose scope hits a changed path. Changes to lockfiles, Dockerfiles, .env*, and similar expand to every pass.

Prompt sections

{
  "global.md": "Audit only the bucket and pass in this slice. Report concrete, evidence-backed defects with exact file paths.",
  "severity_rubric.md": "Rate security findings as critical, high, medium, low, or info. In bug mode, interpret the same values as impact.",
  "output_schema.md": "Emit a JSON array between <THESPIDER_FINDINGS_JSON> and </THESPIDER_FINDINGS_JSON>."
}

Keep that marker in sync with the project's prompt marker. The server stitches these into every slice prompt.

Evidence basis (optional field)

A Security output schema can ask models for an optional evidenceBasis on each finding: "demonstrated" when the cited evidence shows a concrete exploit or failure path in the code, or "speculative" for a hardening observation without one (for example a dependency pinned to a pre-release version with no known vulnerability). Models should omit it when unsure. The bundled Security methodology (version 2) asks for it:

{
  "severity_rubric.md": "… low (limited impact, or a defense-in-depth gap with a concrete path), info (informational). Severity rates impact if the issue is real; it is not a confidence scale. A hardening observation with no demonstrated exploit or failure path … still gets a severity, and is marked evidenceBasis \"speculative\".",
  "output_schema.md": "… Each object: { title, severity, confidence, categories[], files[{path,line}], evidence, exploitScenario, impact, recommendation, cweCodes[]?, evidenceBasis? }. … evidenceBasis is \"demonstrated\" when the cited evidence shows a concrete exploit or failure path in this code, or \"speculative\" for a hardening observation without one; omit it when unsure."
}

The field is optional everywhere: a methodology that does not mention it keeps working, and findings are then classified only when their own text says they are speculative. A finding counts as a hardening note only when every supporting report is speculative; the triage list can hide hardening notes, but they still count toward the PR gate.

Existing projects keep the version they pinned. Authoring a new version does not pick up the version 2 wording on its own: the editor starts from your current version's prompt sections, so a new version keeps the old severity_rubric.md and output_schema.md text until you replace those two sections with the wording above. New projects start on version 2. On-prem, applying the image's bundled methodology as an update installs version 2 as a new config version.

Category codes matter beyond the prompt: the server uses the models' category tags to place a finding in a bucket, and moves it out of its discovery bucket only when every report agrees on another one. A category code listed in two buckets cannot decide a bucket, so prefer each code in exactly one.

Check it

1. Create version. Confirm the Config page shows it.
2. Settings: enable at least one model.
3. Bounded run:

thespider-agent run --profile security --once --max-slices 1

Open Slices. If the runner produced output but no reports, see troubleshooting.

A campaign stays pinned to the version it started with. Findings and triage stay project-global. Cadence: campaign cadence.