---
title: "Check and evaluate"
description: "Lint, 61 review rules, live CLI probes, eval cases and the proof script every pack ships, with the exit codes and score that CI and review read."
canonical: https://harnesscontextprotocol.io/docs/packs/check
---

# Check and evaluate

> Lint, 61 review rules, live CLI probes, eval cases and the proof script every pack ships, with the exit codes and score that CI and review read.

`hcpx lint` proves a pack's three files exist and agree. `hcpx check` asks
whether an agent will pick the right command, fill its arguments, read its
output and be stopped before a risky write, with 61 rules that write down a
careful maintainer's review. Run it with `--probe` after every edit until no
error is left, then ship the [proof script](#the-proof-script) in CI.

```bash
hcpx check ./packs/gh                    # static rules: structure, manifest, Policy, Skill
hcpx check ./packs/gh --probe            # also run the CLI the way an agent does
hcpx check ./packs/gh --probe --agent    # the same, as a fix list for a coding agent
hcpx check ./packs/gh --init-evals       # write a starter hcp.eval.json
hcpx eval  ./packs/gh                    # replay eval cases through the runtime only
```

It exits `0` with nothing to fix, `2` on warnings only and `1` on any error.
Notes (`info`) never fail, and `--strict` fails warnings too, for CI.
`--timeout <ms>` limits each probe and eval case (default 10000), and
`--no-eval` skips the eval layer.

## The four layers

| Layer | Runs | Needs |
|-------|------|-------|
| Structure | [Lint](#lint): three files, Policy coverage, Skill drift | Nothing |
| Static | Quality rules over `surface.json`, `policy.json` and `HARNESS.md` | Nothing |
| Probe | The wrapped CLI with stdin closed, `NO_COLOR=1`, `TERM=dumb` and a timeout: `--help`, an unknown subcommand, `<bin> <subcommand> --help` per command, and every read it can call | `--probe`; the CLI on `PATH` or in the pack |
| Eval | [Eval cases](#eval-cases) through the reference runtime with grants, the confirm gate and the sandbox | An eval file and a pack that passes the structure layer |

Probes never run a write. For a pack with `wrap.workspace.mode: project`,
probes and eval cases run in the directory you started `hcpx check` from, so you
can check against a real project.

## Findings and score

Each finding has a stable id, a severity, the file and JSON pointer or heading,
what is wrong and a fix to apply as is.

```text
error  R014             surface.json /verbs/list/cli
       "gh list" is not a command of this CLI (unknown command "list" for "gh").
       fix: Point "list".cli at a subcommand that exists (see "gh --help"), or remove
            "list" from surface.json, policy.json, and HARNESS.md.
```

The score starts at 100. Each error costs 12 points and each warning 4, any
error caps the grade at D, and a pack is ready to share at 90 or more with no
errors. `--json` returns the full report: `score`, `grade`, per-category scores,
every finding with `fix` and `docs`, the probe records (exit code, milliseconds,
approximate tokens, whether stdout was JSON) and eval results with p50 and p95
latency.

## The coding-agent loop

`--agent` prints Markdown for a coding agent: errors, then warnings, each with a
"Do:" line and a link to the reason, then the eval table and the loop
instruction.

```md
# hcpx check: gh

Score 8/100, grade F. 7 error(s), 2 warning(s).

## Fix now (errors)

1. **surface.json `/verbs`** (M018): 6 command(s) still carry the scaffold template …
   Do: Replace the template commands with 5 to 12 real commands from "gh --help", …
```

The [`/wrap.md`](/wrap.md) runbook scaffolds with `hcpx wrap`, runs
`hcpx check --probe --agent`, fixes errors and then warnings, and stops at 90 or
more with no errors. Agents must never weaken an eval expectation to make a case
pass.

## Eval cases

`hcp.eval.json` sits next to the three pack files. Each case names a command
(`verb`), its `args` and what to `expect`.

```json title="packs/gh/hcp.eval.json"
{
  "cases": [
    { "name": "repo.view returns JSON", "verb": "repo.view",
      "args": { "repo": "cli/cli" },
      "expect": { "ok": true, "json": true, "contains": ["cli/cli"], "max_ms": 8000 } },
    { "name": "issue.create is refused without confirm", "verb": "issue.create",
      "args": { "repo": "cli/cli", "title": "probe", "body": "never created" },
      "expect": { "ok": false, "error": "confirm required" } }
  ]
}
```

| `expect` field | Passes when |
|----------------|-------------|
| `ok` | The runtime result's `ok` matches. A case without `expect` expects `ok: true` |
| `error` | The error message contains this text |
| `contains` / `not_contains` | Output includes / excludes each string |
| `json` | A successful call's output parsed as JSON |
| `max_ms` | The call finished within this many milliseconds |
| `max_tokens` | Output stayed under this many tokens (about four characters each) |

Cases run on the reference runtime's `hcpx exec` pipeline as an agent Principal
with every grant, in sandbox mode, with default access settings and the confirm
gate. A confirm-gated write is refused unless the case passes `confirm: true` in
its `args`, so a refusal case and a confirmed run can sit side by side. Output
is the command's stdout, or its JSON result when stdout is empty.

A case fails without running while an argument still starts with `TODO`, and
fails when it runs past the timeout. Results record duration and approximate
tokens, and the report adds the pass count and p50 and p95 latency. A failing
case is finding E005: fix the command, or the case if its expectation is wrong.
The layer runs whenever the file exists and the pack passes the structure layer,
and `hcpx eval` runs it alone, one line per case. The first case that expects
success also supplies a read's arguments to the probes, so probes reach reads
that need arguments.

`--init-evals` writes `hcp.eval.json` unless one exists, with one case per read
expecting `{ "ok": true, "json": true, "max_ms": 3000, "max_tokens": 5000 }` and
one refusal case per confirm-gated write expecting
`{ "ok": false, "error": "confirm required" }`; writes without `confirm` get no
case. Required arguments come from the manifest's first example or stay
`TODO <name>` until you replace them with real values from the CLI.

## Testing

A pack proves itself with a script ([L12](/docs/introduction/design-principles))
after lint passes, and one that passes its proof script against a conforming
runtime is a conforming pack ([Specification: Conformance](/docs/specification/conformance)).
The protocol's own vectors live in the SDK, use `transit` and `ledger` as
fixtures and test the runtime ([Runtime conformance](/docs/runtimes/conformance#vectors)).

### Lint

```bash
hcpx lint ./packs/ledger --json
```

```json
{ "ok": true, "pack_id": "ledger", "issues": [] }
```

A pack fails lint on any of these, and each message names its fix. In
`hcpx check` they appear with ids `L-<kind>`.

| Kind | Message |
|------|---------|
| `missing_harness` | `HCP pack "x" missing HARNESS.md skill` |
| `missing_policy` | `HCP pack "x" missing policy.json` |
| `missing_verb` | `verb "transfer" missing from policy.json` |
| `extra_verb` | `policy verb "refund" not in pack manifest` (remove the row or add the command) |
| `missing_stimulus` | `stimulus "new_item" missing from policy.json` |
| `extra_stimulus` | `policy stimulus "old" not in stimulusTriggers` (remove the row or declare the ref) |
| `write_stimulus` | `policy stimulus "new_item" is a write; a trigger announces a change and is always "read"` |
| `missing_verb` (id) | `policy surface_id "ledgr" !== "ledger"` |
| `wrap` | `wrap requires named commands or pass_through` |
| `binding` | `wrap adapter requires wrap.bin` |
| `binding` | `control plane packs default Binding mothership` (fix the mode or the plane) |
| `invalid_meta` | `openapi adapter requires openapi ref` · `sense_only adapter requires at least one stimulus trigger_ref` · schema errors |

In code:

```ts
import { loadPackFromDir, lintLoadedPack } from '@harnesscontextprotocol/sdk'

const pack = await loadPackFromDir('./packs/ledger')
const issues = lintLoadedPack(pack)   // [] when clean
```

### The proof script

Every pack ships one, and CI runs it against a mock or paper mode of the bin.

```bash title="proof.sh"
set -e
hcpx lint ./packs/ledger --json
hcpx install ./packs/ledger --json
hcpx activate ledger --json

# 1. one read
hcpx exec ledger list --json

# 2. gated write without confirm → live deny
hcpx exec ledger transfer --json -- amount=2.80 payee=demo-transit | grep -q '"confirm required"'

# 3. the same write with confirm, in mock/paper mode
hcpx exec ledger transfer --json -- amount=2.80 payee=demo-transit confirm=true

# 4. availability shows the four flags
hcpx availability ledger --json

# 5. if Sense is declared: announce with grant on, deny with grant off
#    (runtime-specific harness; see Sense delivery)
```

| Step | Proves |
|------|--------|
| lint | Policy coverage, Skill present, wrap shape |
| one read | Adapter path works; sandbox allow-list is right |
| deny without confirm | The gate is enforced before the adapter |
| write with confirm | The happy path, without spending |
| availability | installed · activated · connected · executable, with reasons |
| Sense on/off | Isolation; `sense.enable` is respected |

Steps 1 to 3 can also live in `hcp.eval.json` as [eval cases](#eval-cases).

### The twin-face check

Both faces must return the same result for the same arguments.

```bash
transit plan A B --json                              > cli.json
hcpx exec transit plan --json -- from=A to=B | jq .result > exec.json
diff cli.json exec.json
```

A mismatch means a `cli[]` template is wrong or the bin parses arguments
differently from what the manifest claims.

### Grant deny

A read with no grants must be denied, naming the missing capability.

```ts
const hcp = await Hcp.create({ packDirs: ['./packs/ledger'], grants: [] })
await hcp.activate(['ledger'])
const res = await hcp.exec('ledger', 'list')
// { ok: false, error: 'denied: missing verb.exec.read for ledger.list' }
```

### Sandbox deny

A wrap whose bin is not allow-listed must fail closed instead of falling back
to `PATH`.

```json
{ "ok": false, "error": "sandbox: bin \"ledger\" not allowlisted" }
```

## In CI

```yaml title=".github/workflows/pack.yml"
- run: npx @harnesscontextprotocol/cli check ./packs/gh --probe --strict --json > check.json
```

```yaml title=".github/workflows/pack.yml"
- run: bun install
- run: bun ./packages/cli/src/hcp.ts lint ./packs/ledger --json
- run: bun ./packages/cli/src/hcp.ts eval ./packs/ledger
- run: ./packs/ledger/proof.sh
```

Set `HCP_HOME` to a temp directory so CI never touches a real store, and keep
the `check` JSON report as a build artifact for reviewers.

## Rule catalogue

The rules encode HCP's [design laws](/docs/introduction/design-principles) and
Gate 0 for wrapped CLIs, the [Command Line Interface Guidelines](https://clig.dev)
(exit codes, stdout for data and stderr for messages, no prompts without a TTY,
`NO_COLOR`) and guidance on
[writing tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents)
(unambiguous parameter names, high-signal output, a 25,000-token budget,
actionable errors, evaluation against realistic tasks).

### Manifest

| Id | Severity | Rule |
|----|----------|------|
| M001 | warn, error above 12 | 5 to 12 commands |
| M002 | error | Command names are lower case with dot-separated families |
| M003 | error | No catch-all command (`run`, `exec`, `shell`, …) |
| M004 | error | Every command has a description |
| M005 | warn | The description names input and result, not "Run." |
| M006 | info | Descriptions stay one sentence |
| M007 | warn | Wrap commands declare `cli[]` |
| M008 | warn | `cli[0]` equals `cliBin` |
| M010 | error | No placeholder that forwards arbitrary arguments (`{{argv}}`) |
| M011 | warn | Argument names are unambiguous (`issue_number`, not `x`) |
| M012 | error | No `pass_through` left in a finished pack |
| M013 | error | A Binding mode is declared |
| M014 | warn | Public Binding uses a public hub |
| M015 | error | No credential in the manifest |
| M016 | warn | `wrap.bin` is not a host-absolute path |
| M018 | error | No commands left over from the `hcpx init` / `hcpx wrap` template |
| M019 | error | `wrap.bin` does not point outside the pack; install copies only the pack |
| M020 | info | Every argument has a description and an example |
| M021 | warn | No `TODO` left in a description or argument example |
| M022 | info | Argument descriptions say where the value comes from, not just its name |
| M023 | warn | A CLI on `PATH` runs in the project (`wrap.workspace.mode: project`), not the pack directory |
| M024 | warn | A declared `preview` uses a known kind and only the command's own arguments |
| M025 | info | A confirm command declares a `preview`; the fix is the suggested JSON |
| M026 | warn | A declared [`title`](/docs/concepts/command#titles) uses only the command's arguments, puts `{{result.…}}` only in `done`, and ends each form with a template without placeholders |

### Policy

| Id | Severity | Rule |
|----|----------|------|
| P001 | warn, error for risky names | A command named like a write is not `read` |
| P002 | error | Commands that delete, send, pay, publish, or deploy carry `confirm: true` |
| P003 | error | `confirm` only on writes |
| P004 | warn | A confirm-gated command also passes the CLI's own confirmation flag |
| P005 | warn | `connect` and `setup.*` changes are writes |
| P006 | warn | A pack that writes has a `doctor` or `status` command |
| P007 | info | Commands that pay or transfer use `confirm: "always"`, so access mode `run` still asks |

### Skill

| Id | Severity | Rule |
|----|----------|------|
| S001 | warn | Title is `# HCP — <Name>` |
| S002 | error | A lead follows the title |
| S003 | warn | The lead fits the ~480 characters hosts load |
| S004 | warn | The lead says whether login is needed |
| S005 | error | A hot-path table lists the commands |
| S006 | warn | Examples use `--json` |
| S007 | warn | An anti-patterns section exists |
| S008 | info | 3 to 6 anti-patterns |
| S009 | warn | The Skill fits on one screen |
| S010 | warn | Packs with credentials have a Setup section |
| S011 | info | Triggers are mentioned only when declared |

### Probe

| Id | Severity | Rule |
|----|----------|------|
| R001 | error | `wrap.bin` resolves (in the pack or on `PATH`) |
| R002 | warn | `--help` exits 0 with output |
| R003 | error | An unknown subcommand exits non-zero |
| R004 | warn | Failures explain themselves on stderr or as JSON |
| R005 | error | Reads finish with stdin closed (no prompts, no hangs) |
| R006 | error | JSON mode prints only JSON on stdout |
| R007 | warn | No ANSI colour when stdout is not a TTY or `NO_COLOR` is set |
| R008 | warn | Output stays under ~25,000 tokens |
| R009 | info | Output over ~5,000 tokens suggests a concise default |
| R010 | info | Reads over 3 seconds |
| R011 | warn | Health commands exit 0, 2, or 1 |
| R012 | info | At least one read could be probed |
| R013 | error | Read commands succeed |
| R014 | error | Every command's subcommand exists in the CLI |

### Eval

| Id | Severity | Rule |
|----|----------|------|
| E001 | warn | An eval file exists |
| E002 | error | Cases name real commands |
| E003 | warn | Every confirm-gated write has a refusal case |
| E004 | warn | At least one read succeeds |
| E005 | error | Every case passes |

## Related

- [Best practices](/docs/packs/best-practices)
- [Errors](/docs/reference/errors)

## Sitemap

- [Documentation](https://harnesscontextprotocol.io/docs)
- [Packs](https://harnesscontextprotocol.io/packs)
- [Glossary](https://harnesscontextprotocol.io/docs/reference/glossary)
- [llms.txt](https://harnesscontextprotocol.io/llms.txt)
- [llms-full.txt](https://harnesscontextprotocol.io/llms-full.txt)
- [OpenAPI](https://harnesscontextprotocol.io/openapi.json)
- [MCP](https://harnesscontextprotocol.io/mcp)