Check and evaluate

hcpx lint proves a pack's three files exist and agree. hcpx check asks whether an agent will pick the right command, fill its arguments, read its output and be stopped before a risky write, with 61 rules that write down a careful maintainer's review. Run it with --probe after every edit until no error is left, then ship the proof script in CI.

hcpx check ./packs/gh                    # static rules: structure, manifest, Policy, Skill
hcpx check ./packs/gh --probe            # also run the CLI the way an agent does
hcpx check ./packs/gh --probe --agent    # the same, as a fix list for a coding agent
hcpx check ./packs/gh --init-evals       # write a starter hcp.eval.json
hcpx eval  ./packs/gh                    # replay eval cases through the runtime only

It exits 0 with nothing to fix, 2 on warnings only and 1 on any error. Notes (info) never fail, and --strict fails warnings too, for CI. --timeout <ms> limits each probe and eval case (default 10000), and --no-eval skips the eval layer.

The four layers#

Probes never run a write. For a pack with wrap.workspace.mode: project, probes and eval cases run in the directory you started hcpx check from, so you can check against a real project.

Findings and score#

Each finding has a stable id, a severity, the file and JSON pointer or heading, what is wrong and a fix to apply as is.

error  R014             surface.json /verbs/list/cli
       "gh list" is not a command of this CLI (unknown command "list" for "gh").
       fix: Point "list".cli at a subcommand that exists (see "gh --help"), or remove
            "list" from surface.json, policy.json, and HARNESS.md.

The score starts at 100. Each error costs 12 points and each warning 4, any error caps the grade at D, and a pack is ready to share at 90 or more with no errors. --json returns the full report: score, grade, per-category scores, every finding with fix and docs, the probe records (exit code, milliseconds, approximate tokens, whether stdout was JSON) and eval results with p50 and p95 latency.

The coding-agent loop#

--agent prints Markdown for a coding agent: errors, then warnings, each with a "Do:" line and a link to the reason, then the eval table and the loop instruction.

# hcpx check: gh

Score 8/100, grade F. 7 error(s), 2 warning(s).

## Fix now (errors)

1. **surface.json `/verbs`** (M018): 6 command(s) still carry the scaffold template …
   Do: Replace the template commands with 5 to 12 real commands from "gh --help", …

The /wrap.md runbook scaffolds with hcpx wrap, runs hcpx check --probe --agent, fixes errors and then warnings, and stops at 90 or more with no errors. Agents must never weaken an eval expectation to make a case pass.

Eval cases#

hcp.eval.json sits next to the three pack files. Each case names a command (verb), its args and what to expect.

{
  "cases": [
    { "name": "repo.view returns JSON", "verb": "repo.view",
      "args": { "repo": "cli/cli" },
      "expect": { "ok": true, "json": true, "contains": ["cli/cli"], "max_ms": 8000 } },
    { "name": "issue.create is refused without confirm", "verb": "issue.create",
      "args": { "repo": "cli/cli", "title": "probe", "body": "never created" },
      "expect": { "ok": false, "error": "confirm required" } }
  ]
}

Cases run on the reference runtime's hcpx exec pipeline as an agent Principal with every grant, in sandbox mode, with default access settings and the confirm gate. A confirm-gated write is refused unless the case passes confirm: true in its args, so a refusal case and a confirmed run can sit side by side. Output is the command's stdout, or its JSON result when stdout is empty.

A case fails without running while an argument still starts with TODO, and fails when it runs past the timeout. Results record duration and approximate tokens, and the report adds the pass count and p50 and p95 latency. A failing case is finding E005: fix the command, or the case if its expectation is wrong. The layer runs whenever the file exists and the pack passes the structure layer, and hcpx eval runs it alone, one line per case. The first case that expects success also supplies a read's arguments to the probes, so probes reach reads that need arguments.

--init-evals writes hcp.eval.json unless one exists, with one case per read expecting { "ok": true, "json": true, "max_ms": 3000, "max_tokens": 5000 } and one refusal case per confirm-gated write expecting { "ok": false, "error": "confirm required" }; writes without confirm get no case. Required arguments come from the manifest's first example or stay TODO <name> until you replace them with real values from the CLI.

Testing#

A pack proves itself with a script (L12) after lint passes, and one that passes its proof script against a conforming runtime is a conforming pack (Specification: Conformance). The protocol's own vectors live in the SDK, use transit and ledger as fixtures and test the runtime (Runtime conformance).

Lint#

hcpx lint ./packs/ledger --json

{ "ok": true, "pack_id": "ledger", "issues": [] }

A pack fails lint on any of these, and each message names its fix. In hcpx check they appear with ids L-<kind>.

In code:

import { loadPackFromDir, lintLoadedPack } from '@harnesscontextprotocol/sdk'

const pack = await loadPackFromDir('./packs/ledger')
const issues = lintLoadedPack(pack)   // [] when clean

The proof script#

Every pack ships one, and CI runs it against a mock or paper mode of the bin.

set -e
hcpx lint ./packs/ledger --json
hcpx install ./packs/ledger --json
hcpx activate ledger --json

# 1. one read
hcpx exec ledger list --json

# 2. gated write without confirm → live deny
hcpx exec ledger transfer --json -- amount=2.80 payee=demo-transit | grep -q '"confirm required"'

# 3. the same write with confirm, in mock/paper mode
hcpx exec ledger transfer --json -- amount=2.80 payee=demo-transit confirm=true

# 4. availability shows the four flags
hcpx availability ledger --json

# 5. if Sense is declared: announce with grant on, deny with grant off
#    (runtime-specific harness; see Sense delivery)

Steps 1 to 3 can also live in hcp.eval.json as eval cases.

The twin-face check#

Both faces must return the same result for the same arguments.

transit plan A B --json                              > cli.json
hcpx exec transit plan --json -- from=A to=B | jq .result > exec.json
diff cli.json exec.json

A mismatch means a cli[] template is wrong or the bin parses arguments differently from what the manifest claims.

Grant deny#

A read with no grants must be denied, naming the missing capability.

const hcp = await Hcp.create({ packDirs: ['./packs/ledger'], grants: [] })
await hcp.activate(['ledger'])
const res = await hcp.exec('ledger', 'list')
// { ok: false, error: 'denied: missing verb.exec.read for ledger.list' }

Sandbox deny#

A wrap whose bin is not allow-listed must fail closed instead of falling back to PATH.

{ "ok": false, "error": "sandbox: bin \"ledger\" not allowlisted" }

In CI#

- run: npx @harnesscontextprotocol/cli check ./packs/gh --probe --strict --json > check.json

- run: bun install
- run: bun ./packages/cli/src/hcp.ts lint ./packs/ledger --json
- run: bun ./packages/cli/src/hcp.ts eval ./packs/ledger
- run: ./packs/ledger/proof.sh

Set HCP_HOME to a temp directory so CI never touches a real store, and keep the check JSON report as a build artifact for reviewers.

Rule catalogue#

The rules encode HCP's design laws and Gate 0 for wrapped CLIs, the Command Line Interface Guidelines (exit codes, stdout for data and stderr for messages, no prompts without a TTY, NO_COLOR) and guidance on writing tools for agents (unambiguous parameter names, high-signal output, a 25,000-token budget, actionable errors, evaluation against realistic tasks).

Manifest#

Policy#

Skill#

Probe#

Eval#

Related#

• Best practices
• Errors