abx black-box E2E suite

A language-neutral, black-box end-to-end suite for the abx CLI. It spawns the binary under test exactly like a user would (argv, environment, stdin, exit code, stdout/stderr, files on disk) against the real cloud. Point ABX_BIN at the abx binary you want to test.

中文说明见 README.zh-CN.md.

Quick start

# 1. credentials: environment variables or the gitignored cli/.env (never committed, never printed)
#    ABX_SANDBOX_API_KEY, ABX_SANDBOX_BASE_URL, ABX_SANDBOX_DOMAIN, ALICLOUD_ACCESS_KEY_ID/ALICLOUD_ACCESS_KEY_SECRET, ACR_NAMESPACE, ABX_LLM_API_KEY, ABX_LLM_BASE_URL
cp .env.example .env   # then fill in real values

# 2. run against the abx binary
ABX_BIN=abx                           npm run e2e          # `abx` on PATH (default)
ABX_BIN="node dist/bin/abx.js"        npm run e2e          # after `npm run build`

npm run e2e:offline                   # only the no-credential suite (seconds)
node e2e/run.mjs --suite cloud-lifecycle --filter 'exec|files'
node e2e/run.mjs --list               # all suites and scenarios
node e2e/run.mjs --json out.json --markdown out.md   # machine/markdown reports (secrets masked)
npm run e2e:cleanup                   # dry-run janitor for sandboxes leaked by an interrupted run

Requires Node >= 20. No dependencies are installed by the suite itself.

Guarantees

Suites

Scenario format

Scenarios are plain objects in scenarios/*.mjs (table style). Common fields:

{ id: 'exec-exit-code-propagates', title: '...', requires: ['create-json'],   // skipped unless that scenario passed
  argv: (ctx) => ['exec', ctx.state.id, 'echo out; exit 3'],                   // or a plain array
  env: { CI: 'true' }, stdin: 'n\n', creds: false, freshHome: true,
  exit: 3,                                                                     // number | [numbers] | 'nonzero'
  stdout: ['text', /regex/, { not: 'forbidden' }], stderr: [...], output: [...],   // stdout+stderr
  jsonShape: { ID: 'string', Status: 'string' }, json: (j, ctx, res) => {...},     // JSON assertions
  retries: 1, retryReason: 'cold start', timeoutMs: 180000, needs: ['docker'] }

Multi-step scenarios use run(ctx) with ctx.abx(argv, opts) (masked + normalised result), ctx.rawAbx (masked only, for checks that must see real bytes), ctx.track(id)/ctx.untrack(id) (register sandboxes for cleanup), ctx.eventually(fn), ctx.fail(msg).

Provider tiers

Every provider, including OpenSandbox, is tested in the same six tiers (T0 plugin unit tests through T5 the real Alibaba Cloud platform). The table, the rules, and the commands are in providers/PLAN.md (中文).

Provider plugins (e2e/providers/run.sh)

A bash script that installs the reference plugins into a throwaway HOME and drives real sandboxes through the CLI: npm run e2e:providers or e2e/providers/run.sh [offline|catalog|local-process|opensandbox|e2b-compatible]. local-process and e2b-compatible come from python-sdk/examples/providers. opensandbox is cloned from awesome-providers (opensandbox/).

Variables: ABX_BIN (binary under test, default tsx src/bin/abx.ts in a checkout), PROVIDERS_DIR (local-process and e2b-compatible only), OSB_SRC or ABX_OPENSANDBOX_PLUGIN or ABX_AWESOME_PROVIDERS (opensandbox checkout; otherwise the script clones the catalog), OSB_URL, OSB_KEY, OSB_IMAGE. The script scrubs the child environment, never prints credentials, kills every sandbox it created on exit, and exits non-zero on any failed step.

Environment variables

Results and baseline

BASELINE.md records the pass/fail/flakiness of every scenario against the Python oracle on the real cloud. When you port a command, run the suite against both binaries and compare with the baseline; a scenario that passes for the oracle but fails for the port is a parity bug (or a scenario that over-specifies, in which case fix the scenario in the same change).

The mock-vs-oracle differential harness lives in ../parity/diff/.

Exit code of run.mjs: 0 = no failures (skips allowed), 1 = at least one failure, 2 = binary not runnable.