06 · Testing Strategy

Test runner facts

All tests use tsx directly — there is no vitest/jest runner despite vitest.config.ts existing
Each suite is a standalone test/continuous-test-suite-<name>.ts script; it logs pass/fail and exits with code 0/1
The orchestrator is test/continuous-test-suite.ts (run via pnpm test)
All tests read env vars; missing-credential is treated as skip (not fail) for provider tests. The nightly live matrix is the exception for a provider that HAS its secret: see "The live gate" below

Files to edit

A. `test/continuous-test-suite-providers.ts`

Updated 2026-08-15: the ALL_PROVIDERS array described below no longer exists — continuous-test-suite-providers.ts deleted it once provider coverage moved to two more targeted places:

Structural completeness (zero API keys, runs in CI on every commit): test/continuous-test-suite-provider-structure.ts (pnpm run test:provider-structure) asserts every value in the canonical AIProviderName enum resolves via ProviderFactory, and every src/lib/providers/*.ts module has exactly one dynamic import in providerRegistry.ts.
Live per-provider generate/stream sweep (needs API keys, runs nightly via .github/workflows/live-matrix.yml, not a PR gate): test/continuous-test-suite-provider-matrix.ts (pnpm run test:matrix) iterates test/helpers/providerMatrix.ts's PROVIDERS map — see that file's header comment for the current provider count and coverage gaps.

If you're adding a new provider, add it to providerMatrix.ts's PROVIDERS map so test:matrix picks it up automatically; test:provider-structure needs no edits — it derives its expectations from the AIProviderName enum and the filesystem, not a hand-maintained list.

The live gate. A provider that has its secret (or whose catalog JSON records a liveMatrix run) is expected to answer, so in test:matrix and test:matrix:cli a credential error (401, 403, invalid key), an account error (402, billing, a tier gate) or a rate limit (429) fails its cell instead of skipping. The status is read from the error object (statusCode, then its cause chain) because the SDK often drops it from the message. A known-broken provider is excused only by a dated entry in LIVE_GATE_ALLOWLIST (test/helpers/liveGate.ts): each entry has a reason and an expiry at most 14 days after it was added, and an expired entry excuses nothing. A provider with no secret and no recorded run is skipped in one quiet line without calling the SDK. The nightly sweep prints a coverage table and fails when fewer than MIN_PASSED_PROVIDERS providers passed (LIVE_MATRIX_ENFORCE_FLOOR=1, set by the workflow). The gate throws HardFail (test/helpers/harness.ts), which test() never downgrades: the failure message necessarily contains "401" or "rate limit", which isExpectedProviderError() would turn into a skip. pnpm run test:live-gate shows all of this against a local server that answers 200, 401, 402, 429 and 503.

B. `test/continuous-test-suite-credentials.ts`

For each new provider, add a per-call credential-override test block. Use the existing Mistral block as the template (search for "mistral" in the file):

// DeepSeek credential override
await runTest("deepseek per-call apiKey override", async () => {
  if (!process.env.DEEPSEEK_API_KEY) {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} DEEPSEEK_API_KEY not set`,
    );
    return null; // skipped scenarios MUST return null so the runner records SKIP
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with the word PONG and nothing else." },
    provider: "deepseek",
    credentials: { deepseek: { apiKey: process.env.DEEPSEEK_API_KEY } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

// NVIDIA NIM credential override
await runTest("nvidia-nim per-call apiKey override", async () => {
  if (!process.env.NVIDIA_NIM_API_KEY) {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} NVIDIA_NIM_API_KEY not set`,
    );
    return null;
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with PONG only." },
    provider: "nvidia-nim",
    credentials: { nvidiaNim: { apiKey: process.env.NVIDIA_NIM_API_KEY } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

// LM Studio credential override (baseURL only)
await runTest("lm-studio per-call baseURL override", async () => {
  const url = process.env.LM_STUDIO_BASE_URL || "http://localhost:1234/v1";
  // Skip if server not reachable
  try {
    const r = await fetch(`${url.replace(/\/$/, "")}/models`);
    if (!r.ok) throw new Error("not ok");
  } catch {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} LM Studio not running at ${url}`,
    );
    return null; // SKIP scenarios MUST return null so the runner records SKIP
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with PONG only." },
    provider: "lm-studio",
    credentials: { lmStudio: { baseURL: url } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

// llama.cpp credential override
await runTest("llamacpp per-call baseURL override", async () => {
  const url = process.env.LLAMACPP_BASE_URL || "http://localhost:8080/v1";
  try {
    const r = await fetch(`${url.replace(/\/$/, "")}/models`);
    if (!r.ok) throw new Error("not ok");
  } catch {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} llama-server not running at ${url}`,
    );
    return null;
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with PONG only." },
    provider: "llamacpp",
    credentials: { llamacpp: { baseURL: url } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

C. `test/continuous-test-suite.ts` (orchestrator)

Likely no changes needed — it invokes per-domain suites which are already wired.

D. `package.json`

The canonical entrypoint for the four new providers is pnpm run test:new-providers, which runs the dedicated suite test/continuous-test-suite-new-providers.ts (full feature surface per provider — generate, stream, tools, structured, reasoning, vision-where-supported, abort, timeout, per-call creds, telemetry, error formatting). The existing test:providers and test:credentials are still useful for cross-provider checks but the new suite is the primary coverage for the integration.

NVIDIA NIM-specific test (the only one that needs custom assertions)

NIM has unique behavior (extra-body params, retry-on-400). Add a focused test inside test/continuous-test-suite-providers.ts:

await runTest("nvidia-nim retry strips reasoning_budget on 400", async () => {
  if (!process.env.NVIDIA_NIM_API_KEY) {
    console.log(`  [SKIP] NIM not configured`);
    return;
  }
  const nl = new NeuroLink();
  // Use a model known NOT to support reasoning_budget — should still succeed via retry
  const result = await nl.generate({
    input: { text: "Hi" },
    provider: "nvidia-nim",
    model: "google/gemma-3-27b-it", // doesn't support reasoning_budget
    thinkingLevel: "high", // forces our code to add the field
    maxTokens: 32,
  });
  assert(typeof result?.text === "string", "should succeed via retry");
});

Smoke test scripts

Add (optional) to test/:

# test/test-deepseek.sh
#!/usr/bin/env bash
set -euo pipefail
[ -z "${DEEPSEEK_API_KEY:-}" ] && { echo "DEEPSEEK_API_KEY not set"; exit 1; }
pnpm run cli generate "Reply: PONG" --provider deepseek

# test/test-nvidia-nim.sh
#!/usr/bin/env bash
set -euo pipefail
[ -z "${NVIDIA_NIM_API_KEY:-}" ] && { echo "NVIDIA_NIM_API_KEY not set"; exit 1; }
pnpm run cli generate "Reply: PONG" --provider nvidia-nim
pnpm run cli generate "Solve 17!" --provider nvidia-nim --model nvidia/llama-3.3-nemotron-super-49b-v1 --thinking-level high

# test/test-lm-studio.sh
#!/usr/bin/env bash
set -euo pipefail
echo "Make sure LM Studio is running with a model loaded"
pnpm run cli generate "Reply: PONG" --provider lm-studio

# test/test-llamacpp.sh
#!/usr/bin/env bash
set -euo pipefail
echo "Make sure ./llama-server is running on :8080"
pnpm run cli generate "Reply: PONG" --provider llamacpp

Validation pipeline

After implementing each provider:

pnpm run check                     # type check
pnpm run lint                      # ESLint (rules 1-13)
pnpm run build                     # full build
pnpm run test:providers            # all-provider loop (skips unconfigured)
pnpm run test:credentials          # per-call credential overrides

For the all-provider loop to actually exercise the new provider (vs skip), set the relevant env vars in your local .env. Local providers (LM Studio, llama.cpp) need their servers running; cloud providers (DeepSeek, NVIDIA NIM) need API keys.

CI considerations

Cloud-provider tests with real API calls cost money. Two options:

Skip in CI by default — current pattern. Tests only run if env vars are set; CI can set them as secrets for nightly runs.
Mock the AI SDK — adds complexity; not recommended for v1.

Local-provider tests are fine in CI ONLY if the runner has the local server installed and pre-loaded. For now, expect CI to skip LM Studio and llama.cpp tests.

Manual matrix (run before merging)

Provider	Test
DeepSeek	`--provider deepseek` text gen, then `--model deepseek-reasoner` to verify reasoning
NVIDIA NIM	`--provider nvidia-nim` default model, then `--thinking-level high` on a Nemotron model, then a model that does NOT support reasoning_budget (verify retry)
LM Studio	Start server with a small Llama model, run text gen + tool gen ("write me a 3-line poem")
LM Studio	Stop server, run again — verify error message is the friendly "Open LM Studio app..." form
llama.cpp	Start server with `--jinja`, run text gen and tool gen
llama.cpp	Stop server, run again — verify error message instructs `./llama-server -m ...`
All	Run `pnpm run test:credentials` — all four per-call override tests should pass or skip cleanly

Test runner facts​

Files to edit​

A. test/continuous-test-suite-providers.ts​

B. test/continuous-test-suite-credentials.ts​

C. test/continuous-test-suite.ts (orchestrator)​

D. package.json​

NVIDIA NIM-specific test (the only one that needs custom assertions)​

Smoke test scripts​

Validation pipeline​

CI considerations​

Manual matrix (run before merging)​