06 · Testing Strategy

Test runner facts

• All tests use tsx directly — there is no vitest/jest runner despite vitest.config.ts existing
• Each suite is a standalone test/continuous-test-suite-<name>.ts script; it logs pass/fail and exits with code 0/1
• The orchestrator is test/continuous-test-suite.ts (run via pnpm test)
• All tests read env vars; missing-credential is treated as skip (not fail) for provider tests

Files to edit

A. test/continuous-test-suite-providers.ts

Updated 2026-08-15: the ALL_PROVIDERS array described below no longer exists — continuous-test-suite-providers.ts deleted it once provider coverage moved to two more targeted places:

1. Structural completeness (zero API keys, runs in CI on every commit): test/continuous-test-suite-provider-structure.ts (pnpm run test:provider-structure) asserts every value in the canonical AIProviderName enum resolves via ProviderFactory, and every src/lib/providers/*.ts module has exactly one dynamic import in providerRegistry.ts.
2. Live per-provider generate/stream sweep (needs API keys, runs nightly via .github/workflows/live-matrix.yml, not a PR gate): test/continuous-test-suite-provider-matrix.ts (pnpm run test:matrix) iterates test/helpers/providerMatrix.ts's PROVIDERS map — see that file's header comment for the current provider count and coverage gaps.

If you're adding a new provider, add it to providerMatrix.ts's PROVIDERS map so test:matrix picks it up automatically; test:provider-structure needs no edits — it derives its expectations from the AIProviderName enum and the filesystem, not a hand-maintained list.

B. test/continuous-test-suite-credentials.ts

For each new provider, add a per-call credential-override test block. Use the existing Mistral block as the template (search for "mistral" in the file):

// DeepSeek credential override
await runTest("deepseek per-call apiKey override", async () => {
  if (!process.env.DEEPSEEK_API_KEY) {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} DEEPSEEK_API_KEY not set`,
    );
    return null; // skipped scenarios MUST return null so the runner records SKIP
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with the word PONG and nothing else." },
    provider: "deepseek",
    credentials: { deepseek: { apiKey: process.env.DEEPSEEK_API_KEY } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

// NVIDIA NIM credential override
await runTest("nvidia-nim per-call apiKey override", async () => {
  if (!process.env.NVIDIA_NIM_API_KEY) {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} NVIDIA_NIM_API_KEY not set`,
    );
    return null;
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with PONG only." },
    provider: "nvidia-nim",
    credentials: { nvidiaNim: { apiKey: process.env.NVIDIA_NIM_API_KEY } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

// LM Studio credential override (baseURL only)
await runTest("lm-studio per-call baseURL override", async () => {
  const url = process.env.LM_STUDIO_BASE_URL || "http://localhost:1234/v1";
  // Skip if server not reachable
  try {
    const r = await fetch(`${url.replace(/\/$/, "")}/models`);
    if (!r.ok) throw new Error("not ok");
  } catch {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} LM Studio not running at ${url}`,
    );
    return null; // SKIP scenarios MUST return null so the runner records SKIP
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with PONG only." },
    provider: "lm-studio",
    credentials: { lmStudio: { baseURL: url } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

// llama.cpp credential override
await runTest("llamacpp per-call baseURL override", async () => {
  const url = process.env.LLAMACPP_BASE_URL || "http://localhost:8080/v1";
  try {
    const r = await fetch(`${url.replace(/\/$/, "")}/models`);
    if (!r.ok) throw new Error("not ok");
  } catch {
    console.log(
      `  ${colors.yellow}[SKIP]${colors.reset} llama-server not running at ${url}`,
    );
    return null;
  }
  const nl = new NeuroLink();
  const result = await nl.generate({
    input: { text: "Reply with PONG only." },
    provider: "llamacpp",
    credentials: { llamacpp: { baseURL: url } },
    maxTokens: 16,
  });
  assert(typeof result?.content === "string", "should return content");
});

C. test/continuous-test-suite.ts (orchestrator)

Likely no changes needed — it invokes per-domain suites which are already wired.

D. package.json

The canonical entrypoint for the four new providers is pnpm run test:new-providers, which runs the dedicated suite test/continuous-test-suite-new-providers.ts (full feature surface per provider — generate, stream, tools, structured, reasoning, vision-where-supported, abort, timeout, per-call creds, telemetry, error formatting). The existing test:providers and test:credentials are still useful for cross-provider checks but the new suite is the primary coverage for the integration.

NVIDIA NIM-specific test (the only one that needs custom assertions)

NIM has unique behavior (extra-body params, retry-on-400). Add a focused test inside test/continuous-test-suite-providers.ts:

await runTest("nvidia-nim retry strips reasoning_budget on 400", async () => {
  if (!process.env.NVIDIA_NIM_API_KEY) {
    console.log(`  [SKIP] NIM not configured`);
    return;
  }
  const nl = new NeuroLink();
  // Use a model known NOT to support reasoning_budget — should still succeed via retry
  const result = await nl.generate({
    input: { text: "Hi" },
    provider: "nvidia-nim",
    model: "google/gemma-3-27b-it", // doesn't support reasoning_budget
    thinkingLevel: "high", // forces our code to add the field
    maxTokens: 32,
  });
  assert(typeof result?.text === "string", "should succeed via retry");
});

Smoke test scripts

Add (optional) to test/:

# test/test-deepseek.sh
#!/usr/bin/env bash
set -euo pipefail
[ -z "${DEEPSEEK_API_KEY:-}" ] && { echo "DEEPSEEK_API_KEY not set"; exit 1; }
pnpm run cli generate "Reply: PONG" --provider deepseek

# test/test-nvidia-nim.sh
#!/usr/bin/env bash
set -euo pipefail
[ -z "${NVIDIA_NIM_API_KEY:-}" ] && { echo "NVIDIA_NIM_API_KEY not set"; exit 1; }
pnpm run cli generate "Reply: PONG" --provider nvidia-nim
pnpm run cli generate "Solve 17!" --provider nvidia-nim --model nvidia/llama-3.3-nemotron-super-49b-v1 --thinking-level high

# test/test-lm-studio.sh
#!/usr/bin/env bash
set -euo pipefail
echo "Make sure LM Studio is running with a model loaded"
pnpm run cli generate "Reply: PONG" --provider lm-studio

# test/test-llamacpp.sh
#!/usr/bin/env bash
set -euo pipefail
echo "Make sure ./llama-server is running on :8080"
pnpm run cli generate "Reply: PONG" --provider llamacpp

Validation pipeline

After implementing each provider:

pnpm run check                     # type check
pnpm run lint                      # ESLint (rules 1-13)
pnpm run build                     # full build
pnpm run test:providers            # all-provider loop (skips unconfigured)
pnpm run test:credentials          # per-call credential overrides

For the all-provider loop to actually exercise the new provider (vs skip), set the relevant env vars in your local .env. Local providers (LM Studio, llama.cpp) need their servers running; cloud providers (DeepSeek, NVIDIA NIM) need API keys.

CI considerations

Cloud-provider tests with real API calls cost money. Two options:

1. Skip in CI by default — current pattern. Tests only run if env vars are set; CI can set them as secrets for nightly runs.
2. Mock the AI SDK — adds complexity; not recommended for v1.

Local-provider tests are fine in CI ONLY if the runner has the local server installed and pre-loaded. For now, expect CI to skip LM Studio and llama.cpp tests.

Manual matrix (run before merging)

Provider	Test
DeepSeek	--provider deepseek text gen, then --model deepseek-reasoner to verify reasoning
NVIDIA NIM	--provider nvidia-nim default model, then --thinking-level high on a Nemotron model, then a model that does NOT support reasoning_budget (verify retry)
LM Studio	Start server with a small Llama model, run text gen + tool gen ("write me a 3-line poem")
LM Studio	Stop server, run again — verify error message is the friendly "Open LM Studio app..." form
llama.cpp	Start server with --jinja, run text gen and tool gen
llama.cpp	Stop server, run again — verify error message instructs ./llama-server -m ...
All	Run pnpm run test:credentials — all four per-call override tests should pass or skip cleanly