Skip to main content

NVIDIA NIM Provider Guide

Hundreds of optimised AI models on NVIDIA's GPU-accelerated inference platform — or your own self-hosted NIM deployment

Overview​

NVIDIA NIM (NVIDIA Inference Microservices) is a managed inference platform that hosts a large catalog of open-weight models — Meta Llama, DeepSeek, Mistral, Microsoft Phi, Google Gemma, and more — all GPU-optimised and served through an OpenAI-compatible API. You can also point NeuroLink at a self-hosted NIM cluster by overriding the base URL.

Key Facts​

  • Hosted base URL: https://integrate.api.nvidia.com/v1
  • Protocol: OpenAI-compatible (/v1/chat/completions)
  • Vision: Yes, on supported models (Llama 3.2 Vision, etc.)
  • Reasoning: Yes, model-dependent. The default openai/gpt-oss-20b is a reasoning model and returns its reasoning trace on every request; see Reasoning and thinkingLevel for what NeuroLink's thinkingLevel does on NIM
  • Streaming: Supported
  • Tool calling: Supported on most models
  • Self-hosting: Override NVIDIA_NIM_BASE_URL to point at a private NIM cluster

NIM-Specific Extras​

NIM supports additional generation parameters beyond the standard OpenAI surface: top_k, min_p, repetition_penalty, min_tokens, and per-model chat_template overrides. The NeuroLink provider automatically passes these via the providerOptions.openai.body mechanism. If a model rejects an unsupported parameter with HTTP 400, the provider retries the request with that parameter stripped.

Quick Start​

1. Get an API Key​

Sign up at https://build.nvidia.com and create an API key under API Keys.

2. Configure Environment​

Add to your .env file:

# Required
NVIDIA_NIM_API_KEY=nvapi-...

# Optional: override the default model (default: openai/gpt-oss-20b)
NVIDIA_NIM_MODEL=openai/gpt-oss-20b

# Optional: self-hosted NIM base URL (default: https://integrate.api.nvidia.com/v1)
NVIDIA_NIM_BASE_URL=https://integrate.api.nvidia.com/v1

# Optional: NIM-specific generation parameters
NVIDIA_NIM_TOP_K=40
NVIDIA_NIM_MIN_P=0.05
NVIDIA_NIM_REPETITION_PENALTY=1.1
NVIDIA_NIM_MIN_TOKENS=1
NVIDIA_NIM_CHAT_TEMPLATE=
npm install @juspay/neurolink
# or
pnpm add @juspay/neurolink

4. Generate Your First Response​

import { NeuroLink } from "@juspay/neurolink";

const ai = new NeuroLink();

const result = await ai.generate({
  provider: "nvidia-nim",
  input: { text: "Explain gradient descent in simple terms." },
});

console.log(result.content);

Supported Models​

NIM hosts hundreds of models. NeuroLink ships with these popular models pre-enumerated:

OpenAI gpt-oss (default)​

Model IDContextVisionReasoning
openai/gpt-oss-20b128KNoYes

openai/gpt-oss-20b always reasons: the trace comes back as reasoning_content whether or not you pass a thinking option, and it counts against maxTokens. See Reasoning and thinkingLevel.

Meta Llama​

Model IDContextVisionReasoning
meta/llama-3.3-70b-instruct128KNoNo
meta/llama-3.1-405b-instruct128KNoNo
meta/llama-3.1-70b-instruct128KNoNo
meta/llama-3.2-90b-vision-instruct128KYesNo
meta/llama-3.2-11b-vision-instruct128KYesNo

NVIDIA Nemotron (Reasoning)​

Model IDContextVisionReasoning
nvidia/llama-3.3-nemotron-super-49b-v1128KNoYes
nvidia/llama-3.1-nemotron-nano-8b-v1128KNoYes
nvidia/llama-3.1-nemotron-70b-instruct128KNoYes

DeepSeek (Hosted on NIM)​

Model IDContextVisionReasoning
deepseek-ai/deepseek-r1128KNoYes
deepseek-ai/deepseek-r1-distill-llama-70b128KNoYes

Other Models​

Model IDContextNotes
mistralai/mixtral-8x22b-instruct-v0.164KLarge MoE
mistralai/mixtral-8x7b-instruct-v0.132KEfficient MoE
microsoft/phi-416KCompact, capable
google/gemma-3-27b-it128KGoogle Gemma

Availability, checked 2026-10-24. NVIDIA's GET https://integrate.api.nvidia.com/v1/models lists openai/gpt-oss-20b, meta/llama-3.2-90b-vision-instruct, meta/llama-3.2-11b-vision-instruct and nvidia/llama-3.1-nemotron-70b-instruct from the tables above, and none of the others. A chat request to meta/llama-3.3-70b-instruct or meta/llama-3.1-70b-instruct answers HTTP 410, "has reached its end of life on 2026-08-26T09:00:00Z and is no longer available"; nvidia/llama-3.3-nemotron-super-49b-v1 and mistralai/mixtral-8x22b-instruct-v0.1 also answer 410, and deepseek-ai/deepseek-r1 and meta/llama-3.1-405b-instruct answer 404. nvidia/llama-3.1-nemotron-70b-instruct is listed but answered 404 ("not found for account") to the key used for the check. openai/gpt-oss-20b and meta/llama-3.2-11b-vision-instruct each answered a plain request and a tool-calling request. The NvidiaNimModels enum keeps the retired members so existing code compiles.

Browse the full catalog at https://build.nvidia.com/models. You can pass any model ID via --model or model: — NIM returns 404 for IDs that are not in the catalog.

SDK Usage​

Basic Generation​

import { NeuroLink } from "@juspay/neurolink";

const ai = new NeuroLink();

const result = await ai.generate({
  provider: "nvidia-nim",
  input: { text: "Write a TypeScript utility to deep-clone a plain object." },
});

console.log(result.content);

Using a Specific Model​

const result = await ai.generate({
  provider: "nvidia-nim",
  model: "mistralai/mixtral-8x22b-instruct-v0.1",
  input: { text: "Summarise the key ideas in the CAP theorem." },
});

Reasoning and thinkingLevel​

Two separate questions decide what you get:

  1. Does the model reason? That is a property of the model. openai/gpt-oss-20b, the default, reasons on every request: NIM returns the trace as reasoning_content and NeuroLink exposes it as result.reasoning (and as reasoning chunks when streaming). No option is needed to turn it on.
  2. Can NeuroLink switch it on or tune it? That is a property of the provider code, and on NIM it is narrower than the thinkingLevel type suggests:
    • thinkingLevel is honoured on stream() only. Any level other than minimal adds chat_template_kwargs: { thinking: true, enable_thinking: true, reasoning_budget: <maxTokens> } to the request. generate() does not forward it: the request body carries no chat_template_kwargs.
    • The three active levels are not distinguished. low, medium and high send the same flags, so thinkingLevel is an on/off switch here, with minimal meaning off.
    • NeuroLink never sends reasoning_effort to NIM, so that knob is not available through it.
    • An HTTP 400 that names reasoning_budget or chat_template gets one automatic retry without that field. A 400 about the other thinking flags (thinking, enable_thinking) is not retried and is returned to you.
// The trace is there without any thinking option.
const result = await ai.generate({
  provider: "nvidia-nim",
  model: "openai/gpt-oss-20b",
  input: { text: "What is 17 * 23? Answer with just the number." },
  maxTokens: 512,
});

console.log(result.reasoning); // the model's reasoning trace
console.log(result.content); // the answer

Reasoning tokens count toward maxTokens. With a small budget the model can spend all of it reasoning and stop with finish_reason: "length" and an empty content. On openai/gpt-oss-20b, maxTokens: 16 returned an empty answer for a one-word reply, while 64 and 256 returned it. Leave a few hundred tokens of headroom for short answers, more for hard prompts.

Streaming​

import { NeuroLink } from "@juspay/neurolink";

const ai = new NeuroLink();

const stream = await ai.stream({
  provider: "nvidia-nim",
  model: "openai/gpt-oss-20b",
  input: {
    text: "Walk me through building a REST API with Hono and TypeScript.",
  },
});

for await (const chunk of stream.stream) {
  process.stdout.write(chunk);
}

Per-Call Credential Override​

const result = await ai.generate({
  provider: "nvidia-nim",
  input: { text: "Hello" },
  credentials: {
    nvidiaNim: {
      apiKey: "nvapi-per-user-key",
    },
  },
});

For self-hosted NIM clusters, override the base URL per call:

const result = await ai.generate({
  provider: "nvidia-nim",
  input: { text: "Hello from self-hosted NIM" },
  credentials: {
    nvidiaNim: {
      apiKey: "internal-token",
      baseURL: "https://nim.internal.example.com/v1",
    },
  },
});

CLI Usage​

Basic Commands​

# Generate with the default model
pnpm run cli generate "What is the transformer architecture?" --provider nvidia-nim

# Use provider aliases
pnpm run cli generate "Hello" --provider nim
pnpm run cli generate "Hello" --provider nvidia

# Specify a model
pnpm run cli generate "Explain reinforcement learning" \
  --provider nvidia-nim \
  --model mistralai/mixtral-8x22b-instruct-v0.1

# Reasoning model with thinking enabled
pnpm run cli generate "Solve: what is 17! mod 13?" \
  --provider nvidia-nim \
  --model deepseek-ai/deepseek-r1 \
  --thinking-level high

# Interactive loop
pnpm run cli loop --provider nvidia-nim

Provider Aliases​

AliasExample
nvidia-nim--provider nvidia-nim
nvidia--provider nvidia
nim--provider nim

Configuration Reference​

Environment VariableRequiredDefaultDescription
NVIDIA_NIM_API_KEYYes—NVIDIA NIM API key (starts with nvapi-)
NVIDIA_NIM_MODELNoopenai/gpt-oss-20bDefault model
NVIDIA_NIM_BASE_URLNohttps://integrate.api.nvidia.com/v1Base URL (override for self-hosted NIM)
NVIDIA_NIM_TOP_KNo—Top-K sampling; -1 to disable
NVIDIA_NIM_MIN_PNo—Minimum token probability; 0 to disable
NVIDIA_NIM_REPETITION_PENALTYNo—Anti-repetition factor; 1 is neutral
NVIDIA_NIM_MIN_TOKENSNo—Minimum output length in tokens
NVIDIA_NIM_CHAT_TEMPLATENo—Override the model's default chat template

Self-Hosted NIM​

If you run NIM on your own GPU cluster, set NVIDIA_NIM_BASE_URL to point at your cluster. Authentication is still forwarded via Authorization: Bearer, so set NVIDIA_NIM_API_KEY to any non-empty value if your cluster does not require it (or to your actual cluster token if it does).

NVIDIA_NIM_API_KEY=internal-token
NVIDIA_NIM_BASE_URL=https://nim.cluster.example.com/v1

Feature Support​

FeatureSupportedNotes
Text generationYes
StreamingYes
Tool callingYesMost models; depends on model support
Vision / imagesYesModel-dependent (Llama 3.2 Vision, etc.)
Reasoning traceYesModel-dependent; openai/gpt-oss-20b always returns one. thinkingLevel is an on/off switch on stream() only
EmbeddingsNoUse OpenAI or Bedrock for embeddings

Troubleshooting​

"Invalid NVIDIA NIM API key"​

The NVIDIA_NIM_API_KEY is missing, expired, or incorrect.

echo $NVIDIA_NIM_API_KEY
export NVIDIA_NIM_API_KEY=nvapi-...

Get or rotate keys at https://build.nvidia.com/settings/api-keys.

"NVIDIA NIM rate limit exceeded"​

Your account has hit its request-per-minute or token-per-day limit. Upgrade your account, reduce request frequency, or implement backoff. Check your current usage at https://build.nvidia.com/usage.

"NVIDIA NIM model not available"​

The model ID is not in the NIM catalog, or your account tier does not have access.

# Browse the catalog
open https://build.nvidia.com/models

Use the exact model ID shown on the model's page (e.g., openai/gpt-oss-20b).

HTTP 410 "has reached its end of life"​

The request named a model NVIDIA has retired, for example meta/llama-3.1-70b-instruct or meta/llama-3.3-70b-instruct (retired 2026-08-26). If you did not pass model:, look for a stale NVIDIA_NIM_MODEL in your .env or shell: it takes precedence over the built-in default (openai/gpt-oss-20b). Remove it or set it to a model from the current catalog.

"NVIDIA NIM quota exceeded"​

Account-level token or compute quota reached. Check your NIM dashboard.

HTTP 400 with reasoning_budget or chat_template in the error​

The model does not support one of the NIM-specific extras. The provider automatically retries without the rejected parameter. If you see this error surfaced, it means the second attempt also failed — check the rest of the error message for the root cause.

Empty content with finish_reason: "length"​

A reasoning model spent the whole maxTokens budget on its reasoning before it started the answer. The reasoning is in result.reasoning; raise maxTokens (a few hundred tokens is enough for a short answer on openai/gpt-oss-20b).

Thinking level has no visible effect​

  • On generate(), thinkingLevel is not forwarded to NIM at all; use stream().
  • On stream(), any level except minimal sends the same thinking flags, so low, medium and high behave alike.
  • A model that already reasons on every request, such as openai/gpt-oss-20b, returns its trace with or without the flags.
  • If the model rejects reasoning_budget with a 400 that names it, the provider retries once without it. A rejection of thinking or enable_thinking is not retried and comes back as the 400.

See Also​

Need Help? Join the GitHub Discussions or open an issue.