Production Guide

This page covers the parts of the library that matter once the first happy-path integration already works.

If you want the concrete production setup details for .env, Postgres wiring, and where embedding vectors are actually stored, read Production Setup alongside this guide.

Route Traffic With ModelRouter

ModelRouter lets you centralize model selection logic instead of scattering it across request handlers.

import { LLMClient, ModelRouter } from 'unified-llm-client';

const router = new ModelRouter({
  rules: [
    {
      name: 'tool-traffic',
      match: { hasTools: true },
      target: { provider: 'openai', model: 'gpt-4o' },
      fallback: [{ provider: 'openai', model: 'gpt-4o-mini' }],
    },
    {
      name: 'default-fast-path',
      target: { provider: 'openai', model: 'gpt-4o-mini' },
    },
  ],
});

const client = LLMClient.fromEnv({
  defaultModel: 'gpt-4o-mini',
  modelRouter: router,
});

Common reasons to add a router:

• Send tool-heavy traffic to a model with stronger tool support
• Keep low-value requests on a cheaper model
• Define fallback chains during provider outages
• Run deterministic weighted experiments

Weighted variants must be a non-empty array. Every variant needs a non-empty model ID and a finite weight greater than zero; positive fractional weights are supported and normalized by their total. The router snapshots rule structures at construction, so later mutations to caller-owned variant arrays do not change routing behavior.

Use Budget Policies Intentionally

Both request-level and conversation-level calls accept budget controls.

const client = LLMClient.fromEnv({
  defaultModel: 'gpt-4o',
  budgetExceededAction: 'warn',
  onWarning: (message) => {
    console.warn('llm warning', message);
  },
});

You can also override budget behavior per call:

await client.complete({
  budgetUsd: 0.05,
  budgetExceededAction: 'throw',
  messages: [{ role: 'user', content: 'Write a short answer.' }],
});

Use:

• throw when overspend is unacceptable
• warn when you want observability without interruption
• skip when you want a graceful no-call fallback

Choose The Right Runtime

The core client surface is safe for Edge-style runtimes:

• LLMClient
• Conversation
• SessionApi
• in-memory storage
• routing and utility helpers

Node-only features are loaded lazily:

• PostgresSessionStore
• PostgresUsageLogger

Practical rule:

• Use Edge for stateless request execution and streaming.
• Use Node when you need Postgres-backed persistence or usage aggregation in-process.
• Use Node for OpenAI speech-to-text uploads when your runtime does not provide stable Blob and FormData support.

Logging And Data Hygiene

The built-in ConsoleLogger and PostgresUsageLogger sanitize usage metadata before writing it. They remove credentials, prompts, canonical messages, explicit tool arguments/results, raw request and response payloads, transcripts, and binary or encoded media while retaining attribution and usage metrics. PostgresUsageLogger sanitizes and clones metadata before it enters the in-memory batch queue.

The UsageLogger interface itself receives the original UsageEvent. A custom logger must apply its own allowlist or equivalent sanitization before retaining, forwarding, or printing the event. Treat that interface as a trusted application boundary.

Recommended production posture:

• Log request ids, session ids, tenant ids, model ids, finish reasons, duration, and cost.
• Do not add raw prompts or tool payloads to a custom logger unless you have a clear compliance reason.
• Do not add raw audio, base64 audio, or full transcripts to a custom logger unless your product has explicit retention and consent controls.
• Keep tool results narrow and structured so downstream logging stays predictable.

Speech In Production

Speech is available through client.speak() and client.transcribe() for OpenAI batch endpoints. Keep it separate from conversation persistence: the library returns audio bytes and transcript text, but it does not store audio files or transcript history automatically.

Use PostgresUsageLogger when you need speech cost attribution. Text completions continue to use the normal usage table, while speech events are written to a sibling table named ${tableName}_speech, such as llm_usage_events_speech. Query those totals with client.getSpeechUsage() or export them with client.exportSpeechUsage().

For budgets, pass explicit durations when they cannot be derived from the payload:

• estimatedOutputSeconds or maxOutputSeconds for text-to-speech.
• inputAudioSeconds for compressed speech-to-text inputs.

As with text usage, usage.costUSD is the numeric field for billing logic. usage.cost is display-only.

Testing Without Live Providers

Use LLMClient.mock() for deterministic tests.

Mock operation queues are strict. Queue one entry for every complete, stream, embed, speak, or transcribe call. An empty or exhausted queue throws MockQueueExhaustedError; mock clients no longer synthesize defaults.

import { LLMClient } from 'unified-llm-client';

const client = LLMClient.mock({
  responses: [
    {
      content: [{ type: 'text', text: 'MOCK_OK' }],
      finishReason: 'stop',
      model: 'mock-model',
      provider: 'mock',
      raw: null,
      text: 'MOCK_OK',
      toolCalls: [],
      usage: {
        cachedTokens: 0,
        cost: '$0.0000',
        costUSD: 0,
        inputTokens: 5,
        outputTokens: 2,
      },
    },
  ],
});

const response = await client.complete({
  messages: [{ role: 'user', content: 'Ping' }],
});

Use mock clients for:

• unit tests
• CI checks that must not depend on external APIs
• deterministic examples and snapshots

Testing With Live Providers

Live-provider tests are opt-in because they consume quota and depend on account state. Keep normal CI on pnpm test; run live checks manually or in a protected pipeline with provider keys available.

LIVE_TESTS=1 pnpm test:live
pnpm test:real:health
pnpm test:real:package
pnpm test:real:providers
pnpm test:real:sessions
pnpm test:real:tools
pnpm test:real:budgets
pnpm test:real

The test:real:* suite loads .env with a safe parser and validates package exports, provider adapters, conversations, tools, Session API tenant isolation, budgets, retrieval, Postgres stores, usage logging, and selected security guards. Tests that touch Postgres require DATABASE_URL. Provider account limits are not treated as deterministic code failures; rerun after resolving credit or rate-limit issues.

Keep live-provider tests opt-in and separate from the default test suite.

Versioning And Reuse Across Projects

You can install directly from the GitHub repository:

pnpm add github:07rjain/LLMlibrary

For more stable reuse across projects, create tags and install a specific version:

pnpm add github:07rjain/LLMlibrary#v0.1.0

That gives consumers a pinned dependency instead of tracking main.

Rollout Checklist

• Start with one provider and one model.
• Confirm the first complete() path in production-like logs.
• Add streaming only where it improves user experience.
• Add session persistence only where continuity matters.
• Add tool execution only when prompts alone are insufficient.
• Add usage logging before you need billing or cost attribution.
• Add routing rules after you have real traffic patterns to optimize against.
• Tag versions before multiple projects begin depending on the library.

Supporting Docs

• API reference: ./api/index.html
• Session API contract: SESSION_API_REFERENCE.md
• Production setup: PRODUCTION_SETUP.md
• Provider comparison: PROVIDER_COMPARISON.md
• Cost policy: COST_AND_PRICING.md