User Guide

This guide is for developers who want to use unified-llm-client as an application dependency, not contributors working on the library internals.

If you are opening the repository for the first time, read the pages below in order:

1. Getting Started
2. Completions And Streaming
3. Conversations And Tools
4. Agent Instructions And Skills
5. Chatbot Production Helpers
6. Persistence And Session API
7. Speech
8. Session API Reference
9. Production Setup
10. Production Guide

What This Library Gives You

• One LLMClient surface for OpenAI, Anthropic, and Gemini
• Shared request and response types across providers
• Stateless OpenAI Responses transport while keeping conversation/session state inside this library
• One-off completions and streaming
• Stateful conversations with optional tool execution
• Session persistence in memory, Postgres, or Redis
• Usage and cost tracking
• Live provider model discovery through client.models.listRemote({ provider })
• A framework-agnostic Session API built on Request and Response
• Routing and fallback rules for production traffic
• Google Embedding 2 support through client.embed()
• OpenAI batch speech support through client.speak() and client.transcribe()
• Optional retrieval helpers through unified-llm-client/retrieval
• Fail-closed RAG orchestration through unified-llm-client/chatbot
• Best-effort transcript PII redaction through unified-llm-client/pii
• Reusable chunking helpers through unified-llm-client/chunking
• Node-only agent instruction and skill loading through unified-llm-client/agent-files

Which Page To Read For Which Task

• "I just want one model call to work" Read Getting Started
• "I need streaming output" Read Completions And Streaming
• "I need tool calls or persistent conversation state" Read Conversations And Tools
• "I need users to add agent.md files or skills to their agents" Read Agent Instructions And Skills
• "I need saved history or HTTP endpoints" Read Persistence And Session API
• "I need grounded chatbot answers, citations, or transcript redaction" Read Chatbot Production Helpers
• "I need routing, budgets, testing, or rollout guidance" Read Production Guide
• "I need production env setup, Postgres wiring, or embedding storage details" Read Production Setup
• "I need text-to-speech, speech-to-text, or speech cost tracking" Read Speech

Reference Docs

• API reference: docs/api/index.html
• Session API contract: SESSION_API_REFERENCE.md
• Provider comparison: PROVIDER_COMPARISON.md
• Embeddings integration report: EMBEDDINGS_REPORT.md
• Embeddings review cross-check: EMBEDDINGS_REVIEW_CROSSCHECK.md
• Embeddings follow-up fix plan: EMBEDDINGS_FOLLOW_UP_FIX_PLAN.md
• Prompt caching implementation report: PROMPT_CACHING_REPORT.md
• Reasoning efforts report: REASONING_EFFORTS_REPORT.md
• OpenAI Responses migration report: OPENAI_RESPONSES_MIGRATION_REPORT.md
• Migration notes: MIGRATION_GUIDE.md
• Cost policy and pricing notes: COST_AND_PRICING.md
• Production setup guide: PRODUCTION_SETUP.md
• Speech guide: SPEECH.md
• Agent instructions and skills guide: AGENT_INSTRUCTIONS_AND_SKILLS.md
• Chatbot production helpers: CHATBOT_PRODUCTION_HELPERS.md
• Speech API research report: SPEECH_API_RESEARCH_REPORT.md

Current Provider Notes

• OpenAI requests now use the Responses API in stateless mode with library-owned history replay.
• Prompt caching is exposed through provider-specific options rather than one artificial cross-provider abstraction.
• OpenAI uses providerOptions.openai.promptCaching, including explicit options, plus promptCacheBreakpoint on supported user text/image parts.
• Anthropic uses part-level cacheControl, tool-level cacheControl, and request-level providerOptions.anthropic.cacheControl.
• Gemini uses providerOptions.google.promptCaching.cachedContent, and explicit cache resources can be managed with client.googleCaches.
• Reasoning and thinking controls are also provider-specific: providerOptions.openai.reasoning, providerOptions.anthropic.thinking, providerOptions.anthropic.effort, and providerOptions.google.thinking.
• OpenAI reasoning accepts the model-dependent max effort used by GPT-5.6.
• client.models.listRemote({ provider }) fetches the provider's live model list without changing the local routing registry.
• client.embed() is now available for Google Embedding 2.
• OpenAI batch speech is available through client.speak() and client.transcribe(). Speech usage is tracked separately through client.getSpeechUsage() and client.exportSpeechUsage().
• Retrieval remains explicit; use retrieveAndComplete() when you want the optional fail-closed generation step without changing complete() or conversation().
• The external follow-up review has been cross-checked in EMBEDDINGS_REVIEW_CROSSCHECK.md; the highest-confidence follow-ups are lightweight stores, chunking helpers, and Gemini text batching.
• The detailed follow-up implementation plan is in EMBEDDINGS_FOLLOW_UP_FIX_PLAN.md, including exact fix order and file-level implementation guidance.
• The active implementation tracker is stored in the repository root as prompt_caching_todo.md.
• The embeddings implementation tracker is stored in the repository root as embeddings_todo.md.

Embeddings And Retrieval Quick Start

import { LLMClient } from 'unified-llm-client';
import {
  createDenseRetriever,
  createInMemoryKnowledgeStore,
  createPostgresKnowledgeStore,
  formatRetrievedContext,
} from 'unified-llm-client/retrieval';
import { chunkText, cleanText, stripHtml } from 'unified-llm-client/chunking';

const client = LLMClient.fromEnv({
  defaultEmbeddingModel: 'gemini-embedding-2',
  defaultModel: 'gpt-4o',
});

const knowledgeStore = createPostgresKnowledgeStore({
  connectionString: process.env.DATABASE_URL,
});

await knowledgeStore.ensureSchema();

const retriever = createDenseRetriever({
  embed: client,
  embedding: { model: 'gemini-embedding-2' },
  store: knowledgeStore,
});

const results = await retriever.search({
  filter: {
    botId: 'bot-1',
    embeddingProfileId: 'profile-2026-04-24',
    knowledgeSpaceId: 'kb-support',
    tenantId: 'tenant-1',
  },
  query: 'What is the refund window?',
  topK: 4,
});

const context = formatRetrievedContext(results, {
  maxResults: 4,
  maxTokens: 900,
});

const answer = await client.complete({
  messages: [
    {
      content: `Question: What is the refund window?\n\n${context.text}`,
      role: 'user',
    },
  ],
});

Chunking helpers stay separate from retrieval and generation:

const cleaned = cleanText(
  stripHtml('<h1>Refund Policy</h1><p>Refunds last 30 days.</p>'),
);
const chunks = chunkText(cleaned, {
  chunkSize: 900,
  overlap: 120,
});

The embedding request itself stays separate:

const embedding = await client.embed({
  input: 'Refunds are available for 30 days after purchase.',
  purpose: 'retrieval_document',
  providerOptions: {
    google: {
      title: 'Refund Policy',
    },
  },
});

Embedding inputs must contain at least one non-empty string or canonical content-part array. Supported purpose values are retrieval_document, retrieval_query, semantic_similarity, classification, and clustering. gemini-embedding-2 accepts integer dimensions from 128 through 3072; its recommended sizes are 768, 1536, and 3072. Invalid requests fail locally with ProviderCapabilityError before provider dispatch. Its details provide machine-readable option and constraint fields, plus an optional zero-based itemIndex; use these fields instead of parsing the generic error message.

PostgresKnowledgeStore enforces strict retrieval filters before it will run a dense or lexical search:

• tenantId
• botId
• knowledgeSpaceId
• embeddingProfileId

That is intentional. The store fails closed rather than broadening the search scope. Keep stored chunk embeddings and live query embeddings on the same embedding profile. createDenseRetriever() and createHybridRetriever() now accept optional rerank hooks, while PostgresKnowledgeStore exposes active-profile and reindex helpers such as activateEmbeddingProfile(), getActiveEmbeddingProfile(), listKnowledgeSources(), and markKnowledgeSourcesNeedingReindex(). Chunking, ingestion queues, provider-managed reranking services, and automatic retrieval inside complete() / conversation() remain outside the core library.

activateEmbeddingProfile() throws when the requested knowledge space is missing or when the embedding profile does not belong to that tenant/bot/knowledge-space scope. That is deliberate fail-closed behavior.

For local demos, tests, or single-process apps, you can start with:

const knowledgeStore = createInMemoryKnowledgeStore();

InMemoryKnowledgeStore keeps chunks and vectors in process memory and mirrors the main retrieval helper methods, so you can prototype without a database. It is not durable, so production retrieval should still use PostgresKnowledgeStore.

The opt-in Postgres retrieval lifecycle smoke uses fixed vectors and a typed stub embedder, so it makes no provider calls or paid requests. It requires LIVE_REAL_TESTS=1 and DATABASE_URL and uses a unique temporary schema with exact cleanup:

LIVE_REAL_TESTS=1 DATABASE_URL=... pnpm vitest run test/live-real/retrieval-lifecycle.test.ts
# or:
pnpm test:real:retrieval

formatRetrievedContext() wraps each used result in explicit untrusted-content delimiters, accounts for every omission, and supports uncalibrated score-display modes:

• scoreDisplay: 'raw' prints labeled raw values such as dense similarity, lexical relevance, or fused rank score
• scoreDisplay: 'relative_top_1' prints an uncalibrated display value normalized to the top shown result

Live embedding validation is opt-in:

LIVE_TESTS=1 pnpm test:embeddings:live

Prompt Caching Quick Start

const client = LLMClient.fromEnv({ defaultModel: 'gpt-4o' });

await client.complete({
  messages: [{ content: 'Summarize the FAQ.', role: 'user' }],
  providerOptions: {
    openai: {
      promptCaching: {
        key: 'faq-v1',
        retention: '24h',
      },
    },
  },
});

const cache = await client.googleCaches.create({
  model: 'gemini-2.5-flash',
  messages: [{ content: 'Refunds are available for 30 days.', role: 'user' }],
  ttl: '3600s',
});

await client.complete({
  model: 'gemini-2.5-flash',
  messages: [{ content: 'What is the refund window?', role: 'user' }],
  providerOptions: {
    google: {
      promptCaching: {
        cachedContent: cache.name,
      },
    },
  },
});

client.googleCaches.create() returns cache names in the provider format cachedContents/{id}. Pass that name directly into providerOptions.google.promptCaching.cachedContent when you want to reuse the cache.

Reasoning And Thinking Quick Start

Use provider-specific options when you want a model to spend more or less effort on hidden reasoning or thinking. The library does not expose a single top-level reasoning knob because provider behavior differs too much across model families.

await client.complete({
  model: 'gpt-5',
  maxTokens: 800,
  messages: [{ content: 'Audit this rollout plan.', role: 'user' }],
  providerOptions: {
    openai: {
      reasoning: {
        effort: 'medium',
        summary: 'auto',
      },
    },
  },
});

await client.complete({
  model: 'claude-sonnet-4-6',
  maxTokens: 1200,
  messages: [{ content: 'Review this architecture decision.', role: 'user' }],
  providerOptions: {
    anthropic: {
      effort: 'medium',
      thinking: { type: 'adaptive', display: 'omitted' },
    },
  },
});

await client.complete({
  model: 'gemini-2.5-flash',
  maxTokens: 700,
  messages: [{ content: 'Find edge cases in this workflow.', role: 'user' }],
  providerOptions: {
    google: {
      thinking: { budgetTokens: 0, includeThoughts: false },
    },
  },
});

Provider-reported reasoning or thought-token counts are exposed as usage.reasoningTokens when available. Reasoning summaries and thoughts are not merged into response.text.

Typical Adoption Path

1. Install the package from GitHub or a local path.
2. Add provider keys to the consuming application's .env.
3. Create LLMClient.fromEnv({ defaultModel }).
4. Ship one complete() call.
5. Add stream() or conversation() once the first path works.
6. Add a durable sessionStore and usageLogger when you need persistence or analytics.
7. Add ModelRouter and budget policies after you understand real production traffic.

Before You Start

• Node >=18 is required.
• The consumer project owns the environment variables. The library does not read the .env in this repository unless you run examples from this repository.
• If you want durable conversations, add DATABASE_URL or wire your own RedisSessionStore.
• If you want aggregated usage exports, use PostgresUsageLogger.