LLM Gateway is an intelligent routing layer that sits between your application and multiple LLM providers. It automatically selects the best model for each request based on cost, latency, reliability, and your preferences.

Get up and running in under 5 minutes. Here's everything you need.

1. Get your API key

Sign up and create an API key from the dashboard.

2. Make your first request

Use cURL or any HTTP client to make a request:

curl https://api.llmroute.xyz/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "Hello, world!"}
    ]
  }'

3. Or use the OpenAI SDK

Just change the base URL—everything else stays the same:

from openai import OpenAI

client = OpenAI(
    api_key="your-llm-gateway-api-key",
    base_url="https://api.llmroute.xyz/v1"
)

response = client.chat.completions.create(
    model="auto",  # Let the gateway choose the best model
    messages=[
        {"role": "user", "content": "Hello, world!"}
    ]
)

print(response.choices[0].message.content)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'your-llm-gateway-api-key',
  baseURL: 'https://api.llmroute.xyz/v1',
});

const response = await client.chat.completions.create({
  model: 'auto',
  messages: [
    { role: 'user', content: 'Hello, world!' }
  ],
});

console.log(response.choices[0].message.content);

All API requests require authentication using a Bearer token.

Authorization: Bearer llm_gw_xxxxxxxxxxxxxxxxxxxx

API Key Types

Auto-routing is the core feature of LLM Gateway. When you set model: "auto", the gateway evaluates multiple factors to select the optimal model.

How It Works

Routing Hints

You can provide hints to influence routing decisions:

{
  "model": "auto",
  "messages": [...],
  "x-routing-hints": {
    "priority": "cost",        // "cost" | "latency" | "quality"
    "max_latency_ms": 2000,    // Maximum acceptable latency
    "prefer_providers": ["anthropic", "openai"],
    "exclude_providers": ["cohere"],
    "min_quality_score": 0.8
  }
}

LLM Gateway supports all major LLM providers. You can use auto-routing or specify a model directly.

Supported Providers

Direct Model Access

You can also request a specific model directly:

{
  "model": "openai/gpt-4o",
  "messages": [...]
}

// Or with provider prefix
{
  "model": "anthropic/claude-3-5-sonnet-20241022",
  "messages": [...]
}

LLM Gateway automatically handles failures with intelligent fallback strategies.

Automatic Fallback

When a provider fails, we automatically retry with the next best option:

Circuit Breaker

We implement circuit breakers to prevent cascading failures:

• Closed: Normal operation, requests pass through
• Open: Provider temporarily disabled after repeated failures
• Half-Open: Testing provider recovery with limited traffic

Custom Fallback Chain

{
  "model": "auto",
  "messages": [...],
  "x-fallback-chain": [
    "anthropic/claude-3-5-sonnet",
    "openai/gpt-4o",
    "mistral/mistral-large"
  ]
}

LLM Gateway helps you save 30-50% on LLM costs through intelligent routing and real-time cost analysis.

How We Optimize Costs

Setting Spend Limits

{
  "limits": {
    "daily_usd": 100,
    "monthly_usd": 2500,
    "per_request_usd": 0.50
  },
  "alerts": {
    "threshold_percent": 80,
    "webhook_url": "https://your-app.com/webhook/spend-alert"
  }
}

Semantic caching reduces costs by up to 90% for repeated or similar queries. Unlike exact-match caching, semantic cache understands the meaning of your prompts and returns cached responses for semantically similar requests.

How It Works

Example: Similar Queries Hit Cache

Configuration

{
  "cache_enabled": true,
  "similarity_threshold": 0.95,  // 0.90-0.99, higher = stricter matching
  "ttl_seconds": 3600,           // Cache expiration (1 hour default)
  "exclude_models": []           // Models to never cache
}

Cache is isolated per organization—your cached responses are never shared with other customers.

Connect your own API keys from LLM providers to get direct pricing and use your existing quotas. LLM Gateway manages routing, fallbacks, and observability while you maintain your provider relationships.

Supported Providers

Benefits of BYOK (Bring Your Own Keys)

• Direct pricing: Pay provider rates directly, no markup on API costs
• Use existing quotas: Leverage your negotiated rate limits and credits
• Enterprise agreements: Maintain your BAAs and compliance contracts
• Gradual migration: Test LLM Gateway without changing billing

How to Connect a Provider

1. Navigate to Connections in the dashboard
2. Click Add Connection and select your provider
3. Enter your API key (encrypted with AES-256)
4. Test the connection to verify it works
5. Enable the provider for routing

LLM Gateway analyzes your usage patterns and provides personalized recommendations to optimize costs, improve performance, and enhance reliability.

Types of Recommendations

How Recommendations Work

• We analyze your last 30 days of usage data
• Recommendations are generated weekly or on-demand
• Each recommendation includes estimated savings/impact
• Apply recommendations with one click in the dashboard
• Track the impact after applying changes

Run open-source models on dedicated GPU infrastructure for maximum control, privacy, and cost efficiency at scale.

Available GPU Options

Cost Comparison Example

Deployment Options

• Managed: We handle deployment, scaling, and maintenance
• Self-hosted: Run on your own infrastructure with our container images
• Hybrid: Route between managed GPUs and API providers based on load

The Chat Completions API is fully compatible with the OpenAI specification.

Request Format

{
  "model": "auto",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "temperature": 0.7,
  "max_tokens": 1000,
  "stream": false
}

Response Format

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1704067200,
  "model": "anthropic/claude-3-haiku",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 8,
    "total_tokens": 33
  },
  "x-llm-gateway": {
    "request_id": "req_xyz789",
    "routing_decision": "cost_optimized",
    "cost_usd": 0.00012,
    "latency_ms": 234,
    "fallback_used": false
  }
}

Stream responses in real-time using Server-Sent Events (SSE).

from openai import OpenAI

client = OpenAI(
    api_key="your-api-key",
    base_url="https://api.llmroute.xyz/v1"
)

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Every request generates an execution receipt with full details about the routing decision, cost, and performance.

{
  "request_id": "req_abc123xyz",
  "timestamp": "2024-01-15T10:30:00Z",
  "routing": {
    "strategy": "cost_optimized",
    "candidates_evaluated": 5,
    "winner": "anthropic/claude-3-haiku",
    "reason": "Lowest cost meeting quality threshold"
  },
  "execution": {
    "provider": "anthropic",
    "model": "claude-3-haiku-20240307",
    "latency_ms": 234,
    "routing_overhead_ms": 12,
    "tokens": {
      "prompt": 150,
      "completion": 89,
      "total": 239
    }
  },
  "cost": {
    "provider_cost_usd": 0.00023,
    "platform_fee_usd": 0.00002,
    "total_usd": 0.00025,
    "savings_vs_default_usd": 0.00089
  },
  "fallback": {
    "used": false,
    "attempts": []
  }
}

Access receipts via the dashboard or the x-llm-gateway response header.

Rate limits protect the system and ensure fair usage across all customers.

Rate Limit Headers

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1704067260

The dashboard provides real-time visibility into your LLM usage, costs, and performance.

Manage API keys for different environments and use cases.

Creating Keys

1. Navigate to Settings → API Keys in the dashboard
2. Click "Create New Key"
3. Select environment (Production or Development)
4. Set optional rate limits and expiration
5. Copy and securely store your key

Pay only for what you use with transparent, per-request pricing.

Pricing Model

No monthly minimums. No hidden fees. Cancel anytime.

Customize how the auto-router selects models for your requests.

{
  "default_priority": "cost",
  "quality_threshold": 0.8,
  "max_latency_ms": 3000,
  "allowed_providers": ["openai", "anthropic", "mistral"],
  "blocked_providers": [],
  "allowed_models": [],
  "blocked_models": [],
  "fallback_enabled": true,
  "fallback_chain": ["anthropic/claude-3-haiku", "mistral/mistral-small"]
}

LLM Gateway is built with security-first principles. Your data and API keys are protected by multiple layers of enterprise-grade security.

Infrastructure Security

• Deployed on cloud infrastructure with industry-standard security
• DDoS protection and Web Application Firewall (WAF)
• Automated security updates and patching
• Multi-region deployment for high availability

Access Controls

• Role-based access control (RBAC) for team members
• Per-API-key IP restrictions
• Automatic key expiration and rotation reminders
• Brute force protection with automatic lockout
• Rate limiting at multiple levels (org, key, IP)

We take a privacy-first approach to handling your data. By default, we don't store your prompts or responses.

What We Store

PII Protection

Built-in guardrails can detect and redact sensitive information before it reaches LLM providers:

• Email addresses and phone numbers
• Credit card numbers and SSNs
• API keys and tokens
• Custom patterns via regex

{
  "pii_detection_enabled": true,
  "pii_action": "redact",
  "pii_types": ["email", "phone", "ssn", "credit_card", "api_key"]
}

LLM Gateway is built with security and privacy in mind, providing tools to help you meet your compliance requirements.

Data Residency

Control where your data is processed:

• US region (default)
• EU region available for GDPR compliance
• Custom regions available for Enterprise plans

Audit & Reporting

Comprehensive audit logs for security and compliance:

• API key creation, rotation, and revocation
• Configuration changes and policy updates
• Access attempts and authentication events
• Export logs in JSON or CSV format
• Configurable retention (7-365 days)

Use LLM Gateway as a drop-in replacement for the OpenAI SDK.

from openai import OpenAI

client = OpenAI(
    api_key="your-llm-gateway-api-key",
    base_url="https://api.llmroute.xyz/v1"
)

# All OpenAI SDK methods work the same
response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hello!"}]
)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.LLM_GATEWAY_API_KEY,
  baseURL: 'https://api.llmroute.xyz/v1',
});

const response = await client.chat.completions.create({
  model: 'auto',
  messages: [{ role: 'user', content: 'Hello!' }],
});

Integrate LLM Gateway with LangChain for complex AI workflows.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="auto",
    openai_api_key="your-llm-gateway-api-key",
    openai_api_base="https://api.llmroute.xyz/v1"
)

# Use with chains, agents, etc.
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

prompt = PromptTemplate(
    input_variables=["topic"],
    template="Write a short poem about {topic}."
)

chain = LLMChain(llm=llm, prompt=prompt)
result = chain.run(topic="the moon")

Use LLM Gateway with LlamaIndex for RAG and document Q&A.

from llama_index.llms.openai import OpenAI
from llama_index.core import Settings

Settings.llm = OpenAI(
    model="auto",
    api_key="your-llm-gateway-api-key",
    api_base="https://api.llmroute.xyz/v1"
)

# Now use LlamaIndex as normal
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What is this document about?")