LLM Providers

LLMKit supports multiple LLM providers through a unified API, allowing you to switch between providers and models without changing your application code. This guide explains how to configure and use different providers.

Supported Providers

LLMKit currently supports the following providers:

• OpenRouter (default)
• OpenAI
• Anthropic
• Google (Gemini)
• Mistral AI
• Cohere
• Azure OpenAI
• Local models (via Ollama)

Provider Configuration

Setting Up Provider API Keys

To use a provider, you need to configure its API key:

1. Navigate to "Settings" > "Providers" in the LLMKit interface
2. Click "Add Provider"
3. Select the provider from the dropdown
4. Enter your API key
5. (Optional) Configure provider-specific settings
6. Click "Save"

You can also configure providers through environment variables:

# Required
OPENROUTER_API_KEY=your_openrouter_key

# Optional - Add only the providers you need
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GOOGLE_API_KEY=your_google_key
MISTRAL_API_KEY=your_mistral_key
COHERE_API_KEY=your_cohere_key
AZURE_OPENAI_API_KEY=your_azure_key

Provider-Specific Configuration

Some providers require additional configuration beyond just an API key:

Azure OpenAI

AZURE_OPENAI_API_KEY=your_azure_key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_DEPLOYMENT_NAME=your_deployment_name

Local Models (Ollama)

OLLAMA_HOST=http://localhost:11434

Selecting Providers and Models

Default Provider

You can set a default provider for each prompt:

1. Navigate to the prompt's detail page
2. Click the "Settings" tab
3. Select a default provider and model
4. Click "Save"

Model Selection in API Calls

When calling the API, you can override the default model by specifying a different one:

from openai import OpenAI

client = OpenAI(
    api_key="llmkit_yourkey",
    base_url="http://localhost:8000/v1",
)

# Use specific provider and model
response = client.chat.completions.create(
    model="your-prompt-id",
    messages=[
        {"role": "user", "content": "Hello!"}
    ],
    llmkit_provider="anthropic",  # Override default provider
    llmkit_model="claude-3-opus-20240229"  # Specify model
)

Provider Capabilities

Different providers and models have different capabilities. LLMKit helps you handle these differences:

Model Compatibility

LLMKit automatically adapts your prompts to work with different models:

• Prompt format conversion
• Token limit adjustments
• Feature compatibility checks

Feature Matrix

Provider Fallbacks

LLMKit supports automatic fallbacks between providers:

Configuring Fallbacks

1. Navigate to "Settings" > "Provider Fallbacks"
2. Click "Add Fallback Rule"
3. Configure the rule:
   • Primary provider
   • Fallback provider(s)
   • Conditions for fallback (timeout, error, etc.)
4. Click "Save"

Fallback Example

{
  "primary": "openai",
  "fallbacks": ["anthropic", "google"],
  "conditions": {
    "timeout_ms": 5000,
    "errors": ["rate_limited", "service_unavailable"]
  }
}

This configuration will:

1. Try OpenAI first
2. If OpenAI times out or returns a rate limit error, try Anthropic
3. If Anthropic fails, try Google
4. If all fail, return an error

Cost Management

LLMKit helps you manage costs when working with multiple providers:

Cost Tracking

Monitor your spending across providers:

1. Navigate to "Analytics" > "Cost Dashboard"
2. View cost breakdowns by:
   • Provider
   • Model
   • Prompt
   • Time period

Cost Controls

Set up cost control measures:

1. Navigate to "Settings" > "Cost Controls"
2. Configure limits like:
   • Daily spending cap per provider
   • Maximum tokens per request
   • Rate limits per application

Cost Optimization

LLMKit can help optimize costs:

1. Navigate to "Settings" > "Cost Optimization"
2. Enable features like:
   • Automated model selection based on prompt complexity
   • Caching for repeated queries
   • Token usage optimization

Provider-Specific Optimizations

OpenAI-Specific Settings

client.chat.completions.create(
    model="your-prompt-id",
    llmkit_provider="openai",
    llmkit_model="gpt-4",
    response_format={"type": "json_object"},  # OpenAI-specific
    tools=tools_list  # OpenAI function calling
)

Anthropic-Specific Settings

client.chat.completions.create(
    model="your-prompt-id",
    llmkit_provider="anthropic",
    llmkit_model="claude-3-opus-20240229",
    anthropic_metadata={  # Anthropic-specific
        "user_id": "user123"
    }
)

Best Practices

For effective multi-provider management:

1. Test Across Providers: Verify your prompts work well across different providers
2. Use Provider-Agnostic Features: When possible, stick to features supported by all providers
3. Configure Reasonable Fallbacks: Set up fallbacks to ensure resilience
4. Monitor Costs: Regularly review cost analytics to avoid surprises
5. Benchmark Performance: Compare response quality and latency across providers

Example: Multi-Provider Strategy

Here's an example of a comprehensive multi-provider strategy:

1. Default Configuration:
   • Use OpenRouter as the default provider for flexibility
   • Set up provider-specific API keys for direct access
2. Production Prompt Settings:
   • Critical paths: Use GPT-4 with Claude as fallback
   • High-volume paths: Use Mixtral for cost efficiency
   • RAG applications: Use embedding models from Cohere
3. Development Environment:
   • Use Ollama with local models for development
   • Switch to production providers for final testing
4. Fallback Chain:
   • Primary: OpenAI
   • First fallback: Anthropic
   • Second fallback: Mistral
   • Final fallback: Local model via Ollama

This strategy provides a balance of quality, cost efficiency, and reliability.

Conclusion

LLMKit's multi-provider support gives you flexibility and resilience when working with LLMs. By leveraging this capability, you can optimize for cost, performance, and reliability while maintaining a consistent interface for your applications.