llmkit

Tracing and Debugging

LLMKit provides powerful tracing and debugging tools to help you understand how your prompts are processed, identify issues, and optimize performance. This guide explains how to use these tools effectively.

Trace Overview

Every LLM call in LLMKit generates a detailed trace that captures the entire lifecycle of the request, from template rendering to final response. Traces provide visibility into:

  • Template variable resolution
  • Conditional logic evaluation
  • Prompt rendering
  • Token usage
  • Response generation
  • Error handling

Trace Overview

Accessing Traces

Viewing Recent Traces

To view recent traces:

  1. Navigate to the "Traces" section in the LLMKit interface
  2. Browse the list of recent traces
  3. Click on a trace to view its details

Finding Specific Traces

You can search for traces using various filters:

  • By prompt ID
  • By time range
  • By status (success, error)
  • By containing specific text
  • By user or client application

Trace Export

For detailed analysis, you can export traces in various formats:

bash 
llmkit trace export --id=trace-123abc --format=json

Supported formats include JSON, CSV, and OpenTelemetry.

Trace Details

A typical trace contains the following sections:

1. Request Summary

Overview information about the request:

  • Timestamp
  • Prompt ID and version
  • Execution time
  • Status
  • Token usage
  • Client information

2. Template Processing

Details about how the template was processed:

  • Template variables and their values
  • Conditional branch evaluation
  • Loop iterations
  • Rendered template (final prompt sent to the model)

3. LLM Request

Information about the request to the LLM provider:

  • Provider and model
  • Request parameters (temperature, top_p, etc.)
  • Request headers
  • Token count

4. LLM Response

Details about the response from the LLM:

  • Raw response
  • Completion tokens
  • Response time
  • Token usage

5. Errors and Warnings

Any issues that occurred during processing:

  • Template rendering errors
  • Variable validation errors
  • Provider API errors
  • Token limit warnings

Debugging with Traces

Traces are invaluable for debugging prompt issues:

Template Rendering Issues

If your prompt isn't rendering correctly:

  1. Check the "Template Processing" section
  2. Look for undefined variables or syntax errors
  3. Examine the conditional branch evaluation
  4. Verify the final rendered template

Unexpected Responses

If your LLM responses aren't what you expected:

  1. Check the rendered template to ensure it contains what you think
  2. Review the request parameters (temperature, etc.)
  3. Look at token usage to see if you're hitting limits
  4. Check for any provider-specific warnings

Performance Issues

If your prompts are slow:

  1. Check the execution time breakdown
  2. Look for slow template rendering (complex conditionals)
  3. Check provider response time
  4. Review token usage (larger prompts take longer)

Live Debugging

LLMKit provides tools for live debugging:

Debug Mode

Enable debug mode for a prompt:

  1. Navigate to the prompt's detail page
  2. Click the "Settings" tab
  3. Enable "Debug Mode"
  4. Click "Save"

In debug mode, LLMKit provides extra information in traces and can log debug messages.

Debug Console

Use the debug console for real-time insights:

  1. Navigate to "Developer" > "Debug Console"
  2. Watch live traces as they occur
  3. Filter for specific prompts or clients
  4. Set breakpoints to pause execution

Template Preview

For template debugging:

  1. Navigate to a prompt's detail page
  2. Click the "Template" tab
  3. Enter test values for variables
  4. Click "Preview" to see how the template renders

Trace Analytics

LLMKit provides analytics based on trace data:

Performance Dashboard

Monitor overall performance:

  1. Navigate to "Analytics" > "Performance"
  2. View metrics like:
    • Average response time
    • Token usage trends
    • Error rates
    • Requests per minute

Prompt Performance

Analyze specific prompt performance:

  1. Navigate to a prompt's detail page
  2. Click the "Analytics" tab
  3. View prompt-specific metrics:
    • Usage frequency
    • Average response time
    • Token efficiency
    • Error rates

Custom Reports

Create custom reports based on trace data:

  1. Navigate to "Analytics" > "Custom Reports"
  2. Click "New Report"
  3. Select dimensions and metrics
  4. Choose visualization type
  5. Click "Generate Report"

Optimizing with Trace Data

Use trace data to optimize your prompts:

Token Usage Optimization

Reduce token usage and costs:

  1. Identify high-token prompts in the analytics dashboard
  2. Review trace details to find optimization opportunities
  3. Consider:
    • Shortening instructions
    • Removing redundant context
    • Streamlining examples
    • Using more efficient variable formats

Performance Tuning

Improve response times:

  1. Identify slow prompts in the analytics dashboard
  2. Check trace execution breakdowns
  3. Consider:
    • Simplifying complex template logic
    • Caching static content
    • Using shorter prompts
    • Switching to faster models

Error Reduction

Minimize errors:

  1. Review error trends in the analytics dashboard
  2. Examine traces for common error patterns
  3. Implement:
    • Better variable validation
    • Fallback mechanisms
    • Token limit management
    • Retry logic

Integrating with External Systems

LLMKit's tracing system integrates with external monitoring and observability platforms:

OpenTelemetry Export

Export trace data to OpenTelemetry-compatible systems:

bash 
llmkit configure --tracing.exporter=otlp --tracing.endpoint=https://your-collector:4317

Prometheus Metrics

Export metrics to Prometheus:

bash 
llmkit configure --metrics.exporter=prometheus --metrics.endpoint=:9090

Logging Integration

Send trace logs to external logging systems:

bash 
llmkit configure --logging.exporter=fluentbit --logging.endpoint=http://your-fluentbit:8888

Best Practices

For effective tracing and debugging:

  1. Enable Comprehensive Tracing: In development environments, enable detailed tracing
  2. Use Structured Logging: Add structured context to your traces
  3. Sample in Production: In high-volume production, use sampling to reduce overhead
  4. Set Up Alerts: Configure alerts for unusual patterns or errors
  5. Regularly Review Analytics: Schedule time to review trace analytics for optimization opportunities

Example: Debugging Workflow

Here's an example of a complete debugging workflow:

  1. Identify the Issue:
    • Notice unexpected responses in a specific scenario
    • Locate relevant traces using search filters
  2. Analyze Traces:
    • Examine the rendered template
    • Check variable values
    • Review model parameters
    • Look for warnings or errors
  3. Test Hypotheses:
    • Modify prompt in the template preview
    • Adjust variables or parameters
    • Test different inputs
  4. Implement Solution:
    • Update prompt template
    • Add input validation
    • Adjust model parameters
    • Create a new prompt version
  5. Verify Fix:
    • Run tests with problematic inputs
    • Check new traces to confirm resolution
    • Monitor analytics to ensure the fix holds

By following this workflow and leveraging LLMKit's tracing tools, you can efficiently identify and resolve issues with your prompts.

Conclusion

Effective tracing and debugging are essential for developing reliable, high-quality LLM applications. LLMKit's comprehensive tracing system provides the visibility and tools you need to understand, debug, and optimize your prompts at every stage of the development lifecycle.