Skip to main content

Langfuse Observability

What is Observability?​

Observability is the ability to understand what's happening inside your application by examining its outputs — logs, metrics, and traces. For AI applications, observability answers critical questions:

  • What prompts were sent to the LLM?
  • Why did the agent make that decision?
  • How long did each step take?
  • How much did this request cost?
  • Why did this conversation fail?

Without observability, debugging AI applications is like flying blind — you see inputs and outputs but have no visibility into the complex reasoning happening in between.

What is Langfuse?​

Langfuse is an open-source LLM observability platform that provides comprehensive tracing, debugging, and monitoring for your AI agents and LLM applications. Think of it as "DevTools for AI" — giving you full visibility into every LLM call, agent step, tool invocation, and decision point.

CAIP Langfuse Instance​

CAIP provides a managed Langfuse instance with BMW WebEAM SSO authentication:

RegionEnvironmentURL
ROWProductionhttps://langfuse.caip.bmw.cloud/
CNProductionhttps://langfuse.caip.bmwchina.cloud/
No Separate Account Needed

Login with your BMW WebEAM credentials — the same credentials you use for other BMW systems.

Key Features​

🔍 Distributed Tracing​

Track every operation in your AI application as a hierarchical trace:

Trace: "answer_user_question" (2,340ms)
├── Span: "validate_input" (15ms)
├── Span: "search_knowledge_base" (450ms)
│   ├── Span: "embed_query" (120ms)
│   └── Span: "vector_search" (330ms)
├── Generation: "gpt-4o" (1,750ms)
│   ├── Input: 2,456 tokens ($0.0245)
│   ├── Output: 512 tokens ($0.0154)
│   └── Total Cost: $0.0399
└── Span: "format_response" (125ms)

Benefits:

  • See the complete execution flow at a glance
  • Identify slow operations immediately
  • Understand parent-child relationships between operations

🐛 Debug & Inspect​

Drill into any trace to see exact inputs, outputs, and metadata:

  • LLM Prompts: See exactly what was sent to the model
  • Model Responses: View complete generated text
  • Tool Calls: Inspect function arguments and return values
  • Errors: See stack traces and error context

💰 Cost Tracking​

Monitor token usage and estimated costs across all your models:

MetricDescription
Input TokensTokens in prompts (charged at input rate)
Output TokensTokens in completions (charged at output rate)
Estimated CostCalculated from token counts and model pricing
Cost by ModelBreakdown of spending per model

👤 User & Session Tracking​

Group traces by user and session for conversation-level debugging:

  • User ID: Find all traces for a specific user
  • Session ID: View complete conversation threads
  • Tags: Add custom labels for filtering (e.g., production, beta-feature)

📈 Analytics Dashboard​

Visualize trends and patterns across your application:

  • Request volume over time
  • Latency percentiles (P50, P95, P99)
  • Error rates and failure patterns
  • Token usage trends
  • Cost projections

Architecture Overview​

Langfuse Architecture

How it works:

  1. Your Application — Your CAIP Agents code instrumented with @observe decorators
  2. CAIP Agents SDK — Automatically captures LLM calls, tool invocations, and custom spans)
  3. Langfuse Server — Receives, stores, and indexes trace data
  4. Dashboard — Web UI for viewing and analyzing traces

Traces are sent asynchronously in the background, so observability adds minimal latency to your application.

Integration Options​

Langfuse integrates with CAIP applications in multiple ways:

MethodUse CaseComplexity
CAIP Agents SDK (Full)Building agents with full SDK features + observability. See the Agents SDK Installation Guide.Easiest ⭐
CAIP Agents SDK (Observability Only)Just need @observe without agent features. Install via the Agents SDK Installation Guide.Easy
Langfuse MCP ServerAI coding assistants (Cursor, Claude Code, VS Code) — no code neededEasy
Langfuse Python SDKCustom LLM apps without CAIP SDKModerate
OpenTelemetry (OTLP)Non-Python apps (Java, Go, .NET) or existing OTel setupsModerate

See the Getting Started guide for detailed setup instructions for each method.

Quick Example​

Add observability to your code with a single decorator:

from caip_agents_sdk.observability import observe, propagate_attributes
 
@observe(name="chat_handler")
async def handle_chat(user_id: str, message: str):
    # Automatically tracked in Langfuse
    with propagate_attributes(user_id=user_id, session_id="sess_123"):
        response = await agent.run(message)
        return response

This creates a trace showing:

  • The user who made the request
  • The session it belongs to
  • All nested LLM calls and tool invocations
  • Token counts and costs
  • Execution time for each step

Advanced Features​

Langfuse provides a rich set of capabilities beyond tracing and monitoring. The following features are available in the Langfuse platform and will be offered through CAIP as part of our roadmap.

Explore Langfuse Docs

These features are supported by Langfuse and are being evaluated for CAIP integration. You can explore them today via the official Langfuse documentation.

📊 Scores & Evaluation​

Attach quality scores to traces and observations — either programmatically or via the UI.

FeatureDescription
ScoresAttach numeric, boolean, or categorical scores to any trace or observation
Score ConfigsDefine custom score schemas with data types, categories, and value ranges
Score AnalyticsVisualize score trends over time on custom dashboards

📖 Langfuse Scores Documentation

🤖 LLM-as-a-Judge​

Use LLMs to automatically evaluate the quality of your AI outputs at scale.

FeatureDescription
LLM-as-a-Judge EvaluatorsConfigure LLM-based evaluators that auto-score traces in production
Custom Eval TemplatesDefine evaluation criteria with custom prompts and scoring rubrics
Online EvaluationRun evaluators automatically on live production traces

📖 Langfuse LLM-as-a-Judge Documentation

📋 Datasets & Experiments​

Build reusable test datasets and run experiments to compare prompt, model, or code changes.

FeatureDescription
DatasetsCreate collections of test inputs from production traces or manual curation
ExperimentsRun datasets against different configurations and compare results side-by-side
CI/CD IntegrationBlock deploys on quality regressions with automated experiment checks

📖 Langfuse Datasets Documentation

✍️ Human Annotations​

Enable human review workflows for trace quality assessment.

FeatureDescription
Annotation QueuesCreate review queues for human evaluators to rate traces
Manual Scoring via UIReview and score traces directly in the Langfuse dashboard
Text AnnotationsLeave open-ended notes and comments on any trace

📖 Langfuse Annotation Documentation

📝 Prompt Management​

Manage, version, and deploy prompts separately from your application code.

FeatureDescription
Prompt VersioningStore and version prompts centrally with labels (e.g., production, staging)
Prompt PlaygroundTest prompts directly in the Langfuse UI with different models
SDK IntegrationFetch managed prompts from your application code via SDK

📖 Langfuse Prompt Management Documentation

🔧 Additional Capabilities​

FeatureDescription
PlaygroundTest LLM calls with different models and parameters directly in the UI
Metrics & Custom DashboardsCreate custom dashboards with advanced metric widgets
MCP ServerAccess Langfuse data from AI coding assistants via Model Context Protocol
OpenTelemetry IntegrationSend traces from any language (Java, Go, .NET, etc.) using the OpenTelemetry standard

Documentation​