Langfuse for tracing and evaluating LLM applications

How it works

Instrumentation records observations within traces, connecting model inputs and outputs to the rest of a request. Evaluations and datasets can use examples captured through this workflow.

What you need

Decide where the tracing service will run and how the app authenticates to it. Verify the boundary between local application logs and data sent to the service.

What to try first

Instrument one synthetic request with retrieval and generation steps. Confirm token counts, timing and which fields are visible before trying real traffic.

Record a retry as a distinct span or observation in a test trace. Otherwise the apparent cost or latency of one successful model call can omit the failed attempt.

Where to look for failures

Trace payloads can include sensitive prompts and tool output. Decide which fields to capture and set access and retention policies before shipping instrumentation.

Related Franklin pages

A gateway tells you where a request went, while instrumentation shows what happened around it. Keep IDs across layers so provider errors can be tied to the original application request.

• Use a consistent client interface or proxy across multiple model providers. LiteLLM for model routing and gateways
• Use OpenTelemetry-based traces to inspect model, retrieval and agent activity. Phoenix for inspecting LLM traces and evaluations

Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks

Project documentation

• Current project README
• README at the reviewed revision