AI GitHub
Phoenix for inspecting LLM traces and evaluations
Use OpenTelemetry-based traces to inspect model, retrieval and agent activity.
Key Takeaways
- Phoenix links spans from instrumented applications to trace views and evaluation workflows.
- Start with a documented exporter and a trace destination.
- Send a trace for a known failed query and locate the stage where relevant context was lost.
- Useful traces depend on consistent instrumentation.
How it works
Phoenix links spans from instrumented applications to trace views and evaluation workflows. A developer can examine retrieved documents and model responses in the context of a request.
What you need
Start with a documented exporter and a trace destination. Include identifiers for the retrieval and generation steps so they can be correlated during debugging.
What to try first
Send a trace for a known failed query and locate the stage where relevant context was lost. Preserve the example as an evaluation case.
Try tracing a search result with an intentionally wrong document ID. Confirm that the ID survives instrumentation and can be matched to your evaluation fixture.
Where to look for failures
Useful traces depend on consistent instrumentation. Redact sensitive attributes and keep track of the instrumentation version when comparing runs.
Related Franklin pages
Trace the same synthetic request in each platform. Compare how the search, model call and application steps appear and which data the instrumentation captures by default.
- Capture agent or model traces and turn examples into repeatable evaluations. Opik for tracing and testing model applications
- Collect traces around model calls, prompts and application steps. Langfuse for tracing and evaluating LLM applications
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗