AI GitHub
Opik for tracing and testing model applications
Capture agent or model traces and turn examples into repeatable evaluations.
Key Takeaways
- Opik connects application instrumentation with datasets, experiments and scores.
- Choose an instrumented client or wrapper and a test project.
- Run two versions of a prompt on the same set of fixtures.
- Decide which prompts and outputs can enter the trace store.
How it works
Opik connects application instrumentation with datasets, experiments and scores. Traces make it possible to inspect what happened before a final answer.
What you need
Choose an instrumented client or wrapper and a test project. Use a synthetic dataset to confirm that traces and experiments reach the intended workspace.
What to try first
Run two versions of a prompt on the same set of fixtures. Compare outputs and investigate a failure using the spans for the corresponding call.
Use an evaluation with both a deterministic score and a review rubric. Compare them on a few outputs before using either to approve a change.
Where to look for failures
Decide which prompts and outputs can enter the trace store. Compare experiment scores only when the dataset and rubric are held constant.
Related Franklin pages
Trace the same synthetic request in each platform. Compare how the search, model call and application steps appear and which data the instrumentation captures by default.
- Collect traces around model calls, prompts and application steps. Langfuse for tracing and evaluating LLM applications
- Use OpenTelemetry-based traces to inspect model, retrieval and agent activity. Phoenix for inspecting LLM traces and evaluations
Franklin’s benchmark pages offer examples of evaluating models on specific tasks. Franklin’s AI benchmarks
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗