Noveum.ai: trace, evaluate and investigate production AI agents
Noveum.ai is an evaluation platform for chat, voice and workflow agents. It brings recorded runs, configured scorers and failure analysis into one workflow, helping a team investigate why an agent produced an unwanted result.
From trace to evaluation evidence
The open-source Python SDK records instrumented model calls, tool invocations, retrievals and agent steps as structured spans. The official site lists integrations including LangChain, LangGraph, LiveKit, Pipecat and CrewAI, alongside custom spans for application-specific work.
Teams can turn selected production traces into dataset items and apply compatible scorers from the published library. Voice testing adds configured personas and scenarios. These capabilities make the platform relevant when failures occur across several steps and a final answer alone does not explain what happened.
Recommendations remain an engineering decision
NovaPilot analyzes a selected evaluated dataset and produces a report covering scorer performance, recurring failures, prompt findings and ranked recommendations. NovaPilot recommends changes; engineers implement and verify them. The distinction matters when a proposed fix could affect behavior beyond the examples that triggered it.
A practical workflow is to capture a failed run, check whether the scorer describes the actual problem, review the underlying spans, then evaluate the proposed change on fresh examples as well as the original case.
Deployment and limitations
The vendor offers enterprise deployment options that can include in-VPC, on-premise or bring-your-own ClickHouse arrangements. Their scope depends on the agreement. The site describes SOC 2 Type II as in progress, so that label should not be read as a completed certification.
Treat displayed customer results as vendor-provided case studies rather than an expected outcome for a new agent. Before adopting the platform, confirm which integrations capture the necessary evidence and whether the selected scorers measure the failures your users actually encounter.