Back to AI Research

AI Research

TRACE-CTI: Auditable Post-Extraction Governance of... | AI Research

Key Takeaways

  • TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs Security Operations Centers (SOCs) frequently use automated tools to map...
  • The framework retains native evidence granularity, complete extraction provenance, versioned trust decisions, and non-destructive revocation history.
  • We evaluate TRACE-CTI on two public CTI corpora comprising 65 reports and 5,303 sentences, using a controlled 2 x 3 matrix of retrievers and generator families, incrementally ingested across six GraphVersions.
  • All setups are incorporated without schema modification; provenance paths remain complete, operational scopes remain disjoint, and every trusted GraphAssertion has an active qualifying validation ground.
  • Cross-generator-family setup pairs exhibit greater output diversity than same-family pairs.
Paper AbstractExpand

Security Operations Centers increasingly rely on automated mapping of Cyber Threat Intelligence reports to MITRE ATT&CK, yet extractor outputs remain fallible and are often stored without the evidence, provenance, and validation history needed to decide whether an individual mapping should be trusted. We present TRACE- CTI, a post-extraction claim-governance framework that preserves run-level Predictions, aggregates them into configuration-level GraphAssertions, materializes setup-deduplicated corroboration as ConsensusAssertions, and exposes only GraphAssertions backed by policy-compliant validation grounds. The framework retains native evidence granularity, complete extraction provenance, versioned trust decisions, and non-destructive revocation history. We evaluate TRACE-CTI on two public CTI corpora comprising 65 reports and 5,303 sentences, using a controlled 2 x 3 matrix of retrievers and generator families, incrementally ingested across six GraphVersions. All setups are incorporated without schema modification; provenance paths remain complete, operational scopes remain disjoint, and every trusted GraphAssertion has an active qualifying validation ground. Cross-generator-family setup pairs exhibit greater output diversity than same-family pairs. At the final graph state, increasing setup support from k >= 1 to six-setup unanimity raises gold-aligned precision from 25.3% to 90.6%, while recall decreases from 88.2% to 16.3%. The graph also directly answers seven questions about provenance, trust, versioning, dependency, disagreement, and review-queue that the evaluated minimal flat output cannot fully answer without enrichment or reprocessing. These results support explicit, auditable governance of extracted TTP claims; the observed corroboration trajectory is descriptive and does not establish statistical independence or a causal model-family effect.

TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs
Security Operations Centers (SOCs) frequently use automated tools to map Cyber Threat Intelligence (CTI) reports to the MITRE ATT&CK framework. While these tools are helpful, they are not perfect, and their outputs are often stored without the necessary context, evidence, or history to determine if a mapping is actually reliable. This paper introduces TRACE-CTI, a framework designed to manage these extracted claims after they are generated. Instead of treating model outputs as immediate facts, TRACE-CTI creates a structured, auditable lifecycle that tracks where a claim came from, how it was validated, and whether it should be trusted for operational use.

Managing the Lifecycle of Extracted Claims

The framework functions by organizing data into a hierarchy that separates raw model predictions from trusted knowledge. It begins by preserving "Predictions"—the raw, run-level outputs from an extractor—along with their specific source text and configuration details. These are then aggregated into "GraphAssertions," which represent the claim at a configuration level. Finally, the system uses "ConsensusAssertions" to identify when different extraction setups agree on the same information. By maintaining this structure, the system ensures that every piece of information remains traceable to its original evidence and the specific process that produced it.

Establishing Trust Through Validation

A core feature of TRACE-CTI is its strict policy on what constitutes "trusted" knowledge. A model-derived assertion does not automatically become a fact; it must be backed by a recorded validation event. This validation can be based on gold-standard annotations, human analyst review, or corroboration from multiple, distinct extraction setups. By requiring this explicit validation, the framework prevents fallible model outputs from being silently promoted to trusted intelligence, helping to reduce the risk of propagating errors into downstream security operations.

Evaluating Performance and Operational Utility

The researchers tested the framework using a matrix of two retrievers and three generator families across 65 CTI reports. The results showed that as the system required agreement from more "witnesses" (different extraction setups), the precision of the trusted assertions improved significantly—rising from 25.3% to 90.6% when moving from a single setup to six-setup unanimity. Beyond accuracy, the framework proved capable of answering complex operational questions that standard, flat data storage cannot, such as identifying which model configuration produced a specific claim, tracking the history of trust decisions, and managing the revocation of outdated or incorrect mappings.

Important Considerations

It is important to note that TRACE-CTI is a governance framework rather than a new TTP extraction model. It is designed to be independent of the specific language models or extractors used. While the study observed that increasing the number of supporting setups improves precision, the authors emphasize that this is a descriptive observation. It does not prove that these models are statistically independent or that there is a causal effect related to the model families themselves. The framework serves as a tool for auditability and transparency, ensuring that SOCs can maintain a clear boundary between raw, automated predictions and verified, actionable intelligence.

Comments (0)

No comments yet

Be the first to share your thoughts!