PolicyGuard: From Organizational Policies to Neuro-... | AI Research

PolicyGuard: From Organizational Policies to Neuro-Symbolic Compliance Review Engines
Organizations often struggle to ensure that legal documents, such as Non-Disclosure Agreements (NDAs), align with their internal policies. While Large Language Models (LLMs) can read and analyze these documents, they often struggle to apply complex, evolving company rules consistently. When an LLM is asked to decide compliance in a single step, the logic remains hidden, making it difficult for legal teams to audit, update, or trust the results. PolicyGuard addresses this by creating a "neuro-symbolic" framework that separates the interpretation of document text from the application of formal policy rules.

How PolicyGuard Works

Instead of asking an LLM to make a final "compliant" or "non-compliant" judgment, PolicyGuard breaks the process into two distinct stages. First, it converts organizational policy guidance into an executable "review engine." This engine consists of formal, typed logic rules and specific, atom-level questions.
During the review process, the LLM acts only as an extractor: it answers targeted questions about the document, such as whether a specific clause contains a certain type of obligation or exception. Once these facts are extracted, a symbolic evaluator—a deterministic, rule-based system—applies the formal logic to these facts to reach a final decision. This ensures that the compliance logic is explicit and consistent, rather than being buried inside the LLM’s probabilistic reasoning.

Improving Reliability and Auditability

A major challenge with using LLMs for legal review is that they can provide different answers for the same document across multiple runs. PolicyGuard solves this by confining the LLM’s variability to the initial extraction phase. Because the final decision is made by a symbolic solver, the system is significantly more stable. In testing, PolicyGuard demonstrated high consistency across repeated runs, whereas standard LLM prompting methods showed significant fluctuations in their conclusions. Furthermore, because the rules are formalized, legal experts can inspect, test, and update individual components of the engine without needing to retrain the entire system.

Performance and Results

When tested against company-specific NDA policies, PolicyGuard significantly outperformed traditional prompting methods. It achieved higher accuracy and better detection of non-compliant clauses, proving that pre-compiling policy guidelines into an executable engine is more effective than asking an LLM to interpret raw policy text on the fly. The framework also proved to be portable, performing well across various open-source and closed-source LLMs, which suggests that the architecture itself—rather than the specific model—is the primary driver of its success.

Important Considerations

While PolicyGuard offers a robust way to automate compliance, it has limitations. The current version does not build a comprehensive "fact graph" to link information across an entire contract, meaning it may not catch inconsistencies that span multiple sections. Additionally, the system is designed for specific policy-grounded tasks; its effectiveness in other domains or with different types of documents would require new rule construction and validation. Finally, because the system relies on human-defined rules, the quality of the output depends on the initial formalization of the organization's policies.