Knowledge- and Gradient-Guided Reinforcement Learni... | AI Research

Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes
This paper introduces a new approach to Reinforcement Learning (RL) designed for complex environments where decisions involve both choosing a symbolic action (like "open valve") and setting numerical parameters (like "valve opening"). Traditional RL methods often struggle with these "Parametrized Action Markov Decision Processes" (PAMDPs) because they treat parameter selection as a simple estimation problem, leading to inefficient training and potential safety violations. The authors propose a novel algorithm called KGRL (Knowledge- and Gradient-Guided Reinforcement Learning) that bridges the gap between symbolic reasoning and neural network-based learning to improve both efficiency and safety.

Combining Symbolic Knowledge with Learning

The core innovation of KGRL is the integration of a Datalog knowledge base directly into the RL agent's decision-making process. In many real-world scenarios, developers possess incomplete but useful domain knowledge—such as safety rules or expert heuristics—that can rule out dangerous or impossible actions. KGRL uses this knowledge to evaluate the current state of the environment before the agent makes a move. By doing so, it prunes non-applicable actions and restricts the agent’s parameter choices to feasible regions, ensuring the agent only explores decisions that are known to be valid.

Gradient-Guided Parameter Refinement

Beyond pruning the decision space, KGRL improves how the agent selects numerical parameters. While standard methods rely on "one-shot" estimators that guess parameters in a single pass, KGRL employs a gradient-guided refinement loop. This mechanism uses the gradient of the agent’s action-value function to iteratively adjust and optimize parameter values. By combining this with the constraints derived from the knowledge base, the agent can fine-tune its decisions to be both optimal and compliant with domain-specific requirements, even during the training phase.

Explainability and Performance

Because KGRL evaluates a symbolic knowledge base at every step, it provides a unique benefit: transparency. The algorithm can record which specific rules were triggered and why certain actions were pruned or parameters were constrained. This creates a local, procedural explanation of the agent's behavior, which is highly valuable for safety-critical applications. In empirical evaluations, the authors demonstrate that KGRL outperforms existing state-of-the-art RL baselines, achieving higher episodic returns and reaching optimal performance significantly faster due to its more efficient exploration strategy.