Knowledge-Centric Agents for Workflow Generation

Knowledge-Centric Agents for Workflow Generation | AI Research

Key Takeaways

Knowledge-Centric Agents for Workflow Generation introduces a new framework designed to help AI agents create complex, functional workflows for visual creati...
Workflow generation in visual creation systems such as ComfyUI demands not only syntactic accuracy but also expert-level reasoning over modular compositions.
Existing large language model (LLM) approaches often treat this as a direct text-to-JSON generation task, struggling with structural brittleness and lacking the experiential knowledge required for effective design.
We argue that successful workflow generation requires modeling knowledge itself, including its structure, hierarchy, and reasoning dynamics.
To this end, we propose a knowledge-centric framework that learns to invert, inject, and infer with knowledge across multiple abstraction levels.

Paper AbstractExpand

Workflow generation in visual creation systems such as ComfyUI demands not only syntactic accuracy but also expert-level reasoning over modular compositions. Existing large language model (LLM) approaches often treat this as a direct text-to-JSON generation task, struggling with structural brittleness and lacking the experiential knowledge required for effective design. We argue that successful workflow generation requires modeling knowledge itself, including its structure, hierarchy, and reasoning dynamics. To this end, we propose a knowledge-centric framework that learns to invert, inject, and infer with knowledge across multiple abstraction levels. We first perform knowledge inversion to distill hierarchical representations, ranging from full pseudo-codes and skeletons to high-level strategies, from large collections of real-world workflows. We then conduct knowledge injection through supervised fine-tuning, teaching the model to reason from task descriptions to strategies and from strategies to executable structures. During inference, the model performs reversible reasoning to synthesize executable workflows, augmented by self-refinement for structural coherence. Extensive experiments demonstrate that our method produces workflows with richer node diversity, more coherent structures, and higher execution success rates than existing systems, establishing a new foundation for knowledge-driven, agentic workflow generation.

Knowledge-Centric Agents for Workflow Generation introduces a new framework designed to help AI agents create complex, functional workflows for visual creation systems like ComfyUI. While existing AI models often struggle to generate these workflows because they treat them as simple text-to-code tasks, this research argues that effective workflow generation requires a deeper understanding of the "knowledge" behind the design—including the structural hierarchy, expert reasoning, and strategic planning involved in building a visual pipeline.

From Raw Data to Expert Knowledge

The core of this framework is a process called "knowledge inversion." Instead of training an AI on raw, unorganized workflow files, the researchers first break these files down into a hierarchical structure. They transform complex, messy workflows into "pseudo-code" that outlines the essential connections between nodes. From there, they extract the high-level strategies and reasoning traces that explain why an expert designer would choose a specific sequence of modules. This turns unannotated data into a structured library of expert-level design logic.

Teaching the Agent to Reason

Once this knowledge is distilled, the researchers use it to train an AI model through supervised fine-tuning. The model learns to perform a two-step reasoning process: first, it analyzes a user’s task description to develop a high-level strategy; second, it translates that strategy into a concrete, executable graph structure. By teaching the model to think in terms of strategies and structural skeletons rather than just raw JSON code, the system becomes much more reliable at producing workflows that actually work.

Inference and Self-Refinement

When a user provides a new task, the agent follows a "reversible" reasoning path. It starts by planning the overall strategy, generates the corresponding pseudo-code, and finally reconstructs the full, executable workflow. To ensure the final output is accurate, the agent includes a self-refinement step where it reviews its own work to correct potential errors in logic or connectivity. This iterative approach helps the agent maintain structural coherence, even when dealing with complex or novel requests.

Performance and Impact

Experiments show that this knowledge-centric approach significantly outperforms existing methods. By focusing on the underlying logic of workflow design, the agent produces workflows with greater node diversity and more coherent structures. Most importantly, the system achieves a higher success rate in executing these workflows, proving that modeling the "knowledge" behind a task is more effective than simply trying to predict code patterns. This research establishes a new foundation for building agentic systems that can handle the sophisticated demands of modern visual programming.