Enhancing Virtual Agents through SLMs and Edge-Comp...

Enhancing Virtual Agents through SLMs and Edge-Comp... | AI Research

Key Takeaways

This paper explores how to make virtual agents in Metaverse-style environments more intelligent by using Small Language Models (SLMs) to handle core cognitiv...
Embodied intelligent virtual agents are expected to operate as persistent, adaptive, and context-aware entities within complex virtual and Metaverse worlds.
However, implementing cognitively capable agents in such environments is conceptually and technologically challenging.
This paper explores how to make virtual agents in Metaverse-style environments more intelligent by using Small Language Models (SLMs) to handle core cognitive processes locally on edge hardware.
## Implementing Cognitive Processes on the Edge

Paper AbstractExpand

Embodied intelligent virtual agents are expected to operate as persistent, adaptive, and context-aware entities within complex virtual and Metaverse worlds. However, implementing cognitively capable agents in such environments is conceptually and technologically challenging. Among a range of blueprints and development approaches, the Cognitive Embodied Agent Architecture (CEAA) has been developed as an implementation-oriented framework for architecting components of perception, memory, reasoning, planning, and embodied action. Considering the recent advances in edge computing and generative AI language models, this paper explores the use of Small Language Models (SLMs) to support edge-based operation of selected CEAA components, focusing on "Think" and "Memory" as processes central to cognitive orchestration and persistence of virtual agents in interactive virtual worlds. An edge-based virtual agent gateway system was developed and evaluated on an NVIDIA Jetson Orin NX using Qwen2.5 models of different sizes, exploring the system's capability to process service requests and handle memory-driven conversations. A series of simulation experiments evaluated routing accuracy, memory-read performance, and latency, demonstrating an SLM-driven prototype agent system that partially implements selected CEAA processes to support the development of embodied agents whose cognitive "brain" can operate efficiently and contextually for interactive experiences in immersive virtual worlds.

This paper explores how to make virtual agents in Metaverse-style environments more intelligent by using Small Language Models (SLMs) to handle core cognitive processes locally on edge hardware. The authors, Aimilios Hadjiliasi and Louis Nisiotis, investigate whether SLMs can effectively manage "Think" and "Memory" components—two essential parts of the Cognitive Embodied Agent Architecture (CEAA)—to allow agents to act with better context and persistence without relying on cloud-based processing.

Implementing Cognitive Processes on the Edge

The researchers developed an edge-based gateway system designed to run on an NVIDIA Jetson Orin NX. This system uses Qwen2.5 models (in 0.5B, 1.5B, and 3.0B parameter sizes) to act as the agent's "brain." The goal is to enable the agent to interpret user input, decide whether to store or retrieve information, and route requests to specific backend services—such as 3D model generation, motion creation, or text-to-speech—without sending data to the cloud.

How the System Handles Thinking and Memory

The architecture divides agent tasks into two primary functions:

Think: The system uses the SLM to classify user prompts and route them to the correct service. This acts as a cognitive coordinator, ensuring the agent knows whether to engage in conversation, perform a technical task, or access its database.
Memory: The system uses a combination of the SLM, LangMem, and an SQLite database. Every interaction is recorded, and the SLM extracts durable facts from these conversations. When a user asks a question, the system retrieves relevant facts and recent session history to generate a context-aware response.

Experimental Results

The authors evaluated the system using a simulation of a collaborative game-development environment. They tested the models on 1,000 routing prompts and 250 memory-based interactions. The experiments measured routing accuracy, memory-read performance, and latency. The findings suggest that SLMs can perform these orchestration tasks locally, providing a way to manage agent behavior in resource-constrained environments where privacy and low latency are priorities.

Considerations for Future Development

While the study shows that SLMs can partially operationalize CEAA components, the authors note several limitations. SLMs are generally less capable than larger models when it comes to complex, open-ended reasoning, long-term planning, and handling ambiguous requests. Additionally, the researchers point out that while these models are efficient, their smaller size can lead to reduced consistency during extended interactions. The study serves as an exploratory look at how to bridge the gap between simple, rule-based virtual agents and more advanced, cognitively capable entities in interactive virtual worlds.