Talking Head Synthesis with Facial Landmark Guidanc... | AI Research

Talking Head Synthesis with Facial Landmark Guidance via 3D Gaussian Splatting addresses a fundamental challenge in digital human generation: how to create realistic, audio-driven facial animations that maintain accurate geometry and movement. While current technologies like 3D Gaussian Splatting (3DGS) allow for efficient rendering, they often struggle because audio signals lack the spatial information needed to precisely control facial expressions. This research introduces a framework that uses facial landmarks as a structural guide to ensure that mouth movements and expressions are both accurate and stable.

Bridging Audio and Geometry

The core issue in speech-driven synthesis is that audio features describe sounds over time, not the physical layout of a face. To solve this, the authors developed a Spatial Enhancement Module. By using a pretrained model to predict 3D facial landmarks from audio, the system identifies "expression-sensitive" regions—such as the mouth and eyes—and injects landmark-based structural information into the 3D representation. This ensures the model knows exactly which parts of the face should move in response to specific sounds. The robotics story also surfaces in NVIDIA Launches Cosmos 3 Edge for..., adding another angle.

Global Structural Correction

Beyond local adjustments, the system employs a global landmark compensation mechanism. It takes the full set of predicted facial keypoints and encodes them into a single conditioning vector. This vector acts as a global guide, refining the 3D Gaussian attributes (such as position, rotation, and scale) across the entire face. By combining this global structural awareness with local point-wise enhancements, the model achieves more coherent and natural facial motion, preventing the artifacts often seen in simpler systems.

Improving Visual Fidelity

To ensure the generated faces look realistic and maintain consistent geometry, the researchers incorporated a geometry regularizer. This component uses depth and surface-normal constraints to keep the 3D structure stable, even when the model is viewed from different angles. Additionally, the team designed a lightweight audio encoder that processes speech features efficiently, ensuring the system can handle complex phonetic transitions without the high computational costs associated with older, more complex models. The robotics story also surfaces in MIT Researchers Develop Method to Make..., adding another angle.

Performance and Results

The proposed method was tested against several existing 2D and 3D talking-head frameworks. In both self-driven and cross-driven scenarios—where the model is tested on audio it has not encountered during training—the system demonstrated superior performance. It achieved higher visual quality, better structural accuracy, and more precise lip synchronization compared to previous baselines. These results confirm that integrating explicit landmark guidance into 3D Gaussian Splatting effectively resolves the mismatch between acoustic patterns and facial geometry. The robotics story also surfaces in Black Forest Labs Unveils FLUX 3..., adding another angle. as detailed in the full paper on Arxiv